Visual positioning method and device, computer device and storage medium

Abstract

The application relates to a visual positioning method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: inputting a video to be positioned into a visual encoder, extracting video frame features, inputting a query text into a text encoder, and extracting text features; extracting an image aggregation feature set of the video to be positioned and a text aggregation feature of the query text through a feature aggregation network; inputting the image aggregation feature set into a space-time aggregation network to extract global visual features; inputting the image aggregation feature set and the text aggregation feature into a cross-modal encoder to extract cross-modal visual features and cross-modal text features; inputting the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder to obtain video frame query results; and inputting the video frame query results into a coordinate prediction head and a time detection head to obtain spatial positioning results and time positioning results. The method can improve the accuracy of video information extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular, to a visual positioning method and device, a computer device, a storage medium, and a computer program product. BACKGROUND

[0002] The visual positioning task aims to locate the target and time period corresponding to the input text query in the video. The research and development of this technology can promote the interaction between visual content and natural language, optimize the video analysis process, and provide better experiences and services for users by mining the potential value of video data. Application scenarios include searching for specific people, objects, or event video clips in multimedia information retrieval, quickly locating suspicious behavior or events in monitoring systems, and assisting video editing and summarization for news reports, movies, etc.

[0003] There are two common model paradigms, namely, an end-to-end method and a two-stage method. The two-stage method first generates pre-extracted candidate boxes or candidate time segments by a detector, and then matches the text to obtain the best result. The disadvantage of this method is that it is difficult to jointly optimize the two stages, lacks global modeling, and accumulates possible errors, resulting in misalignment of video context features, inconsistency of positioning targets before and after in the time dimension, and large differences between recognition results and text semantics. The end-to-end method may also have a mismatch between the target and the query content if the scene in the video is complex or contains multiple entities with similar appearances.

[0004] Therefore, the traditional technology has the problem of low accuracy when performing video query extraction based on text. SUMMARY

[0005] Therefore, it is necessary to provide a visual positioning method, device, computer device, computer readable storage medium, and computer program product that can improve the accuracy of video query extraction based on text.

[0006] A visual positioning method comprises:

[0007] Obtaining a video to be positioned, and obtaining a query text for the video to be positioned;

[0008] Inputting the video to be positioned into a visual encoder of a visual positioning model to extract video frame features of the video to be positioned, and inputting the query text into a text encoder of the visual positioning model to extract text features of the query text;

[0009] Extracting an image aggregation feature set of the video to be positioned and a text aggregation feature of the query text based on the video frame features and the text features by a feature aggregation network of the visual positioning model;

[0010] inputting the image aggregated feature set into a spatio-temporal aggregation network of the visual localization model to extract global visual features, and inputting the image aggregated feature set and the text aggregated feature into a cross-modal encoder of the visual localization model to extract cross-modal visual features and cross-modal text features;

[0011] inputting the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder of the visual localization model to obtain a video frame query result;

[0012] inputting the video frame query result into a coordinate prediction head and a time detection head of the visual localization model to obtain a spatial localization result and a temporal localization result; the spatial localization result represents a picture position of the target in the video to be localized; and the temporal localization result represents an appearance time of the target in the video to be localized.

[0013] In one of the embodiments, the text encoder has a first preset number of text encoding layers, the text features of the query text include text encoding features output by each text encoding layer of the text encoder for the query text, and the image aggregated feature set of the video to be localized and the text aggregated feature of the query text are extracted based on the video frame features and the text features by the feature aggregation network of the visual localization model, including:

[0014] grouping and aggregating the text encoding features output by each text encoding layer for the query text by the feature aggregation network to obtain a plurality of text encoding feature aggregation results corresponding to the query text;

[0015] splicing the plurality of text encoding feature aggregation results and the text encoding features output by the last text encoding layer of the text encoder for the query text by the feature aggregation network to obtain a feature splicing result corresponding to the query text;

[0016] inputting the feature splicing result corresponding to the query text into a multi-layer perception of the feature aggregation network to obtain the text aggregated feature corresponding to the query text.

[0017] In one of the embodiments, the visual encoder has a second preset number of visual encoding layers, the video frame features of the video to be localized include image encoding feature sets corresponding to each sampling video frame of the video to be localized, the image encoding feature set corresponding to any sampling video frame includes image encoding features output by each visual encoding layer of the visual encoder for the sampling video frame, and the image aggregated feature set of the video to be localized and the text aggregated feature of the query text are extracted based on the video frame features and the text features by the feature aggregation network of the visual localization model, including:

[0018] grouping and aggregating the image encoding features output by each visual encoding layer for any sampling video frame by the feature aggregation network to obtain a plurality of image encoding feature aggregation results corresponding to any sampling video frame;

[0019] The image encoding feature aggregation result corresponding to any sampled video frame and the image encoding feature output by the last visual encoding layer of the visual encoder for any sampled video frame are spliced by the feature aggregation network to obtain a feature splicing result corresponding to any sampled video frame;

[0020] The feature splicing result corresponding to any sampled video frame is input into the multi-layer perception of the feature aggregation network to obtain image aggregation features corresponding to any sampled video frame;

[0021] Based on the image aggregation features corresponding to each sampled video frame, an image aggregation feature set is generated.

[0022] In one of the embodiments, the video to be positioned is input into the visual encoder of the visual positioning model to extract video frame features of the video to be positioned, including:

[0023] The video to be positioned is sparsely and uniformly sampled to obtain each sampled video frame corresponding to the video to be positioned;

[0024] Each sampled video frame is input into the visual encoder of the visual positioning model, and the visual encoder is used to perform visual encoding processing on each sampled video frame to obtain image encoding features corresponding to each sampled video frame;

[0025] The image encoding features corresponding to each sampled video frame are taken as the video frame features of the video to be positioned.

[0026] In one of the embodiments, the image aggregation feature set is input into the spatio-temporal aggregation network of the visual positioning model to extract global visual features, including:

[0027] The image aggregation feature set is input into the spatio-temporal aggregation network of the visual positioning model in the form of a sequence, and the first convolutional layer in the spatio-temporal aggregation network is used for feature fusion to obtain fused features;

[0028] The fused features are input into the space-time block in the spatio-temporal aggregation network to obtain spatial features and temporal features; the space-time block is formed by alternately connecting a plurality of spatial self-attention layers and a plurality of temporal self-attention layers;

[0029] The spatial features and the temporal features are input into the second convolutional layer in the spatio-temporal aggregation network for feature aggregation to obtain global visual features.

[0030] In one of the embodiments, the video frame query result is input into the coordinate prediction head and the time detection head of the visual positioning model to obtain spatial positioning results and temporal positioning results, and the method includes:

[0031] The video frame query result is input into a coordinate prediction head and a time detection head of the visual positioning model to obtain probability distribution information corresponding to each target detection frame and start and end time of each target detection frame;

[0032] According to the probability distribution information corresponding to each target detection frame and start and end time of each target detection frame, a joint probability between any target detection frame and start and end time of any target detection frame is determined.

[0033] The start and end time with the maximum joint probability is taken as a time positioning result, and a target detection frame associated with the time positioning result is taken as a spatial positioning result.

[0034] A visual positioning device comprises:

[0035] An acquisition module is configured to acquire a video to be positioned and acquire a query text for the video to be positioned.

[0036] A feature extraction module is configured to input the video to be positioned into a visual encoder of a visual positioning model to extract video frame features of the video to be positioned, and input the query text into a text encoder of the visual positioning model to extract text features of the query text.

[0037] A feature aggregation module is configured to extract, by a feature aggregation network of the visual positioning model, an image aggregation feature set of the video to be positioned and a text aggregation feature of the query text based on the video frame features and the text features.

[0038] A feature encoding module is configured to input the image aggregation feature set into a spatio-temporal aggregation network of the visual positioning model to extract global visual features, and input the image aggregation feature set and the text aggregation feature into a cross-modal encoder of the visual positioning model to extract cross-modal visual features and cross-modal text features.

[0039] A feature decoding module is configured to input the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder of the visual positioning model to obtain a video frame query result.

[0040] A prediction module is configured to input the video frame query result into a coordinate prediction head and a time detection head of the visual positioning model to obtain a spatial positioning result and a time positioning result; the spatial positioning result represents a picture position of a target in the video to be positioned; and the time positioning result represents an appearance time of the target in the video to be positioned.

[0041] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0042] A computer readable storage medium, having stored thereon a computer program, the computer program being executed by a processor to implement the steps of the method.

[0043] A computer program product, comprising a computer program, the computer program being executed by a processor to implement the steps of the method.

[0044] The visual positioning method, device, computer device, storage medium and computer program product, by acquiring the to-be-positioned video, and acquiring the query text for the to-be-positioned video; inputting the to-be-positioned video into the visual encoder of the visual positioning model, extracting the video frame features of the to-be-positioned video, and inputting the query text into the text encoder of the visual positioning model, extracting the text features of the query text; based on the video frame features and the text features, extracting the image aggregation features set of the to-be-positioned video and the text aggregation features of the query text through the feature aggregation network of the visual positioning model; inputting the image aggregation features set into the spatio-temporal aggregation network of the visual positioning model, extracting the global visual features, and inputting the image aggregation features set and the text aggregation features into the cross-modal encoder of the visual positioning model, extracting the cross-modal visual features and the cross-modal text features; inputting the global visual features, the cross-modal visual features and the cross-modal text features into the cross-modal decoder of the visual positioning model, obtaining the video frame query result; inputting the video frame query result into the coordinate prediction head and the time detection head of the visual positioning model, obtaining the spatial positioning result and the time positioning result; the spatial positioning result represents the position of the target in the picture of the to-be-positioned video; the time positioning result represents the appearance time of the target in the to-be-positioned video; in this way, the query matching performance can be improved and the computing overhead can be reduced by strengthening the features in the two dimensions of vision and language, time and space, the processing capability of the model on complex queries can be covered by enhancing the visual features and the text features, a more fine-grained positioning can be realized, the possible ambiguity in video understanding can be eliminated by enhancing the spatio-temporal dependence of the video context, the performance loss caused by the video length expansion can be reduced, and the video segment matched with the query text can be accurately extracted according to the input query text. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0046] Figure 1 An application environment diagram of a visual positioning method in an embodiment;

[0047] Figure 2A flowchart of a visual positioning method in an embodiment;

[0048] Figure 3 A schematic diagram of a model overall architecture of a visual positioning model in an embodiment;

[0049] Figure 4 A schematic diagram of a network structure of a feature aggregation network in an embodiment;

[0050] Figure 5 A schematic diagram of a network structure of a spatio-temporal aggregation network in an embodiment;

[0051] Figure 6 A flowchart of a visual positioning method in another embodiment;

[0052] Figure 7 A structural block diagram of a visual positioning device in an embodiment;

[0053] Figure 8 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0055] Visual grounding is a video understanding task that spatially and temporally localizes a specific event or action in a given video. Typically, the model is required to localize and identify the corresponding target object or event in the video according to the given natural language description, including the spatial position (bounding box) and time span (start and end time) of the target.

[0056] The visual localization task aims to locate the target and time period in the video corresponding to the input text query. The research and development of this technology can promote the interaction between visual content and natural language, optimize the video analysis process through automation, tap the potential value of video data, and provide better experience and service for users. The application scenarios it is suitable for include searching for specific people, objects or event video clips in multimedia information retrieval; quickly locating suspicious behavior or events in monitoring systems; assisting video editing and summarization for news reports, movies, etc. There are two common model paradigms, two-stage method and end-to-end method. The two-stage method first generates pre-extracted candidate boxes or candidate time segments by the detector, and then matches the text to obtain the best result. The disadvantage of this method is that it is difficult to realize the joint optimization of the two stages, lacks global modeling, and may accumulate errors, resulting in misalignment of video context features, inconsistency of target positioning in time dimension, and large gap between recognition results and text semantics. However, even if the end-to-end modeling is used, if the scene in the video is complex or there are multiple entities with similar appearances, the target may not match the query content. At the same time, existing methods mainly involve positioning queries for short video segments, and there is a lack of support for long videos in terms of computational cost and performance. In order to solve the above problems, the present application proposes a visual localization method based on dense feature aggregation trained in an end-to-end manner.

[0057] The visual localization method provided by the embodiments of the present application can be applied to, for example Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains the video to be located, and obtains the query text for the video to be located; the server 104 inputs the video to be located into the visual encoder of the visual positioning model, extracts the video frame features of the video to be located, and inputs the query text into the text encoder of the visual positioning model, extracts the text features of the query text; the server 104 extracts the image aggregation feature set of the video to be located and the text aggregation feature of the query text based on the video frame features and the text features through the feature aggregation network of the visual positioning model; the server 104 inputs the image aggregation feature set into the spatio-temporal aggregation network of the visual positioning model, extracts the global visual features, and inputs the image aggregation feature set and the text aggregation feature into the cross-modal encoder of the visual positioning model, extracts the cross-modal visual features and the cross-modal text features; the server 104 inputs the global visual features, the cross-modal visual features and the cross-modal text features into the cross-modal decoder of the visual positioning model, obtains the video frame query result; the server 104 inputs the video frame query result into the coordinate prediction head and the time detection head of the visual positioning model, obtains the spatial positioning result and the time positioning result; the spatial positioning result represents the position of the target in the picture of the video to be located; the time positioning result represents the appearance time of the target in the video to be located. Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be realized by an independent server or a server cluster composed of multiple servers.

[0058] In an exemplary embodiment, as shown in Figure 2 A visual positioning method is provided, which is applied to the server 104 in Figure 1 for example, including the following steps S202 to S206. Among them:

[0059] Step S202, obtaining the video to be located, and obtaining the query text for the video to be located.

[0060] Among them, the video to be located refers to the video that needs to extract the video segment, for example, when the video segment of a specific person, object or event in a multimedia video A needs to be extracted, the multimedia video A is the video to be located, when the video segment of the suspicious behavior or suspicious event in the monitoring video B needs to be located, the monitoring video B is the video to be located.

[0061] The query text can be the text corresponding to the target to be found. With the assistance of the query text, the target in the video to be located can be quickly located.

[0062] Optionally, the server obtains the video to be located and the query text for the video to be located, for example, the server obtains a movie, and obtains the query text "female lead" for the movie. The query text "female lead" can assist in locating the female lead in the movie in the subsequent process, so as to extract the clip corresponding to the female lead.

[0063] In step S204, the video to be located is input into the visual encoder of the visual positioning model to extract the video frame features of the video to be located, and the query text is input into the text encoder of the visual positioning model to extract the text features of the query text.

[0064] The overall architecture of the visual positioning model is shown in Figure 3 When the visual positioning model is built, a double-encoder-single-decoder structure model is used as a basic model, which includes a visual encoder, a text encoder, a feature aggregation network, a space-time aggregation network, a cross-modal encoder, a cross-modal decoder, a coordinate detection head, and a time detection head. The weights of the visual encoder and the text encoder are always in a frozen state, and the coordinate detection head and the time detection head are MLP structures.

[0065] The visual encoder can be a network structure for visual encoding of a video frame screenshot, and the visual encoder can include multiple visual encoding layers. The video encoder can use a trained ViT-L model, and the model weights are fixed.

[0066] The text encoder can be a network structure for text encoding of the query text, and the text encoder can include multiple text encoding layers. The text encoder uses a trained Bert-L model, and the model weights are fixed.

[0067] The video frame features can include a set of image encoding features corresponding to each sampled video frame of the video to be located. The set of image encoding features of any sampled video frame includes image encoding features output by each visual encoding layer in the visual encoder for the sampled video frame.

[0068] The text features can include text encoding features output by each text encoding layer in the text encoder for the query text.

[0069] Optionally, the server inputs each sample video frame of the video to be positioned into a visual encoder of the visual positioning model, for any sample video frame, a first visual encoding layer in the visual encoder encodes a screenshot of the sample video frame to obtain first image encoding features output by the first visual encoding layer, then the first image encoding features output by the first visual encoding layer are input into a second visual encoding layer to obtain second image encoding features output by the second visual encoding layer, when the visual encoder includes N visual encoding layers, then the (N-1)th image encoding features output by the (N-1)th visual encoding layer are input into the Nth visual encoding layer to obtain Nth image encoding features output by the Nth visual encoding layer, in this way, a set of image encoding features corresponding to the sample video frame can be obtained, the same processing manner is adopted for each sample video frame, and a set of image encoding features corresponding to each sample video frame can be obtained, and the set of image encoding features corresponding to each sample video frame is formed into video frame features of the video to be positioned; and the server inputs the query text into a text encoder of the visual positioning model, a first text encoding layer in the text encoder encodes the query text to obtain first text encoding features output by the first text encoding layer, then the first text encoding features output by the first text encoding layer are input into a second text encoding layer to obtain second text encoding features output by the second text encoding layer, when the text encoder includes M text encoding layers, then (M-1)th text encoding features output by the (M-1)th text encoding layer are input into the Mth text encoding layer to obtain Mth text encoding features output by the Mth text encoding layer, in this way, text encoding features output by each text encoding layer for the query text can be obtained, that is, text features of the query text are obtained.

[0070] In step S206, the feature aggregation network of the visual positioning model extracts a set of image aggregation features of the video to be positioned and a set of text aggregation features of the query text based on the video frame features and the text features.

[0071] The feature aggregation network can be used to aggregate the image encoding features output by each visual encoding layer in the visual encoder for any sample video frame, that is, to aggregate the video frame features of the video to be positioned, and to aggregate the text encoding features output by each text encoding layer in the text encoder for the query text, that is, to aggregate the text features of the video to be positioned.

[0072] The set of image aggregation features can be represented as is the image aggregation feature obtained by aggregating the image encoding features output by each visual encoding layer for the ith sample video frame, and the set of image aggregation features is composed of image aggregation features corresponding to T sample video frames.

[0073] ​​The text aggregation feature can be represented as .

[0074] Optionally, the image encoding features output by each visual encoding layer in the visual encoder for any sampled video frame are aggregated by a feature aggregation network of the visual positioning model to obtain image aggregation features corresponding to any sampled video frame, an image aggregation feature set is generated based on the image aggregation features corresponding to each sampled video frame, and the text encoding features output by each text encoding layer in the text encoder for the query text are aggregated by the feature aggregation network of the visual positioning model to obtain text aggregation features corresponding to the query text.

[0075] In step S208, the image aggregation feature set is input into a spatio-temporal aggregation network of the visual positioning model to extract global visual features, and the image aggregation feature set and the text aggregation features are input into a cross-modal encoder of the visual positioning model to extract cross-modal visual features and cross-modal text features.

[0076] The spatio-temporal aggregation network can be used to aggregate the image aggregation feature set in spatial dimension features and temporal dimension features.

[0077] The global visual features can be visual encoding features describing global context .

[0078] The cross-modal encoder can be a cross-modal spatio-temporal encoder in a Video Grounding-DINO model.

[0079] The cross-modal visual features can be cross-modal features learned by adding cross-frame temporal information to original single-modal visual features.

[0080] The cross-modal text features can be cross-modal features learned by adding cross-frame temporal information to original single-modal text features.

[0081] Optionally, the server inputs the image aggregation feature set into the spatio-temporal aggregation network of the visual positioning model to aggregate the image aggregation feature set in spatial dimension features and temporal dimension features, to obtain global visual features describing global context , and inputs the image aggregation feature set and the text aggregation features ​The cross-modal encoder input to the visual positioning model adds time information across frames on the basis of original single-modal visual features and text features, models and learns cross-modal features, to improve time correlation and positioning accuracy, and outputs cross-modal text features and cross-modal visual features with features highly relevant to text prompts in each frame object detection.

[0082] In step S210, the global visual features, the cross-modal visual features and the cross-modal text features are input to a cross-modal decoder of the visual positioning model to obtain a video frame query result.

[0083] The cross-modal decoder can be a cross-modal spatio-temporal decoder in a Video Grounding-DINO model.

[0084] Optionally, the server inputs the global visual features, the cross-modal visual features and the cross-modal text features to the cross-modal decoder of the visual positioning model, embeds global event features into a query for global semantic consistency and local semantic specificity, and obtains the video frame query result, which can instantly utilize the entire visual features and text features as context, and is helpful for the model to locate the region and time domain of interest.

[0085] In step S212, the video frame query result is input to a coordinate prediction head and a time detection head of the visual positioning model to obtain a spatial positioning result and a temporal positioning result; the spatial positioning result represents a picture position of a target in a video to be positioned; and the temporal positioning result represents an occurrence time of the target in the video to be positioned.

[0086] The coordinate prediction head is composed of three layers of MLP.

[0087] The time detection head is composed of two layers of MLP.

[0088] The spatial positioning result can be detection bounding boxes of all detected targets. , The spatial positioning result is a collection of all object detection results.

[0089] The temporal positioning result can be a distribution of start and end events of all detected targets in the video to be positioned. .

[0090] Optionally, the server inputs the video frame query result to the coordinate prediction head and the time detection head of the visual positioning model to obtain the spatial positioning result and the temporal positioning result, and obtains detection bounding boxes of all targets and probability distribution of start and end times .

[0091] ​In the above visual positioning method, the video to be positioned is obtained, and a query text for the video to be positioned is obtained; the video to be positioned is input into a visual encoder of the visual positioning model to extract video frame features of the video to be positioned, and the query text is input into a text encoder of the visual positioning model to extract text features of the query text; the image aggregation feature set of the video to be positioned and the text aggregation feature of the query text are extracted based on the video frame features and the text features by a feature aggregation network of the visual positioning model; the global visual features are extracted by inputting the image aggregation feature set into a spatio-temporal aggregation network of the visual positioning model, and the cross-modal visual features and the cross-modal text features are extracted by inputting the image aggregation feature set and the text aggregation feature into a cross-modal encoder of the visual positioning model; the video frame query result is obtained by inputting the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder of the visual positioning model; the spatial positioning result and the temporal positioning result are obtained by inputting the video frame query result into a coordinate prediction head and a time detection head of the visual positioning model; the spatial positioning result represents the position of the target in the picture of the video to be positioned; the temporal positioning result represents the appearance time of the target in the video to be positioned; in this way, the query matching performance can be improved and the calculation overhead can be reduced by strengthening the features in two dimensions of vision and language, time and space, the processing capability of the model on complex queries can be covered by enhancing the visual features and the text features, a more fine-grained positioning can be realized, the possible ambiguity in video understanding can be eliminated by enhancing the spatio-temporal dependence of the video context, the performance loss caused by the expansion of the video length can be reduced, and the video segment matched with the query text can be accurately extracted according to the input query text.

[0092] In an example embodiment, the text encoder has a first preset number of text encoding layers, the text features of the query text include text encoding features output by each text encoding layer of the text encoder for the query text, and the image aggregation feature set of the video to be positioned and the text aggregation feature of the query text are extracted based on the video frame features and the text features by the feature aggregation network of the visual positioning model, including: grouping and aggregating the text encoding features output by each text encoding layer of the text encoder for the query text to obtain a plurality of text encoding feature aggregation results corresponding to the query text by the feature aggregation network; the feature splicing result corresponding to the query text is obtained by splicing the plurality of text encoding feature aggregation results and the text encoding features output by the last text encoding layer of the text encoder for the query text by the feature aggregation network; and the text aggregation feature corresponding to the query text is obtained by inputting the feature splicing result corresponding to the query text into a multi-layer perception of the feature aggregation network.

[0093] The first preset number can be set according to actual needs.

[0094] Among them, the number of text encoding feature aggregation results in multiple text encoding feature aggregation results is related to the feature aggregation method.

[0095] Optionally, see Figure 4 The network structure of the feature aggregation network shown is used by the server to combine the text encoding layers for the query text. The output text encoding features are grouped and aggregated. The text encoding features output by each text encoding layer are divided into: Groups, where each group contains The two adjacent text encoding features, namely If the first The encoded vector output by the layer is , will the The text encoding features within the group are added together to obtain the fusion vector. Ultimately, it will produce The text feature representation after group fusion can be expressed as: ;in, It can be set to 24. Setting it to 2 means that the text encoding features from layers 1-12 will be aggregated together, and the text encoding features from layers 13-24 will be aggregated together, thus obtaining the corresponding query text. The server then aggregates the group text encoding features and then... The text encoding feature aggregation results of the intermediate layer of the group are concatenated with the text encoding features output from the last text encoding layer, and then passed to the MLP layer to output the aggregated text encoding features. As query text The corresponding text aggregation features can be represented as:

[0096]

[0097] In this embodiment, the text encoding features output by each text encoding layer for the query text are grouped and aggregated by the feature aggregation network to obtain multiple text encoding feature aggregation results corresponding to the query text. The multiple text encoding feature aggregation results are then concatenated with the text encoding features output by the last text encoding layer of the text encoder for the query text to obtain the feature concatenation result corresponding to the query text. The feature concatenation result corresponding to the query text is then input into the multilayer perceptron of the feature aggregation network to obtain the text aggregation features corresponding to the query text. In this way, the dense features of adjacent layers in the text encoder can be integrated by the feature aggregation network, and the text features can be implicitly enhanced based on the information of the intermediate layers in the text encoder, so as to extract accurate text feature information.

[0098] In an exemplary embodiment, the visual encoder has a second preset number of visual coding layers. The video frame features of the video to be located include image coding feature sets corresponding to each sampled video frame of the video to be located. The image coding feature set corresponding to any sampled video frame includes the image coding features output by each visual coding layer in the visual encoder for the sampled video frame. Based on the video frame features and text features, the feature aggregation network of the visual localization model extracts the image aggregation feature set of the video to be located and the text aggregation features of the query text. This includes: grouping and aggregating the image coding features output by each visual coding layer for any sampled video frame through the feature aggregation network to obtain multiple image coding feature aggregation results corresponding to any sampled video frame; concatenating the multiple image coding feature aggregation results corresponding to any sampled video frame and the image coding features output by the last visual coding layer of the visual encoder for any sampled video frame through the feature aggregation network to obtain the feature concatenation result corresponding to any sampled video frame; inputting the feature concatenation result corresponding to any sampled video frame into the multilayer perceptron of the feature aggregation network to obtain the image aggregation features corresponding to any sampled video frame; and generating an image aggregation feature set based on the image aggregation features corresponding to each sampled video frame.

[0099] The second preset quantity can be set according to actual needs.

[0100] Among them, the number of image coding feature aggregation results in multiple image coding feature aggregation results is related to the feature aggregation method.

[0101] Optionally, see Figure 4 The network structure of the feature aggregation network shown describes how, for any sampled video frame, the server uses the feature aggregation network to group and aggregate the image coding features output by each visual coding layer for that sampled video frame. The features of the layer visual coding layer are divided into: Groups, where each group contains The image coding features of adjacent images, i.e. If the first The image encoding vector output by the layer is , will the The image encoding vectors within the group are added together to obtain the fusion vector. Ultimately, it will produce The image feature representation of group fusion can be expressed as: , here It can be set to 24. Setting it to 2 means that the feature vectors from layers 1-12 will be aggregated together, and the feature vectors from layers 13-24 will be aggregated together. The middle layer image encoding vector of the group is spliced with the image encoding vector output by the last visual encoding layer of the visual encoder in the channel dimension, and then is transmitted to the MLP layer to output the aggregated visual encoding feature , which can be expressed as , which can be expressed as The image aggregation feature of each sampling video frame is expressed as .

[0102] In this embodiment, the image encoding features output by each visual encoding layer for any sampling video frame are grouped and aggregated by the feature aggregation network to obtain a plurality of image encoding feature aggregation results corresponding to any sampling video frame; the plurality of image encoding feature aggregation results corresponding to any sampling video frame and the image encoding feature output by the last visual encoding layer of the visual encoder for any sampling video frame are spliced by the feature aggregation network to obtain a feature splicing result corresponding to any sampling video frame; the feature splicing result corresponding to any sampling video frame is input into the multi-layer perception of the feature aggregation network to obtain an image aggregation feature corresponding to any sampling video frame; and an image aggregation feature set is generated based on the image aggregation features corresponding to each sampling video frame; in this way, the adjacent layer dense features in the visual encoder can be integrated by the feature aggregation network, the image features can be implicitly enhanced based on the information of the middle layer of the visual encoder, and accurate image feature information can be extracted.

[0103] The above two embodiments realize dense aggregation of visual feature information and text feature information. The feature aggregation network integrates adjacent layer dense features, the model obtains richer semantic information with minimal computational overhead, and thus the expression ability of the network is improved. In a common model structure, only the last layer features of the frozen visual and text encoders are used for further processing, while in the present application, the information of the middle layer of the Transformer is used for implicit enhancement of the features, and a small amount of additional calculation is introduced. The feature aggregation network has good scalability and compatibility, the parameter configuration in the feature aggregation network allows flexible feature fusion according to task requirements, and can adapt to most existing visual and language tasks based on the Transformer architecture.

[0104] In an exemplary embodiment, the video to be positioned is input into the visual encoder of the visual positioning model to extract video frame features of the video to be positioned, including: uniformly sampling the video to be positioned sparsely to obtain each sampling video frame corresponding to the video to be positioned; inputting each sampling video frame into the visual encoder of the visual positioning model, and performing visual encoding processing on each sampling video frame by the visual encoder to obtain image encoding features corresponding to each sampling video frame; and taking the image encoding features corresponding to each sampling video frame as the video frame features of the video to be positioned.

[0105] The sampling video frame can refer to T frames of video frames obtained by uniformly and sparsely sampling the video to be positioned.

[0106] Optionally, for a video to be positioned, after uniformly and sparsely sampling the video to be positioned, T frames of sampling video frames are obtained. The image of each frame of the sampling video frame is encoded by the visual encoder to obtain image encoding features output by each visual encoding layer for each sampling video frame, and the image encoding features output by each visual encoding layer for each sampling video frame are taken as video frame features of the video to be positioned. Subsequently, the image encoding features output by each visual encoding layer for each sampling video frame are aggregated by a sampling feature aggregation network to obtain a visual feature sequence corresponding to each frame of the sampling video frame .

[0107] In this embodiment, by uniformly and sparsely sampling the video to be positioned, T frames of sampling video frames corresponding to the video to be positioned are obtained; the T frames of sampling video frames are input into a visual encoder of the visual positioning model, and the T frames of sampling video frames are visually encoded by the visual encoder to obtain image encoding features corresponding to the T frames of sampling video frames; the image encoding features corresponding to the T frames of sampling video frames are taken as video frame features of the video to be positioned; in this way, the T frames of sampling video frames can be accurately extracted and the image encoding features of the T frames of sampling video frames can be extracted.

[0108] In an exemplary embodiment, the image aggregated feature set is input into a spatio-temporal aggregation network of the visual positioning model to extract global visual features, including: inputting the image aggregated feature set in the form of a sequence into the spatio-temporal aggregation network of the visual positioning model, performing feature fusion by a first convolutional layer in the spatio-temporal aggregation network to obtain fused features; inputting the fused features into a spatial-temporal block in the spatio-temporal aggregation network to obtain spatial features and temporal features; the spatial-temporal block is formed by alternately connecting a plurality of spatial self-attention layers and a plurality of temporal self-attention layers in series; inputting the spatial features and the temporal features into a second convolutional layer in the spatio-temporal aggregation network to perform feature aggregation to obtain the global visual features.

[0109] The first convolutional layer can refer to a CNN layer on the input side of the spatio-temporal aggregation network, the second convolutional layer can refer to a CNN layer on the output side of the spatio-temporal aggregation network, and the spatial-temporal block can refer to a spatial-temporal block Spatial-Termporal Block formed by connecting a spatial self-attention layer and a temporal self-attention layer in series in the spatio-temporal aggregation network, see Figure 5 .

[0110] The fused features can be features obtained by the first convolutional layer performing feature fusion on the image aggregated feature set, i.e., the visual feature sequence .

[0111] Optionally, the server will sequence the visual features The spatial and temporal dimension features are aggregated through the spatio-temporal aggregation network to obtain global visual features describing the global context The spatio-temporal aggregation network is composed of multiple temporal and spatial attention. Specifically, the video feature sequence is input into the spatio-temporal aggregation network, first through the first convolutional layer for feature stacking fusion, and the fused features are input into the space-time block composed of multiple spatial and temporal self-attention layers in series. Here, the space-time block is set to 2, the spatial self-attention layer focuses on each time point, i.e. spatial dimension features, and the temporal self-attention focuses on temporal period, i.e. temporal dimension features, and finally input into the second convolutional layer to obtain the aggregated global visual features .

[0112] In this embodiment, the image aggregation feature set is input into the spatio-temporal aggregation network of the visual positioning model in the form of a sequence, the features are fused through the first convolutional layer in the spatio-temporal aggregation network, and the fused features are input into the space-time block in the spatio-temporal aggregation network to obtain spatial features and temporal features. The space-time block is composed of multiple spatial self-attention layers and multiple temporal self-attention layers in series. The spatial features and temporal features are input into the second convolutional layer in the spatio-temporal aggregation network for feature aggregation to obtain global visual features. In this way, the spatial dimension self-attention layer and the temporal dimension self-attention layer can be stacked to densely aggregate spatial and temporal feature information, so that the spatio-temporal aggregation network has the ability to extract dynamic dependency relationships of spatio-temporal features, and the expression of long-time sequence features is improved. Compared with traditional time series models such as Kalman filter, which is only suitable for short-term prediction, or graph neural network, which has an advantage in capturing time correlation but lacks the ability to handle complex scenes in highly dynamic space, the present embodiment can learn high-dimensional latent feature space in space-time by cascading spatial dimension and temporal dimension self-attention modules, obtain hidden information and global dependency relationships of long-time changes from local, and obtain more rich context semantic information, i.e. global visual features.

[0113] In an example embodiment, the video frame query result is input into a coordinate prediction head and a time detection head of the visual positioning model to obtain a spatial positioning result and a time positioning result, and the method comprises: inputting the video frame query result into the coordinate prediction head and the time detection head of the visual positioning model to obtain a probability distribution information corresponding to each target detection frame and a start and end time of each target detection frame; determining a joint probability between any target detection frame and a start and end time of any target detection frame according to the probability distribution information corresponding to each target detection frame and the start and end time of each target detection frame; taking the start and end time with the maximum joint probability as the time positioning result, and taking a target detection frame associated with the time positioning result as the spatial positioning result.

[0114] wherein the target detection frame can refer to a picture frame of a target to be detected in the video frame, and all target corresponding detection prediction frames can be obtained by inputting the video frame query result into the coordinate prediction head and the time detection head of the visual positioning model. forming a target detection frame set .

[0115] wherein the probability distribution information corresponding to the start and end time can refer to a probability distribution of the start and end time of the target appearing in the video, and can be expressed as .

[0116] Optionally, the server inputs the video frame query result into the coordinate prediction head and the time detection head, the coordinate prediction head is composed of three layers of MLP, the time detection head is composed of two layers of MLP, and all target detection prediction frames set and the distribution of the start and end time , wherein is the horizontal and vertical coordinates of the target frame center point, is the target frame index in the image, and a maximum of target detection is supported in an image, and is set to 900.

[0117] In this embodiment, the video frame query result is input into the coordinate prediction head and the time detection head of the visual positioning model to obtain the probability distribution information corresponding to each target detection frame and the start and end time of each target detection frame; the joint probability between any target detection frame and the start and end time of any target detection frame is determined according to the probability distribution information corresponding to each target detection frame and the start and end time of each target detection frame; the start and end time with the maximum joint probability is taken as the time positioning result, and a target detection frame associated with the time positioning result is taken as the spatial positioning result; in this way, complex spatiotemporal positioning of multiple targets, actions or relationships can be realized, for example, positioning a specific part of a target or identifying a specific action being performed by a target, and the positioning accuracy can be improved, the deviation and error of spatial and temporal positioning can be reduced, and the ambiguity that can occur at the appearance and disappearance time can be eliminated.

[0118] In an exemplary embodiment, a method for training a visual localization model is provided, and the specific steps include: inputting the training data into the visual localization model in the form of video-text pairs, and the Ground-Truth of each query is composed of a series of bounding boxes, including coordinates and start and end time stamps , and Gaussian distribution of the start point and the end point is constructed by the time stamps . For the spatial localization loss, the L1 loss of the coordinate box and the generalized intersection loss GIoU are divided, which can be expressed as . For the time localization loss, the KL divergence is used to measure the distance of the start and end time distribution, which can be expressed as . The linear combination of the two parts of the loss is used as the optimization function, which can be expressed as .

[0119] In an exemplary embodiment, the trained visual localization model needs to be model-inferred, and the specific steps include: inputting the test data into the visual localization model in the form of video-text pairs, obtaining a series of target bounding boxes and probability distribution of start and end time, locating the time segment according to the joint probability, and obtaining the target queried by the query text, wherein the time localization needs to be determined first, that is, the time period with the maximum joint probability of the two is obtained as the final time prediction result , and the target box in the time period is used as the spatial localization result.

[0120] The visual localization method of each embodiment in the present application can achieve the following beneficial effects as a whole:

[0121] (1) The visual localization method of the present application has less computational overhead than other models: in the multi-scale feature fusion stage of the feature aggregation network, the linear addition and splicing process of the features do not add new learnable parameters, and have almost no impact on the computational complexity; the MLP layer of the last connection has a fixed number of nodes due to the consistency of the output dimensions of the original text encoder and the visual encoder, and the additional computational amount is limited; in the space-time feature fusion stage of the space-time aggregation network, compared with other modules with the same effect, the computational cost is lower and the training speed is faster, and it is independent of the processing video time; therefore, under the same expression ability, the visual localization model of the present application has less computational cost than other models, thereby improving the optimization iteration speed of the model.

[0122] (2) The visual positioning method of the present application has higher feature expression capability than other models: the multi-dimensional feature aggregation network can help the visual positioning model better analyze the structure of the query and decompose it into subparts that can be matched with the video content, adapt to complex spatiotemporal positioning containing multiple targets, actions or relationships, etc., such as positioning the specific parts of the target or identifying the specific action the target is performing; and improve the positioning accuracy, reduce the deviation and error of spatial and temporal positioning, and eliminate the ambiguity that may occur at the appearance and disappearance moments; by fusing dense features of different levels of single modal, the influence of input data noise on the performance of the model is reduced, thereby improving the generalization; by fusing spatiotemporal features, the understanding of the context of the video is enhanced, including continuous dynamic changes and static information. At the same time, redundant information is removed during the training process, making the model more efficient and robust.

[0123] (3) The visual positioning method of the present application has scalability in processing the length of the video compared to other models: the visual positioning method of the present application alleviates the difficulties in long video tasks from two dimensions of computational complexity and time dependence; sparse sampling and spatiotemporal aggregation module controls the consumption of memory of the visual positioning model during the training process, and the calculation time and complexity will not increase significantly with the length of the video; the single modal feature aggregation network and the spatiotemporal aggregation network solve the problem of long-time information dependence to a certain extent, effectively capture and utilize the time domain information without sacrificing computational efficiency, thereby improving the efficiency and effect of long video processing.

[0124] In another embodiment, as shown in Figure 6 , a visual positioning method is provided, which is applied to the server 104 in Figure 1 for example, and includes the following steps:

[0125] Step S602, obtaining a video to be positioned, and obtaining a query text for the video to be positioned.

[0126] Step S604, inputting the video to be positioned into the visual encoder of the visual positioning model to extract the video frame features of the video to be positioned, and inputting the query text into the text encoder of the visual positioning model to extract the text features of the query text.

[0127] Step S606, extracting the image aggregation features set of the video to be positioned and the text aggregation features of the query text based on the video frame features and the text features by the feature aggregation network of the visual positioning model.

[0128] Step S608, inputting the image aggregation features set into the spatiotemporal aggregation network of the visual positioning model in the form of a sequence, performing feature fusion by the first convolutional layer in the spatiotemporal aggregation network to obtain the fused features.

[0129] Step S610, input the fused features into a space-time block in the space-time aggregation network to obtain spatial features and temporal features; the space-time block is formed by alternately connecting a plurality of spatial self-attention layers and a plurality of temporal self-attention layers.

[0130] Step S612, input the spatial features and the temporal features into a second convolutional layer in the space-time aggregation network to aggregate the features to obtain global visual features.

[0131] Step S614, input the image aggregated feature set and the text aggregated feature into a cross-modal encoder of the visual positioning model to extract cross-modal visual features and cross-modal text features.

[0132] Step S616, input the global visual features, the cross-modal visual features, and the cross-modal text features into a cross-modal decoder of the visual positioning model to obtain a video frame query result.

[0133] Step S618, input the video frame query result into a coordinate prediction head and a time detection head of the visual positioning model to obtain a spatial positioning result and a time positioning result; the spatial positioning result represents a picture position of the target in the video to be positioned; the time positioning result represents an occurrence time of the target in the video to be positioned.

[0134] It should be noted that the specific definitions of the above steps can refer to the specific definitions of the visual positioning method described above.

[0135] It should be understood that, although each step in the flowchart involved in each embodiment described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps has no strict sequence limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0136] Based on the same inventive concept, the embodiments of the present application also provide a visual positioning device for implementing the visual positioning method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific definitions in one or more visual positioning device embodiments provided below can refer to the definitions of the visual positioning method described above, and will not be repeated here.

[0137] In an exemplary embodiment, as Figure 7As shown, a visual positioning apparatus is provided, comprising: an acquisition module 702, a feature extraction module 704, a feature aggregation module 706, a feature encoding module 708, a feature decoding module 710 and a prediction module 712, wherein:

[0138] The acquisition module 702 is configured to acquire a video to be positioned, and acquire a query text for the video to be positioned.

[0139] The feature extraction module 704 is configured to input the video to be positioned into a visual encoder of a visual positioning model, extract video frame features of the video to be positioned, and input the query text into a text encoder of the visual positioning model, extract text features of the query text.

[0140] The feature aggregation module 706 is configured to extract, by a feature aggregation network of the visual positioning model, based on the video frame features and the text features, an image aggregation feature set of the video to be positioned and a text aggregation feature of the query text.

[0141] The feature encoding module 708 is configured to input the image aggregation feature set into a spatio-temporal aggregation network of the visual positioning model, extract global visual features, and input the image aggregation feature set and the text aggregation feature into a cross-modal encoder of the visual positioning model, extract cross-modal visual features and cross-modal text features.

[0142] The feature decoding module 710 is configured to input the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder of the visual positioning model, to obtain a video frame query result.

[0143] The prediction module 712 is configured to input the video frame query result into a coordinate prediction head and a time detection head of the visual positioning model, to obtain a spatial positioning result and a temporal positioning result; the spatial positioning result represents a picture position of a target in the video to be positioned; and the temporal positioning result represents an appearance time of the target in the video to be positioned.

[0144] In one embodiment, the text encoder has a first preset number of text encoding layers, the text features of the query text include text encoding features output by each text encoding layer of the text encoder for the query text, the feature aggregation module 706 is specifically configured to group and aggregate the text encoding features output by each text encoding layer of the text encoder for the query text through the feature aggregation network, to obtain a plurality of text encoding feature aggregation results corresponding to the query text; splice the plurality of text encoding feature aggregation results and the text encoding feature output by the last text encoding layer of the text encoder for the query text through the feature aggregation network, to obtain a feature splicing result corresponding to the query text; and input the feature splicing result corresponding to the query text into a multi-layer perception of the feature aggregation network, to obtain the text aggregation feature corresponding to the query text.

[0145] In one of the embodiments, the visual encoder has a second preset number of visual encoding layers, the video frame features of the video to be located include image encoding feature sets corresponding to each sampled video frame of the video to be located, the image encoding feature set corresponding to any sampled video frame includes image encoding features output by each visual encoding layer of the visual encoder for the sampled video frame, the feature aggregation module 706 is specifically configured to group and aggregate the image encoding features output by each visual encoding layer for any sampled video frame to obtain a plurality of image encoding feature aggregation results corresponding to the sampled video frame through a feature aggregation network; splice the plurality of image encoding feature aggregation results corresponding to any sampled video frame and the image encoding features output by the last visual encoding layer of the visual encoder for the sampled video frame to obtain a feature splicing result corresponding to the sampled video frame through the feature aggregation network; and input the feature splicing result corresponding to any sampled video frame into a multi-layer perception of the feature aggregation network to obtain image aggregation features corresponding to the sampled video frame; and generate an image aggregation feature set based on the image aggregation features corresponding to each sampled video frame.

[0146] In one of the embodiments, the feature extraction module 704 is specifically configured to perform sparse uniform sampling on the video to be located to obtain each sampled video frame corresponding to the video to be located; input each sampled video frame into a visual encoder of the visual positioning model to perform visual encoding processing on each sampled video frame through the visual encoder to obtain image encoding features corresponding to each sampled video frame; and use the image encoding features corresponding to each sampled video frame as the video frame features of the video to be located.

[0147] In one of the embodiments, the feature encoding module 708 is specifically configured to input the image aggregation feature set into a space-time aggregation network of the visual positioning model in a sequence form, perform feature fusion through a first convolutional layer in the space-time aggregation network to obtain fused features; input the fused features into a space-time block in the space-time aggregation network to obtain spatial features and temporal features; the space-time block is formed by alternately connecting a plurality of spatial self-attention layers and a plurality of temporal self-attention layers; and input the spatial features and the temporal features into a second convolutional layer in the space-time aggregation network to perform feature aggregation to obtain global visual features.

[0148] In one of the embodiments, the prediction module 712 is specifically configured to input the video frame query result into a coordinate prediction head and a time detection head of the visual positioning model to obtain probability distribution information corresponding to each target detection frame and start and end time of each target detection frame; determine a joint probability between any target detection frame and start and end time of the target detection frame according to the probability distribution information corresponding to each target detection frame and start and end time of each target detection frame; use the start and end time with the maximum joint probability as the time positioning result, and use the target detection frame associated with the time positioning result as the spatial positioning result.

[0149] The modules in the visual positioning apparatus can be implemented by software, hardware, or a combination thereof. The modules can be embedded in a processor in a computer device or independent of the processor, or stored in a memory in the computer device to be invoked by the processor to perform the operations of the modules.

[0150] In an example embodiment, a computer device, which can be a server, is provided. An internal structure diagram of the computer device can be as shown in Figure 8 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected by a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store visual positioning data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is configured to be executed by the processor to implement a visual positioning method.

[0151] Those skilled in the art can understand that Figure 8 The structure shown in the above

[0152] In an example embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The computer program is configured to be executed by the processor to cause the processor to perform the steps of the visual positioning method. The steps of the visual positioning method can be the steps of the visual positioning method in any of the above embodiments.

[0153] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is configured to be executed by a processor to cause the processor to perform the steps of the visual positioning method. The steps of the visual positioning method can be the steps of the visual positioning method in any of the above embodiments.

[0154] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, causes the processor to perform the steps of a visual positioning method as described above. The steps of a visual positioning method can be the steps of a visual positioning method in any of the embodiments described above.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0156] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0157] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method of visual positioning, characterized by The method comprises: acquiring a video to be positioned and acquiring query text for the video to be positioned; inputting the video to be positioned into a visual encoder of a visual positioning model to extract video frame features of the video to be positioned, and inputting the query text into a text encoder of the visual positioning model to extract text features of the query text; extracting, by a feature aggregation network of the visual positioning model, image aggregated features of the video to be positioned and text aggregated features of the query text based on the video frame features and the text features; inputting the image aggregated features into a spatio-temporal aggregation network of the visual positioning model to extract global visual features, and inputting the image aggregated features and the text aggregated features into a cross-modal encoder of the visual positioning model to extract cross-modal visual features and cross-modal text features; the inputting the image aggregated features into the spatio-temporal aggregation network comprises: inputting the image aggregated features into the spatio-temporal aggregation network in a sequence form, performing feature fusion by a first convolutional layer in the spatio-temporal aggregation network to obtain fused features; inputting the fused features into a spatial-temporal block in the spatio-temporal aggregation network to obtain spatial features and temporal features; the spatial-temporal block is formed by a plurality of spatial self-attention layers and a plurality of temporal self-attention layers which are alternately connected in series; and inputting the spatial features and the temporal features into a second convolutional layer in the spatio-temporal aggregation network to perform feature aggregation to obtain the global visual features; inputting the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder of the visual positioning model to obtain video frame query results; inputting the video frame query results into a coordinate prediction head and a time detection head of the visual positioning model to obtain spatial positioning results and time positioning results; the spatial positioning results represent a picture position of a target in the video to be positioned; and the time positioning results represent an occurrence time of the target in the video to be positioned.

2. The method of claim 1, wherein, The text encoder has a first preset number of text encoding layers, the text features of the query text comprise text encoding features output by each text encoding layer of the text encoder for the query text, and the extracting, by the feature aggregation network of the visual positioning model, the image aggregated features of the video to be positioned and the text aggregated features of the query text based on the video frame features and the text features comprises: grouping and aggregating, by the feature aggregation network, the text encoding features output by each text encoding layer of the text encoder for the query text to obtain a plurality of text encoding feature aggregation results corresponding to the query text; splicing, by the feature aggregation network, the plurality of text encoding feature aggregation results and a text encoding feature output by a last text encoding layer of the text encoder for the query text to obtain a feature splicing result corresponding to the query text; and inputting the feature splicing result corresponding to the query text into a multi-layer perceptron of the feature aggregation network to obtain the text aggregated features corresponding to the query text.

3. The method of claim 1, wherein, The visual encoder has a second preset number of visual encoding layers, the video frame features of the video to be located include image encoding feature sets corresponding to each sample video frame of the video to be located, the image encoding feature set corresponding to any sample video frame includes image encoding features output by each visual encoding layer in the visual encoder for the sample video frame, and the feature aggregation network of the visual positioning model extracts image aggregation features of the video to be located and text aggregation features of the query text based on the video frame features and the text features, including: Grouping and aggregating the image encoding features output by each visual encoding layer for any sample video frame through the feature aggregation network to obtain a plurality of image encoding feature aggregation results corresponding to any sample video frame; Splicing the plurality of image encoding feature aggregation results corresponding to any sample video frame and the image encoding features output by the last visual encoding layer of the visual encoder for any sample video frame through the feature aggregation network to obtain a feature splicing result corresponding to any sample video frame; Inputting the feature splicing result corresponding to any sample video frame into the multi-layer perception of the feature aggregation network to obtain image aggregation features corresponding to any sample video frame; Generating the image aggregation feature set based on the image aggregation features corresponding to each sample video frame.

4. The method of claim 1, wherein, The visual encoder of the visual positioning model is used to input the video to be located to extract video frame features of the video to be located, including: Performing sparse uniform sampling on the video to be located to obtain each sample video frame corresponding to the video to be located; Inputting each sample video frame into the visual encoder of the visual positioning model to perform visual encoding processing on each sample video frame through the visual encoder to obtain image encoding features corresponding to each sample video frame; Taking the image encoding features corresponding to each sample video frame as the video frame features of the video to be located.

5. The method of claim 1, wherein, The video frame query result is input into the coordinate prediction head and the time detection head of the visual positioning model to obtain a spatial positioning result and a time positioning result, and the method includes: Inputting the video frame query result into the coordinate prediction head and the time detection head of the visual positioning model to obtain each target detection frame and probability distribution information corresponding to the start and end time of each target detection frame; Determining the joint probability between any target detection frame and the start and end time of any target detection frame according to the probability distribution information corresponding to each target detection frame and the start and end time of each target detection frame; Taking the start and end time with the maximum joint probability as the time positioning result, and taking the target detection frame associated with the time positioning result as the spatial positioning result.

6. A visual positioning device, characterized by The device includes: An acquisition module is configured to acquire a video to be located and a query text for the video to be located. The feature extraction module is configured to input the video to be located into a visual encoder of a visual localization model, extract video frame features of the video to be located, and input the query text into a text encoder of the visual localization model, extract text features of the query text. The feature aggregation module is configured to extract, by a feature aggregation network of the visual localization model, an image aggregation feature set of the video to be located and text aggregation features of the query text based on the video frame features and the text features. The feature encoding module is configured to input the image aggregation feature set into a spatio-temporal aggregation network of the visual localization model, extract global visual features, and input the image aggregation feature set and the text aggregation features into a cross-modal encoder of the visual localization model, extract cross-modal visual features and cross-modal text features. The feature encoding module is specifically configured to input the image aggregation feature set into the spatio-temporal aggregation network of the visual localization model in a sequence form, perform feature fusion by a first convolutional layer in the spatio-temporal aggregation network to obtain fused features, input the fused features into a spatial-temporal block in the spatio-temporal aggregation network to obtain spatial features and temporal features, and perform feature aggregation by a second convolutional layer in the spatio-temporal aggregation network to obtain the global visual features. The feature decoding module is configured to input the global visual features, the cross-modal visual features and the cross-modal text features into a cross-modal decoder of the visual localization model to obtain video frame query results. The prediction module is configured to input the video frame query results into a coordinate prediction head and a time detection head of the visual localization model to obtain spatial localization results and time localization results. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images