Video positioning method and device, video positioning model training method and device, equipment and medium

Abstract

The invention discloses a video positioning method and device, a video positioning model training method and device, equipment and a medium. The method comprises the following steps: performing feature extraction on to-be-queried information of a target video to obtain text features; performing feature extraction on the target video to obtain video features; encoding the text feature to obtain a first shared feature; encoding the video feature to obtain a second shared feature aligned with the first shared feature in dimension; fusing the first shared feature and the second shared feature to obtain a fused feature; and performing regression detection on the fusion feature to obtain the positioning time of the to-be-queried information in the target video. According to the embodiment of the invention, the video information positioning accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a video positioning method and device, a video positioning model training method and device, equipment and a medium. BACKGROUND

[0002] Moment retrieval refers to searching and locating the specific start and end moments of an event or action in a video given a text description of the event or action.

[0003] Existing moment retrieval is usually a simple superposition of video information and text, which loses a lot of detailed information, resulting in low accuracy of the detection result. SUMMARY

[0004] The present application provides a video positioning method and device, a video positioning model training method and device, and equipment and a medium, which can improve the accuracy of video information positioning.

[0005] According to an aspect of the present application, the present application embodiment provides a video positioning method, which comprises:

[0006] extracting features of the to-be-queried information of the target video to obtain text features;

[0007] extracting features of the target video to obtain video features;

[0008] encoding the text features to obtain first shared features;

[0009] encoding the video features to obtain second shared features aligned with the dimension of the first shared features;

[0010] fusing the first shared features and the second shared features to obtain fused features;

[0011] performing regression detection on the fused features to obtain the positioning time of the to-be-queried information in the target video.

[0012] According to another aspect of the present application, the present application embodiment further provides a video positioning model training method, which comprises:

[0013] extracting features of the to-be-queried information of the sample video through a text feature extraction module of a video positioning model to obtain text features;

[0014] extracting features of the sample video through a video feature extraction module of the video positioning model to obtain video features;

[0015] encoding the text features through a text encoding module of the video positioning model to obtain first shared features;

[0016] The video features are encoded using the video encoding module of the video localization model to obtain a second shared feature aligned with the dimension of the first shared feature.

[0017] The fusion module of the video positioning model fuses the first shared feature and the second shared feature to obtain the fused feature.

[0018] The regression module of the video positioning model performs regression detection on the fused features to obtain the positioning time of the information to be queried in the sample video.

[0019] The parameters of the video positioning model are adjusted based on the positioning time.

[0020] According to one aspect of the present invention, an embodiment of the present invention provides a video positioning device, the device comprising:

[0021] The text feature extraction module of the video localization model is used to extract features from the query information of the target video to obtain text features;

[0022] The video feature extraction module of the video localization model is used to extract features from the target video to obtain video features;

[0023] The text encoding module of the video localization model is used to encode the text features to obtain the first shared features;

[0024] The video encoding module of the video localization model is used to encode the video features to obtain a second shared feature aligned with the first shared feature dimension.

[0025] The fusion module of the video positioning model is used to fuse the first shared feature and the second shared feature to obtain fused features;

[0026] The regression module of the video positioning model is used to perform regression detection on the fused features to obtain the positioning time of the information to be queried in the target video.

[0027] According to another aspect of the present invention, embodiments of the present invention also provide a training apparatus for a video localization model, the apparatus comprising:

[0028] The text feature extraction module of the video localization model is used to extract features from the query information of the sample video to obtain text features;

[0029] The video feature extraction module of the video localization model is used to extract features from the sample video to obtain video features;

[0030] The text encoding module of the video localization model is used to encode the text features to obtain the first shared features;

[0031] The video encoding module of the video localization model is used to encode the video features to obtain a second shared feature aligned with the first shared feature dimension.

[0032] The fusion module of the video positioning model is used to fuse the first shared feature and the second shared feature to obtain fused features;

[0033] The regression module of the video localization model is used to perform regression detection on the fused features to obtain the localization time of the information to be queried in the sample video.

[0034] The adjustment module is used to adjust the parameters of the video positioning model according to the positioning time.

[0035] According to another aspect of the present invention, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0036] At least one processor; and

[0037] A memory that is communicatively connected to at least one processor; wherein,

[0038] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to execute the video localization method or the training method of the video localization model according to any embodiment of the present invention.

[0039] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the video localization method or the training method of the video localization model of any embodiment of the present invention.

[0040] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the video positioning method or the training method for the video positioning model described in any embodiment of the present invention.

[0041] The technical solution of this invention extracts features from the target video and the query information. The extracted video and text features are encoded and aligned to obtain a first shared feature and a second shared feature with the same dimension. These two shared features are then fused to obtain a fused feature. Regression detection is performed on the fused feature to locate the query information in the target video. This achieves the mapping and fusion of text and video features into the same space, solving the problem of low retrieval accuracy caused by the loss of details due to simple superposition of video and text in existing technologies. It fully fuses text and video features, increasing the richness of feature content and improving the representativeness of the fused features. Furthermore, video localization based on the fused features can improve the accuracy of video localization.

[0042] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a video positioning method provided according to an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of a training method for a video localization model according to an embodiment of the present invention;

[0046] Figure 3 This is a scene diagram illustrating a training method for a video localization model according to an embodiment of the present invention;

[0047] Figure 4 This is a structural diagram of a video positioning device provided according to an embodiment of the present invention;

[0048] Figure 5 This is a structural diagram of a training device for a video localization model according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of an electronic device that implements a video positioning method or a video positioning model training method provided in an embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] The acquisition, storage, and application of driving trajectory points and other related technologies in the technical solutions of this invention comply with relevant laws and regulations and do not violate public order and good morals.

[0053] Figure 1 This is a flowchart illustrating a video localization method provided by an embodiment of the present invention. This embodiment is applicable to situations where the location of a text-described event is determined within a video. The method can be executed by a video localization device, which can be implemented in hardware and / or software.

[0054] See Figure 1 The video localization method shown includes:

[0055] S101. Extract features from the query information of the target video to obtain text features.

[0056] The target video can refer to captured images of a specific scene. The query information can refer to descriptive content describing the object to be queried. In some embodiments, the target video can refer to security surveillance video or user authentication video, etc. The query information can refer to information about an object performing an action. The query information can refer to natural language content, which is usually text information. The query information can be a single character, a word, or a sentence. A text feature extraction module can be used to extract text features from the query information to obtain text features. The dimensions of the text features can include one-dimensional, two-dimensional, or three-dimensional features.

[0057] In some embodiments, feature extraction can be performed on the query information to obtain a one-dimensional vector. The text feature of a text fragment (character or word, etc.) is represented as an n×1 two-dimensional vector, equivalent to an n one-dimensional vector. The text feature of a sentence composed of multiple text fragments can be represented as an n×m two-dimensional vector, equivalent to a combined two-dimensional vector of text features. The text feature after adding time information can be represented as an n×m×t three-dimensional vector. Here, n represents the number of rows, m represents the number of columns, and t represents time.

[0058] S102. Extract features from the target video to obtain video features.

[0059] This process involves splitting the target video into multiple images, extracting features from each image, and obtaining video features. Video features include image features and temporal features. The dimensions of video features can be two-dimensional, three-dimensional, or four-dimensional.

[0060] In some embodiments, feature extraction is performed directly on the video to obtain a two-dimensional vector of video features. The features of an image can be obtained as an n×m two-dimensional vector, while the video features of a video are obtained by combining the features of multiple images to obtain an n×m×k three-dimensional vector. The video features after adding time information can be represented as an n×m×k×t four-dimensional vector. Here, n represents the number of rows, m represents the number of columns, k represents the number of channels (number of images), and t represents time.

[0061] S103. Encode the text features to obtain the first shared features.

[0062] In this process, text features are encoded to map text features and video features to the same vector space. The first shared feature can refer to the mapped text features.

[0063] S104. Encode the video features to obtain a second shared feature that is aligned with the dimension of the first shared feature.

[0064] In this process, video features are encoded to map text and video features into the same vector space. The second shared feature can refer to the mapped video features.

[0065] In some embodiments, feature extraction and encoding can be implemented using an encoder with the same structure, or encoding can be implemented using an encoder while feature extraction can be implemented using a feature extraction network with a different structure than the encoder.

[0066] S105. The first shared feature and the second shared feature are fused to obtain the fused feature.

[0067] In this embodiment, the first shared feature and the second shared feature have the same dimension, and can be fused to obtain a fused feature. In some embodiments, the first shared feature and the second shared feature can be concatenated and fused. In some embodiments, the first shared feature and the second shared feature can be cumulatively fused along the same dimension. In some embodiments, an attention mechanism can be used to fuse the fused feature.

[0068] S106. Perform regression detection on the fused features to obtain the location time of the information to be queried in the target video.

[0069] The fused features include rich, multi-dimensional information from multiple modalities. Temporal localization detection based on these fused features can improve localization accuracy. Localization time can refer to the video time within the target video. In some embodiments, the target video is actually a series of video frames arranged in a temporal sequence, or the localization time can be represented by the identification information of the video frames within the target video.

[0070] This invention combines multimodal data of textual and visual information to perform video localization of textual information, thereby enabling natural language localization to pinpoint specific actions or activities.

[0071] In some embodiments, within the financial industry, video positioning can enable more personalized financial services. By integrating natural language descriptions and visual information, it can uncover more correlations and trends between data points, helping financial institutions obtain more information through services and thus make more informed decisions. Simultaneously, video positioning can also be applied to intelligent security monitoring systems, enabling rapid location and automatic detection of potential security risks through employee verbal descriptions and video surveillance, and issuing warnings.

[0072] The technical solution of this invention extracts features from the target video and the query information. The extracted video and text features are encoded and aligned to obtain a first shared feature and a second shared feature with the same dimension. These two shared features are then fused to obtain a fused feature. Regression detection is performed on the fused feature to locate the query information in the target video. This achieves the mapping and fusion of text and video features into the same space, solving the problem of low retrieval accuracy caused by the loss of details due to simple superposition of video and text in existing technologies. It fully fuses text and video features, increasing the richness of feature content and improving the representativeness of the fused features. Furthermore, video localization based on the fused features can improve the accuracy of video localization.

[0073] In an optional embodiment, the step of extracting features from the target video to obtain video features includes: splitting the target video to obtain multiple video images; extracting features from each of the video images to obtain image features of each video image; and combining the image features of each of the video images to obtain video features.

[0074] The process involves sampling the target video to obtain multiple video images and the temporal sequence of each video image within the target video. Feature extraction is performed on each video image individually, yielding its image features. These image features are then sequentially concatenated along the channels according to their temporal sequence within the target video to form the video features.

[0075] As can be seen, by splitting the target video, features can be extracted from each video image and combined to form video features. Detail information can be extracted from images at multiple times in the target video, enriching the content of the video features and increasing their representativeness.

[0076] In an optional embodiment, the video features are four-dimensional, the text features are three-dimensional, and both the first shared feature and the second shared feature are three-dimensional.

[0077] The image features of a single video image can be represented as an n×m two-dimensional vector. Combining the image features of various video images yields an n×m×k three-dimensional vector. Adding temporal information, the video features of the target video can be represented as an n×m×k×t four-dimensional vector. Encoding the video features is equivalent to dimensionality reduction, resulting in a three-dimensional second shared feature. The query information is segmented into words, yielding at least one text fragment. The features formed by these text fragments are one-dimensional vectors. Fusing these text fragments yields the sentence features, which are two-dimensional vectors. Adding temporal information increases the feature dimension, resulting in an n×m×t three-dimensional text feature vector.

[0078] The dimensionality of video features can be reduced to the same level as that of text features. At the same time, the dimensions of video features and text features can be aligned. The aligned video features are the second shared features, and the aligned text features are the first shared features.

[0079] It is evident that by configuring video features as four-dimensional, video feature fusion achieves both global and temporal features; by configuring text features as three-dimensional, text feature fusion achieves temporal features; and by configuring the first and second shared features as three-dimensional, video features and text features are mapped to the same vector space, increasing the information richness of the fused features and thus improving the accuracy of video localization.

[0080] In an optional embodiment, the step of splitting the target video to obtain multiple video images includes: splitting the target video to obtain multiple video frames; clustering each video frame to obtain at least one video group and a center video frame in each video group; and filtering each video frame according to each video group and the center video frame in each video group to obtain multiple video images.

[0081] In this context, a video frame can refer to an image directly obtained by splitting the target video. The target video can be sampled to obtain video frames. These video frames are then clustered to obtain at least one video group. A video group is typically a combination of video frames with similar content. Video frames not belonging to a video group are usually independent and considered invalid, and can be filtered out. The center video frame within a video group can be the video frame representing the group. The center video frame can also be the cluster center.

[0082] In some embodiments, video groups can be pre-filtered based on the content of the central video frame to remove redundant video groups, resulting in filtered video groups. In some embodiments, for the retained video groups, similar video frames are filtered out from the video frames other than the central video frame, and the retained video frames are used as video images. In some embodiments, the central video frame in the retained video group is used as a video image. In some embodiments, video frames that do not belong to a video group can be filtered out.

[0083] It is evident that by filtering video frames from the target video segmentation to obtain video images, redundant data can be reduced, the proportion of effective information in video features can be increased, and the representativeness of video features can be improved, thereby enhancing the accuracy of video positioning.

[0084] In an optional embodiment, encoding the text features to obtain a first shared feature includes: obtaining historical encoding results of at least one historical query information; encoding the text features to obtain a new encoding result; and fusing the historical encoding result and the new encoding result to obtain the first shared feature.

[0085] This allows for continuous video localization. Historical query information refers to past queries targeting the target video. This historical query information can be added as context to the first shared feature. A single query can be understood as input for one round of dialogue. Multiple queries represent multiple rounds of dialogue.

[0086] For the current query information, the text features extracted from the query information are encoded to obtain a new encoding result. This new encoding result can be obtained by aligning the text features of the query information with the video features.

[0087] After locating the target video based on the query information, the query information can be identified as historical query information, and the first shared feature generated from the query information can be identified as the historical encoding result of the historical query information. The historical encoding result is also the encoding result after being aligned with the video features of the target video.

[0088] In some embodiments, the weights of each historical query information and the current query information can be preset. The historical encoding results and the new encoding results are then weighted and fused according to their respective weights to obtain the first shared feature. In some embodiments, the current query information has the highest weight. The smaller the input time difference between the historical query information and the current query information, the higher its weight. The larger the input time difference between the historical query information and the current query information, the lower its weight. Here, the input time difference can refer to the time difference between inputs of query information.

[0089] It is evident that by adding historical query information and fusing the historical encoding results of the historical query information with the new encoding results of the query information to obtain the first shared feature, the contextual information in the first shared feature is added to enrich the first shared feature, and video localization is performed based on the rich first shared feature, thereby improving the accuracy of video localization.

[0090] In an optional embodiment, the information to be queried includes: a description of the action, and the location time includes the start time and the end time.

[0091] The action description text refers to the description of the action of the object being queried. The start time is before the end time. The time period between the start and end times exists in the corresponding video segment of the target video, containing the video content describing the information to be queried.

[0092] In some embodiments, the information to be queried is "entering through the door".

[0093] In some embodiments, it can be applied to target videos collected by monitoring and acquisition modules to locate query information on abnormal behavior in the target videos. It can quickly find video segments with abnormal behavior in the target videos, thereby performing anomaly handling, such as issuing security warnings to improve scene security.

[0094] As can be seen, by configuring the information to be queried as action description text and the positioning time as start and end time, the application scenarios of video positioning can be further refined, enabling accurate content positioning and expanding the scope of application scenarios.

[0095] Figure 2This is a flowchart illustrating a training method for a video localization model provided in an embodiment of the present invention. The embodiment of the present invention is applicable to training scenarios for video localization models used to determine the location of an event described in text within a video. This method can be executed by a training device for the video localization model, which can be implemented in hardware and / or software.

[0096] See Figure 2 The training method for the video localization model shown includes:

[0097] S201. The text feature extraction module of the video positioning model extracts features from the information to be queried in the sample video to obtain text features.

[0098] The video localization model processes the input target video and query information, outputting the location time of the query information within the target video. A large number of training samples are used to train the video localization model. Each training sample includes a sample video and the query information for that video. In supervised training scenarios, the training sample may also include the time annotations in the sample video.

[0099] The video localization model can include a text feature extraction module, a video feature extraction module, a text encoding module, a video encoding module, a fusion module, and a regression module. The text feature extraction module is used to extract features from the query information to obtain text features.

[0100] S202. The video feature extraction module of the video positioning model is used to extract features from the sample video to obtain video features.

[0101] The video feature extraction module is used to extract features from sample videos to obtain video features.

[0102] S203. The text features are encoded using the text encoding module of the video positioning model to obtain the first shared features.

[0103] The text encoding module is used to encode text features so that the text features are aligned with video features to obtain the first shared features after alignment.

[0104] S204. The video features are encoded by the video encoding module of the video positioning model to obtain a second shared feature aligned with the dimension of the first shared feature.

[0105] The video encoding module is used to encode video features so that the video features are aligned with the text features, resulting in an aligned second shared feature.

[0106] S205. The first shared feature and the second shared feature are fused through the fusion module of the video positioning model to obtain the fused feature.

[0107] The fusion module is used to fuse the first shared feature and the second shared feature, and output the fused feature.

[0108] S206. Using the regression module of the video positioning model, regression detection is performed on the fused features to obtain the positioning time of the information to be queried in the sample video.

[0109] The regression module processes the fused features and outputs the localization time. In some embodiments, the regression module performs nonlinear processing on the fused features to obtain the localization time.

[0110] S207. Adjust the parameters of the video positioning model according to the positioning time.

[0111] In some embodiments, the video localization model can be trained unsupervised based on the localization time. In some embodiments, the video localization model can be trained in a supervised manner based on the difference between the localization time and the annotation time.

[0112] This invention extracts features from sample videos and the information to be queried. The extracted video and text features are encoded and aligned to obtain first and second shared features of the same dimension. These first and second shared features are then fused to obtain fused features. Regression detection is performed on the fused features to locate the information to be queried in the sample video. This process maps text and video features to the same space for fusion detection, yielding the location time. Based on the location time, the parameters of the video localization model are adjusted, allowing the model to acquire richer features and improving the time-based localization accuracy of the video localization model.

[0113] In an optional embodiment, adjusting the parameters of the video positioning model based on the positioning time includes: calculating a first difference between the positioning time and the annotation time of the information to be queried in the sample video; performing data augmentation on the sample video and processing the augmented sample video using the video positioning model to obtain the augmentation time output by the video positioning model; calculating a second difference between the positioning time and the augmentation time; and adjusting the parameters of the video positioning model based on the first difference and the second difference.

[0114] The first difference describes the discrepancy between the localization time output by the video localization model and the ground truth. Data augmentation refers to adjusting the input data and feeding it into the video localization model so that the model can still correctly predict even with augmented input data. Data augmentation enhances the generalization ability of the video localization model. Augmentation time refers to the time of the query information output by the video localization model in the augmented sample video. The second difference describes the discrepancy between the output of the original data and the output of the augmented data from the video localization model. In practice, if the sample video is augmented by scrambling the order, although the time of each video image in the augmented sample video will change, the time of the video image containing the query information in the sample video before scrambling should still be the same as the time in the sample video before scrambling. Essentially, the localization time of the query information in the augmented sample video and the sample video are the same. Therefore, the augmentation time should be the same as the localization time, allowing the calculation of the second difference to enable the video localization model to learn the ability to locate the query information in different forms of sample videos.

[0115] In some embodiments, the first difference and the second difference can be fused to obtain the loss value of the video localization model, and the parameters of the video localization model can be adjusted based on the loss value. In some embodiments, fusion may include accumulation or weighted summation.

[0116] In some embodiments, data augmentation can be performed on the video images segmented from the sample video. Data augmentation methods may include at least one of the following: scrambling the video images segmented from the sample video, rotating at least one video image, removing foreground from at least one video image, and removing background from at least one video image.

[0117] There can be at least one data augmentation method, and each data augmentation method will result in an augmentation time. The number of corresponding augmentation times is at least one, and the number of second differences is at least one. The loss value can be calculated by accumulating or weighting the second differences.

[0118] It is evident that by calculating the first difference between the localization time and the annotation time of the sample video, performing data augmentation on the sample video, predicting the augmentation time based on the data-augmented sample video, calculating the second difference between the augmentation time and the localization time, and training the video localization model based on the first and second differences, the generalization ability of the video localization model can be enhanced, enabling the video localization model to have high video localization accuracy for videos in different situations.

[0119] In an optional embodiment, the sample video is data augmented, and the augmented sample video is processed using the video localization model; the parameters of the video localization model are then adjusted. Specifically, data augmentation of the sample video and unsupervised training of the video localization model can also enhance the generalization ability of the video localization model.

[0120] In some embodiments, the step of extracting features from the sample video to obtain video features includes: splitting the sample video to obtain multiple video images; extracting features from each of the video images to obtain image features of each video image; and combining the image features of each video image to obtain video features.

[0121] In an optional embodiment, splitting the sample video to obtain multiple video images includes: splitting the sample video to obtain multiple video frames; clustering each video frame to obtain at least one video group and a center video frame in each video group; and filtering each video frame according to each video group and the center video frame in each video group to obtain multiple video images.

[0122] In an optional embodiment, encoding the text features to obtain a first shared feature includes: obtaining historical encoding results of at least one historical query information; encoding the text features to obtain a new encoding result; and fusing the historical encoding result and the new encoding result to obtain the first shared feature.

[0123] In an optional embodiment, the video features are four-dimensional, the text features are three-dimensional, and both the first shared feature and the second shared feature are three-dimensional.

[0124] In an optional embodiment, the information to be queried includes: a description of the action, and the location time includes the start time and the end time.

[0125] In a scenario, such as Figure 3As shown, the text is the query information, and the video is the sample video. The query information is processed through text feature extraction to obtain text features. These text features are then encoded to map to a shared feature space, resulting in the first shared feature. The sample video is processed through video feature extraction to obtain video features. These video features are then encoded to map to a shared feature space, resulting in the second shared feature. The first and second shared features are fused to obtain the fused feature. Regression prediction is performed on the fused feature to obtain the start and end times, which are then determined as the localization time. The video localization model is then trained unsupervised, or supervised based on the first difference between the localization time and the labeled time. Alternatively, data augmentation can be performed on the video. Video features are extracted from the data-augmented sample video to obtain video features. These video features are then encoded to map to a shared feature space, resulting in the third shared feature. The first and third shared features are then fused to obtain the fused feature. Regression prediction is performed on the fused feature to obtain the start and end times, which are then determined as the augmentation time. The video localization model is then trained unsupervised, or supervised based on the second difference between the localization time and the augmentation time and the previously calculated first difference.

[0126] In fact, since a video is composed of multiple video images, the images are cut into multiple blocks and input into the feature encoder in various orders during the processing of the video images. After processing by the feature encoder, multiple feature vectors are obtained. In order to better describe the features of the video, the multiple feature vectors are jointly processed into a feature vector containing global features, local features and temporal features, which can obtain richer contextual knowledge.

[0127] In the feature encoder section, to align video and text features, the feature vectors extracted in the previous step need to be projected into the same vector space. Video vectors can be input into the video encoding module in different dimensions (via matrix transpose) to obtain video features capable of parsing multiple dimensions of information. After the first and second shared features extracted by encoding, a self-attention layer is added to capture the relationships between and within features, aiming to obtain richer contextual information.

[0128] After acquiring rich contextual information, two feature vectors are fused to obtain multimodal fusion features. Based on these multimodal fusion features, multi-scale evaluation criteria are set to calculate the loss function, aiming to obtain more accurate time predictions. This includes supervised regression prediction as well as data augmentation through unsupervised or self-supervised methods to further enhance prediction accuracy. This invention can better facilitate the rapid location of a specific moment in high-quality long videos even with limited computing power.

[0129] In one example, the collected surveillance video is used as the target video, and the information to be queried is: a person pushes a door open and enters. The trained video localization model outputs the start time (xx minutes xx seconds) and end time (xx minutes xx seconds) of the action of pushing the door open and entering. By using multimodal data, the action can be quickly located, avoiding the cost of manual retrieval.

[0130] Figure 4 This is a schematic diagram of a video positioning device provided in an embodiment of the present invention. The present invention is applicable to situations where the location of a text-described event in a video is determined. The device can execute a video positioning method and can be implemented in hardware and / or software.

[0131] See Figure 4 The video positioning device shown includes:

[0132] The text feature extraction module 401 of the video localization model is used to extract features from the query information of the target video to obtain text features;

[0133] The video feature extraction module 402 of the video localization model is used to extract features from the target video to obtain video features;

[0134] The text encoding module 403 of the video positioning model is used to encode the text features to obtain the first shared features;

[0135] The video encoding module 404 of the video localization model is used to encode the video features to obtain a second shared feature aligned with the first shared feature dimension.

[0136] The fusion module 405 of the video positioning model is used to fuse the first shared feature and the second shared feature to obtain fused features;

[0137] The regression module 406 of the video positioning model is used to perform regression detection on the fused features to obtain the positioning time of the information to be queried in the target video.

[0138] The technical solution of this invention extracts features from the target video and the query information. The extracted video and text features are encoded and aligned to obtain a first shared feature and a second shared feature with the same dimension. These two shared features are then fused to obtain a fused feature. Regression detection is performed on the fused feature to locate the query information in the target video. This achieves the mapping and fusion of text and video features into the same space, solving the problem of low retrieval accuracy caused by the loss of details due to simple superposition of video and text in existing technologies. It fully fuses text and video features, increasing the richness of feature content and improving the representativeness of the fused features. Furthermore, video localization based on the fused features can improve the accuracy of video localization.

[0139] Optionally, the video feature extraction module 402 is specifically used for:

[0140] The target video is split into multiple video images;

[0141] Feature extraction is performed on each of the video images to obtain the image features of each of the video images;

[0142] The image features of each video image are combined to obtain video features.

[0143] Optionally, the video feature extraction module 402 is specifically used for:

[0144] The target video is split into multiple video frames;

[0145] Cluster the video frames to obtain at least one video group and the central video frame in each video group;

[0146] Based on each video group and the central video frame in each video group, each video frame is filtered to obtain multiple video images.

[0147] Optional, text encoding module 403, specifically used for:

[0148] Retrieve the historical encoding result of at least one historical query information;

[0149] The text features are encoded to obtain a new encoding result;

[0150] The historical encoding results and the new encoding results are fused to obtain the first shared feature.

[0151] Optionally, the video features are four-dimensional, the text features are three-dimensional, and both the first shared feature and the second shared feature are three-dimensional.

[0152] Optionally, the information to be queried includes: a description of the action, and the location time includes the start time and the end time.

[0153] The video positioning device provided in the embodiments of the present invention can execute the video positioning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the video positioning method.

[0154] Figure 5 This is a schematic diagram of a training device for a video localization model provided in an embodiment of the present invention. The embodiment of the present invention is applicable to the training of video localization models. This device can execute the training method for the video localization model and can be implemented in hardware and / or software.

[0155] See Figure 5 The training device for the video localization model shown includes:

[0156] The text feature extraction module 501 of the video localization model is used to extract features from the query information of the sample video to obtain text features;

[0157] The video feature extraction module 502 of the video localization model is used to extract features from the sample video to obtain video features;

[0158] The text encoding module 503 of the video positioning model is used to encode the text features to obtain the first shared features;

[0159] The video encoding module 504 of the video localization model is used to encode the video features to obtain a second shared feature aligned with the first shared feature dimension.

[0160] The fusion module 505 of the video positioning model is used to fuse the first shared feature and the second shared feature to obtain fused features;

[0161] The regression module 506 of the video positioning model is used to perform regression detection on the fused features to obtain the positioning time of the information to be queried in the sample video.

[0162] The adjustment module 507 is used to adjust the parameters of the video positioning model according to the positioning time.

[0163] This invention extracts features from sample videos and the information to be queried. The extracted video and text features are encoded and aligned to obtain first and second shared features of the same dimension. These first and second shared features are then fused to obtain fused features. Regression detection is performed on the fused features to locate the information to be queried in the sample video. This process maps text and video features to the same space for fusion detection, yielding the location time. Based on the location time, the parameters of the video localization model are adjusted, allowing the model to acquire richer features and improving the time-based localization accuracy of the video localization model.

[0164] Optionally, adjustment module 507 is specifically used for:

[0165] Calculate the first difference between the location time and the annotation time of the information to be queried in the sample video;

[0166] The sample video is augmented with data, and the augmented sample video is processed using the video localization model to obtain the augmentation time output by the video localization model.

[0167] Calculate the second difference between the positioning time and the enhancement time;

[0168] Based on the first difference and the second difference, the parameters of the video positioning model are adjusted.

[0169] The training device for the video localization model provided in this embodiment of the invention can execute the training method for the video localization model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the training method for the video localization model.

[0170] Figure 6 A schematic diagram of the structure of an electronic device 600 that can be used to implement an embodiment of the present invention is shown.

[0171] like Figure 6As shown, the electronic device 600 includes at least one processor 601 and a memory, such as a read-only memory 602 or a random access memory 603, communicatively connected to the at least one processor 601. The memory stores computer programs executable by the at least one processor. The processor 601 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 602 or loaded from storage unit 608 into the random access memory 603. The random access memory 603 may also store various programs and data required for the operation of the electronic device 600. The processor 601, read-only memory 602, and random access memory 603 are interconnected via a bus 604. An input and / or output interface 605 is also connected to the bus 604.

[0172] Multiple components in electronic device 600 are connected to input and / or output interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] Processor 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 601 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 601 performs the various methods and processes described above, such as video localization methods or methods for training video localization models.

[0174] In some embodiments, the video localization method or the training method for a video localization model may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 600 via read-only memory 602 and / or communication unit 609. When the computer program is loaded into random access memory 603 and executed by processor 601, one or more steps of the video localization method or the training method for a video localization model described above may be performed. Alternatively, in other embodiments, processor 601 may be configured to perform the video localization method or the training method for a video localization model by any other suitable means (e.g., by means of firmware).

[0175] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0177] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on an operational detection device. This electronic device includes: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0179] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0180] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system. This addresses the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0181] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0182] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video positioning method, characterized in that, The method includes: Feature extraction is performed on the query information of the target video to obtain text features; Feature extraction is performed on the target video to obtain video features; The text features are encoded to obtain the first shared features; The video features are encoded to obtain a second shared feature aligned with the dimension of the first shared feature; The first shared feature and the second shared feature are fused to obtain the fused feature; Regression detection is performed on the fused features to obtain the location time of the information to be queried in the target video.

2. The method according to claim 1, characterized in that, The step of extracting features from the target video to obtain video features includes: The target video is split into multiple video images; Feature extraction is performed on each of the video images to obtain the image features of each of the video images; The image features of each video image are combined to obtain video features.

3. The method according to claim 2, characterized in that, The process of splitting the target video to obtain multiple video images includes: The target video is split into multiple video frames; Cluster the video frames to obtain at least one video group and the central video frame in each video group; Based on each video group and the central video frame in each video group, each video frame is filtered to obtain multiple video images.

4. The method according to claim 1, characterized in that, The encoding of the text features to obtain the first shared feature includes: Retrieve the historical encoding result of at least one historical query information; The text features are encoded to obtain a new encoding result; The historical encoding results and the new encoding results are fused to obtain the first shared feature.

5. The method according to claim 1, characterized in that, The video features are four-dimensional, the text features are three-dimensional, and both the first shared feature and the second shared feature are three-dimensional.

6. The method according to claim 1, characterized in that, The information to be queried includes: a description of the action, and the location time includes: start time and end time.

7. A training method for a video localization model, characterized in that, The method includes: The text feature extraction module of the video localization model extracts features from the query information of the sample video to obtain text features. The video feature extraction module of the video localization model extracts features from the sample video to obtain video features; The text features are encoded using the text encoding module of the video localization model to obtain the first shared feature; The video features are encoded using the video encoding module of the video localization model to obtain a second shared feature aligned with the dimension of the first shared feature. The fusion module of the video positioning model fuses the first shared feature and the second shared feature to obtain the fused feature. The regression module of the video positioning model performs regression detection on the fused features to obtain the positioning time of the information to be queried in the sample video. The parameters of the video positioning model are adjusted based on the positioning time.

8. The method according to claim 7, characterized in that, The step of adjusting the parameters of the video positioning model based on the positioning time includes: Calculate the first difference between the location time and the annotation time of the information to be queried in the sample video; The sample video is augmented with data, and the augmented sample video is processed using the video localization model to obtain the augmentation time output by the video localization model. Calculate the second difference between the positioning time and the enhancement time; Based on the first difference and the second difference, the parameters of the video positioning model are adjusted.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video localization method of any one of claims 1-6, or the training method of the video localization model of any one of claims 7-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video localization method of any one of claims 1-6, or the training method of the video localization model of any one of claims 7-8.

Images