Video text retrieval method based on temporal token merging

Abstract

The application discloses a video text retrieval method and device based on timing token merging, equipment and storage medium, belongs to cross-modal retrieval technical field, including: obtaining video data to be matched and text data to be matched, and extracting a plurality of continuous image frames from the video data; determining the block token data sequence of the video data according to each image frame, and determining the text feature data of the text data; the block token data sequence is used as a sampling data set, the statistical mean of the video data is determined as the block token average data of the video data; the cross-modal similarity of the block token average data and the text feature data is determined as the matching result of the video data and the text data. The problem of too large image data token merging data operation in the video text retrieval process is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cross-modal retrieval, and particularly relates to a video text retrieval method and device based on time sequence token merging, equipment and a storage medium. BACKGROUND

[0002] In the field of video language understanding, text video retrieval is a key task aiming to match videos corresponding to a specific query text or vice versa. With the progress of multi-modal learning, especially the success of text-image pre-training models in cross-modal tasks, text video retrieval technology has attracted extensive attention. By applying pre-trained image and text encoders to video data, researchers can achieve significant results in the text video retrieval task.

[0003] In the prior art, many studies use image text encoders (Contrastive Language-Image Pre-Training, CLIP) as the backbone and optimize the text video retrieval performance through a comprehensive fine-tuning method. And in order to improve efficiency, a token compression method is used to reduce the computational burden by reducing the number of tokens while retaining key information.

[0004] However, due to the large number of tokens in the video, the existing token compression method still faces the problem of high inference cost when processing video data, and fails to effectively handle the temporal redundancy in the video, resulting in high computational overhead. SUMMARY

[0005] The application aims to provide a video text retrieval method and device based on time sequence token merging, equipment and a storage medium, which at least solves the problem of slow matching speed caused by large data operation amount of image data token merging in the video text retrieval process.

[0006] In a first aspect, the embodiments of the application disclose a video text retrieval method based on time sequence token merging, comprising:

[0007] obtaining video data to be matched and text data to be matched, and extracting a plurality of continuous image frames from the video data;

[0008] determining a block token data sequence of the video data according to each of the image frames, and determining text feature data of the text data; the block token data sequence contains a plurality of block token data of the video data;

[0009] determine a statistical mean of the video data as block token average data of the video data, by taking the block token data sequence as a sampling data set; a statistical weight of the statistical mean is determined according to a token similarity between block token data in the block token data sequence; the token similarity is negatively related to a data difference degree between two block token data;

[0010] determine a cross-modal similarity between the block token average data and the text feature data as a matching result of the video data and the text data; the cross-modal similarity is positively related to a matching degree of the video data and the text data.

[0011] In a second aspect, an embodiment of the present application further discloses a video-text retrieval device based on time sequence token merging, comprising:

[0012] a data acquisition module, configured to acquire video data to be matched and text data to be matched, and extract a plurality of continuous image frames from the video data;

[0013] a feature generation module, configured to determine a block token data sequence of the video data according to each of the image frames, and determine text feature data of the text data; the block token data sequence comprises a plurality of block token data of the video data;

[0014] an averaging module, configured to determine a statistical mean of the video data as block token average data of the video data, by taking the block token data sequence as a sampling data set; a statistical weight of the statistical mean is determined according to a token similarity between block token data in the block token data sequence; the token similarity is negatively related to a data difference degree between two block token data;

[0015] a data matching module, configured to determine a cross-modal similarity between the block token average data and the text feature data as a matching result of the video data and the text data; the cross-modal similarity is positively related to a matching degree of the video data and the text data.

[0016] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0017] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0018] In summary, in the embodiment of the present application, the statistical weight is determined according to the similarity between the block token data, which can more accurately reflect the feature distribution of the video data. Then, the statistical mean of the block token data sequence is calculated by using the statistical weight, which effectively reduces the redundancy of the data and extracts the representative video features. Finally, the cross-modal similarity between the block token average data and the text feature data is calculated, and the cross-modal similarity is positively correlated with the matching degree. Therefore, the reliability and accuracy of the matching result are improved, and the effective matching of the video and text data is realized. Thus, the token synthesis speed is accelerated based on the method of the embodiment of the present application, and the problem of slow matching speed caused by large image data token merging data operation in the related art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the drawings:

[0020] Figure 1 is a step flow chart of a video text retrieval method based on time sequence token merging provided by an embodiment of the present application;

[0021] Figure 2 is a step flow chart of another video text retrieval method based on time sequence token merging provided by an embodiment of the present application;

[0022] Figure 3 is an execution logic schematic diagram of a data processing model under the method provided by an embodiment of the present application;

[0023] Figure 4 is a further explanation of the execution logic of the data processing model in the embodiment of the present application;

[0024] Figure 5 is a block diagram of a video text retrieval device based on time sequence token merging provided by an embodiment of the present application;

[0025] Figure 6 is a block diagram of an electronic device of one embodiment provided by an embodiment of the present application;

[0026] Figure 7 is a block diagram of an electronic device of another embodiment provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.

[0029] Figure 1 is a video text retrieval method based on timing token merging provided by the present embodiment.

[0030] The method can include the following steps:

[0031] Step 101, obtaining video data to be matched and text data to be matched, and extracting a plurality of continuous image frames from the video data.

[0032] In some embodiments of the present application, the video data and text data to be matched are obtained, and a plurality of continuous image frames are extracted from the video data. The video data is usually composed of a series of continuous image frames, which capture the temporal changes of the video content. The text data is descriptive text related to the video content. By extracting continuous image frames from the video, the video content can be decomposed into multiple static images, facilitating subsequent feature extraction and matching processing.

[0033] For example, the video clip and the corresponding text description can be obtained first, such as "in the forest, birds are singing on the branches". Then, 300 continuous image frames (10 seconds x 30 frames / second = 300 frames) are extracted from some video clips. These image frames will be used for subsequent determination of block token data sequence and feature extraction of text data to achieve matching of video and text.

[0034] Step 102, determining the block token data sequence of the video data according to each image frame, and determining the text feature data of the text data.

[0035] The block token data sequence contains a plurality of block token data of the video data.

[0036] In some embodiments of the present application, a block token data sequence of the video data is determined according to each image frame, and text feature data of the text data is determined. The block token data sequence is generated by dividing each image frame into a plurality of fixed-size image blocks and converting the image blocks into corresponding block token data. The text feature data is obtained by tokenizing and encoding the text data. The block token data sequence and the text feature data are used for subsequent cross-modal similarity calculation to achieve matching of the video and the text.

[0037] For example, assume that a frame of image extracted from a video has a size of 224x224 pixels. The image can be first divided into 14x14 image blocks, each having a size of 16x16 pixels. Then, each image block is converted into a block token data, generating a block token data sequence containing 196 block token data. For the text data "In the forest, birds are singing on the branches of the trees", it is converted into a series of text feature data by a text encoder. Finally, the obtained block token data sequence of the video data and the text feature data of the text data are used for subsequent matching process.

[0038] In step 103, the block token data sequence is taken as a sampling data set, and a statistical mean of the video data is determined as block token average data of the video data.

[0039] The statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively related to the data difference between two block token data.

[0040] In some embodiments of the present application, the block token data sequence is taken as a sampling data set, and a statistical mean of the video data is determined as block token average data of the video data. The calculation of the statistical mean is based on the similarity between the block token data, and the statistical weight is determined by the similarity between the block token data. The higher the similarity, the greater the statistical weight; the lower the similarity, the smaller the statistical weight. In this way, the feature distribution of the video data can be more accurately reflected, data redundancy can be reduced, and representative video features can be extracted.

[0041] For example, assume that the block token data sequence extracted from a video contains 8 image frames, each containing 196 block token data, and a total of 1568 block token data. First, the similarity between each pair of block token data is calculated using cosine similarity measure. Then, the statistical weight of each block token data is determined according to the similarity. Next, the statistical mean of the block token data sequence is calculated using the statistical weights, obtaining the block token average data of the video data. This average data will be used for subsequent cross-modal similarity calculation to improve the accuracy and efficiency of video and text matching.

[0042] At step 104, the cross-modal similarity between the block token average data and the text feature data is determined as the matching result of the video data and the text data.

[0043] The cross-modal similarity is positively correlated with the matching degree of the video data and the text data.

[0044] In some embodiments of the present application, the cross-modal similarity between the block token average data and the text feature data is determined as the matching result of the video data and the text data. The cross-modal similarity is an index for measuring the matching degree between the video data and the text data. By calculating the similarity between the block token average data and the text feature data, the correlation between the video content and the text description can be evaluated. The higher the similarity, the higher the matching degree of the video data and the text data, thereby improving the reliability and accuracy of the matching result. In the actual retrieval process, multiple videos can also be matched with a certain text, or multiple texts can be matched with a certain video, and the matching degree can be sorted according to the cross-modal similarity to obtain a selectable retrieval result.

[0045] For example, assuming that the block token average data of the video data is a vector v avg , and the text feature data is a vector t feat . Then the similarity between v avg and t feat can be calculated using the cosine similarity, and the formula is as follows:

[0046] Similarity = (v avg · t feat ) / ||v avg ||· ||t feat ||;

[0047] Where v avg · t feat represents the dot product of two vectors, and ||v avg || and ||t feat || represent the lengths of the two vectors, respectively. The calculated similarity value is used as the matching result of the video data and the text data. Assuming that the calculation result is 0.85, it indicates that the matching degree of the video content and the text description is high, and thus it can be considered that the video and the text description are related.

[0048] In summary, in the embodiments of the present application, the statistical weight is determined according to the similarity between the block token data, which can more accurately reflect the feature distribution of the video data. Then, the statistical mean of the block token data sequence is calculated by using the statistical weight, which effectively reduces the redundancy of the data and extracts the representative video features. Finally, the cross-modal similarity between the block token average data and the text feature data is calculated, and the cross-modal similarity is positively correlated with the matching degree. Therefore, the reliability and accuracy of the matching result are improved, and the effective matching of the video and text data is realized. Thus, the token synthesis speed is accelerated based on the method of the embodiments of the present application, and the problem of slow matching speed caused by large data operation amount of image data token merging in the related art is solved.

[0049] Figure 3 Another video text retrieval method based on time sequence token merging is provided for the embodiments of the present application. The method can include the following steps:

[0050] In step 201, the video data to be matched and the text data to be matched are obtained, and a plurality of continuous image frames are extracted from the video data.

[0051] The method shown in this step has been described in step 101, and will not be repeated here.

[0052] In step 202, the block token data sequence of the video data is determined according to each image frame, and the text feature data of the text data is determined.

[0053] The block token data sequence includes a plurality of block token data of the video data.

[0054] The method shown in this step has been described in step 102, and will not be repeated here.

[0055] Optionally, in order to determine the block token data sequence of the video data according to each image frame, step 202 includes the following sub-steps:

[0056] In sub-step 2021, each image frame is divided into a plurality of image blocks.

[0057] In some embodiments of the present application, each image frame is divided into a plurality of image blocks. The image block is to divide the image frame into a plurality of rectangular regions of fixed size, and each region is called an image block. Through this division, the large image frame can be converted into a plurality of small blocks, which is convenient for subsequent feature extraction and processing. Each image block will be part of the subsequent block token data, which is used for feature representation of the video data.

[0058] For example, assume that a frame of image extracted from a video has a size of 224x224 pixels. First, the image frame is divided into 14x14 image blocks, each with a size of 16x16 pixels. In this way, the entire image frame is divided into 196 image blocks. Each image block will be treated as an independent unit for subsequent convolution processing and feature extraction to form block token data. This segmentation method helps to capture local features in the image frame and lays the foundation for the generation of subsequent block token data sequences.

[0059] Sub-step 2022, each image block is subjected to convolution processing to obtain the block feature value corresponding to each image block, and each block feature value corresponding to the image frame is taken as a component of the block token data corresponding to the image frame to obtain the block token data corresponding to the image frame.

[0060] In some embodiments of the present application, each image block will be subjected to convolution processing to obtain the block feature value corresponding to each image block. Convolution processing is a commonly used image processing technique that extracts features from image blocks using a convolution kernel to generate block feature values. The block feature value of each image block will be taken as a component of the block token data corresponding to the image frame. In this way, local information in the image frame can be converted into a feature vector, facilitating subsequent feature matching and processing.

[0061] For example, assume that a frame of image extracted from a video has been divided into 14x14 image blocks, each with a size of 16x16 pixels. Each image block is subjected to convolution processing, for example, a 3x3 convolution kernel can be used with a step size of 1 and a padding of 1. After convolution processing, each image block generates a feature vector, assuming the dimension of the feature vector is 128. The feature vector of each image block is taken as part of the block token data, and finally a block token data containing 196 block feature values is formed. These block token data will be used in the subsequent generation and matching process of the block token data sequence.

[0062] Sub-step 2023, sort all block token data in the time order of the corresponding image frame to obtain a block token data sequence.

[0063] In some embodiments of the present application, all block token data will be sorted in the time order of the corresponding image frame to obtain a block token data sequence. By sorting in time order, the temporal relationship between video frames can be maintained, ensuring that the block token data sequence accurately reflects the temporal changes of the video content. This step is crucial for subsequent video feature extraction and matching processes, as it preserves the temporal information of the video and helps improve the accuracy of cross-modal similarity calculation.

[0064] For example, assume that the chunk token data extracted from the video contains 8 image frames, each containing 196 chunk token data. Then the chunk token data of each image frame can be arranged in the time order of the frames first. Assume that the time order of the image frames is frame 1, frame 2, frame 3, …, frame 8. The 196 chunk token data of frame 1 is arranged at the front, followed by the 196 chunk token data of frame 2, and so on, until the 196 chunk token data of frame 8. Finally, a chunk token data sequence containing 1568 chunk token data is formed, which will be used in the subsequent video feature extraction and matching process.

[0065] Optionally, to determine the text feature data of the text data, step 202 includes the following sub-steps:

[0066] Sub-step 2024, taking the number of each token in the text data as a component of the text vector of the text data to obtain the text vector of the text data.

[0067] In some embodiments of the present application, the number of each token in the text data is taken as a component of the text vector of the text data to obtain the text vector of the text data. The text vector is a numerical representation of the text data, and by counting the number of occurrences of each token in the text data, a vector can be generated, where each component corresponds to the number of a token. This representation method helps to capture the word frequency information of the text data, providing a basis for subsequent feature extraction and matching.

[0068] For example, assume that the text data is "In the forest, birds sing on the branches". First, the text data is tokenized to obtain the token list: ["in", "forest", "in", "birds", "on", "branches", "sing"]. Then, count the number of occurrences of each token in the text data to generate the text vector: [2, 1, 1, 1, 2, 1, 1, 1]. Among them, "in" appears 2 times, and other tokens each appear 1 time. This text vector will be used as a numerical representation of the text data for subsequent text feature extraction and matching process.

[0069] Sub-step 2025, inputting the text vector into the pre-trained text feature model to obtain the text feature data of the text data.

[0070] In some embodiments of the present application, the text vector is input into the pre-trained text feature model to obtain the text feature data of the text data. The pre-trained text feature model is a deep learning model trained on a large amount of text data, which can convert the text vector into high-dimensional feature representation. These feature representations capture the semantic information of the text, which helps to improve the accuracy of text and video data matching. By inputting the text vector, the model can generate text feature data for subsequent cross-modal similarity calculation.

[0071] For example, assume that the generated text vector is [2, 1, 1, 1, 2, 1, 1, 1], representing the text "In the forest, birds are singing on the branches of the trees". The text vector is input into a pre-trained text feature model, such as Bidirectional Encoder Representations from Transformers (BERT). The model processes the text vector and generates a high-dimensional feature vector, with a dimension of 768, for example. This high-dimensional feature vector is the text feature data of the text data, which contains the semantic information of the text and will be used in the subsequent video-text matching process.

[0072] In step 203, the block token data sequence is taken as a sampling data set, and the statistical mean of the video data is determined as the block token average data of the video data.

[0073] The statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively related to the data difference degree between the two block token data.

[0074] The method shown in this step has been described in step 103, and will not be repeated here.

[0075] Optionally, step 203 includes the following sub-steps:

[0076] In sub-step 2031, according to the statistical weight of the block token data in the block token data sequence, the first target block token data pair in the block token data sequence is merged to obtain a first updated sequence.

[0077] The first target block token data pair is two block token data in the block token data sequence with a data difference degree ranking greater than a preset first ranking threshold.

[0078] In some embodiments of the present application, the first target block token data pair in the block token data sequence is merged according to the statistical weight of the block token data in the block token data sequence to obtain a first updated sequence. The first target block token data pair refers to two block token data in the block token data sequence with a data difference degree ranking greater than a preset first ranking threshold. The data difference degree is an index for measuring the similarity between two block token data, and the smaller the difference degree, the higher the similarity. By merging these similar block token data, data redundancy can be reduced and computational efficiency can be improved.

[0079] For example, assume that the block token data sequence contains 10 block token data, T1, T2, …, T10, and the statistical weight of each block token data is 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, respectively. 10Then the difference degree between each pair of block token data will be calculated first, and ranked according to the difference degree. Assuming that the preset first ranking threshold is 2, which means selecting the block token data pairs with the top 2 difference degrees. Assuming that the block token data pairs with the top 5 difference degrees are (T1, T2), (T3, T4). Then, these block token data pairs are merged respectively to generate new block token data T 1,2 3,4 , and form the first update sequence [T 1,2 3,4 , T5, T6, …, T 10 ]. This update sequence will be used for further merging and feature extraction in the subsequent.

[0080] Optionally, substep 2031 includes the following substeps:

[0081] Substep 20311, according to the preset grouping number, divide the block token data in the block token data sequence into multiple block token data subsequences, and randomly divide the block token data in each block token data subsequence into two groups to obtain the same number of first subgroups and second subgroups as the number of block token data subsequences.

[0082] In some embodiments of the present application, according to the preset grouping number, the block token data in the block token data sequence will be divided into multiple block token data subsequences, and the block token data in each block token data subsequence will be randomly divided into two groups to obtain the same number of first subgroups and second subgroups as the number of block token data subsequences. Through this grouping, the large-scale block token data sequence can be decomposed into smaller subsequences, which is convenient for subsequent processing and calculation. Randomly dividing into two groups ensures that the data in each subsequence is evenly distributed, which helps to improve the efficiency and accuracy of data processing.

[0083] For example, assuming that the block token data sequence contains 16 block token data, and the preset grouping number is 4. Then first, the 16 block token data will be divided into 4 block token data subsequences, each containing 4 block token data. Then, the 4 block token data in each subsequence will be randomly divided into two groups, namely the first subgroup and the second subgroup. Then, the 4 block token data in the first subsequence are randomly divided, and the first subgroup contains 2 block token data, and the second subgroup also contains 2 block token data. In this way, 4 first subgroups and 4 second subgroups are obtained, which lays the foundation for subsequent difference degree calculation and data merging.

[0084] Substep 20312, determine the first data difference degree between each block token data in each first subgroup and each block token data in each second subgroup, and determine the same number of target first data difference degrees as the preset first ranking index number in the first data difference degree. ​​

[0085] In some embodiments of the present application, the first data difference degrees between each block token data in each first sub-group and each block token data in each second sub-group are determined, and in the first data difference degrees, target first data difference degrees same as the preset first ranking index number are determined. The data difference degree is an index for measuring the similarity between two block token data, which is usually calculated using methods such as cosine similarity or Euclidean distance. By determining the target first data difference degrees, the most similar block token data pairs can be selected, which provides a basis for subsequent data merging.

[0086] For example, assume that the first sub-group contains block token data A1, A2, …, A8, and the second sub-group contains block token data B1, B2, …, B8. Then, the first data difference degrees between each A i and each B j are calculated first, for example, the cosine similarity calculation formula can be used:

[0087] First data difference degree (A i , B j ) = (A i ·B j ) / (||A i ||·||B j ||);

[0088] Then, 28 difference degree values are obtained. Assume that the preset first ranking index number is 4, indicating that the first 4 target first data difference degrees with the smallest difference degrees are selected. Assume that the block token data pairs corresponding to the 4 target first data difference degrees are (A1, B3), (A2, B5), and (A4, B1). These data pairs will be used as target block token data pairs for subsequent data merging.

[0089] Sub-step 20313, the block token data of the first sub-group and the block token data of the second sub-group corresponding to each target first data difference degree are selected as a group of first target block token data.

[0090] In some embodiments of the present application, the block token data of the first sub-group and the block token data of the second sub-group corresponding to each target first data difference degree are selected as a group of first target block token data. By selecting these data pairs, the most similar block token data pairs can be determined, which provides a basis for subsequent data merging. Each group of first target block token data contains one block token data from the first sub-group and one block token data from the second sub-group, and these data pairs will be merged in subsequent steps.

[0091] For example, assume that the target first data difference degree determined in the previous step corresponds to the block token data pairs (A1, B3), (A2, B5), and (A4, B1). Then these data pairs will be selected as the first target block token data. Specifically, the first target block token data will include the following data pairs:

[0092] First target block token data pair 1: A1 and B3,

[0093] First target block token data pair 2: A2 and B5,

[0094] First target block token data pair 3: A4 and B1;

[0095] These data pairs will serve as the basis for subsequent data merging, further reducing data redundancy and improving computational efficiency.

[0096] In sub-step 20314, according to the corresponding block token data sub-sequence, the time sequence correction parameter for each block token data in the block token data sub-sequence is generated, and the block token data in each block token sub-sequence is corrected through the corresponding time sequence correction parameter.

[0097] wherein the correction parameter is obtained by training the pre-trained frame position embedding engine.

[0098] In some embodiments of the present application, according to the corresponding block token data sub-sequence, the time sequence correction parameter for each block token data in the block token data sub-sequence is generated, and the block token data in each block token sub-sequence is corrected through the corresponding time sequence correction parameter. The time sequence correction parameter is a parameter for adjusting the time information of the block token data, ensuring that the temporal relationship between video frames is preserved during the merging process. The correction parameter is obtained by training the pre-trained frame position embedding engine, which can capture the temporal features of video frames and improve the accuracy of video feature extraction.

[0099] For example, assume that the block token data sub-sequence contains 6 block token data, T1, T2, …, T6. Then first use the pre-trained frame position embedding engine to generate time sequence correction parameters P1, P2, …, P6, which correspond to the time positions of each block token data. Then, correct each block token data, as follows: i T i ′=T i (i=1, 2, …, 6), where T i ′ is the corrected block token data. Through this correction, the time information of the block token data can be preserved, ensuring that the temporal relationship between video frames is preserved during the subsequent merging process. The corrected block token data will be used in the subsequent merging and feature extraction process.

[0100] Sub-step 20315, using each of the modified first target block token data as sample data to obtain a second arithmetic mean vector, and using the obtained second arithmetic mean vector as the first updated block token data.

[0101] In some embodiments of the present application, each of the modified first target block token data is used as sample data to obtain a second arithmetic mean vector, and the obtained second arithmetic mean vector is used as the first updated block token data. By calculating the arithmetic mean vector of each of the first target block token data, a vector representing the overall characteristics of the data in the group can be generated. This arithmetic mean vector will be used as the first updated block token data for subsequent data processing and feature extraction.

[0102] For example, assuming that the pairs of modified first target block token data in the modified block token data (T1', T2',..., T 16 ′) are (T1', T3'), (T2', T5'), and (T4', T6'). Then, first, the average of the two block token data in each data pair is calculated as follows:

[0103] T 1,3 = (T1' + T3') / 2,

[0104] T 2,5 = (T2' + T5') / 2,

[0105] T 4,6 = (T4' + T6') / 2.

[0106] Through this calculation, three second arithmetic mean vectors T 1,3 , T 2,5 , and T 4,6 are obtained. These average vectors will be used as the first updated block token data to represent the overall characteristics of the modified first target block token data for subsequent data merging and feature extraction processes.

[0107] Sub-step 20316, deleting all the first target block token data from the modified block token data sequence, and adding all the first updated block token data to the deleted block token data sequence to obtain a first updated sequence.

[0108] In some embodiments of the present application, all the first target block token data is deleted from the modified block token data sequence, and all the first updated block token data is added to the deleted block token data sequence to obtain a first updated sequence. By deleting the merged target block token data and adding new updated block token data to the sequence, data redundancy can be effectively reduced, the data structure can be optimized, and a simplified data sequence can be provided for subsequent processing steps.

[0109] For example, assume the modified block token data sequence is (T1', T2',..., T 16 ), and the first target block token data pairs are (T1', T3'), (T2', T5'), (T4', T6'), and the corresponding first update block token data are T 1,3 , T 2,5 , T 4,6 . Then first, the first target block token data T1', T2',..., T6' are deleted from the sequence, and the remaining sequence is (T7', T8',..., T 16 '). Then, the first update block token data, i.e., T 1,3 , T 2,5 , T 4,6 are added to the sequence to form the first update sequence (T' 1,3 , T' 2,5 , T' 4,6 , T7', T8',..., T 16 '). This update sequence will be used for subsequent data processing and feature extraction processes.

[0110] Sub-step 2032, merge the second target block token data pairs in the first update sequence according to the statistical weights of the block token data in the first update sequence to obtain a second update sequence.

[0111] Wherein, the second target block token data pairs are two block token data in the first update sequence whose data difference degree ranking is greater than a preset second ranking threshold.

[0112] In some embodiments of the present application, the second target block token data pairs in the first update sequence will be merged according to the statistical weights of the block token data in the first update sequence to obtain a second update sequence. The second target block token data pairs refer to two block token data in the first update sequence whose data difference degree ranking is greater than a preset second ranking threshold. By further merging these similar block token data, data redundancy can be further reduced and computing efficiency can be improved.

[0113] For example, assume that the first update sequence contains 8 block token data, T 1,2 , T 3,4 , T5, T6,..., T 10 . Then first, the difference degree between each pair of block token data is calculated and sorted according to the difference degree. Assume that the preset second ranking threshold is 2, indicating that the block token data pairs with the top 2 difference degrees are selected. Assume that the block token data pairs with the top 2 difference degrees are (T5, T 3,4 ) and (T6, T 10 ). Then, these block token data pairs are merged respectively to generate new block token data T1,2 T 3,4,5 , and T 6,10 T7, T8, T9, and form a second update sequence [T 1,2 T 3,4,5 T 6,10 T7, T8, T9]. This update sequence will be used for subsequent further merging and feature extraction.

[0114] Optionally, sub-step 2032 comprises the following sub-steps:

[0115] Sub-step 20321, randomly divide the block token data in the first update sequence into two groups to obtain a third sub-group and a fourth sub-group.

[0116] In some embodiments of the present application, the block token data in the first update sequence will be randomly divided into two groups to obtain a third sub-group and a fourth sub-group. By random division, the distribution of each group of data can be ensured to be uniform, reducing data bias. Dividing the data into two groups helps subsequent difference calculation and data merging, improving processing efficiency and accuracy.

[0117] For example, assuming that the first update sequence contains 8 block token data, K1, K2…K8. First, randomly divide the 8 block token data into two groups, assuming the grouping result is: third sub-group: [K1,…,K4], fourth sub-group: [K5,…,K8]. Through such random division, the distribution of each group of data can be ensured to be uniform, laying a foundation for subsequent difference calculation and data merging.

[0118] Sub-step 20322, determine the second data difference between each block token data in each third sub-group and each block token data in each fourth sub-group, and in the second data difference, determine the target second data difference same as the preset second sorting index number.

[0119] In some embodiments of the present application, the second data difference between each block token data in each third sub-group and each block token data in each fourth sub-group will be determined, and in the second data difference, the target second data difference same as the preset second sorting index number will be determined. The second data difference is an index for measuring the similarity between two block token data, which is usually calculated using cosine similarity or Euclidean distance method. By determining the target second data difference, the most similar block token data pair can be selected, providing a basis for subsequent data merging.

[0120] For example, assuming that the third sub-group contains block token data K1,…,K4, and the fourth sub-group contains block token data K5,…,K8. Then, the cosine similarity between each K i , (i = 1,…,4) and each K j, (j = 5, …, 8), for example, the cosine similarity calculation formula can be used to calculate:

[0121] The second data difference degree (K i ,K j ) = (K i ·K j ) / (||K i ||·||K j ||), (i = 1, …, 4, j = 5, …, 8), then, 6 difference degree values are obtained. Assuming that the preset second sorting index number is 3, indicating that the first 3 target second data difference degrees with the smallest difference degree are selected. Assuming that the 3 target second data difference degrees correspond to (K1, K5), (K2, K6), (K3, K 10 ). These data pairs will be used as target block token data pairs for subsequent data merging.

[0122] Sub-step 20323, the block token data of the third sub-group and the block token data of the fourth sub-group corresponding to each target second data difference degree are selected as a group of second target block token data, and each selected group of second target block token data is used as sample data to obtain a third arithmetic mean vector, and the obtained third arithmetic mean vector is used as second update block token data.

[0123] In some embodiments of the present application, the block token data of the third sub-group and the block token data of the fourth sub-group corresponding to each target second data difference degree are selected as a group of second target block token data, and each selected group of second target block token data is used as sample data to obtain a third arithmetic mean vector. By calculating the arithmetic mean vector of each group of second target block token data, a vector representing the overall characteristics of the data in the group can be generated. This arithmetic mean vector will be used as second update block token data for subsequent data processing and feature extraction.

[0124] For example, assuming that in the first update sequence determined in the previous step, the block token data pairs corresponding to the target second data difference degree among K1, K2 … K8 are (K1, K5), (K2, K6), (K3, K 10 ). Then these data pairs will be selected as second target block token data. Specifically, the second target block token data will include the following data pairs:

[0125] Second target block token data pair 1: K1, K5, second target block token data pair 2: K2, K6, second target block token data pair 3: K3, K 10 ; then, the two block token data in each group of data pairs are averaged, and the formula is as follows:

[0126] K 1,5= (K1+K5) / 2,

[0127] K 2,6 = (K2+K6) / 2,

[0128] K 3,10 = (K3+K 10 ) / 2,

[0129] By this calculation, three third arithmetic average vectors K 1,5 ,K 2,6 , K 3,10 are obtained. These average vectors will be used as second update block token data, representing the overall features of the second target block token data, for subsequent data merging and feature extraction processes.

[0130] Sub-step 20324, delete all the second target block token data from the first update sequence, and add all the second update block token data to the first update sequence after deletion, to obtain a second update sequence.

[0131] In some embodiments of the present application, all the second target block token data will be deleted from the first update sequence, and all the second update block token data will be added to the first update sequence after deletion, to obtain a second update sequence. By deleting the merged target block token data and adding new update block token data to the sequence, data redundancy can be effectively reduced, data structure can be optimized, and a simplified data sequence can be provided for subsequent processing steps.

[0132] For example, assuming that in the first update sequence K1, K2…K8, the second target block token data pairs are (K1, K5), (K2, K6), (K3, K 10 ), and the corresponding second update block token data are K 1,5 , K 2,6 , K 3,10 . Then delete the second target block token data K1, K2, K3, K5, K6, K 10 from the sequence, and the remaining sequence is K4, K7, K8. Then, add the vectors K 1,5 , K 2,6 , K 3,10 to the sequence to form the second update sequence [K 1,5 , K 2,6 , K 3,10 , K4, K7, K8]. This update sequence will be used for subsequent data processing and feature extraction processes.

[0133] Sub-step 2033, take all the block token data in the second update sequence as sample data to obtain a first arithmetic average vector, and take the obtained first arithmetic average vector as block token average data.

[0134] In some embodiments of the present application, all the block token data in the second update sequence are taken as sample data to obtain the first arithmetic mean vector, and the obtained first arithmetic mean vector is taken as the block token mean data. The arithmetic mean vector is a vector obtained by averaging all the sample data, and can represent the overall characteristics of the sample data. By calculating the arithmetic mean vector of the block token data, the data redundancy can be effectively reduced, and the representative video features can be extracted.

[0135] For example, it is assumed that the second update sequence contains 6 block token data, T 1,2 ,T 3,4,5 ,T 6,10 ,T7, T8, and T9. Then, the 6 block token data are taken as sample data. Then, the arithmetic mean vector of the 6 block token data is calculated, and the formula is as follows:

[0136] The arithmetic mean vector = (T 1,2 + T 3,4,5 + T 6,10 + T7 + T8 + T9) / 6.

[0137] The first arithmetic mean vector is obtained by element-by-element addition of the 6 block token data and division by the sample number 6. The arithmetic mean vector is the block token mean data, which contains the overall characteristics of the sample data and will be used in the subsequent video and text matching process.

[0138] Step 204, determining the cross-modal similarity between the block token mean data and the text feature data as the matching result of the video data and the text data.

[0139] The cross-modal similarity is positively correlated with the matching degree of the video data and the text data.

[0140] The method shown in this step has been described in step 104, and will not be repeated here.

[0141] Optionally, step 204 includes the following sub-steps:

[0142] Sub-step 2041, taking the ratio of the inner product of the block token mean data and the text feature data to the product of the modulus as the cross-modal similarity between the block token mean data and the text feature data.

[0143] In some embodiments of the present application, the ratio of the inner product of the block token average data and the text feature data to the product of their norms is taken as the cross-modal similarity of the block token average data and the text feature data. The inner product (dot product) is the result of multiplying two vectors and summing the products, and the norm (length of a vector) is the square root of the sum of the squares of the components of the vector. By calculating the ratio of the inner product to the product of the norms, the cosine similarity between the two vectors can be obtained, which measures the degree of similarity between them. The higher the cosine similarity value, the more similar the two vectors are, thus reflecting the matching degree of the video data and the text data.

[0144] For example, assume that the block token average data is a vector v avg , and the text feature data is a vector t feat . Then the inner product of the two vectors will be calculated first:

[0145]

[0146] where v avg,i and t feat,i are the i-th components of the vectors v avg and t feat , respectively.

[0147] Next, the norms of the two vectors will be calculated:

[0148]

[0149] Finally, the ratio of the inner product to the product of the norms, i.e., the cosine similarity, will be calculated:

[0150] Cosine similarity = (v avg · t feat ) / (||v avg || · ||t feat ||);

[0151] Assuming that the result of the calculation is 0.85, it indicates that the similarity of the block token average data and the text feature data is high, and thus it can be considered that the matching degree of the video data and the text data is high. This similarity value will be taken as the matching result of the video data and the text data, which will be used for subsequent retrieval and analysis.

[0152] Reference Figure 3The execution flow diagram in the large language model with a low-rank adaptation of a large language model (LoRA) is established under the method provided in the embodiment of the present application. In order to implement the efficient video text retrieval model, only the pre-trained CLIP (the snowflake symbol in the figure represents the image encoder layer in the frozen state) is frozen in the embodiment of the present application, and only the LoRA (the flame symbol in the figure represents the LoRA in the trainable state) is trained in the image and text encoder. The specific execution process is performed in the direction of steps S1 to S3, where A represents the block token sequence, B represents the first update sequence, C represents the second update sequence, and D represents the block token average data. The steps S1 to S3 will be explained respectively as follows:

[0153] Step S1: In this step, the image encoder layer of the pre-trained CLIP model remains in the frozen state, and the LoRA module is in the trainable state. In this way, efficient fine-tuning can be performed without changing the parameters of the pre-trained model, and the features of the video and the text can be extracted.

[0154] Step S2: In this step, the LoRA module is continuously applied to train and optimize the data to generate the second update sequence (C). Through this step-by-step updating method, data redundancy can be gradually reduced, and more representative features can be extracted.

[0155] Step S3: In this step, the arithmetic mean vector of all block token data in the second update sequence is calculated to extract the block token average data representing the features of the entire video. This average data will be used in the matching process of the video and the text.

[0156] The complete CLIP model described above is composed of an image encoder and a text encoder, which adopts a transformer architecture (Transformer). The architecture is composed of alternating multi-head self-attention (Multi-Head Self-Attention, MHSA) and multilayer perceptron (Multilayer Perceptron, MLP) (x12) blocks.

[0157] Among them, the attention module in MHSA is as follows:

[0158] Attention(x)=softmax((xW q ) T (xW k ) / d 1 / 2 )(xW v );

[0159] Among them, W q ,W k ,W vare three mapping matrices, respectively. Given an input text t, the text encoder first tokenizes the text description into word tokens. Then, these tokens are passed through 12 text encoder layers to extract the final text feature t. For an input video v, each frame image I i is processed by the image encoder, respectively. The image I is segmented into fixed-size image patches and projected into patch tokens, which are input into 12 image encoder layers to extract frame features f. The final video feature v is obtained by averaging all frame features i} F i＝1 . Finally, a cross-modal contrastive loss is applied to simultaneously optimize both text-to-video and video-to-text directions. This loss function maximizes the similarity of the corresponding text-video pairs.

[0160] To only train a small number of parameters, the LoRA low-rank adaptation method is adopted to migrate CLIP to the video text retrieval task. LoRA injects a trainable rank decomposition matrix in the attention module of each Transformer layer. For the pre-trained matrix LoRA optimizes its rank decomposition change, ΔW=W down W up , where W down ∈R D×r , W down ∈R r×d , and the rank r is much smaller than the minimum of D and d. For x bar =xW, at this time the forward pass is modified to x bar =xW+xΔW. The trainable matrix ΔW can be merged with the frozen weight W without introducing additional inference latency.

[0161] As a further embodiment of Figure 3 , the model provided by the embodiments of the present application is further described as follows: Figure 4

[0162] In step S1, the following method is specifically performed:

[0163] Step S11: Randomly divide the patch token data into two groups to ensure uniform data distribution and reduce data bias.

[0164] Step S12: In each group of data, find similar token pairs and calculate their similarity.

[0165] Step S13: Select the target token pair according to the similarity as the object of subsequent merging.

[0166] Step S14: Add timing information to each target token pair to preserve the temporal relationship between video frames. ​

[0167] Step S15: merging the target token pair to generate updated block token data.

[0168] In step S2, the following method is specifically used

[0169] Step S21: randomly divide the block token data in the first update sequence into two groups to ensure uniform data distribution.

[0170] Step S22: in each group of data, find similar token pairs and calculate their similarity.

[0171] Step S23: select the target token pair according to the similarity as the object of subsequent merging.

[0172] Step S24: merging the target token pair to generate updated block token data.

[0173] In summary, in the embodiments of the present application, by determining the statistical weight according to the similarity between the block token data, the feature distribution of the video data can be more accurately reflected, and then the statistical mean of the block token data sequence is calculated using the statistical weight, which effectively reduces the redundancy of the data and extracts the representative video features. Finally, by calculating the cross-modal similarity between the block token average data and the text feature data, and using the characteristic that the cross-modal similarity is positively correlated with the matching degree, the reliability and accuracy of the matching result are improved, and the effective matching of video and text data is realized. Therefore, based on the method of the embodiments of the present application, the synthesis speed of the token is accelerated, and the problem of slow matching speed caused by large data operation amount of image data token merging in related technologies is solved.

[0174] Reference Figure 5 It shows a video text retrieval device 30 based on time sequence token merging provided by the embodiments of the present application, which comprises:

[0175] The data acquisition module 301 is configured to acquire video data to be matched and text data to be matched, and extract a plurality of continuous image frames from the video data.

[0176] The feature generation module 302 is configured to determine a block token data sequence of the video data according to each image frame, and determine text feature data of the text data; the block token data sequence comprises a plurality of block token data of the video data.

[0177] The averaging module 303 is configured to determine a statistical mean of the video data as block token average data of the video data by taking the block token data sequence as a sampling data set; the statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between the two block token data.

[0178] The data matching module 304 is configured to determine a cross-modal similarity between the block token average data and the text feature data as a matching result of the video data and the text data; the cross-modal similarity is positively correlated with a matching degree of the video data and the text data.

[0179] Optionally, the feature generation module 302 comprises:

[0180] The image segmentation sub-module is configured to segment each image frame into a plurality of image blocks.

[0181] The token generation sub-module is configured to perform convolution processing on each image block respectively to obtain a block feature value corresponding to each image block, and take the block feature value corresponding to each image frame as a component of the block token data corresponding to the image frame, so as to obtain the block token data corresponding to the image frame.

[0182] The sequence generation sub-module is configured to sort all the block token data according to a time sequence of the corresponding image frames to obtain a block token data sequence.

[0183] Optionally, the feature generation module 302 comprises:

[0184] The word segmentation sub-module is configured to take a number of each word segmentation in the text data as a component of a text vector of the text data, so as to obtain the text vector of the text data.

[0185] The text feature sub-module is configured to input the text vector into a pre-trained text feature model to obtain text feature data of the text data.

[0186] Optionally, the averaging module 303 comprises:

[0187] The inter-fragment merging sub-module is configured to merge a first target block token data pair in the block token data sequence according to a statistical weight of the block token data in the block token data sequence to obtain a first updated sequence; the first target block token data pair is two block token data in the block token data sequence whose data difference degrees are ranked greater than a preset first ranking threshold.

[0188] The intra-fragment merging sub-module is configured to merge a second target block token data pair in the first updated sequence according to a statistical weight of the block token data in the first updated sequence to obtain a second updated sequence; the second target block token data pair is two block token data in the first updated sequence whose data difference degrees are ranked greater than a preset second ranking threshold.

[0189] The averaging sub-module is configured to take all the block token data in the second updated sequence as sample data to obtain a first arithmetic average vector, and take the obtained first arithmetic average vector as the block token average data.

[0190] Optionally, the inter-fragment merging submodule comprises:

[0191] The first grouping unit is configured to divide the block token data in the block token data sequence into a plurality of block token data subsequences according to a preset grouping quantity, and randomly divide the block token data in each block token data subsequence into two groups to obtain a same number of first subgroups and second subgroups as the number of block token data subsequences;

[0192] The first difference degree unit is configured to determine a first data difference degree between each block token data in each first subgroups and each block token data in each second subgroups, and determine a same number of target first data difference degrees as a preset first ranking index quantity in the first data difference degree;

[0193] The first selection unit is configured to select the block token data of the first subgroups and the block token data of the second subgroups corresponding to each target first data difference degree as a group of first target block token data;

[0194] The correction unit is configured to generate a time sequence correction parameter for the block token data in each block token data subsequence according to the corresponding block token data subsequence, and correct the block token data in each block token subsequence through the corresponding time sequence correction parameter; the correction parameter is obtained through training of a pre-trained frame position embedding engine;

[0195] The first averaging unit is configured to take the corrected first target block token data in each group as sample data to obtain a second arithmetic mean vector, and take the obtained second arithmetic mean vector as first updated block token data;

[0196] The first updating unit is configured to delete all the first target block token data from the corrected block token data sequence, and add all the first updated block token data to the deleted block token data sequence to obtain a first updated sequence.

[0197] Optionally, the intra-fragment merging submodule comprises:

[0198] The second grouping unit is configured to randomly divide the block token data in the first updated sequence into two groups to obtain a third subgroups and a fourth subgroups;

[0199] The second difference degree unit is configured to determine a second data difference degree between each block token data in each third subgroups and each block token data in each fourth subgroups, and determine a same number of target second data difference degrees as a preset second ranking index quantity in the second data difference degree;

[0200] The second selection unit is configured to select the block token data of the third sub-group and the block token data of the fourth sub-group corresponding to the difference degree of each target second data as a group of second target block token data, and take each selected group of second target block token data as sample data to obtain a third arithmetic mean vector, and take the obtained third arithmetic mean vector as second update block token data.

[0201] The second update unit is configured to delete all the second target block token data from the first update sequence, and add all the second update block token data to the first update sequence after the deletion, to obtain a second update sequence.

[0202] Optionally, the data matching module 304 comprises:

[0203] The cross-modal similarity calculation sub-module is configured to take the ratio of the inner product of the block token average data and the text feature data to the product of the inner product and the modulus as the cross-modal similarity of the block token average data and the text feature data.

[0204] To sum up, in the embodiment of the present application, the statistical weight is determined according to the similarity between the block token data, which can more accurately reflect the feature distribution of the video data, and then the statistical mean of the block token data sequence is calculated by using the statistical weight, which effectively reduces the redundancy of the data and extracts representative video features. Finally, the cross-modal similarity of the block token average data and the text feature data is calculated, and the reliability and accuracy of the matching result are improved by using the characteristic that the cross-modal similarity is positively correlated with the matching degree, and the effective matching of the video and text data is realized. Therefore, based on the method of the embodiment of the present application, the synthesis speed of the token is accelerated, and the problem of slow matching speed caused by large data operation amount of image data token merging in related technologies is solved.

[0205] Referring to Figure 6 The electronic device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0206] The processing component 502 usually controls the overall operation of the electronic device 500, such as operations associated with displaying, making phone calls, data communications, camera operations and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of the steps of the methods described above. In addition, the processing component 502 can include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0207] Memory 504 is used for storing data or code that supports operations of electronic device 500. Examples of such data include instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, multimedia, and the like. Memory 504 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0208] Power supply component 506 supplies power to various components of electronic device 500. Power supply component 506 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 500.

[0209] Multimedia component 508 includes an interface that provides an output interface between electronic device 500 and a user. In some embodiments, the interface can include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensor can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. The front-facing camera and / or the rear-facing camera can receive external multimedia data when electronic device 500 is in an operation mode, such as a photographing mode or a multimedia mode. Each of the front-facing camera and the rear-facing camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0210] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) for receiving an external audio signal when electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0211] Input / output (I / O) interface 512 provides an interface between processing component 502 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0212] The sensor component 514 includes one or more sensors for providing status assessments for various aspects of the electronic device 500. For example, the sensor component 514 can detect an open / closed position of the electronic device 500, relative positioning of components, such as a display and keypad of the electronic device 500, a change in position of the electronic device 500 or a component of the electronic device 500, presence or absence of user contact with the electronic device 500, orientation or acceleration / deceleration / rotation of the electronic device 500, and temperature changes of the electronic device 500. The sensor component 514 can include an optical sensor for detecting ambient light, a proximity sensor configured to detect proximity of an object, a motion sensor, a temperature sensor, a magnetic sensor, an acceleration sensor, a gyroscope sensor, or a pressure sensor.

[0213] The communication component 516 facilitates wired or wireless communication in relation to the electronic device 500. The electronic device 500 can access a wireless network based on a corresponding communication standard, such as WiFi, Bluetooth®, a cellular communication protocol, or a combination thereof. In an example embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system using a broadcast channel. In an example embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0214] In an example embodiment, the electronic device 500 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, to implement methods provided by embodiments of the present application.

[0215] In an example embodiment, a non-transitory computer readable storage medium, such as the memory 504 including instructions, is also provided, which can be executed by the processor 520 of the electronic device 500 to implement the methods described above. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, and the like.

[0216] Figure 7FIG. 6 is a block diagram of an electronic device 600 according to another embodiment of the present application. For example, the electronic device 600 can be provided as a server. Referring to FIG. 6, the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions, such as an application program, executable by the processing component 622. The application program stored in the memory 632 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute the instructions to perform the methods provided by the embodiments of the present application. Figure 7 The electronic device 600 can further include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 can operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0217] The electronic device 600 can further include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 can operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0218] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0219] It is to be understood that the application is not limited to the precise details of construction and the arrangement of components described above and illustrated in the drawings. The scope of the application should be determined not with reference to the above description nor with reference to the appended claims, but with reference to the claims appended hereto, taking their legal equivalents into consideration as if written.

Claims

1. A method for video text retrieval based on temporal token merging, characterized in that, The method comprises: obtaining video data to be matched and text data to be matched, and extracting a plurality of continuous image frames from the video data; determining a block token data sequence of the video data according to each of the image frames, and determining text feature data of the text data; the block token data sequence comprises a plurality of block token data of the video data; determining a statistical mean of the video data as block token average data of the video data by taking the block token data sequence as a sample data set; a statistical weight of the statistical mean is determined according to token similarity between block token data in the block token data sequence; the token similarity is negatively correlated with a data difference degree between two block token data; determining a cross-modal similarity between the block token average data and the text feature data as a matching result of the video data and the text data; the cross-modal similarity is positively correlated with a matching degree of the video data and the text data; the determining of the block token data sequence of the video data according to each of the image frames comprises: segmenting each of the image frames into a plurality of image blocks; respectively performing convolution processing on each of the image blocks to obtain a block feature value corresponding to each of the image blocks, and taking each of the block feature values corresponding to the image frames as a component of the block token data corresponding to the image frames to obtain the block token data corresponding to the image frames; sorting all the block token data according to a time sequence of the corresponding image frames to obtain the block token data sequence; the determining of the statistical mean of the video data as the block token average data of the video data by taking the block token data sequence as the sample data set comprises: merging first target block token data pairs in the block token data sequence according to statistical weights of the block token data in the block token data sequence to obtain a first update sequence; the first target block token data pairs are two block token data in the block token data sequence whose data difference degrees are greater than a preset first ranking threshold; merging second target block token data pairs in the first update sequence according to statistical weights of the block token data in the first update sequence to obtain a second update sequence; the second target block token data pairs are two block token data in the first update sequence whose data difference degrees are greater than a preset second ranking threshold; taking all the block token data in the second update sequence as sample data to obtain a first arithmetic mean vector, and taking the obtained first arithmetic mean vector as the block token average data.

2. The method of claim 1, wherein, the determining of the text feature data of the text data comprises: taking a number of each word segmentation in the text data as a component of a text vector of the text data to obtain the text vector of the text data; inputting the text vector into a pre-trained text feature model to obtain the text feature data of the text data.

3. The method of claim 1, wherein, the merging of the first target block token data pairs in the block token data sequence according to the statistical weights of the block token data in the block token data sequence to obtain the first update sequence comprises: According to a preset number of groups, the block token data in the block token data sequence is divided into a plurality of block token data subsequences, and the block token data in each of the block token data subsequences is randomly divided into two groups to obtain a same number of first subgroups and second subgroups as the number of block token data subsequences; Determine the first data difference between each block token data in each of the first subgroups and each block token data in each of the second subgroups, and determine a same number of target first data differences as a preset first ranking index number in the first data difference; Select the block token data of the first subgroup and the block token data of the second subgroup corresponding to each of the target first data differences as a group of first target block token data; According to the corresponding block token data subsequence, generate a timing correction parameter for the block token data in each of the block token data subsequences, and correct the block token data in each of the block token data subsequences through the corresponding timing correction parameter; The correction parameter is obtained by training a pre-trained frame position embedding engine; The corrected first target block token data in each group is used as sample data to obtain a second arithmetic mean vector, and the obtained second arithmetic mean vector is used as first update block token data; Delete all the first target block token data from the corrected block token data sequence, and add all the first update block token data to the deleted block token data sequence to obtain the first update sequence.

4. The method of claim 1, wherein, The second target block token data in the first update sequence is combined according to the statistical weight of the block token data in the first update sequence to obtain a second update sequence, including: Randomly divide the block token data in the first update sequence into two groups to obtain a third subgroup and a fourth subgroup; Determine the second data difference between each block token data in each of the third subgroups and each block token data in each of the fourth subgroups, and determine a same number of target second data differences as a preset second ranking index number in the second data difference; Select the block token data of the third subgroup and the block token data of the fourth subgroup corresponding to each of the target second data differences as a group of second target block token data, and use each of the selected second target block token data as sample data to obtain a third arithmetic mean vector, and use the obtained third arithmetic mean vector as second update block token data; Delete all the second target block token data from the first update sequence, and add all the second update block token data to the deleted first update sequence to obtain the second update sequence.

5. The method of claim 1, wherein, The determination of the cross-modal similarity between the block token average data and the text feature data includes: The ratio of the inner product of the block token average data and the text feature data to the product of the modulus is taken as the cross-modal similarity between the block token average data and the text feature data.

6. A video text retrieval device based on time-series token merging, characterized in that, Including: The data acquisition module is configured to acquire video data to be matched and text data to be matched, and extract a plurality of continuous image frames from the video data. The feature generation module is configured to determine a block token data sequence of the video data according to each of the image frames, and determine text feature data of the text data. The block token data sequence comprises a plurality of block token data of the video data. The averaging module is configured to determine a statistical mean of the video data as block token average data of the video data by taking the block token data sequence as a sample data set. The statistical weight of the statistical mean is determined according to a token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with a data difference degree between two block token data. The data matching module is configured to determine a cross-modal similarity between the block token average data and the text feature data as a matching result of the video data and the text data. The cross-modal similarity is positively correlated with a matching degree of the video data and the text data. The feature generation module comprises: The image segmentation submodule is configured to segment each of the image frames into a plurality of image blocks. The token generation submodule is configured to respectively perform convolution processing on each of the image blocks to obtain a block feature value corresponding to each of the image blocks, and take each of the block feature values corresponding to the image frames as a component of the block token data corresponding to the image frames to obtain the block token data corresponding to the image frames. The sequence generation submodule is configured to sort all of the block token data in a time sequence of corresponding image frames to obtain the block token data sequence. The averaging module comprises: The inter-fragment merging submodule is configured to merge a first target block token data pair in the block token data sequence according to the statistical weight of the block token data in the block token data sequence to obtain a first update sequence; the first target block token data pair is two block token data in the block token data sequence with a data difference degree ranking greater than a preset first ranking threshold. The intra-fragment merging submodule is configured to merge a second target block token data pair in the first update sequence according to the statistical weight of the block token data in the first update sequence to obtain a second update sequence; the second target block token data pair is two block token data in the first update sequence with a data difference degree ranking greater than a preset second ranking threshold. The averaging submodule is configured to take all of the block token data in the second update sequence as sample data to obtain a first arithmetic mean vector, and take the obtained first arithmetic mean vector as the block token average data.

7. An electronic device, comprising: The computer readable storage medium comprises: The processor and a memory for storing instructions executable by the processor. The processor is configured to execute the instructions to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can perform the method of any one of claims 1 to 5.

Images