Video text retrieval method based on time sequence token combination
Through the method based on time-series token merging, statistical weights are calculated using the similarity between block token data, reducing data redundancy and extracting video features, solving the problem of slow matching speed in video text retrieval and achieving more efficient video Match text data.
Patent Information
- Application Number
- CN202411209352.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-30
AI Technical Summary
In the prior art, in video text retrieval, the matching speed is slow due to the large amount of image data token merge data.
Using a time-series token merging method, statistical weights are calculated by determining the similarity between block token data, reducing data redundancy, extracting representative video features, and using cross-modal similarity to improve the reliability of matching results. Sex and accuracy.
It speeds up the speed of token synthesis, improves the accuracy and efficiency of matching video and text data, and solves the problem of slow matching speed.
Smart Images

Figure CN119066222A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of cross-modal retrieval technology, and specifically relates to a video text retrieval method, device, equipment and storage medium based on temporal token merging. Background Art
[0002] In the field of video language understanding, text-to-video retrieval is a key task that aims to match videos corresponding to a specific query text or vice versa. With the progress of multimodal learning, especially the success of text-image pre-trained models in cross-modal tasks, text-to-video retrieval technology has gained widespread attention. By applying pre-trained image and text encoders to video data, researchers are able to achieve remarkable results in text-to-video retrieval tasks.
[0003] In the prior art, many studies use the image text encoder (Contrastive Language-Image Pre-Training, CLIP) as the backbone and optimize the text video retrieval performance through comprehensive fine-tuning methods. And in order to improve efficiency, token compression methods are used to reduce the computational burden by reducing the number of tokens while retaining key information.
[0004] However, due to the large number of tokens in videos, existing token compression methods still face the problem of high inference cost when processing video data and fail to effectively handle temporal redundancy in videos, resulting in high computational overhead. Summary of the invention
[0005] The present application aims to provide a video text retrieval method, apparatus, device and storage medium based on time-series token merging, at least to solve the problem of slow matching speed caused by large amount of data calculation of image data token merging in the video text retrieval process.
[0006] In a first aspect, an embodiment of the present application discloses a video text retrieval method based on time sequence token merging, comprising:
[0007] Acquire video data to be matched and text data to be matched, and extract multiple continuous image frames from the video data;
[0008] Determine a block token data sequence of the video data according to each of the image frames, and determine text feature data of the text data; the block token data sequence includes a plurality of block token data of the video data;
[0009] The block token data sequence is used as a sampling data set, and a statistical mean of the video data is determined as the block token average data of the video data; the statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between two block token data;
[0010] Determine the cross-modal similarity between the block token average data and the text feature data as a matching result between the video data and the text data; the cross-modal similarity is positively correlated with the matching degree between the video data and the text data.
[0011] In a second aspect, the embodiment of the present application further discloses a video text retrieval device based on time sequence token merging, comprising:
[0012] A data acquisition module, used to acquire video data to be matched and text data to be matched, and extract a plurality of continuous image frames from the video data;
[0013] A feature generation module, used for determining a block token data sequence of the video data according to each of the image frames, and determining text feature data of the text data; the block token data sequence comprises a plurality of block token data of the video data;
[0014] An averaging module is used to use the block token data sequence as a sampling data set to determine the statistical mean of the video data as the block token average data of the video data; the statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between two block token data;
[0015] A data matching module is used to determine the cross-modal similarity between the block token average data and the text feature data as a matching result between the video data and the text data; the cross-modal similarity is positively correlated with the matching degree between the video data and the text data.
[0016] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
[0017] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0018] In summary, in the embodiment of the present application, by determining the statistical weight according to the similarity between the block token data, the characteristic distribution of the video data can be more accurately reflected, and then the statistical weight is used to calculate the statistical mean of the block token data sequence, which effectively reduces the redundancy of the data and extracts representative video features; finally, by calculating the cross-modal similarity between the block token average data and the text feature data, the cross-modal similarity is positively correlated with the matching degree, and the reliability and accuracy of the matching results are improved, and the effective matching of video and text data is achieved. Therefore, the method based on the embodiment of the present application speeds up the synthesis speed of the tokens, and solves the problem of slow matching speed caused by the large amount of image data token merging data calculation in the related art. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In the attached picture:
[0020] Figure 1 It is a flowchart of the steps of a video text retrieval method based on time token merging provided by an embodiment of the present application;
[0021] Figure 2 It is a flowchart of another method for video text retrieval based on time token merging provided by an embodiment of the present application;
[0022] Figure 3 It is a schematic diagram of the execution logic of the data processing model under the method provided in the embodiment of the present application;
[0023] Figure 4 It is a further explanation of the execution logic of the data processing model in the embodiment of the present application;
[0024] Figure 5 It is a block diagram of a video text retrieval device based on time sequence token merging provided by an embodiment of the present application;
[0025] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application;
[0026] Figure 7 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0029] Figure 1 This embodiment provides a video text retrieval method based on time token merging.
[0030] The method may include the following steps:
[0031] Step 101 : acquiring video data to be matched and text data to be matched, and extracting a plurality of continuous image frames from the video data.
[0032] In some embodiments of the present application, video data and text data to be matched are obtained, and multiple continuous image frames are extracted from the video data. Video data is usually composed of a series of continuous image frames, which capture the time changes of the video content. The text data is descriptive text related to the video content. By extracting continuous image frames in the video, the video content can be decomposed into multiple static images, which is convenient for subsequent feature extraction and matching processing.
[0033] For example, the video clip and the corresponding text description, such as "In the forest, birds are singing on branches", can be first obtained. Then, 300 consecutive image frames (10 seconds x 30 frames / second = 300 frames) are extracted from some video clips. These image frames will be used for the subsequent determination of block token data sequences and feature extraction to achieve matching between video and text.
[0034] Step 102, determining a block token data sequence of the video data according to each image frame, and determining text feature data of the text data.
[0035] The block token data sequence includes a plurality of block token data of the video data.
[0036] In some embodiments of the present application, a block token data sequence of video data is determined based on each image frame, and text feature data of text data is determined. The block token data sequence is generated by dividing each image frame into multiple fixed-size image blocks and converting these image blocks into corresponding block token data. The text feature data is obtained by tokenizing and encoding the text data. The block token data sequence and the text feature data will be used for subsequent cross-modal similarity calculations to achieve matching between video and text.
[0037] For example, suppose that the size of a frame of image extracted from a video is 224×224 pixels. Then, the image can be first divided into 14×14 image blocks, each of which is 16×16 pixels in size. Then, each image block is converted into a block token data, generating a block token data sequence containing 196 block token data. For the text data "In the forest, birds are singing on the branches", it is converted into a series of text feature data through a text encoder. Finally, the obtained block token data sequence of video data and the text feature data of text data will be used in the subsequent matching process.
[0038] Step 103, taking the block token data sequence as a sample data set, and determining the statistical mean of the video data as the block token average data of the video data.
[0039] The statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between the two block token data.
[0040] In some embodiments of the present application, the block token data sequence is used as a sampling data set to determine the statistical mean of the video data as the block token average data of the video data. The calculation of the statistical mean is based on the similarity between the block token data, and the statistical weight is determined by the similarity between the block token data. The higher the similarity, the greater the statistical weight; the lower the similarity, the smaller the statistical weight. In this way, the feature distribution of the video data can be more accurately reflected, data redundancy can be reduced, and representative video features can be extracted.
[0041] For example, suppose that the block token data sequence extracted from the video contains 8 image frames, each frame contains 196 block token data, and a total of 1568 block token data. First, the similarity between each pair of block token data is calculated, using the cosine similarity metric. Then, the statistical weight of each block token data is determined based on the similarity. Next, the statistical mean of the block token data sequence is calculated using these statistical weights to obtain the block token average data of the video data. This average data will be used for subsequent cross-modal similarity calculations to improve the accuracy and efficiency of video and text matching.
[0042] Step 104 , determining the cross-modal similarity between the block token average data and the text feature data as a matching result between the video data and the text data.
[0043] Among them, cross-modal similarity is positively correlated with the matching degree between video data and text data.
[0044] In some embodiments of the present application, the cross-modal similarity between the block token average data and the text feature data is determined as the matching result of the video data and the text data. The cross-modal similarity is an indicator to measure the degree of matching between the video data and the text data. By calculating the similarity between the block token average data and the text feature data, the correlation between the video content and the text description can be evaluated. The higher the similarity, the higher the degree of matching between the video data and the text data, thereby improving the reliability and accuracy of the matching results. In the actual retrieval process, multiple videos and a certain text, or multiple texts and a certain video can also be sorted by matching degree, and sorted according to the cross-modal similarity to obtain a selective retrieval result.
[0045] For example, suppose the block token average data of the video data is a vector v avg , the text feature data is a vector t feat Then we can use cosine similarity to calculate v avg With t feat The similarity between them is as follows:
[0046] Similarity = (v avg ·t feat ) / ||v avg ||·||t feat ||;
[0047] Among them, v avg ·t feat represents the dot product of two vectors, ||v avg || and ||t feat || respectively represent the modulus lengths of the two vectors. The calculated similarity value is used as the matching result of the video data and the text data. Assuming that the calculation result is 0.85, it means that the video content and the text description have a high degree of matching, so it can be considered that the video and the text description are related.
[0048] In summary, in the embodiment of the present application, by determining the statistical weight according to the similarity between the block token data, the characteristic distribution of the video data can be more accurately reflected, and then the statistical weight is used to calculate the statistical mean of the block token data sequence, which effectively reduces the redundancy of the data and extracts representative video features; finally, by calculating the cross-modal similarity between the block token average data and the text feature data, the cross-modal similarity is positively correlated with the matching degree, and the reliability and accuracy of the matching results are improved, and the effective matching of video and text data is achieved. Therefore, the method based on the embodiment of the present application speeds up the synthesis speed of the tokens, and solves the problem of slow matching speed caused by the large amount of image data token merging data calculation in the related art.
[0049] Figure 3 Another video text retrieval method based on time sequence token merging provided in the embodiment of the application may include the following steps:
[0050] Step 201 : acquiring video data to be matched and text data to be matched, and extracting a plurality of continuous image frames from the video data.
[0051] The method shown in this step has been explained in step 101 and will not be repeated here.
[0052] Step 202, determining a block token data sequence of the video data according to each image frame, and determining text feature data of the text data.
[0053] The block token data sequence includes a plurality of block token data of the video data.
[0054] The method shown in this step has been explained in step 102 and will not be repeated here.
[0055] Optionally, in order to determine the block token data sequence of the video data according to each image frame, step 202 includes the following sub-steps:
[0056] Sub-step 2021, dividing each image frame into multiple image blocks.
[0057] In some embodiments of the present application, each image frame is segmented into multiple image blocks. Image blocks are image frames divided into a number of rectangular areas of fixed size, each area is called an image block. Through this segmentation, a large image frame can be converted into multiple small blocks, which is convenient for subsequent feature extraction and processing. Each image block will be used as part of the subsequent block token data for feature representation of video data.
[0058] For example, suppose that a frame of image extracted from a video is 224×224 pixels in size. First, the image frame is segmented into 14×14 image blocks, each of which is 16×16 pixels in size. In this way, the entire image frame is segmented into 196 image blocks. Each image block will be treated as an independent unit for subsequent convolution processing and feature extraction to form block token data. This segmentation method helps to capture local features in the image frame and lays the foundation for the generation of subsequent block token data sequences.
[0059] In sub-step 2022, convolution processing is performed on each image block to obtain a block feature value corresponding to each image block, and each block feature value corresponding to an image frame is used as a component of the block token data corresponding to the image frame to obtain the block token data corresponding to the image frame.
[0060] In some embodiments of the present application, each image block is subjected to convolution processing to obtain a block feature value corresponding to each image block. Convolution processing is a commonly used image processing technology, which extracts features from image blocks through a convolution kernel to generate block feature values. The block feature value of each image block will be used as a component of the block token data corresponding to the image frame. In this way, the local information in the image frame can be converted into a feature vector, which is convenient for subsequent feature matching and processing.
[0061] For example, suppose that a frame of image extracted from a video has been segmented into 14×14 image blocks, each of which is 16×16 pixels in size. Each image block is convolved separately, for example, a 3×3 convolution kernel can be used, with a step size of 1 and a padding of 1. After convolution, each image block generates a feature vector, assuming that the dimension of the feature vector is 128. The feature vector of each image block is used as part of the block token data, and finally forms block token data containing 196 block feature values. These block token data will be used in the subsequent generation and matching process of block token data sequences.
[0062] Sub-step 2023, sorting all block token data according to the time sequence of the corresponding image frames to obtain a block token data sequence.
[0063] In some embodiments of the present application, all block token data are sorted in the time order of the corresponding image frames to obtain a block token data sequence. By sorting in time order, the time relationship between video frames can be maintained, ensuring that the block token data sequence can accurately reflect the time changes of the video content. This step is crucial for the subsequent video feature extraction and matching process because it retains the time information of the video and helps to improve the accuracy of cross-modal similarity calculation.
[0064] For example, suppose the block token data extracted from the video contains 8 image frames, and each frame contains 196 block token data. Then first, the block token data of each image frame can be arranged in the time order of the frames. Assume that the time order of the image frames is frame 1, frame 2, frame 3, ..., frame 8. Put the 196 block token data of frame 1 in the front, followed by the 196 block token data of frame 2, and so on, until the 196 block token data of frame 8. Finally, a block token data sequence containing 1568 block token data is formed, which will be used in the subsequent video feature extraction and matching process.
[0065] Optionally, in order to determine text feature data of the text data, step 202 includes the following sub-steps:
[0066] Sub-step 2024, taking the number of each word segment in the text data as a component of the text vector of the text data, to obtain the text vector of the text data.
[0067] In some embodiments of the present application, the number of each word segment in the text data is used as a component of the text vector of the text data to obtain the text vector of the text data. The text vector is a numerical representation of the text data. By counting the number of times each word segment appears in the text data, a vector can be generated, in which each component corresponds to the number of words segment. This representation method helps to capture the word frequency information of the text data and provides a basis for subsequent feature extraction and matching.
[0068] For example, suppose the text data is "In the forest, birds are singing on branches". First, the text data is segmented to obtain a list of segmented words: ["in", "forest", "in", "birds", "on", "branch", "on", "singing"]. Then, the number of times each segmented word appears in the text data is counted to generate a text vector: [2,1,1,1,2,1,1,1]. Among them, "in" appears 2 times, and the other segmented words appear once each. This text vector will be used as a numerical representation of the text data for subsequent text feature extraction and matching processes.
[0069] Sub-step 2025, inputting the text vector into a pre-trained text feature model to obtain text feature data of the text data.
[0070] In some embodiments of the present application, the text vector is input into a pre-trained text feature model to obtain text feature data of the text data. The pre-trained text feature model is a deep learning model trained with a large amount of text data, which can convert text vectors into high-dimensional feature representations. These feature representations capture the semantic information of the text and help improve the accuracy of matching text with video data. By inputting the text vector, the model can generate text feature data for subsequent cross-modal similarity calculations.
[0071] For example, suppose the generated text vector is [2,1,1,1,2,1,1,1], which represents the text "In the forest, birds are singing on branches". This text vector is input into a pre-trained text feature model, such as the Bidirectional Encoder Representations from Transformers (BERT). The model processes the text vector and generates a high-dimensional feature vector. Assume that the dimension of the feature vector is 768. This high-dimensional feature vector is the text feature data of the text data, which contains the semantic information of the text and will be used in the subsequent video and text matching process.
[0072] Step 203: Use the block token data sequence as a sample data set to determine the statistical mean of the video data as the block token average data of the video data.
[0073] The statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between the two block token data.
[0074] The method shown in this step has been explained in step 103 and will not be repeated here.
[0075] Optionally, step 203 includes the following sub-steps:
[0076] Sub-step 2031, merging the first target block token pair data in the block token data sequence according to the statistical weight of the block token data in the block token data sequence to obtain a first update sequence.
[0077] The first target block token data pair is two block token data in the block token data sequence whose data difference ranking is greater than a preset first ranking threshold.
[0078] In some embodiments of the present application, the first target block token pair data in the block token data sequence is merged according to the statistical weight of the block token data in the block token data sequence to obtain a first update sequence. The first target block token data pair refers to two block token data in the block token data sequence whose data difference ranking is greater than a preset first ranking threshold. The data difference is an indicator to measure the similarity between two block token data. The smaller the difference, the higher the similarity. By merging these similar block token data, data redundancy can be reduced and computing efficiency can be improved.
[0079] For example, suppose the block token data sequence contains 10 block token data, namely T 1 ,T 2 ,…,T 10Then the difference between each pair of block token data will be calculated first, and then sorted according to the difference. Assume that the preset first ranking threshold is 2, which means that the top 2 block token data pairs with the highest difference are selected. Assume that the top 5 block token data pairs with the highest difference are (T 1 ,T 2 ),(T 3 ,T 4 Then, these block token data pairs are merged to generate new block token data T 1,2 ,T 3,4 , and form the first update sequence [T 1,2 ,T 3,4 ,T 5 ,T 6 ,…,T 10 ]. This update sequence will be used for further merging and feature extraction later.
[0080] Optionally, sub-step 2031 includes the following sub-steps:
[0081] Sub-step 20311, according to a preset number of groupings, divide the block token data in the block token data sequence into multiple block token data sub-sequences, and randomly divide the block token data in each block token data sub-sequence into two groups to obtain a first sub-group and a second sub-group with the same number as the block token data sub-sequences.
[0082] In some embodiments of the present application, the block token data in the block token data sequence is divided into a plurality of block token data subsequences according to a preset number of groups, and the block token data in each block token data subsequence is randomly divided into two groups to obtain a first subgroup and a second subgroup having the same number as the block token data subsequences. Through this grouping, a large-scale block token data sequence can be decomposed into smaller subsequences to facilitate subsequent processing and calculation. The random division into two groups ensures that the data in each subsequence is evenly distributed, which helps to improve the efficiency and accuracy of data processing.
[0083] For example, suppose that a block token data sequence contains 16 block token data, and the preset number of groups is 4. Then, the 16 block token data will first be divided into 4 block token data subsequences, each subsequence containing 4 block token data. Then, the 4 block token data in each subsequence are randomly divided into two groups, namely the first subgroup and the second subgroup. Then, after the 4 block token data in the first subsequence are randomly divided, the first subgroup contains 2 block token data, and the second subgroup also contains 2 block token data. In this way, 4 first subgroups and 4 second subgroups are obtained, which lays the foundation for the subsequent difference calculation and data merging.
[0084] Sub-step 20312, determining the first data difference between each block token data in each first subgroup and each block token data in each second subgroup, and determining, among the first data differences, a target first data difference that is the same as the preset first sorting index number.
[0085] In some embodiments of the present application, the first data difference between each block token data in each first subgroup and each block token data in each second subgroup is determined, and in the first data difference, a target first data difference equal to the number of preset first sorting indicators is determined. The data difference is an indicator to measure the similarity between two block token data, and is usually calculated using methods such as cosine similarity or Euclidean distance. By determining the target first data difference, the most similar block token data pair can be selected to provide a basis for subsequent data merging.
[0086] For example, assuming that the entire first subgroup contains block token data A 1 ,A 2 ,…,A 8 , the second subgroup of all contains block token data B 1 ,B 2 ,…,B 8 Then we first calculate each A i With each B j The first data difference between them can be calculated using the cosine similarity formula, for example:
[0087] The first data difference (A i ,B j )=(A i ·B j ) / (||A i ||·||B j ||);
[0088] Then, 28 difference values are obtained. Assume that the number of preset first sorting indicators is 4, indicating that the first 4 target first data differences with the smallest difference are selected. Assume that the block token data pairs corresponding to the 4 target first data differences are (A 1 ,B 3 ),(A 2 ,B 5 ),(A 4 ,B 1 ). These data pairs will serve as the target block token data pairs for subsequent data merging.
[0089] Sub-step 20313, selecting the block token data of the first subgroup and the block token data of the second subgroup corresponding to the difference degree with each target first data as a group of first target block token data.
[0090] In some embodiments of the present application, the block token data of the first subgroup and the block token data of the second subgroup corresponding to the difference degree of each target first data are selected as a group of first target block token data. By selecting these data pairs, the most similar block token data pairs can be determined to provide a basis for subsequent data merging. Each group of first target block token data will contain a block token data from the first subgroup and a block token data from the second subgroup, and these data pairs will be merged in subsequent steps.
[0091] For example, assuming that the block token data pair corresponding to the target first data difference determined in the previous step is (A 1 ,B 3 ),(A 2 ,B 5 ),(A 4 ,B 1 ). Then these data pairs will be selected as the first target block token data. Specifically, the first target block token data will include the following data pairs:
[0092] First target block token data pair 1 : A 1 and B 3 ,
[0093] First target block token data pair 2 : A 2 and B 5 ,
[0094] First target block token data pair 3 : A 4 and B 1 ;
[0095] These data pairs will serve as the basis for subsequent data merging, further reducing data redundancy and improving computing efficiency.
[0096] Sub-step 20314, generating a timing correction parameter for the block token data in each block token data sub-sequence according to the corresponding block token data sub-sequence, and correcting the block token data in each block token sub-sequence by using the corresponding timing correction parameter.
[0097] The correction parameters are obtained by training the pre-trained frame position embedding engine.
[0098] In some embodiments of the present application, a timing correction parameter for the block token data in each block token data subsequence is generated according to the corresponding block token data subsequence, and the block token data in each block token subsequence is corrected by the corresponding timing correction parameter. The timing correction parameter is a parameter used to adjust the time information of the block token data to ensure that the time relationship between video frames is preserved during the merging process. The correction parameter is obtained by training a pre-trained frame position embedding engine, which can capture the time characteristics of the video frame and improve the accuracy of video feature extraction.
[0099] For example, suppose the block token data subsequence contains 6 block token data, namely T 1 ,T 2 ,…,T 6 Then we first use the pre-trained frame position embedding engine to generate the timing correction parameter P 1 ,P 2 ,…,P 6 , these parameters correspond to the time position of each block token data. Then, each block token data is corrected, the formula is as follows: T i ′=T i +P i (i=1,2,…,6), where T i ′ is the modified block token data. Through this modification, the time information of the block token data can be retained, ensuring that the time relationship between video frames is preserved in the subsequent merging process. The modified block token data will be used in the subsequent merging and feature extraction process.
[0100] Sub-step 20315, using each group of the modified first target block token data as sample data to obtain a second arithmetic mean vector, and using the obtained second arithmetic mean vector as the first update block token data.
[0101] In some embodiments of the present application, each group of the first target block token data after correction is used as sample data to obtain a second arithmetic mean vector, and the obtained second arithmetic mean vector is used as the first update block token data. By calculating the arithmetic mean vector of each group of the first target block token data, a vector representing the overall characteristics of the group of data can be generated. This arithmetic mean vector will be used as the first update block token data for subsequent data processing and feature extraction.
[0102] For example, assuming that in the modified block token data (T 1 ′,T 2 ′…,T 16 The modified first target block token data pair in (T 1 ′,T 3 ′),(T 2 ′,T 5′),(T 4 ′,T 6 ′). Then the two block token data in each data pair will be averaged, and the formula is as follows:
[0103] T 1,3 =(T 1 ′+T 3 ′) / 2,
[0104] T 2,5 =(T 2 ′+T 5 ′) / 2,
[0105] T 4,6 =(T 4 ′+T 6 ′) / 2;
[0106] Through this calculation, three second arithmetic mean vectors are obtained: T 1,3 ,T 2,5 ,T 4,6 These average vectors will be used as the first update block token data, representing the overall features of the corrected first target block token data, for subsequent data merging and feature extraction processes.
[0107] Sub-step 20316, delete all first target block token data from the modified block token data sequence, and add all first update block token data to the deleted block token data sequence to obtain a first update sequence.
[0108] In some embodiments of the present application, all first target block token data are deleted from the modified block token data sequence, and all first update block token data are added to the deleted block token data sequence to obtain a first update sequence. By deleting the merged target block token data and adding the new update block token data to the sequence, data redundancy can be effectively reduced, data structure can be optimized, and a simplified data sequence can be provided for subsequent processing steps.
[0109] For example, assuming that the modified block token data sequence is (T 1 ′,T 2 ′…,T 16 ′), and the first target block token data pair is (T 1 ′,T 3 ′),(T 2 ′,T 5 ′),(T 4 ′,T 6 '), and the corresponding first update block token data is T' 1,3 ,T′ 2,5 ,T′ 4,6Then first, the first target block token data T 1 ′,T 2 ′,…,T6′ are deleted from the sequence, and the remaining sequence is (T 7 ′,T 8 ′…,T 16 ′). Then, the first update block token data, namely T 1,3 ,T 2,5 ,T 4,6 Join the sequence to form the first update sequence (T′ 1,3 ,T′ 2,5 ,T′ 4,6 , T 7 ′,T 8 ′…,T 16 ′). This update sequence will be used in the subsequent data processing and feature extraction process.
[0110] Sub-step 2032, merging the second target block token data pairs in the first update sequence according to the statistical weights of the block token data in the first update sequence to obtain a second update sequence.
[0111] The second target block token data pair is two block token data in the first update sequence whose data difference ranking is greater than a preset second ranking threshold.
[0112] In some embodiments of the present application, the second target block token data pair in the first update sequence is merged according to the statistical weight of the block token data in the first update sequence to obtain the second update sequence. The second target block token data pair refers to two block token data in the first update sequence whose data difference ranking is greater than a preset second ranking threshold. By further merging these similar block token data, data redundancy can be further reduced and computing efficiency can be improved.
[0113] For example, suppose the first update sequence contains 8 blocks of token data, namely T 1,2 ,T 3,4 ,T 5 ,T 6 ,…,T 10 . Then first, the difference between each pair of block token data is calculated and sorted according to the difference. Assume that the preset second ranking threshold is 2, which means that the top 2 block token data pairs with the highest difference are selected. Assume that the top 2 block token data pairs with the highest difference are (T 5 ,T 3,4 ) and (T 6 ,T 10 Then, these block token data pairs are merged to generate new block token data T 1,2 ,T 3,4,5 , and T 6,10 ,T7 ,T 8 ,T 9 , and form the second update sequence [T 1,2 ,T 3,4,5 ,T 6,10 ,T 7 ,T 8 ,T 9 ]. This update sequence will be used for further merging and feature extraction later.
[0114] Optionally, sub-step 2032 includes the following sub-steps:
[0115] Sub-step 20321, randomly divide the block token data in the first update sequence into two equal groups to obtain a third sub-group and a fourth sub-group.
[0116] In some embodiments of the present application, the block token data in the first update sequence is randomly divided into two groups to obtain the third subgroup and the fourth subgroup. By randomly dividing, it is possible to ensure that each group of data is evenly distributed and reduce data deviation. Dividing the data into two groups facilitates subsequent difference calculation and data merging, and improves processing efficiency and accuracy.
[0117] For example, suppose the first update sequence contains 8 blocks of token data, namely K 1 ,K 2 …K 8 First, the 8 blocks of token data are randomly divided into two groups. Assume that the grouping results are: The third subgroup: [K 1 ,…,K 4 ], the fourth subgroup: [K 5 ,…,K 8 ]Through this random division, we can ensure that each group of data is evenly distributed, laying the foundation for subsequent difference calculation and data merging.
[0118] Sub-step 20322, determining the second data difference between each block token data in each third subgroup and each block token data in each fourth subgroup, and determining, among the second data differences, a target second data difference that is the same as the preset second sorting index number.
[0119] In some embodiments of the present application, the second data difference between each block token data in each third subgroup and each block token data in each fourth subgroup is determined, and in the second data difference, a target second data difference equal to the preset second sorting index is determined. The second data difference is an indicator to measure the similarity between two block token data, and is usually calculated using methods such as cosine similarity or Euclidean distance. By determining the target second data difference, the most similar block token data pair can be selected to provide a basis for subsequent data merging.
[0120] For example, suppose the third subgroup contains block token data K 1 ,…,K 4 , the fourth subgroup contains the block token data K 5 ,…,K 8 Then we first calculate each K i , (i=1,…,4) and each K j , (j=5,…,8), the second data difference between them can be calculated using the cosine similarity calculation formula, for example:
[0121] The second data difference (K i ,K j )=(K i ·K j ) / (||K i ||·||K j ||), (i=1,…,4,j=5,…,8), then, 6 difference values are obtained. Assume that the preset number of second sorting indicators is 3, which means that the first 3 target second data differences with the smallest difference are selected. Assume that the block token data pairs corresponding to the differences of these 3 target second data are (K 1 ,K 5 ),(K 2 ,K 6 ),(K 3 ,K 10 ). These data pairs will serve as the target block token data pairs for subsequent data merging.
[0122] In sub-step 20323, the block token data of the third subgroup and the block token data of the fourth subgroup corresponding to the difference degree with each target second data are selected as a group of second target block token data, and each selected group of second target block token data is used as sample data to obtain the third arithmetic mean vector, and the obtained third arithmetic mean vector is used as the second updated block token data.
[0123] In some embodiments of the present application, the block token data of the third subgroup and the block token data of the fourth subgroup corresponding to each target second data difference are selected as a group of second target block token data, and each selected group of second target block token data is used as sample data to obtain a third arithmetic mean vector. By calculating the arithmetic mean vector of each group of second target block token data, a vector representing the overall characteristics of the group of data can be generated. This arithmetic mean vector will be used as the second update block token data for subsequent data processing and feature extraction.
[0124] For example, suppose that in the first update sequence determined in the previous step, K 1 ,K 2 …K 8The block token data pair corresponding to the target second data difference is (K 1 ,K 5 ),(K 2 ,K 6 ),(K 3 ,K 10 ). Then these data pairs will be selected as the second target block token data. Specifically, the second target block token data will include the following data pairs:
[0125] Second target block token data pair 1 :K 1 ,K 5 , the second target block token data pair 2 :K 2 ,K 6 , the second target block token data pair 3 :K 3 ,K 10 ; Then, the two block token data in each data pair are averaged and the formula is as follows:
[0126] K 1,5 =(K 1 +K 5 ) / 2,
[0127] K 2,6 =(K 2 +K 6 ) / 2,
[0128] K 3,10 =(K 3 +K 10 ) / 2,
[0129] Through this calculation, we get three third arithmetic mean vectors K 1,5 ,K 2,6 , K 3,10 These average vectors will be used as the second update block token data, representing the overall features of the second target block token data, for subsequent data merging and feature extraction processes.
[0130] Sub-step 20324, delete all second target block token data from the first update sequence, and add all second update block token data to the deleted first update sequence to obtain a second update sequence.
[0131] In some embodiments of the present application, all second target block token data are deleted from the first update sequence, and all second update block token data are added to the deleted first update sequence to obtain a second update sequence. By deleting the merged target block token data and adding the new update block token data to the sequence, data redundancy can be effectively reduced, data structure can be optimized, and a simplified data sequence can be provided for subsequent processing steps.
[0132] For example, suppose that in the first update sequence K 1 ,K 2 …K 8 The second target block token data pair is (K 1 ,K 5 ), (K 2 ,K 6 ), (K 3 ,K 10 ), the corresponding second update block token data is K 1,5 ,K 2,6 ,K 3,10 Then the second target block token data K 1 ,K 2 ,K 3 ,K 5 ,K 6 ,K 10 Delete from the sequence, the remaining sequence is K 4 ,K 7 ,K 8 Then, vector K 1,5 ,K 2,6 ,K 3,10 Join the sequence to form the second update sequence [K 1,5 ,K 2,6 ,K 3,10 ,K 4 ,K 7 ,K 8 ]. This update sequence will be used in subsequent data processing and feature extraction processes.
[0133] Sub-step 2033, using all block token data in the second update sequence as sample data to obtain a first arithmetic mean vector, and using the obtained first arithmetic mean vector as block token average data.
[0134] In some embodiments of the present application, all block token data in the second update sequence are used as sample data to obtain a first arithmetic mean vector, and the obtained first arithmetic mean vector is used as the block token average data. The arithmetic mean vector is a vector obtained by averaging all sample data and can represent the overall characteristics of the sample data. By calculating the arithmetic mean vector of the block token data, data redundancy can be effectively reduced and representative video features can be extracted.
[0135] For example, suppose the second update sequence contains 6 blocks of token data, namely T 1,2 ,T 3,4,5 ,T 6,10 ,T 7 ,T 8 ,T 9 . Then, these 6 blocks of token data will be used as sample data. Then, the arithmetic mean vector of these 6 blocks of token data is calculated, and the formula is as follows:
[0136] Mean vector = (T 1,2 +T 3,4,5 +T 6,10 +T 7 +T 8 +T 9 ) / 6;
[0137] The first arithmetic mean vector is obtained by adding the six block token data element by element and dividing by the number of samples 6. This arithmetic mean vector is the block token average data, which contains the overall characteristics of the sample data and will be used in the subsequent video and text matching process.
[0138] Step 204 , determining the cross-modal similarity between the block token average data and the text feature data as a matching result between the video data and the text data.
[0139] Among them, cross-modal similarity is positively correlated with the matching degree between video data and text data.
[0140] The method shown in this step has been described in step 104 and will not be repeated here.
[0141] Optionally, step 204 includes the following sub-steps:
[0142] Sub-step 2041 , taking the ratio of the inner product of the block token average data and the text feature data to the product of the modulus as the cross-modal similarity between the block token average data and the text feature data.
[0143] In some embodiments of the present application, the ratio of the inner product of the block token average data and the text feature data to the product of the modulus is used as the cross-modal similarity between the block token average data and the text feature data. The inner product (dot product) is the result of multiplying and summing two vectors, and the modulus (the length of the vector) is the square root of the sum of the squares of the components of the vector. By calculating the ratio of the inner product to the product of the modulus, the cosine similarity between the two vectors can be obtained to measure their similarity. The higher the cosine similarity value, the more similar the two vectors are, thereby reflecting the degree of matching between the video data and the text data.
[0144] For example, suppose the block token average data is a vector v avg , the text feature data is vector t feat . Then the inner product of the two vectors will be calculated first:
[0145]
[0146] Among them, v avg,i and t feat,i They are vector v avg and t feat The i-th component of .
[0147] Next, we will calculate the magnitude of the two vectors:
[0148]
[0149] Finally, the ratio of the inner product to the product of the modulus is calculated, i.e. the cosine similarity:
[0150] Cosine similarity = (v avg ·t feat ) / (||v avg ||·||t feat ||);
[0151] Assuming the calculation result is 0.85, it means that the block token average data and the text feature data have a high similarity, so it can be considered that the video data and the text data have a high degree of matching. This similarity value will be used as the matching result of the video data and the text data for subsequent retrieval and analysis.
[0152] refer to Figure 3It is a schematic diagram of the execution flow in a large language model with a low-rank adaptation model (Low-Rank Adaptation of Large Language Models, LoRA) established under the method provided in the embodiment of the present application. In order to realize an efficient video text retrieval model, in the embodiment of the present application, only the pre-trained CLIP is frozen (the snowflake symbol in the figure represents that the image encoder layer is frozen), and only the LoRA is trained in the image and text encoder (the flame symbol in the figure represents that LoRA is in a trainable state). The specific execution process is carried out in the direction of steps S1 to S3, wherein A represents a block token sequence, B represents a first update sequence, C represents a second update sequence, and D represents block token average data. Steps S1 to S3 are explained separately below:
[0153] Step S1: In this step, the image encoder layer of the pre-trained CLIP model remains frozen, while the LoRA module is in a trainable state. In this way, efficient fine-tuning can be performed to extract features of video and text without changing the parameters of the pre-trained model.
[0154] Step S2: In this step, the LoRA module is continuously applied to train and optimize the data to generate a second update sequence (C). Through this gradual update method, data redundancy can be gradually reduced and more representative features can be extracted.
[0155] Step S3: In this step, the average block token data representing the features of the entire video is extracted by calculating the arithmetic mean vector of all block token data in the second update sequence. This average data will be used in the video and text matching process.
[0156] The complete CLIP model above consists of an image encoder and a text encoder, using a Transformer architecture that consists of alternating Multi-Head Self-Attention (MHSA) and Multilayer Perceptron (MLP) (×12) blocks.
[0157] Among them, the attention module in MHSA is as follows:
[0158] Attention(x) = softmax((xW q ) T (xW k ) / d 1 / 2 )(xW v );
[0159] Among them, W q ,W k ,W vare three mapping matrices respectively. Given an input text t, the text encoder first tokenizes the text description into word tokens. Then, these tokens pass through 12 text encoder layers to extract the final text features t. For an input video v, each frame image I i (i=1,…,F) are processed separately by the image encoder. The image I is segmented into fixed-size image blocks and projected into block tokens, which are input into 12 image encoder layers to extract frame features f. The final video feature v is obtained by averaging all frame features {f i} F i=1 Finally, the cross-modal contrast loss is applied to optimize both the text-to-video and video-to-text directions. This loss function maximizes the similarity of the corresponding text-video pair.
[0160] In order to train only a small number of parameters and realize CLIP migration to video text retrieval tasks, this application adopts the LoRA low-rank adaptation method. LoRA injects a trainable rank decomposition matrix into the attention module of each Transformer layer. For the pre-trained matrix in formula (1) LoRA optimizes its rank decomposition change, ΔW = W down W up , where W down ∈R D×r ,W down ∈R r×d , and the rank r is much smaller than the minimum value of D and d. bar =xW, the forward pass is modified to x bar =xW+xΔW. The trainable matrix ΔW can be merged with the frozen weight W without introducing additional inference latency.
[0161] As Figure 3 A further embodiment of Figure 4 This is a further description of the model provided in the embodiments of the present application:
[0162] In step S1, it is specifically performed in the following manner:
[0163] Step S11: randomly divide the block token data into two groups to ensure uniform data distribution and reduce data deviation.
[0164] Step S12: In each set of data, find similar token pairs and calculate the similarity between them.
[0165] Step S13: Select a target token pair according to the similarity as the object for subsequent merging.
[0166] Step S14: Add timing information to each target token pair to preserve the temporal relationship between video frames.
[0167] Step S15: Merge the target token pairs to generate updated block token data.
[0168] In step S2, the following steps are performed:
[0169] Step S21: randomly divide the block token data in the first update sequence into two groups to ensure uniform data distribution.
[0170] Step S22: In each set of data, find similar token pairs and calculate the similarity between them.
[0171] Step S23: Select a target token pair according to the similarity as the object for subsequent merging.
[0172] Step S24: Merge the target token pairs to generate updated block token data.
[0173] In summary, in the embodiment of the present application, by determining the statistical weight according to the similarity between the block token data, the characteristic distribution of the video data can be more accurately reflected, and then the statistical weight is used to calculate the statistical mean of the block token data sequence, which effectively reduces the redundancy of the data and extracts representative video features; finally, by calculating the cross-modal similarity between the block token average data and the text feature data, the cross-modal similarity is positively correlated with the matching degree, and the reliability and accuracy of the matching results are improved, and the effective matching of video and text data is achieved. Therefore, the method based on the embodiment of the present application speeds up the synthesis speed of the tokens, and solves the problem of slow matching speed caused by the large amount of image data token merging data calculation in the related art.
[0174] refer to Figure 5 , which shows a video text retrieval device 30 based on time token merging provided by an embodiment of the present application, comprising:
[0175] The data acquisition module 301 is used to acquire the video data to be matched and the text data to be matched, and extract a plurality of continuous image frames from the video data;
[0176] A feature generation module 302 is used to determine a block token data sequence of the video data according to each image frame, and determine text feature data of the text data; the block token data sequence includes a plurality of block token data of the video data;
[0177] The averaging module 303 is used to use the block token data sequence as a sampling data set to determine the statistical mean of the video data as the block token average data of the video data; the statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between two block token data;
[0178] The data matching module 304 is used to determine the cross-modal similarity between the block token average data and the text feature data as the matching result of the video data and the text data; the cross-modal similarity is positively correlated with the matching degree of the video data and the text data.
[0179] Optionally, the feature generation module 302 includes:
[0180] An image segmentation submodule, used for segmenting each image frame into multiple image blocks;
[0181] The token generation submodule is used to perform convolution processing on each image block to obtain a block feature value corresponding to each image block, and use each block feature value corresponding to the image frame as a component of the block token data corresponding to the image frame to obtain the block token data corresponding to the image frame;
[0182] The sequence generation submodule is used to sort all the block token data according to the time sequence of the corresponding image frames to obtain a block token data sequence.
[0183] Optionally, the feature generation module 302 includes:
[0184] A word segmentation submodule, used for taking the number of each word segmentation in the text data as a component of a text vector of the text data, so as to obtain the text vector of the text data;
[0185] The text feature submodule is used to input the text vector into the pre-trained text feature model to obtain text feature data of the text data.
[0186] Optionally, the averaging module 303 includes:
[0187] The inter-fragment merging submodule is used to merge the first target block token pair data in the block token data sequence according to the statistical weight of the block token data in the block token data sequence to obtain a first update sequence; the first target block token data pair is two block token data in the block token data sequence whose data difference ranking is greater than a preset first ranking threshold;
[0188] The intra-segment merging submodule is used to merge the second target block token data pairs in the first update sequence according to the statistical weights of the block token data in the first update sequence to obtain a second update sequence; the second target block token data pair is two block token data in the first update sequence whose data difference ranking is greater than a preset second ranking threshold;
[0189] The averaging submodule is used to use all the block token data in the second update sequence as sample data to obtain a first arithmetic average vector, and use the obtained first arithmetic average vector as the block token average data.
[0190] Optionally, the inter-fragment merging submodule includes:
[0191] A first grouping unit is used to divide the block token data in the block token data sequence into a plurality of block token data subsequences according to a preset grouping number, and randomly divide the block token data in each block token data subsequence into two groups to obtain first subgroups and second subgroups with the same number as the block token data subsequences;
[0192] A first difference unit, used to determine a first data difference between each block token data in each first subgroup and each block token data in each second subgroup, and determine, among the first data differences, a target first data difference that is the same as the number of preset first sorting indicators;
[0193] A first selection unit is used to select the block token data of the first subgroup and the block token data of the second subgroup corresponding to the difference degree with each target first data as a group of first target block token data;
[0194] A correction unit, used to generate a timing correction parameter for the block token data in each block token data subsequence according to the corresponding block token data subsequence, and correct the block token data in each block token subsequence by the corresponding timing correction parameter; the correction parameter is obtained according to the pre-trained frame position embedding engine training;
[0195] A first averaging unit is used to use each group of the modified first target block token data as sample data to obtain a second arithmetic average vector, and use the obtained second arithmetic average vector as the first update block token data;
[0196] The first updating unit is used to delete all first target block token data from the modified block token data sequence, and add all first update block token data to the deleted block token data sequence to obtain a first update sequence.
[0197] Optionally, the intra-fragment merging submodules include:
[0198] a second grouping unit, configured to randomly and equally divide the block token data in the first update sequence into two groups to obtain a third subgroup and a fourth subgroup;
[0199] A second difference unit, configured to determine a second data difference between each block token data in each third subgroup and each block token data in each fourth subgroup, and determine, among the second data differences, a target second data difference that is the same as the number of preset second sorting indicators;
[0200] A second selection unit is used to select the block token data of the third subgroup and the block token data of the fourth subgroup corresponding to the difference degree with each target second data as a group of second target block token data, and use each group of the selected second target block token data as sample data to obtain a third arithmetic mean vector, and use the obtained third arithmetic mean vector as the second update block token data;
[0201] The second updating unit is used to delete all the second target block token data from the first update sequence, and add all the second update block token data to the deleted first update sequence to obtain a second update sequence.
[0202] Optionally, the data matching module 304 includes:
[0203] The cross-modal similarity calculation submodule is used to take the ratio of the inner product of the block token average data and the text feature data to the product of the modulus as the cross-modal similarity between the block token average data and the text feature data.
[0204] In summary, in the embodiment of the present application, by determining the statistical weight according to the similarity between the block token data, the characteristic distribution of the video data can be more accurately reflected, and then the statistical weight is used to calculate the statistical mean of the block token data sequence, which effectively reduces the redundancy of the data and extracts representative video features; finally, by calculating the cross-modal similarity between the block token average data and the text feature data, the cross-modal similarity is positively correlated with the matching degree, and the reliability and accuracy of the matching results are improved, and the effective matching of video and text data is achieved. Therefore, the method based on the embodiment of the present application speeds up the synthesis speed of the tokens, and solves the problem of slow matching speed caused by the large amount of image data token merging data calculation in the related art.
[0205] Reference Figure 6 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .
[0206] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.
[0207] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0208] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.
[0209] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0210] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.
[0211] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0212] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0213] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0214] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.
[0215] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0216] Figure 7600 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server. Figure 7 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.
[0217] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.
[0218] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0219] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A video text retrieval method based on temporal token merging, characterized in that: include: Acquire video data to be matched and text data to be matched, and extract multiple continuous image frames from the video data; Determine a block token data sequence of the video data according to each of the image frames, and determine text feature data of the text data; The block token data sequence includes a plurality of block token data of the video data; Taking the block token data sequence as a sample data set, determining a statistical mean of the video data as block token average data of the video data; The statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between two block token data; Determine the cross-modal similarity between the block token average data and the text feature data as a matching result between the video data and the text data; the cross-modal similarity is positively correlated with the matching degree between the video data and the text data.
2. The method according to claim 1, characterized in that The step of determining a block token data sequence of the video data according to each of the image frames comprises: Dividing each of the image frames into a plurality of image blocks; Performing convolution processing on each of the image blocks respectively to obtain a block feature value corresponding to each of the image blocks, and using each block feature value corresponding to the image frame as a component of the block token data corresponding to the image frame to obtain the block token data corresponding to the image frame; All block token data are sorted according to the time sequence of the corresponding image frames to obtain the block token data sequence.
3. The method according to claim 1, characterized in that The determining of the text feature data of the text data comprises: Taking the number of each word segment in the text data as a component of a text vector of the text data, so as to obtain a text vector of the text data; The text vector is input into a pre-trained text feature model to obtain text feature data of the text data.
4. The method according to claim 1, characterized in that The method of using the block token data sequence as a sample data set and determining the statistical mean of the video data as the block token average data of the video data includes: According to the statistical weights of the block token data in the block token data sequence, the first target block token pair data in the block token data sequence are merged to obtain a first update sequence; the first target block token data pair is two block token data in the block token data sequence whose data difference ranking is greater than a preset first ranking threshold; According to the statistical weight of the block token data in the first update sequence, the second target block token data pair in the first update sequence is merged to obtain a second update sequence; the second target block token data pair is two block token data in the first update sequence whose data difference ranking is greater than a preset second ranking threshold; All block token data in the second update sequence are used as sample data to obtain a first arithmetic mean vector, and the obtained first arithmetic mean vector is used as the block token average data.
5. The method according to claim 4, characterized in that The step of merging the first target block token pair data in the block token data sequence according to the statistical weight of the block token data in the block token data sequence to obtain a first update sequence comprises: According to a preset number of groupings, the block token data in the block token data sequence is divided into a plurality of block token data subsequences, and the block token data in each of the block token data subsequences is randomly and equally divided into two groups to obtain a first subgroup and a second subgroup having the same number as the block token data subsequences; Determine a first data difference between each block token data in each of the first subgroups and each block token data in each of the second subgroups, and determine, among the first data differences, target first data differences that are equal to the number of preset first sorting indicators; Selecting the block token data of the first subgroup and the block token data of the second subgroup corresponding to the difference degree with each of the target first data as a group of first target block token data; According to the corresponding block token data subsequence, a timing correction parameter for the block token data in each of the block token data subsequences is generated, and the block token data in each of the block token subsequences is corrected by the corresponding timing correction parameter; the correction parameter is obtained according to the pre-trained frame position embedding engine training; Using the modified token data of each group of the first target block as sample data to obtain a second arithmetic mean vector, and using the obtained second arithmetic mean vector as the first update block token data; All of the first target block token data are deleted from the modified block token data sequence, and all of the first update block token data are added to the deleted block token data sequence to obtain the first update sequence.
6. The method according to claim 4, characterized in that The step of merging the second target block token data pairs in the first update sequence according to the statistical weights of the block token data in the first update sequence to obtain a second update sequence comprises: randomly dividing the block token data in the first update sequence into two equal groups to obtain a third subgroup and a fourth subgroup; Determine a second data difference between each block token data in each of the third subgroups and each block token data in each of the fourth subgroups, and determine, among the second data differences, a target second data difference that is the same as the number of preset second sorting indicators; The block token data of the third subgroup and the block token data of the fourth subgroup corresponding to the difference degree of each target second data are selected as a group of second target block token data, and each group of the selected second target block token data is used as sample data to obtain a third arithmetic mean vector, and the obtained third arithmetic mean vector is used as the second update block token data; All the second target block token data are deleted from the first update sequence, and all the second update block token data are added to the deleted first update sequence to obtain the second update sequence.
7. The method according to claim 1, characterized in that Determining the cross-modal similarity between the block token average data and the text feature data includes: The ratio of the inner product of the block token average data and the text feature data to the product of the modulus is taken as the cross-modal similarity between the block token average data and the text feature data.
8. A video text retrieval device based on time token merging, characterized in that: include: A data acquisition module, used to acquire video data to be matched and text data to be matched, and extract a plurality of continuous image frames from the video data; A feature generation module, used for determining a block token data sequence of the video data according to each of the image frames, and determining text feature data of the text data; The block token data sequence includes a plurality of block token data of the video data; an averaging module, configured to use the block token data sequence as a sampling data set, and determine a statistical mean of the video data as block token average data of the video data; The statistical weight of the statistical mean is determined according to the token similarity between the block token data in the block token data sequence; the token similarity is negatively correlated with the data difference between two block token data; A data matching module is used to determine the cross-modal similarity between the block token average data and the text feature data as a matching result between the video data and the text data; the cross-modal similarity is positively correlated with the matching degree between the video data and the text data.
9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.
Citation Information
Patent Citations
Cross-modal text-video retrieval method based on space-time relationship enhancement
CN114048351A
Video text retrieval method based on pre-training model
CN116109960A
Video processing method, electronic equipment, storage medium and computer program product
CN116343079A
Token compression and bidirectional asymmetric matching multi-modal query image retrieval method
CN116910287A
Video text retrieval method based on BEiT-3 multi-mode large model
CN118377930A
Cited By
Text video retrieval method based on human brain scene memory pathway inspiration
CN119938985A
Video feature extraction model training method and system based on spatio-temporal information aggregation and feature extraction method
CN120219761A
A video feature extraction model training method, system and feature extraction method based on spatiotemporal information aggregation
CN120219761B
Multi-label cross-modal video-text retrieval method based on Mamba and storage medium
CN120780871A
