Data processing method and apparatus, computer device, computer-readable storage medium, and computer program product

Abstract

The present application discloses a data processing method and apparatus, a computer device, a computer-readable storage medium, and a computer program product. The method comprises: performing feature extraction on video data to be matched, so as to obtain a compressed visual token vector; on the basis of the compressed visual token vector, generating a text feature vector to be matched having a vector length being an alignment length; acquiring a service text feature vector corresponding to service video data, wherein the service text feature vector comprises semantics for representing a content description text of the service video data; and generating a text similarity between the service text feature vector and the text feature vector to be matched, and determining service video data associated with a service text feature vector having a text similarity greater than or equal to a text similarity threshold as a text-retrieved video set, wherein the text-retrieved video set is used for determining a similar video retrieval result for the video data to be matched.

Description

Data processing methods, apparatus, computer equipment, computer-readable storage media and computer program products

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202411405640.0, filed on October 10, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0004] Visual similarity retrieval technology is primarily based on content-based video retrieval (CBVR). This technology takes user input as query content (such as images or videos), uses a visual feature extractor to extract visual features, or performs structured analysis on the query content to obtain its visual features, and then compares the visual features of the query content with those in a video library to identify videos that are similar to the query content.

[0005] Video retrieval based on visual content often involves complex image content within video frames. The model may focus too much on local features and ignore the global contextual information of the video content, thus reducing the accuracy of video retrieval.

[0006] In video review applications, users may use adversarial attacks to alter the visual representation of a video through subtle perturbations, such as cutting the video into multiple segments, adding borders, texture effects, or obscuring emojis. The model may fail to correctly identify these visual features, resulting in false positives (classifying different videos as similar videos) or false negatives (classifying similar videos as different videos). This leads to low accuracy, incomplete retrieval, and poor resistance to perturbations in video retrieval. Summary of the Invention

[0007] This application provides a data processing method, apparatus, device, and readable storage medium that can improve the accuracy and completeness of video retrieval.

[0008] One embodiment of this application provides a data processing method, including:

[0009] Feature extraction is performed on the video data to be matched to obtain a compressed visual label vector;

[0010] The process involves acquiring business text information, concatenating it with compressed visual marker vectors to obtain a fused feature vector, aligning the fused feature vector to obtain a text feature vector of the target text with a length equal to the alignment length. The business text information is a text prompt indicating the content description text corresponding to the extracted video data. The fused feature vector includes semantics representing the content description text of the target video data, and its alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0011] Obtain the business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0012] The text similarity between the business text feature vector and the text feature vector to be matched is generated. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as the text retrieval video set. The text retrieval video set is used to determine the similar video retrieval results for the video data to be matched.

[0013] This application provides another data processing method, including:

[0014] Obtain sample video data and labeled video data, and input the sample video data and labeled video data into the initial video retrieval model;

[0015] The initial video retrieval model extracts features from sample video data to obtain a first visual marker vector. Business text information is then acquired, and the business text information is concatenated with the first visual marker vector to obtain a first fusion feature vector. Features are extracted from labeled video data to obtain a second visual marker vector, and the business text information is concatenated with the second visual marker vector to obtain a second fusion feature vector. The business text information is a text prompt indicating the extraction of video content description text. The first fusion feature vector includes semantics representing the content description text of the sample video data, and the second fusion feature vector includes semantics representing the content description text of the labeled video data.

[0016] The alignment length is determined based on the vector lengths of the first and second fused feature vectors. The first fused feature vector is aligned to obtain a first text feature vector with a vector length equal to the alignment length. The second fused feature vector is aligned to obtain a second text feature vector with a vector length equal to the alignment length.

[0017] The text similarity between the first and second text feature vectors is generated. A model loss value is generated based on the text similarity. The model parameters of the initial video retrieval model are adjusted based on the model loss value. When the initial video retrieval model meets the model training convergence condition, the target video retrieval model is obtained. The target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0018] One embodiment of this application provides a data processing apparatus, including:

[0019] The visual marker extraction module is configured to extract features from the video data to be matched, and obtain a compressed visual marker vector.

[0020] The visual marker processing module is configured to acquire business text information, concatenate the business text information with compressed visual marker vectors to obtain a fused feature vector, and align the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length. The business text information is a text prompt used to indicate the content description text corresponding to the extracted video data. The fused feature vector includes the semantics of the content description text used to characterize the video data to be matched, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0021] The text feature acquisition module is configured to acquire the business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0022] The similarity calculation module is configured to generate text similarity between business text feature vectors and text feature vectors to be matched, and to identify business video data associated with business text feature vectors whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0023] One embodiment of this application provides another data processing apparatus, including:

[0024] The sample acquisition module is configured to acquire sample video data and labeled video data, and input the sample video data and labeled video data into the initial video retrieval model.

[0025] The fusion vector generation module is configured to extract features from sample video data using an initial video retrieval model to obtain a first visual marker vector, acquire business text information, concatenate the business text information with the first visual marker vector to obtain a first fusion feature vector, extract features from labeled video data to obtain a second visual marker vector, and concatenate the business text information with the second visual marker vector to obtain a second fusion feature vector. The business text information is a text prompt used to indicate the extraction of video content description text. The first fusion feature vector includes semantics used to characterize the content description text of the sample video data, and the second fusion feature vector includes semantics used to characterize the content description text of the labeled video data.

[0026] The vector alignment module is configured to determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, perform alignment processing on the first fused feature vector to obtain a first text feature vector with a vector length equal to the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector with a vector length equal to the alignment length.

[0027] The model adjustment module is configured to generate the text similarity between the first text feature vector and the second text feature vector, generate the model loss value based on the text similarity, and adjust the model parameters of the initial video retrieval model based on the model loss value. When the initial video retrieval model meets the model training convergence condition, the target video retrieval model is obtained. The target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0028] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface;

[0029] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, and the memory is used to store computer programs. When the computer program is executed by the processor, the computer device performs the method provided in the embodiments of this application.

[0030] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0031] One embodiment of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application embodiment.

[0032] This embodiment of the application extracts features from the video data to be matched to obtain a compressed visual marker vector, acquires business text information, and concatenates the business text information and the compressed visual marker vector to obtain a fused feature vector. The business text information is a text prompt indicating the content description text corresponding to the extracted video data, and the fused feature vector includes semantics representing the content description text of the video data to be matched. The fused feature vector can ignore visual elements in the video data that are unrelated to the video content, such as borders, texture effects, or occluded emojis added by adversarial methods, which can improve the robustness and perturbation resistance of the model. By aligning the fused feature vector, a text feature vector to be matched with a vector length equal to the alignment length is obtained. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. The business text feature vector corresponding to the business video data is acquired, wherein the business text feature vector includes semantics representing the content description text of the business video data. By calculating the text similarity between the feature vectors of the business text and the feature vectors of the text to be matched, the text similarity between them is obtained. Specifically, text similarity calculation is performed on feature vectors of the same length to resolve the padding problem caused by inconsistent vector lengths, reducing computational costs and improving the efficiency of similar video retrieval. Business video data associated with business text feature vectors whose text similarity is greater than or equal to a text similarity threshold are defined as a text retrieval video set. Using this set to determine similar video retrieval results for the video data to be matched effectively avoids the problem of missed or false detections caused by misleading similarity scores due to visual elements, thereby improving the accuracy and completeness of similar video retrieval. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 is a schematic diagram of a network architecture provided in an embodiment of this application;

[0035] Figure 2 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0036] Figure 3 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0037] Figure 4 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0038] Figure 5 is a schematic diagram of the model structure of a target video retrieval model provided in an embodiment of this application;

[0039] Figure 6 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0040] Figure 7 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0041] Figure 8 is a schematic diagram of a model structure of an encoding and decoding network provided in an embodiment of this application;

[0042] Figure 9 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0043] Figure 10 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0044] Figure 11 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] It is understood that in the specific embodiments of this application, the user (object or player) data involved requires user permission or consent when the above and below embodiments of this application are applied to specific products or technologies, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0047] If this application requires the collection of object data (such as user data), a prompt interface or pop-up window will be displayed before and during the collection process. This prompt interface or pop-up window is used to inform the user that certain data is being collected. The data acquisition steps will only begin after the user confirms the prompt interface or pop-up window; otherwise, the process will end. Furthermore, the acquired user data will be used in reasonable and legal scenarios or for legitimate purposes. In scenarios where user data needs to be used but user authorization has not been obtained, authorization can be requested from the user, and the user data will only be used after authorization is granted.

[0048] Please refer to Figure 1, which is a schematic diagram of a network architecture provided in an embodiment of this application. As shown in Figure 1, the network architecture may include a service server 100 and a cluster of terminal devices. The cluster of terminal devices may include terminal devices 10a, 10b, ..., 10n. Any terminal device in the cluster may have a communication connection with the service server 100. For example, terminal device 10a may have a communication connection with the service server 100, and terminal device 10b may have a communication connection with the service server 100. The communication connection is not limited to a specific method. It may be directly or indirectly connected via wired communication, or directly or indirectly connected via wireless communication, or through other methods. This application does not impose any restrictions on this method.

[0049] Each terminal device in the terminal device cluster can include: smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, in-vehicle terminals, aircraft, and other intelligent terminals with data processing capabilities. In-vehicle terminals can be terminal devices for intelligent transportation scenarios and assisted driving scenarios. It should be understood that each terminal device in the terminal device cluster shown in Figure 1 can have an application client with data processing capabilities installed. When the application client runs on each terminal device, it can interact with the business server 100 shown in Figure 1.

[0050] Specifically, the application client may include: in-vehicle client, smart home client, entertainment client (e.g., game client), multimedia client (e.g., video client), social client, and information client (e.g., news client). In this embodiment, the application client may be integrated into a client (e.g., a social client) or may be a standalone client (e.g., a news client). This embodiment does not limit the type of application client.

[0051] Among them, the business server 100 can be the server corresponding to the application client. The business server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0052] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology applied to the cloud computing business model. It can form resource pools, available on demand, and offers flexibility and convenience. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support to be realized through cloud computing.

[0053] As shown in Figure 1, for ease of understanding, terminal device 10a will be used as an example. Users can upload video data to be matched through the application client in terminal device 10a, and send the video data to be matched to the business server 100.

[0054] The service server 100 can acquire the video data to be matched sent by the terminal device 10a, perform feature extraction on the video data to be matched, and obtain a fused feature vector. The fused feature vector includes semantics representing the content description text (caption) of the video data to be matched. The fused feature vector can be a feature vector used to generate the content description text for the video data to be matched. The fused feature vector can be extracted using a multimodal large language model (LLM).

[0055] The business server 100 can perform alignment processing on the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length. It then compares the similarity between the text feature vector to be matched and the corresponding business text feature vector in the video text feature database to obtain the text similarity between the business text feature vector and the text feature vector to be matched. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as the text retrieval video set.

[0056] The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. The business text feature vector includes the semantics of the content description text used to characterize the business video data. The text retrieval video set is used to determine the similar video retrieval results for the video data to be matched.

[0057] It is understandable that terminal device 10a and business server 100 can provide users with a more personalized and coherent viewing experience through similar video search results. The recommendation system can recommend content similar to videos previously watched by the user, improving user satisfaction and content discovery rate. During the video review process, similar video search results can help identify and filter out duplicate or illegal content, ensuring the diversity and compliance of platform content.

[0058] This application embodiment extracts features from the video data to be matched to obtain a fused feature vector. The business text information is a text prompt indicating the content description text corresponding to the extracted video data. The fused feature vector includes semantics representing the content description text of the video data to be matched. The fused feature vector can ignore visual elements in the video data that are unrelated to the video content, such as borders, texture effects, or occluded emojis added by adversarial methods, thus improving the robustness and perturbation resistance of the model. Generating the fused feature vector through a multimodal large language model can more effectively resist interference and maintain the accuracy and stability of the description. When describing a pair of videos from the same source but processed by different enhancement methods, the multimodal large model tends to ignore elements in the video that are not highly relevant to the video content, such as emojis and borders. It outputs semantically more consistent text feature vectors for videos from the same source with identical content. Therefore, the similarity between them can be calculated more accurately through text feature vectors, thereby improving the accuracy of similar video retrieval.

[0059] Please refer to Figure 2, which is a schematic diagram of a data processing scenario provided by an embodiment of this application. As shown in Figure 2, a user can upload a video to be matched through a terminal device, and the business server can perform a similar video search on the obtained video to be matched through a video text database. The video to be matched can be as shown in video frame sequence 110, which may include video frames 111, 112, and 113. Taking a video text database that includes business video data 1 and business video data 2 as an example, business video data 1 can be as shown in video frame sequence 120, which may include video frames 121, 122, and 123. Business video data 2 can be as shown in video frame sequence 130, which may include video frames 131, 132, and 133. The terminal device can be any of the terminal devices in the embodiment corresponding to Figure 1, such as terminal device 10a, and the business server can be the business server 100 in the embodiment corresponding to Figure 1. The video data to be matched can be user-generated content (UGC), which is interactive media created by users, including short videos, long videos, and other forms.

[0060] It is understandable that the video data to be matched can be obtained by the user modifying the business video data 1 using adversarial methods. These adversarial methods can include editing video frames, deleting or modifying text, adding textures or mosaics, etc. For example, video frame 111 can be obtained by changing the video title 2 of video frame 121 to video title 1, and adding a mosaic to the main subject A2 in video frame 121 to obtain the main subject A1 in video frame 111; video frame 112 can be obtained by adding a mosaic to the main subject B2 in video frame 122 to obtain the main subject B1 in video frame 112, and adding texture D; video frame 113 can be obtained by adding a mosaic to the main subject C2 in video frame 123 to obtain the main subject C1 in video frame 113.

[0061] The business server can extract features from the video data to be matched, obtaining a compressed visual marker vector. The business server can also acquire business text information, which can be a text prompt indicating the content description text corresponding to the extracted video data. For example, it could be a precise and concise annotation of the input video data to be matched. The business server can extract features from the business text information, obtaining a text prompt feature vector. This text prompt feature vector is then concatenated with the compressed visual marker vector to obtain a fused feature vector. The fused feature vector is then aligned to obtain a text feature vector of the matchable data with a length equal to the aligned length. The fused feature vector includes semantics representing the content description text of the video data to be matched, and its alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0062] The business server can calculate text similarity between the feature vector of the text to be matched and the feature vector in the video text feature database. For example, it can calculate the text similarity between the feature vector of the text to be matched and text feature vector 1, and between the feature vector of the text to be matched and text feature vector 2. Here, text feature vector 1 can be obtained by extracting features from business video data 1, and text feature vector 2 can be obtained by extracting features from business video data 2. The generation method of text feature vector 1 and text feature vector 2 can be the same as the generation method of the feature vector of the text to be matched, thus ensuring that the vector length of the feature vector of the text to be matched is the same as the vector length of the text feature vector.

[0063] The business server can generate a text similarity score 1 between text feature vector 1 and the text feature vector to be matched, and a text similarity score 2 between text feature vector 2 and the text feature vector to be matched. For example, if the text similarity score 1 is greater than the text similarity threshold, the business server can determine the business video data 1 associated with the text similarity score 1 as the text retrieval video set, and determine the similar video retrieval results for the video data to be matched through the text retrieval video set. Since the fused feature vector can only include the semantics of the text representing the content description, various deformations of visual elements in the video frame sequence 110 can be ignored. The fused feature vector mainly contains the semantics of the plot content of the video data to be matched, so it can more accurately detect the similarity between the video data to be matched and the business video data 1. The embodiments of this application aim to improve the accuracy of similar video retrieval and can be applied to copyright detection systems, video recommendation systems, and video deduplication systems. By improving the accuracy of similar video detection, the embodiments of this application can effectively combat various review attack methods, such as adding texture effects, emojis, and borders to videos. Even after the video has been processed by these attack methods, it can still match the same source videos in the video library well, thereby improving the retrieval accuracy and reducing false detections and false negatives. In a copyright detection system, this application embodiment can more accurately search for similar videos in the copyright video library when an input video is uploaded, thereby better blocking the upload and recommendation of copyright-free videos to prevent users from uploading copyright-free content; or it can search for too many videos identical to the current video in the library of other recently uploaded videos to reduce the upload and recommendation of too much homogeneous content. In a video recommendation system, this application embodiment can reduce the situation where completely identical videos are repeatedly recommended in a short period of time due to missed detection of similar videos, and reduce the number of videos with completely identical content recommended to users in a short period of time, thereby improving user experience. In a video deduplication system, this application embodiment can more accurately search for whether newly uploaded videos by users have the same source videos in the video library, improve the review process, and reduce the excessive appearance of identical videos in the video library.

[0064] Please refer to Figure 3, which is a flowchart illustrating a data processing method according to an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1 or any terminal device in a terminal device cluster, such as terminal device 10a. The following description will use the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S101-S104:

[0065] Step S101: Extract features from the video data to be matched to obtain a compressed visual label vector;

[0066] Specifically, the computer device can sample the video data to be matched, obtaining sampled video frames. For example, if the video data to be matched is 60 seconds long and has a frame rate of 24 frames per second, the computer device can sample one video frame per second as a sampled video frame, obtaining 60 sampled video frames. The computer device can then extract features from the sampled video frames to obtain compressed visual label vectors.

[0067] It is understandable that computer devices can extract visual token vectors to be matched from video data to be matched, and then perform attention processing on the visual token vectors to be matched based on compressed query vectors to obtain compressed visual token vectors. The compressed visual token vector can be a vector generated by performing attention processing on the visual tokens corresponding to the sampled video frames through a set of fixed query vectors, such as compressed query vectors. The compressed visual token vector can be used to represent higher-level semantic features in visual tokens.

[0068] Step S102: Obtain business text information, concatenate the business text information with the compressed visual marker vector to obtain a fused feature vector, and align the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length. The business text information is a text prompt used to indicate the content description text corresponding to the extracted video data. The fused feature vector includes semantics used to characterize the content description text of the video data to be matched. The fused feature vector can be used to generate a feature vector for the content description text of the video data to be matched. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0069] Specifically, the business server can obtain business text information, which can be text prompts used to indicate the content description text corresponding to the extracted video data. For example, it can be a prompt that you need to make accurate and concise annotations on the input video data to be matched. The business text information can be used to instruct and guide the model to generate content description text that conforms to the text prompts.

[0070] Computer devices can use a tokenizer to segment and encode business text information into tokens. For example, the business text information is "Let's do tokenization!", and the corresponding token sequence can be represented as [Let, 's, do, token, ization]. Here, [Let, 's], [do], [token], and [ization] are all tokens. The text segmentation method can be word-based, character-based, or subword-based; this embodiment does not impose any limitations. The tokens for the business text information can include [CLS], [Let, 's], [do], [token], [ization], and [SEP]. A token in the token sequence refers to the smallest basic unit obtained after segmenting the business text information. [CLS] can be added to the beginning of the tokenizer's segmentation result and is a special token used for classification tasks. [SEP] can be added to the end of the Tokenizer segmentation result and is a special token used for segmenting sentence pairs. All tokens in the business text information can be embedded into vectors of the same size to obtain the text prompt feature vector corresponding to the business text information. The text prompt feature vector and the compressed visual marker vector are concatenated to obtain the fused feature vector. The fused feature vector is then aligned to obtain the text feature vector to be matched with a vector length equal to the alignment length. The fused feature vector includes the semantics of the content description text representing the video data to be matched. The fused feature vector can be a feature vector used to generate the content description text for the video data to be matched. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Since the fused feature vector only needs to contain the semantics representing the content description text, it does not need to retain all visual marker-related features, thus reducing the length of the fused feature vector and lowering the computational cost.

[0071] It is understandable that the video length and content of the video data to be matched are not fixed, and the fused feature vector generated from the video data to be matched is also not fixed. Directly calculating the similarity with the business text feature vector of the business video data will require padding due to the inconsistency in vector length. For example, padding the business text feature vector and the fused feature vector to the same length may lead to problems such as reduced computational efficiency, increased memory consumption, and the influencing of the model's capture of real data and reduced generalization. The embodiments of this application, through the above alignment processing, can effectively generate a text feature vector to be matched with a vector length equal to the alignment length from the fused feature vector. Calculating the similarity between the text feature vector to be matched and the business text feature vector can reduce the computational cost of similarity calculation.

[0072] Step S103: Obtain the business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data.

[0073] Specifically, the computer equipment can acquire the business text feature vectors corresponding to the business video data in the video text feature database. These business text feature vectors can be obtained by feature extraction from the business video data. They include semantics of the content description text representing the business video data, and can be used to generate feature vectors for the content description text of the video data to be matched. The generation method of the business text feature vectors can be the same as the generation method of the feature vectors of the text to be matched, thus ensuring that the vector length of the feature vectors of the text to be matched is the same as the vector length of the business text feature vectors.

[0074] Step S104: Generate the text similarity between the business text feature vector and the text feature vector to be matched. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as the text retrieval video set. The text retrieval video set is used to determine the similar video retrieval results for the video data to be matched.

[0075] Specifically, the business server can calculate the text similarity between the feature vector of the text to be matched and the feature vector in the video text feature database, obtaining the text similarity for each business video data. The business video data associated with the business text feature vectors whose text similarity is greater than or equal to a text similarity threshold are then identified as the text retrieval video set. This text retrieval video set can then be used to determine the similar video retrieval results for the video data to be matched.

[0076] Computer devices can leverage similar video search results to provide users with a more personalized and coherent viewing experience. Recommendation systems can suggest content similar to videos previously viewed by users, improving user satisfaction and content discovery rates. During video review, similar video search results can help identify and filter out duplicate or illegal content, ensuring the diversity and compliance of platform content. In copyright detection systems, computer devices can use similar video search results to more accurately check whether similar videos exist in the copyright video library, thereby better blocking the upload and recommendation of copyright-free videos and preventing users from uploading copyright-free content.

[0077] Computer equipment can calculate the visual similarity between video data to be matched and business video data based on visual elements. This is achieved by extracting features from the business video data to obtain a business visual feature vector, and by extracting features from the video data to be matched to obtain a visual feature vector to be matched. The business visual feature vector includes the semantics representing visual elements in the business video data, and the visual feature vector to be matched includes the semantics representing visual elements in the video data to be matched. The computer equipment can generate a visual similarity score between the business visual feature vector and the visual feature vector to be matched. Business video data associated with business visual feature vectors whose visual similarity is greater than or equal to a video similarity threshold are then used to determine the visual retrieval video set.

[0078] It is understandable that computer devices can use the text-based video retrieval set obtained by fusing feature vectors as a supplement to visual element similarity video retrieval. The computer device can determine the intersection of the visually retrieved video set and the text-based video set as the similarity video retrieval result for the video data to be matched, avoiding the model's overemphasis on local features and neglect of the global contextual information of the video content, which would lead to a decrease in video retrieval accuracy and improve the accuracy of similarity video retrieval. Alternatively, the computer device can determine the union of the visually retrieved video set and the text-based video set as the similarity video retrieval result for the video data to be matched. This avoids false positives or false negatives caused by adversarial attacks, improving the accuracy and robustness of similarity video retrieval. Integrating the embodiments of this application into a visual element similarity video retrieval system can significantly reduce the decrease in visual feature similarity caused by various attack methods, thereby effectively alleviating the problem of false negatives caused by reduced feature similarity during the retrieval process and improving retrieval accuracy. Because the problems of false negatives and false negatives during the retrieval process are solved, similar videos can be retrieved more comprehensively, thus ensuring the integrity of similarity video retrieval. This embodiment of the application extracts features from the video data to be matched to obtain a compressed visual marker vector, acquires business text information, and concatenates the business text information and the compressed visual marker vector to obtain a fused feature vector. The business text information is a text prompt indicating the content description text corresponding to the extracted video data, and the fused feature vector includes semantics representing the content description text of the video data to be matched. The fused feature vector can ignore visual elements in the video data that are unrelated to the video content, such as borders, texture effects, or occluded emojis added by adversarial methods, which can improve the robustness and perturbation resistance of the model. By aligning the fused feature vector, a text feature vector to be matched with a vector length equal to the alignment length is obtained. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. The business text feature vector corresponding to the business video data is acquired, wherein the business text feature vector includes semantics representing the content description text of the business video data. By calculating the text similarity between the feature vectors of the business text and the feature vectors of the text to be matched, the text similarity between them is obtained. Specifically, calculating the text similarity between business text feature vectors and the text feature vectors of the text to be matched with the same vector length avoids the padding problem caused by inconsistent vector lengths, reduces computational costs, and improves the efficiency of similar video retrieval. The business video data associated with business text feature vectors whose text similarity is greater than or equal to a text similarity threshold is defined as the text retrieval video set. Using this text retrieval video set to determine the similar video retrieval results for the video data to be matched avoids the problem of missed detections and false detections caused by misleading similarity scores due to visual elements, thus improving the accuracy and completeness of similar video retrieval.

[0079] Please refer to Figure 4, which is a flowchart illustrating a data processing method provided in this application embodiment. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1 or any terminal device in a terminal device cluster, such as terminal device 10a. The following description will use the execution of this data processing method by a computer device as an example. Feature extraction is achieved through a multimodal text description layer in the target video retrieval model. The multimodal text description layer includes a preprocessing layer, a visual encoding layer, and a visual abstraction layer. The data processing method can include at least the following steps S201-S207:

[0080] Step S201: Input the video data to be matched into the target video retrieval model; the target video retrieval model includes a multimodal text description layer, which includes a preprocessing layer, a visual coding layer, and a visual abstraction layer; in the preprocessing layer, based on the video duration and frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames, and image segmentation is performed on the T sampled video frames to obtain S unit sampled images; T is a positive integer determined by the video duration and frame rate, and S is a positive integer greater than T; in the visual coding layer, feature extraction is performed on the S unit sampled images to obtain S unit visual marker vectors, and based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual marker vectors are remapped to the feature channels to obtain S visual marker vectors to be matched; the rearrangement factor is determined based on the number S of unit visual marker vectors, and the vector length of the unit visual marker vector is greater than the vector length of the visual marker vector to be matched;

[0081] Specifically, the computer device can input the video data to be matched into the target video retrieval model. Please also refer to Figure 5, which is a schematic diagram of the model structure of a target video retrieval model proposed in this embodiment. As shown in Figure 5, the target video retrieval model includes a multimodal text description layer, a feature alignment layer, and a recall matching layer. The multimodal text description layer includes a preprocessing layer, a visual encoding layer, a visual abstraction layer, and a text description layer.

[0082] The computer device can input the video data to be matched into the preprocessing layer of the target video retrieval model. In the preprocessing layer, the computer device can uniformly sample the video data to be matched based on its video duration and frame rate, obtaining T sampled video frames. For example, if the video data to be matched is 60 seconds long and has a frame rate of 24 frames per second, the computer device can sample one video frame per second, resulting in 60 sampled video frames. When the video length is 30 seconds and the frame rate is 50 frames per second, the computer device can sample two video frames per second, resulting in 60 sampled video frames. Here, T can be a positive integer determined by the video duration, video frame rate, and the context window length of the target video retrieval model. The context window length refers to the maximum data length that the target video retrieval model can input when processing input information.

[0083] Computer equipment can perform unified preprocessing on the sampled video frames. Preprocessing can include mean normalization and bicubic interpolation. Normalization adjusts the pixel values ​​of the image data to a specific range, typically [0, 1] or [-1, 1]. Bicubic interpolation scales the image size to a specific range determined by the context window length of the target video retrieval model, for example, 448×448 pixels. The computer equipment can then perform image segmentation on the preprocessed T sampled video frames, dividing them into several fixed-size unit sample images. Each unit sample image can be flattened and mapped to a fixed-dimensional vector (e.g., 14×14 pixels), while adding positional encoding to preserve the relative position information of the unit sample images, resulting in S unit sample images. S is a positive integer greater than T.

[0084] In the visual coding layer, the computer can extract features from S unit sampled images to obtain S unit visual label vectors. The computer can then perform a pixel shuffle on these S unit visual label vectors, rearranging the elements and redistributing high-dimensional spatial channel information to the feature channel dimensions. For example, the shape of the S unit sampled images can be a tensor of B×T×H×W×C. Here, T×H×W represents S, B (Batchsize, the number of data points processed by the model at one time), H (Height, image height), and W (Width, image width) represent the spatial channels, and C (Channels, the number of feature channels) represents the feature channels. The visual coding layer can be trained using an InternViT-300M (ViT (Vision Transformer) model with 300M parameters) model.

[0085] Taking the rearrangement factor r as an example, the rearrangement factor is determined based on the number S of unit visual label vectors, where r is a positive integer less than S. Specifically, a computer device can rearrange S unit visual label vectors according to the rearrangement factor r, remapping the vector elements representing spatial channels to feature channels. More specifically, the computer device can rearrange a unit sampled image according to the rearrangement factor r, reducing the image width and height to 1 / r of their original values, and then rearranging these reduced image patches into new feature channels, thus increasing the number of feature channels to r. 2 For example, the shape of the S visual token vectors to be matched can be a tensor of B×T×(H / r)×(W / r)×(C×r²), where the length of a single visual token vector is greater than the length of the visual token vector to be matched. For example, if a rearrangement factor of 2 is chosen, for a 10-frame video A, each frame will be scaled to 448×448, and each frame will be divided into 1024 visual tokens. Each frame can be processed by pixel shuffle to obtain 256 visual tokens. Therefore, video A can include 10×256 visual tokens.

[0086] It is understandable that Pixel Shuffle can reduce the number of visual markers by rearranging the pixels of an image to reduce the vector dimension representing the height and width of the image per unit sample.

[0087] In step S202, in the visual abstraction layer, attention processing is performed on the visual tag vector to be matched based on the compressed query vector to obtain a compressed visual tag vector; the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector.

[0088] In some embodiments, the visual abstraction layer is a feature processing module that abstracts the raw visual information of the video (such as pixels, object outlines, motion trajectories, etc.). Simply put, it extracts more crucial and concise features from complex visual details, filtering out redundant information. The compressed query vector is a pre-defined guiding vector, acting like a filter or magnifying glass, defining the direction of visual features to focus on (e.g., focusing more on human actions than background color), and it has a fixed vector length. The visual marker vector to be matched interacts with the compressed query vector—by calculating the correlation between the two (similar to which visual features are more relevant to the query direction), higher weights are assigned to important visual features, while irrelevant features are assigned lower weights or even ignored. After attention processing, the final compressed visual marker vector is obtained, which has two key characteristics: Preservation of core information: Only key visual features related to the compressed query vector are retained, achieving feature compression and abstraction. Length matching: Its vector length is exactly the same as the compressed query vector, providing a unified metric for subsequent operations such as concatenation and comparison with other vectors (e.g., text feature vectors), avoiding computational problems caused by inconsistent lengths.

[0089] In some embodiments, step S202 can be implemented as follows: In the visual abstraction layer, a compressed query vector is obtained; the visual tag vector to be matched and the compressed query vector are concatenated to obtain a visual tag vector sequence; the compressed query vector is a learnable parameter; the compressed query vector and the visual tag vector sequence are assigned to N unit processing spaces; cross-attention processing is performed on the compressed query vector and the visual tag vector sequence in each unit processing space to obtain N unit attention result vectors; based on the attention weights corresponding to the N unit attention result vectors and the N unit processing spaces, a compressed visual tag vector is generated; N is a positive integer.

[0090] Specifically, the computer device can input S visual label vectors to be matched into a visual abstraction layer. In the visual abstraction layer, a compressed query vector is obtained. The S visual label vectors to be matched and the compressed query vector are concatenated to obtain a sequence of visual label vectors. The compressed query vector is a learnable parameter obtained through training, and its length can be R, where R is a positive integer less than S. The compressed query vector can be a fixed set of query vectors used to extract higher-level semantic features from the visual labels of multiple frames of images by learning a fixed set of query vectors.

[0091] In some embodiments, the core inputs are prepared as follows: Compressed query vector and raw visual features: Obtaining the compressed query vector: This is a vector with learnable parameters (not a fixed value, but continuously optimized during model training). It acts like a dynamic filter—as training progresses, it gradually learns to focus on visual features more important to the task (e.g., in video retrieval tasks, it may focus more on the subject's actions than background details). Visual tag vector to be matched: The raw visual feature vector extracted from the video to be matched (containing information about various visual elements in the video, such as objects, colors, motion trajectories, etc.). Vector concatenation to obtain a visual tag vector sequence: The visual tag vector to be matched is concatenated with the compressed query vector to form a longer sequence. This step is to initially fuse the information of both, providing a foundation for subsequent cross-attention processing. Multi-space parallel processing: The role of N units of processing space is distributed across N units of processing space: The compressed query vector and the visual tag vector sequence are simultaneously copied to N independent unit processing spaces (which can be understood as N parallel working sub-modules). N is a positive integer, such as N=8, which means that 8 experts analyze the same set of visual features from different angles—some may focus on color, some on shape, and some on motion, capturing more comprehensive information through division of labor.

[0092] In some embodiments, the core operation for each space is cross-attention processing. Cross-attention processing involves calculating the correlation (i.e., attention) between a compressed query vector as the query and a sequence of visual marker vectors as the key and value within each unit processing space. Simply put, each submodule, based on its own compressed query vector (although the original query vectors are the same, different spaces may have subtle emphases), determines which parts of the visual marker vector sequence are more important, then highlights this important information, filters redundant content, and finally obtains the unit attention result vector for that space. After this step, N spaces will output N different result vectors (each focusing on a different dimension of visual features). Integration result: Generating the final compressed visual marker vector. The role of attention weights: Each unit processing space corresponds to an attention weight (also a learnable parameter), representing the importance of that space's result in the final integration (e.g., a space that judges motion features more accurately will have a higher weight). Generating compressed visual marker vectors: The N unit attention result vectors are weighted and merged according to their respective weights, ultimately obtaining a compressed visual marker vector that integrates information from N spaces.

[0093] As an example, the visual marker vector to be matched: Suppose we are detecting a short video uploaded by a user. The raw visual feature vector extracted from the video (i.e., the visual marker vector to be matched) will contain various details: such as two people in the scene, an indoor scene, fallen furniture, a person throwing a punch, a red object (possibly paint), noisy background music, etc. (This information is encoded as a numerical vector). Vector concatenation to obtain a visual marker vector sequence: The visual marker vector to be matched (raw video features) and the compressed query vector (detection-oriented) are concatenated into a longer sequence. This step is equivalent to telling the model: Please carefully analyze the raw visual features of this video, taking into account the detection focus.

[0094] Continuing the previous example, assume N = 3 (3 parallel unit processing spaces, equivalent to 3 specialized recognizers): Space 1: focuses on analyzing the state of ingredient processing (e.g., whether tomatoes are chopped, whether eggs are beaten evenly); Space 2: focuses on analyzing cooking operation steps (e.g., whether eggs are fried first, then tomatoes, whether seasonings are added); Space 3: focuses on analyzing kitchen utensils and environment (e.g., whether an iron pot is used, whether it is in a kitchen scene). The vector sequence is assigned to these 3 spaces, processing features of different dimensions in parallel. Space 1 (Ingredient Processing Analysis): Guided by the compressed query vector (focusing on scrambled eggs with tomatoes), attention is calculated on the original features. It finds that chopped tomatoes and beaten eggs are highly relevant to the target, giving these features high weights, filtering out secondary information such as scallions, and outputting a unit attention result vector focusing on ingredient processing. Space 2 (Cooking Step Analysis): Based on the retrieval-guided vector, it focuses on the operation sequence of pouring oil → frying eggs → frying tomatoes → adding seasonings (these are the core steps of scrambled eggs with tomatoes), ignoring irrelevant information such as pre-cooking preparations, and outputting a unit attention result vector focusing on standard steps. Space 3 (Kitchenware and Environment Analysis): Focuses on features such as the iron wok kitchen scene (a typical environment for home cooking), filtering out irrelevant information such as clothing, and outputting a unit attention result vector that emphasizes the matching of cooking scenes. Attention Weights: After training, the system assigns weights to the three spaces (reflecting the importance of tomato and scrambled eggs retrieval): Space 2 (Cooking Steps) has the highest weight (0.5) – correct steps are the core value of the instructional video; Space 1 (Ingredient Processing) has the second highest weight (0.3) – ingredient processing affects the finished product and is a key detail; Space 3 (Kitchenware and Environment) has a lower weight (0.2) – the scene has a smaller impact on the relevance of the instruction. Weighted Merging: The three unit result vectors are merged according to the above weights to obtain the final compressed visual label vector. This vector highlights core features such as tomato cutting + scrambled eggs steps + iron wok usage, filters redundant information, and has the same length as the compressed query vector.

[0095] In this way, the learnable compressed query vector can be continuously optimized as the model is trained, making it more accurately focus on the core requirements of the task and improving the targeting of feature extraction. Concatenating the visual marker vector to be matched with the compressed query vector to form a visual marker vector sequence allows for the initial fusion of information between the two, laying the foundation for subsequent processing. Through parallel processing of N unit processing spaces, visual features can be analyzed from different dimensions and angles, avoiding the limitations of a single perspective and comprehensively capturing key information. Cross-attention processing in each unit processing space can highlight important visual features related to the task, filter redundant information, and enhance the effectiveness of features. Integrating the attention result vectors of N units based on attention weights allows for the reasonable allocation of weights according to the importance of each space to the task, so that the generated compressed visual marker vector retains the core visual information, achieves dimensional compression, and has the same length as the compressed query vector, providing convenience for subsequent vector operations, matching, and other operations. Ultimately, this significantly improves the accuracy, efficiency, and adaptability of the entire system in feature extraction and processing.

[0096] Computer devices can perform attention processing on S visual label vectors to be matched based on a compressed query vector. For example, they can perform multi-head attention processing on the compressed query vector and the visual label vector sequence, distributing the compressed query vector and the visual label vector sequence to N units of processing space (head). i Process space head in each unit i The compressed query vector and the visual label vector sequence are subjected to cross-attention processing to obtain N unit attention result vectors. Multi-head attention processing can include N unit processing spaces (heads). i Each unit processes head space i All can include a query parameter matrix Key parameter matrix AND-value parameter matrix The learnable parameters in different unit processing spaces are independent of each other, and each unit processing space has a head. i Each can perform attention calculations independently, with each unit processing space head. i Both can compress the query vector and the query parameter matrix. Perform a dot product operation to obtain the query vector Q, and then combine the visual tag vector sequence with the key parameter matrix. Perform a dot product operation to obtain the key vector K, and then combine the visual marker vector sequence with the value parameters. Performing a dot product on the matrix yields a value vector V. The query parameter matrix is ​​included. Key parameter matrix AND-value parameter matrix Each matrix is ​​composed of learnable parameters in the processing space of each unit.

[0097] Computer devices can allocate compressed query vectors and visual tag vector sequences to N units of processing space (head). i Process space head in each unit i The compressed query vector and the visual tag vector sequence are subjected to cross-attention processing to obtain N unit attention result vectors Attn. i Based on N unit attention result vectors Attn i and N units of processing space head i The corresponding attention weights are used to generate compressed visual label vectors. The multi-head attention process can be shown in formula (1): MHAttn = Concat(head1, ..., head2) N W O Formula (1)

[0098] Here, MHAtn is the compressed visual label vector, and the Concat function can be used to process different unit spaces (head). i The output unit attention result vector Attn i By concatenating them together, a higher-dimensional vector representation is obtained. W O This is a multi-head attention weight matrix, comprising attention weights corresponding to N units of processing space, W. O Used to process each unit of space head i The output unit attention result vector is weighted to obtain the attention vector output by the multi-head attention layer.

[0099] It is understandable that multi-head attention computation can divide the attention mechanism into multiple heads, each of which can learn the representation of the input data in different subspaces. This enables the model to capture richer feature information from different subspaces and allows the model to process information from multiple subspaces in parallel, thereby improving the model's computational efficiency.

[0100] In some embodiments, the N unit processing spaces include a target unit processing space; the cross-attention processing of the compressed query vector and the visual marker vector sequence in each unit processing space to obtain N unit attention result vectors can be implemented as follows: In the target unit processing space, the compressed query vector is multiplied by the query parameter matrix to obtain a query vector; the visual marker vector sequence is multiplied by the key parameter matrix to obtain a key vector; the visual marker vector sequence is multiplied by the value parameter matrix to obtain a value vector; an attention score vector is generated based on the query vector and the key vector; the attention score vector is reduced in dimensionality based on the adjacent lengths of the key vector; the reduced attention score vector is normalized to obtain the attention weights corresponding to the target unit processing space; and the attention weights corresponding to the target unit processing space are multiplied by the value vectors to obtain the unit attention result vectors corresponding to the target unit processing space.

[0101] In some embodiments, N units of processing space head i This includes the target unit processing space. The process of performing cross-attention processing in the target unit processing space can be as follows: In the target unit processing space, perform a dot product operation between the compressed query vector and the query parameter matrix to obtain the query vector; perform a dot product operation between the visual tag vector sequence and the key parameter matrix to obtain the key vector; perform a dot product operation between the visual tag vector sequence and the value parameter matrix to obtain the value vector; generate an attention score vector based on the query vector and the key vector; perform dimensionality reduction processing on the attention score vector based on the adjacent length of the key vector; normalize the dimensionality-reduced attention score vector to obtain the attention weight; and perform a dot product operation between the attention weight and the value vector to obtain the unit attention result vector corresponding to the target unit processing space.

[0102] Specifically, the computer device can perform a transpose operation between the query matrix Q and the key matrix, resulting in the result K. T Perform a dot product operation to obtain the attention score matrix. This is based on the dimension value d of the key matrix K. K square root The attention score vector is dimensionality reduced, and then normalized to obtain the attention weights. The attention weights are then multiplied by the value vector to obtain the unit attention result vector Attn corresponding to the target unit processing space. i The process can be shown in formula (2):

[0103] The softmax function is used for normalization. The computer device can then base its analysis on N unit attention result vectors Attn. iGenerate a compressed visual label vector MHAttn.

[0104] In some embodiments, the original vector is transformed through matrix operations: the compressed query vector (which has learned the features related to the tomato and egg stir-fry step) is multiplied by the query parameter matrix (a learnable filter, similar to a step recognition template) specific to the space to obtain the query vector (which is equivalent to refining the original query into the step feature directions of interest in the space); at the same time, the visual label vector sequence (containing the original visual features of the video) is multiplied by the key parameter matrix and value parameter matrix (which are also learnable and used to extract feature labels and feature values, respectively) of the space to obtain the key vector (labeling the original visual features, such as step labels like adding oil, frying eggs, and tomatoes) and the value vector (corresponding to the specific feature information of these steps, such as the amount of oil and the cooking temperature of the eggs). The original vector is transformed through matrix operations: the compressed query vector (which has learned the features related to the tomato and egg scrambled egg step) is multiplied by the query parameter matrix (a learnable filter, similar to a step recognition template) specific to this space to obtain the query vector (equivalent to refining the original query into the feature directions of the steps that this space focuses on); simultaneously, the visual label vector sequence (containing the original visual features of the video) is multiplied by the key parameter matrix and value parameter matrix (also learnable, used to extract feature labels and feature values, respectively) of this space to obtain the key vector (labeling the original visual features, such as step labels like pouring oil, scrambling eggs, and adding tomatoes) and the value vector (corresponding to the specific feature information of these steps, such as the amount of oil poured and the cooking time of scrambling eggs). The attention weight is multiplied by the value vector (step details) to extract key information according to importance (emphasizing high-scoring features such as the cooking time of scrambling eggs and the timing of adding tomatoes, while weakening secondary details), finally obtaining the unit attention result vector of the target unit processing space (a condensed version of the features focusing on the core task of this space).

[0105] As an example, the relevance is calculated using the query vector (focusing on egg coagulation and frying) and the key vector (labels for each step) to obtain a score vector. For example, the coagulation start step has the highest match with the query (score 0.9), followed by the frying step (0.8), and the pouring oil step has the lowest (0.2), resulting in a score vector of [0.2, 0.5, 0.9, 0.8, 0.7, 0.3]. The original key vector may have a high dimension (e.g., containing 6 steps). Dimensionality is reduced based on adjacent lengths (e.g., grouped by preparation-core operation-finishing, merging adjacent 2 steps), simplifying it to a 3-dimensional score: [0.35 (pouring oil + pouring egg liquid), 0.85 (coagulation + frying), 0.5 (forming into pieces + serving)]. The dimensionality-reduced scores are converted into weights that sum to 1, resulting in the attention weights for the target space: [0.15, 0.6, 0.25] (i.e., the core operation steps account for 60%, making them the primary focus). Multiply the attention weights [0.15, 0.6, 0.25] with the value vector (details of each step): focus on retaining the proportion of solidified area and the speed of stir-frying (details of core operation steps, weight 0.6); secondarily retain the size of the pieces and the timing of serving (final step, weight 0.25); weaken the temperature of the oil and the amount of egg liquid (preliminary preparation, weight 0.15).

[0106] Thus, by multiplying the compressed query vector with the query parameter matrix to obtain the query vector, the focus on a specific task can be strengthened by combining space-specific parameters, improving the targeting of feature extraction. Multiplying the visual marker vector sequence with the key parameter matrix and the value parameter matrix respectively to obtain the key vector and value vector can effectively separate the feature's identifier information from its detailed information, making subsequent processing more hierarchical. Generating attention score vectors based on the query vector and key vectors can accurately measure the degree of correlation between each visual feature and the task requirements, providing a basis for selecting important features. Dimensionality reduction of the attention score vector based on the adjacent lengths of the key vectors can simplify calculations while retaining core correlation information, improving processing efficiency. Normalizing the dimensionality-reduced attention score vector to obtain attention weights can clarify the relative importance of each feature, ensuring that key features receive higher weights. Finally, multiplying the attention weights with the value vector to obtain the unit attention result vector can focus on and condense the core features highly relevant to the task, filter redundant information, and make the generated vector more representative, thereby improving the accuracy, efficiency, and adaptability of the entire feature processing, laying a solid foundation for subsequent multi-space result integration and the completion of the overall task.

[0107] Since the fused feature vector only needs to contain the semantics representing the content description text, it does not need to retain all visual marker-related features, thus reducing the length of the fused feature vector and lowering computational costs. This embodiment concatenates the visual marker vectors to be matched and the compressed query vector, which can better capture multimodal information and help the target video retrieval model understand and abstract image information at different levels and granularities, improving the processing performance of the target video retrieval model. Through the visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector. The compressed visual marker vector is obtained by performing multi-head attention processing on the compressed query vector and the visual marker vector sequence. The vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector, both being R. This effectively reduces the number of visual markers, significantly reducing memory load and computational resource consumption.

[0108] Computer equipment can use a multilayer perceptron (MLP) to perform linear transformation and nonlinear activation function processing on the compressed visual label vector MHAttn. The MLP can first perform layer normalization on the compressed visual label vector output by the visual layer, then perform linear processing on the compressed visual label vector through the linear layer Linear1, then process it through the SwiGLU activation function, and finally perform linear processing through the linear layer Linear2 to obtain the compressed visual label vector after MLP processing. The process can be shown in Equation (3): MHAttn=(Linear2(σ(Linear2(MHAttn)))) Equation (3)

[0109] Here, σ represents the SwiGLU activation function, and Linear1 and Linear2 represent the linear layer operations in the multilayer perceptron.

[0110] This application proposes a method for connecting a visual model InternViT-300M and a language model Phi-3 using a multilayer perceptron. This method converts visual symbols into feature representations suitable for language model processing, ensuring that visual information can be effectively utilized by the language model, thereby achieving multimodal information processing. Through this method, the present invention achieves efficient fusion of visual information and a language model, improving the performance of multimodal information processing.

[0111] In some embodiments, the concatenation is achieved through the text description layer in the target video retrieval model. The above-mentioned concatenation of the business text information and the compressed visual marker vector to obtain the fused feature vector can be achieved as follows: In the text description layer, feature extraction is performed on the business text information to obtain a text prompt feature vector, and the text prompt feature vector and the compressed visual marker vector are concatenated to obtain the fused feature vector.

[0112] Step S203: In the text description layer, obtain business text information, extract features from the business text information to obtain text prompt feature vector, and concatenate the text prompt feature vector with the compressed visual mark vector to obtain a fused feature vector.

[0113] Specifically, the computer device can input compressed visual label vectors into the text description layer of the multimodal text description layer. In the text description layer, business text information is obtained, and features are extracted from this information to obtain text prompt feature vectors. The business text information can be text prompts used to indicate the content description text corresponding to the extracted video data. For example, it could be a need for accurate and concise annotations of the input video data to be matched. The business text information can be used to instruct and guide the model to generate content description text that conforms to the text prompts.

[0114] In some embodiments, features are extracted from the business text information to obtain a text prompt feature vector: the business text information is prompt text used to describe the video content (e.g., in food video retrieval, it might be the detailed recipe for scrambled eggs with tomatoes, the steps for egg coagulation, etc.). The text description layer converts these texts into text prompt feature vectors through feature extraction (e.g., using pre-trained text models such as BERT, Word2Vec, etc.)—this string of numerical codes contains the semantic information of the text (e.g., the meaning of scrambled eggs with tomatoes, the direction of the coagulation steps, etc.), enabling it to be processed in the same numerical language as the visual feature vector. The text prompt feature vector and the compressed visual marker vector are concatenated to obtain a fused feature vector: the compressed visual marker vector is a condensed encoding of the video's visual features (e.g., containing visual information such as the color of the tomatoes, the shape of the eggs, and the action of stir-frying). Vector concatenation combines the text prompt feature vector (text semantics) and the compressed visual marker vector (visual features) sequentially into a longer vector, i.e., the fused feature vector. For example, the text prompt feature vector is [0.2, 0.5, 0.8] (representing the semantics of the egg coagulation step), and the compressed visual label vector is [0.3, 0.6, 0.1] (representing the visual features of the egg liquid turning white and forming clumps at the edges). The fused feature vector obtained after splicing may be [0.2, 0.5, 0.8, 0.3, 0.6, 0.1], which contains both the intent of the text description and the visual details of the video.

[0115] Computer devices can use a tokenizer to segment and encode business text information into tokens. For example, the business text information is "Let's do tokenization!", and the corresponding token sequence can be represented as [Let, 's, do, token, ization]. Here, [Let, 's], [do], [token], and [ization] are all tokens. The text segmentation method can be word-based, character-based, or subword-based; this embodiment does not impose any limitations. The tokens for the business text information can include [CLS], [Let, 's], [do], [token], [ization], and [SEP]. A token in the token sequence refers to the smallest basic unit obtained after segmenting the business text information. [CLS] can be added to the beginning of the tokenizer's segmentation result and is a special token used for classification tasks. [SEP] can be added to the end of the tokenizer's segmentation results; it's a special token used for segmenting sentence pairs. All tokens in the business text information can be embedded into a vector of the same size to obtain the text prompt feature vector corresponding to the business text information.

[0116] It's understandable that the target video retrieval model can acquire business text information and its contextual information. If the contextual information is empty, meaning the model doesn't carry information from previous rounds of dialogue, it initializes an empty list and generates image tags, which can include image start tags, image context tags, and image end tags. Then, it appends the business text information after the image tags. If the contextual information is not empty, it merges the historical dialogue as contextual information with the business text information. The merged contextual information and business text information are then appended after the image tags. This enables the target video retrieval model to have multi-turn dialogue capabilities, capture long-distance dependencies between contextual information and business text information, and generate content description text that better matches the text prompts.

[0117] The computer device can concatenate the text prompt feature vector and the compressed visual marker vector to obtain a fused feature vector. This fused feature vector includes semantics representing the content description text of the video data to be matched, and can be a feature vector used to generate the content description text for the video data to be matched. The process of cross-attention processing can be found in the detailed description of step S202 above.

[0118] The text description layer can be a large language model, such as the Phi-3-mini-128k-instruct model (an open-source, lightweight model), designed to generate high-quality text and perform various natural language processing tasks. Using a multimodal large language model to generate fused feature frames can more effectively resist interference and maintain the accuracy and stability of the description. When describing a pair of videos that are from the same source but have undergone different enhancement techniques, the multimodal large model tends to ignore elements in the videos that are not highly relevant to the video content, such as emojis and borders. For videos from the same source with identical content, it outputs semantically more consistent descriptive text, thereby improving the accuracy of similar video retrieval.

[0119] In some embodiments, the above alignment processing is implemented through the feature alignment layer in the target video retrieval model. The alignment processing of the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length can be implemented as follows: In the feature alignment processing layer, the fused feature vector is aligned to obtain a text feature vector to be matched with a vector length equal to the alignment length.

[0120] Step S204: In the feature alignment processing layer, the fused feature vector is aligned to obtain a text feature vector to be matched with a vector length equal to the alignment length.

[0121] Specifically, the computer device can input the fused feature vector into the feature alignment layer. In this layer, the fused feature vector is aligned to obtain a text feature vector of length equal to the alignment length. The feature alignment layer can employ three different alignment methods: First, it can use a Long Short-Term Memory Autoencoder (LSTMAutoencoder) for alignment; second, it can use term frequency and inverse document frequency for alignment; and third, it can use the last unit prediction vector in the fused feature vector as the text feature vector of length equal to the alignment length.

[0122] In some embodiments, if the feature alignment processing layer adopts a first alignment method, that is, the feature alignment processing layer can be a long short-term memory network autoencoder, the feature alignment processing layer includes an autoencoder layer and a bottleneck reconstruction layer, the bottleneck reconstruction layer is trained based on the alignment length, and the length of the vector output by the bottleneck reconstruction layer is fixed to the alignment length; then the computer device can perform alignment processing through the long short-term memory network autoencoder, the process of which may include: in the autoencoder layer, encoding the fused feature vector to obtain an encoded feature vector, and downsampling the encoded feature vector based on the time step to obtain a downsampled feature vector; in the bottleneck reconstruction layer, generating a text feature vector to be matched with a vector length equal to the alignment length based on the dependency relationship of the downsampled feature vector in each time step.

[0123] In some embodiments, the autoencoding layer compresses and simplifies features. The fused feature vector is sequence data containing textual semantics and visual features (e.g., features arranged in video frame order, with each frame corresponding to a sub-vector). Encoding process: The autoencoding layer uses the encoding part of LSTM to "compress and encode" the fused feature vector, transforming the original high-dimensional or long sequence features into a more compact "encoded feature vector" (preserving core information and filtering redundancy). For example, the fused features of 100 frames in a "tomato and egg stir-fry video" are compressed into a more refined 50-frame encoded feature vector. Downsampling process: The encoded feature vector is "downsampled" based on the "time step" (i.e., the length of the feature sequence, such as 100 frames)—for example, taking one frame every other frame, or merging features from two adjacent frames to obtain a shorter "downsampled feature vector" (e.g., compressing from 50 frames to 20 frames). This effectively reduces the amount of data while preserving key temporal information (e.g., the order of "pouring oil → pouring egg liquid → solidification").

[0124] In some embodiments, the bottleneck reconstruction layer generates a fixed-length alignment feature. This layer acts as the "output gate" of the entire alignment process, with its output vector length strictly limited to the "alignment length" (consistent with the feature vector length of the business video in the database). It leverages the temporal dependency capture capability of LSTM to analyze the "relationship between the features at each time step and the preceding and following steps" in the downsampled feature vector (e.g., the feature of "pouring egg liquid" depends on the feature of the previous step "pouring oil," which in turn affects the feature of the subsequent step "solidification"). Based on this dependency, it reconstructs the features, ultimately generating a "text feature vector to be matched" with a length exactly equal to the "alignment length." For example, if the alignment length is 10, the bottleneck layer will transform the downsampled features of 20 frames into a 10-dimensional fixed-length vector by integrating temporal dependencies, thus preserving the core temporal information while meeting the length requirements for subsequent comparisons.

[0125] Specifically, the computer device can input the fused feature vector into the feature alignment processing layer, which includes an autoencoder layer and a bottleneck reconstruction layer. The feature alignment processing layer can be trained by an LSTMAutoencoder. The length of the vector output by the bottleneck reconstruction layer is fixed to the alignment length.

[0126] The LSTMAutoencoder consists of a pair of symmetric encoders and decoders, and a reconstruction layer. Its goal is to reconstruct the original input as accurately as possible from the input data after passing through the encoder and decoder. By minimizing the error between the input and the reconstructed output, the LSTMAutoencoder learns the feature representation of the input data. The autoencoder layer is trained from the encoding layer of the Long Short-Term Memory (LSTM) autoencoder, and the bottleneck reconstruction layer is trained from the initial reconstruction layer of the LSTM autoencoder. The vector lengths of the output vectors from both the initial and bottleneck reconstruction layers are fixed to the alignment length.

[0127] In the autoencoder layer, the computer encodes the fused feature vector to obtain an encoded feature vector. At several time steps, the encoded feature vector is downsampled to obtain a downsampled feature vector. In the bottleneck reconstruction layer, based on the dependencies of the downsampled feature vector at each time step, a text feature vector of the aligned length is generated. In other words, the computer can select the feature output of the LSTMAutoencoder reconstruction layer as the unified-dimensional feature vector after dimensionality reduction, i.e., a text feature vector of the aligned length.

[0128] In some embodiments, if the feature alignment processing layer adopts a second alignment method, the feature alignment processing layer includes a term frequency processing layer and a weighted alignment layer; then the computer device can perform alignment processing through term frequency and inverse document frequency, the process of which may be as follows: In the term frequency processing layer, based on the fused feature vector, a predicted probability corresponding to each word in the vocabulary is generated; based on the predicted probability, a content description text of the video data to be matched is generated; based on the alignment length, the content description text is segmented to obtain F text segments; the alignment length is F, where F is a positive integer; the term frequencies and inverse document frequencies corresponding to the F text segments in the content description text are obtained respectively; F text weights are generated based on the F term frequencies and F inverse document frequencies; in the weighted alignment layer, the F text weights are multiplied by the fused feature vector respectively to obtain F weighted feature vectors; and a text feature vector to be matched with a vector length equal to the alignment length is generated based on the F weighted feature vectors.

[0129] Specifically, the computer device can input the fused feature vector into the word frequency processing layer in the feature alignment processing layer. In the word frequency processing layer, based on the fused feature vector, the predicted probability corresponding to each word in the vocabulary is generated, and based on the predicted probability, the content description text of the video data to be matched is generated.

[0130] In some embodiments, the term frequency processing layer generates text fragments and weights, and generates content description text: Based on the fused feature vector (which already contains the visual features of the video and the semantics of the text), the model first predicts the probability of occurrence of each word in the vocabulary (e.g., "tomato", "scrambled eggs", "stir-fry" have a high probability, while "car" and "sky" have a low probability), and then generates the "content description text" of the video to be matched based on these probabilities (e.g., "cut tomatoes into chunks and stir-fry them with eggs, stir-frying quickly over high heat until the egg liquid solidifies"). F text fragments are obtained through word segmentation: Based on the "alignment length F" (a preset fixed length, such as F=3), the content description text is segmented into F key fragments. For example, when F=3, it may be divided into 3 fragments: "cut tomatoes into chunks", "stir-fry eggs", and "egg liquid solidifies". TF and IDF are calculated: Term Frequency (TF): The frequency of occurrence of each fragment in the current description text (e.g., "stir-fry eggs" appears 2 times, and the total number of fragments is 5, then TF=2 / 5). Inverse Document Frequency (IDF): The rarity of each segment in the set of descriptive texts across all videos (e.g., "egg yolk solidifies" appears only in a few tomato and egg videos, resulting in a high IDF value; "cut into chunks" appears in many recipes, resulting in a low IDF value). Generating Text Weights: Multiplying the TF and IDF of each segment (i.e., TF-IDF value) yields F text weights (higher weights indicate greater criticality of the segment to the current video description). For example, "egg yolk solidifies" might have a weight of 0.8, "tomato chunks" 0.5, and "eggs and eggs stir-fried together" 0.7. Weighted Alignment Layer: Generating fixed-length alignment features. Generating Weighted Feature Vectors: Multiplying each of the F text weights by the fused feature vector—features corresponding to segments with higher weights are strengthened, while those with lower weights are weakened. For example, multiplying the visual and textual features related to "egg yolk solidifies" in the fused features by a weight of 0.8 yields the first weighted feature vector; multiplying the features related to "tomato chunks" by a weight of 0.5 yields the second, and so on, resulting in a total of F weighted feature vectors. The F weighted feature vectors are combined sequentially to generate a "text feature vector to be matched" with a length of F (alignment length). This vector retains the core features of the video (filtered by TF-IDF) and strictly matches the preset alignment length, allowing for direct similarity calculation with the feature vectors of business videos in the database (both with the same length F).

[0131] As an example, the word frequency processing layer generates text fragments and weights, and generates content description text by fusing feature vectors containing visual features of the video (such as "dumbbell, arm movement, standing posture") and text semantics (such as "beginner, bicep curl, muscle activation"). Based on this vector, the word frequency processing layer predicts the probability of words in the vocabulary ("dumbbell," "bicep curl," and "arm" have high probabilities, while "running" and "squat" have low probabilities), and then generates the content description text: "Beginners hold dumbbells and perform bicep curls, keeping their elbows fixed while bending their arms, feeling the biceps muscles engage." Word segmentation yields F text fragments: with an alignment length F = 3, the description text is split into 3 key fragments (splitting according to semantic relevance): F1 = "holding dumbbells," F2 = "bicep curl movement," and F3 = "biceps muscle activation." Calculate term frequency (TF): Count the frequency of each segment in the descriptive text (assuming the total length of the text is 10 units): F1 "holding dumbbells" appears 2 times → TF1 = 2 / 10 = 0.2; F2 "bicep curl" appears 3 times → TF2 = 3 / 10 = 0.3; F3 "biceps exertion" appears 2 times → TF3 = 2 / 10 = 0.2. Calculate Inverse Document Frequency (IDF): Calculate the rarity of each segment in the "collection of all fitness instruction video description texts" (assuming a total of 1000 documents): F1 "holding dumbbells" appears in 600 documents → IDF1 = log(1000 / 600) ≈ 0.51; F2 "bicep curl" appears only in 200 documents (mostly arm training videos) → IDF2 = log(1000 / 200) = 1.61; F3 "biceps activation" appears in 300 documents → IDF3 = log(1000 / 300) ≈ 1.20. Generate Text Weights (TF-IDF): Weight = TF × IDF: Weight 1 = 0.2 × 0.51 ≈ 0.10; Weight 2 = 0.3 × 1.61 ≈ 0.48; Weight 3 = 0.2 × 1.20 ≈ 0.24.

[0132] Continuing from the previous example, the weighted alignment layer generates the feature vector of the text to be matched; generates the weighted feature vector: the fusion feature vector is a high-dimensional vector containing "visual + text" (assumed to be [V1,V2,V3,...,Vn]), and multiplies the three weights by the vector respectively: weighted vector 1 = weight 1 × fusion feature vector → weakens the features related to holding dumbbells (because the weight is low); weighted vector 2 = weight 2 × fusion feature vector → strengthens the features related to bicep curl (because the weight is the highest, it is the core movement); weighted vector 3 = weight 3 × fusion feature vector → moderately strengthens the features related to bicep muscle exertion. Generate a fixed-length vector: concatenate the three weighted feature vectors in order to obtain the feature vector of the text to be matched with a length of F = 3. This vector retains the core information of the video (bicep curl > bicep muscle exertion > holding dumbbells) and is consistent with the length (3) of the feature vector of fitness teaching videos in the database, and can be directly used for subsequent similarity calculation.

[0133] Computer devices can segment the content description text based on the alignment length to obtain F text segments (terms). The segmentation can be done by rule matching or probability matching. Taking the text description text C segmentation by using the text set A (the text set A can be a predefined text library) as an example, the text segmentation of the content description text C will have a segmentation scheme S. The task of text segmentation is to find the scheme in the segmentation scheme S that has the highest probability P(S|C) of the segmented words appearing in the text set A. According to Bayes' theorem, the probability P(S|C) can be calculated as shown in formula (4):

[0134] Where P(C) is the probability that the content description text C appears in the text set A, and P(C|S) is the probability that the word segmentation scheme restores the content description text C. The word segmentation scheme S divides the content description text set into M text segments (text segment W1, text segment W2, ..., text segment W...). M If ), then P(S) = ∑ M＝i P(W i ), P(W i ) is a text fragment W i The ratio of the number of times a word appears in text set A to the total number of words in text set A.

[0135] The computer device can obtain the term frequencies (TFs) of F text segments in the content description text. The term frequency (TF) of the target text segment i is calculated as shown in formula (5):

[0136] Wherein, the target text fragment i can be any one of the F text fragments, TF is the word frequency, C is the content description text, and frequency is the number of times the target text fragment i appears in the content description text.

[0137] The computer device can obtain the inverse document frequencies (IVF) of F text fragments in the content description text, obtain a document set including the content description text, and perform the following processing for each text fragment: determine the total number of documents in the document set, identify the documents in the document set that include the text fragment as target documents, and determine the number of target documents; determine the inverse document probability of the text fragment based on the ratio of the total number of documents to the number of target documents, and calculate the inverse document frequency of the target text fragment i as shown in formula (6):

[0138] Wherein, D represents the set of documents corresponding to the content description text. This set of documents may also include only one content description text C, and this embodiment of the application does not impose any restrictions.s d represents the total number of documents in the document collection. t This represents the number of documents containing the target text fragment i, which is also the number of target documents in the document set.

[0139] The computer device can generate F text weights based on F word frequencies and F inverse document frequencies. The text weights are calculated as shown in formula (7): TF-IDF(i, C, D)=TF(i, C)×IDF(i, D) Formula (7)

[0140] In the weighted alignment layer, the computer device can perform dot product operations on the F text weights and the fused feature vector respectively to obtain F weighted feature vectors. The sum of the F weighted feature vectors is determined as the text weighted feature vector. Each element of the text weighted feature vector is divided by the sum of the text weights corresponding to the F weighted feature vectors respectively to obtain the text feature vector to be matched with a vector length equal to the alignment length. The text feature vector to be matched with a vector length equal to the alignment length is generated based on the F weighted feature vectors, and the calculation is shown in formula (8):

[0141] Wherein, feature is the fused feature vector, WeightedAverage is the feature vector of the text to be matched with a vector length equal to the alignment length, and TF-IDF feature is the weighted feature vector.

[0142] If the feature alignment processing layer adopts the third alignment method, the computer device can obtain the last unit prediction vector in the fused feature vector through the alignment processing layer, and determine the last unit prediction vector in the fused feature vector as the text feature vector to be matched with a vector length equal to the alignment length. The fused feature vector can be used to generate content description text for the video data to be matched. The fused feature vector includes G unit prediction vectors, and one unit prediction vector can correspond to a text fragment or term in the content description text. The vector length of each of the G unit prediction vectors is the alignment length. The last unit prediction vector in the G unit prediction vectors includes the global semantics obtained by concatenating the business text information and the compressed visual marker vector. In the feature alignment processing layer, the computer device can determine the last unit prediction vector in the G unit prediction vectors as the text feature vector to be matched with a vector length equal to the alignment length.

[0143] In some embodiments, the fused feature vector is the result of concatenating business text information (text prompts) and compressed visual marker vectors (visual features). One of its core functions is to generate the "content description text" of the video to be matched (e.g., "detailed steps for making scrambled eggs with tomatoes: chop tomatoes, fry eggs, mix and stir-fry"). Structurally, the fused feature vector consists of G unit prediction vectors (G is a positive integer), each unit prediction vector corresponding to a text fragment or term in the content description text (e.g., the first unit corresponds to chopping tomatoes, the second to frying eggs, and the third to mixing and stir-frying). A key feature is that the lengths of these G unit prediction vectors are all aligned (consistent with the length of the business video feature vectors in the database), providing a basis for subsequent direct selection. Among the G unit prediction vectors, the last unit prediction vector plays the role of integrating global information—it not only contains the information of its corresponding text fragment / term but also integrates the semantics of all the preceding unit prediction vectors (i.e., the global semantics after concatenating the business text information and the compressed visual marker vectors). For example, in the descriptive text of scrambled eggs with tomatoes, the first few units might correspond to local steps such as chopping tomatoes, beating eggs, and heating oil, while the last unit prediction vector integrates the global semantics of the complete cooking process from ingredient preparation to stir-frying, better representing the core content of the entire video. In the feature alignment processing layer, the computer does not need complex encoding or weighted calculations; instead, it directly selects the last unit prediction vector from the G unit prediction vectors of the fused feature vector and determines it as the feature vector of the text to be matched. Since the length of this vector is equal to the alignment length (an inherent property of all unit prediction vectors), it naturally satisfies the requirement that the vector length is consistent with the feature vector of the business video and can be directly used for subsequent similarity calculations.

[0144] As an example, let's take "coffee latte art tutorial videos" as a specific illustration: Suppose the video to be matched is "basic heart-shaped coffee latte art tutorial," its fused feature vector is generated by concatenating "business text information (such as 'coffee latte art steps and techniques')" with "compressed visual marker vectors (such as visual features like 'milk pouring angle, milk foam fineness, heart-forming process', etc.)." This fused feature vector contains G = 4 unit prediction vectors (each vector's length is the alignment length, assumed to be 5), and can be used to generate content description text: "1. Extract espresso; 2. Froth dense milk foam; 3. Pour milk at a 45-degree angle; 4. Shake the latte art pitcher to form a heart shape." The four unit prediction vectors correspond to four segments in the text: Unit prediction vector 1 corresponds to "extracting espresso," containing the local semantics and visual features of this step; Unit prediction vector 2 corresponds to "frothing milk foam," focusing on key information about milk foam processing; Unit prediction vector 3 corresponds to "pouring milk at a 45-degree angle," emphasizing the visual feature of the pouring angle; Unit prediction vector 4 (the last one) not only corresponds to "shaking the latte art pitcher to form a heart shape," but also integrates the information from the first three vectors, containing the global semantics of "the complete process from extracting coffee to completing a heart-shaped latte art," reflecting both the "steps and techniques" emphasized in the business text and integrating the core information of all visual features. In the feature alignment layer, the computer directly selects this last unit prediction vector (with a length of 5, consistent with the alignment length) as the text feature vector to be matched. Because it contains global semantics, can completely represent the core content of the entire coffee latte art tutorial video, and its length meets the requirements, it can directly perform similarity calculations with the business text feature vectors of "coffee latte art tutorial videos" in the database, efficiently achieving accurate retrieval.

[0145] It's understandable that Large Language Models (LLMs) generate video descriptions character by character. When generating each character, the model infers based on previously generated content and the input image information. Therefore, when the model generates the last character, the last unit prediction vector can capture the semantic information of all previously generated characters. Thus, the last unit prediction vector out of G unit prediction vectors can be determined as the feature vector of the text to be matched, with a vector length equal to the alignment length. This feature vector is then input into the recall and fine-matching module for similar video retrieval. That is, the alignment length is the vector length of the last unit prediction vector.

[0146] It is understandable that the video length and content of the video data to be matched are not fixed, and the fused feature vector generated from the video data to be matched is also not fixed. Directly calculating the similarity with the business text feature vector of the business video data will require padding due to the inconsistency in vector length. For example, padding the business text feature vector and the fused feature vector to the same length may lead to problems such as reduced computational efficiency, increased memory consumption, and the influencing of the model's capture of real data and reduced generalization. The embodiments of this application, through the above alignment processing, can effectively generate a text feature vector to be matched with a vector length equal to the alignment length from the fused feature vector. Calculating the similarity between the text feature vector to be matched and the business text feature vector can reduce the computational cost of similarity calculation.

[0147] Step S205: Obtain the business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0148] Specifically, the computer equipment can acquire the business text feature vector corresponding to the business video data. This business text feature vector includes semantics representing the content description text of the business video data. The business text feature vector can be a feature vector used to generate the content description text for the business video data. The generation method of the business text feature vector is the same as the generation method of the text feature vector to be matched, thus ensuring that the vector length of the text feature vector to be matched is the same as the vector length of the text feature vector.

[0149] Computer devices can retrieve video text feature databases using the IndexIVFFlat index in the FAISS library. FAISS (Facebook AI Similarity Search) is a highly efficient similarity search library. This embodiment utilizes the IndexIVFFlat index type in the FAISS library to initialize the business text feature vector for each business video data in the video text feature database and add the business text feature vector to the index.

[0150] The video text feature database can include cluster centers generated based on the IndexIVFFlat index. Each business text feature vector can be assigned to a different cluster center. During the recall operation, the computer device can use the text feature vector to be matched as the query vector. The FAISS library first finds the closest cluster center based on the query vector. Then, it performs an exact search within the closest cluster centers to find the feature vector most similar to the query vector. This step ensures the efficiency and accuracy of the recall because searching only within the most relevant cluster centers reduces computational cost.

[0151] In some embodiments, text similarity is achieved through a recall matching layer in the target video retrieval model. The aforementioned generation of the text similarity between the business text feature vector and the text feature vector to be matched can be achieved as follows: In the recall matching layer, a first parameter of the business text feature vector is generated based on each vector element value in the business text feature vector, and a second parameter of the text feature vector to be matched is generated based on each vector element value in the text feature vector to be matched; the business text feature vector and the text feature vector to be matched are multiplied by a dot product to obtain a matching parameter; and the text similarity between the business text feature vector and the text feature vector to be matched is generated based on the matching parameter, the first parameter, and the second parameter.

[0152] Step S206: In the recall matching layer, a first parameter of the business text feature vector is generated based on the value of each vector element in the business text feature vector, and a second parameter of the text feature vector to be matched is generated based on the value of each vector element in the text feature vector to be matched; the business text feature vector and the text feature vector to be matched are multiplied by a dot product to obtain the matching parameter; and the text similarity between the business text feature vector and the text feature vector to be matched is generated based on the matching parameter, the first parameter, and the second parameter.

[0153] In some embodiments, a first parameter and a second parameter are generated: the first parameter is calculated based on all element values ​​of the "business text feature vector" (feature vectors of existing videos in the database), and the second parameter is calculated based on all element values ​​of the "text feature vector to be matched" (feature vector of the video to be retrieved by the user). These two parameters are typically the "magnitude" of the vector (i.e., the length of the vector, obtained by taking the square root of the sum of the squares of each element in the vector), used for subsequent standardization processing. For example: if the business text feature vector is [0.2, 0.4, 0.6], its first parameter (magnitude) = √(0.2...). 2 +0.4 2 +0.6 2 =√(0.04+0.16+0.36) =√0.56≈0.75; the feature vector of the text to be matched is [0.3,0.5,0.7], and its second parameter (magnitude) =√(0.3). 2 +0.5 2 +0.7 2=√(0.09+0.25+0.49) =√0.83≈0.91. Calculate the matching parameter: Perform a "dot product" operation on the feature vector of the business text and the feature vector of the text to be matched—multiply the corresponding elements and sum the results. The result is the matching parameter, used to measure the "similarity" of the two vectors (the closer the directions, the larger the matching parameter). For example, the dot product of the two vectors above = 0.2×0.3+0.4×0.5+0.6×0.7 = 0.06+0.2+0.42 = 0.68 (matching parameter = 0.68). Generate text similarity: Divide the matching parameter by the product of the first and second parameters to obtain the final text similarity (this is essentially the calculation logic of "cosine similarity"). For example: Text similarity = 0.68÷(0.75×0.91) = 0.68÷0.68≈1.0 (a value close to 1 indicates that the two vectors are highly similar).

[0154] As an example, taking "yoga instruction video retrieval" as an example, suppose a user wants to find videos similar to "Beginner Sun Salutation Yoga Tutorial". The process of calculating text similarity in the recall matching layer is explained as follows: Business text feature vector: There is already a feature vector for a "Basic Sun Salutation Yoga Tutorial" video in the database, denoted as V1 = [0.8, 0.6, 0.7] (vector length is 3, representing the feature codes of the three core movements: "Mountain Pose", "Upward Stretch", and "Forward Bend with Foot Touch"). Matching text feature vector: The feature vector of the "Beginner Sun Salutation Yoga Tutorial" video retrieved by the user, denoted as V2 = [0.7, 0.5, 0.8] (also containing the feature codes of the three core movements, highly correlated with the movement sequence of V1). Calculation steps in the recall matching layer: Generating the first and second parameters: The first parameter is the modulus (vector length) of V1, calculated as the square root of the sum of the squares of each element: First parameter = √(0.8...). 2 +0.6 2 +0.7 2 )=√(0.64+0.36+0.49)=√1.49≈1.22; The second parameter is the modulus of V2: second parameter=√(0.7 2 +0.5 2 +0.8 2 =√(0.49+0.25+0.64) =√1.38≈1.17; Calculate the matching parameter (dot product operation): Multiply the corresponding elements of V1 and V2 and sum them: Matching parameter = 0.8×0.7+0.6×0.5+0.7×0.8 = 0.56+0.3+0.56 = 1.42; Generate text similarity: Divide the matching parameter by the product of the first parameter and the second parameter (i.e., the cosine similarity formula): Text similarity = 1.42÷(1.22×1.17) = 1.42÷1.43≈0.99.

[0155] Specifically, the computer device can input the business text feature vector and the text feature vector to be matched into the recall matching layer of the target video retrieval model. The computer device can use methods such as cosine similarity, Jaccard similarity coefficient, and Hamming distance to determine the similarity between the business text feature vector and the text feature vector to be matched. Taking cosine similarity as an example, the computer device generates the first parameter of the business text feature vector based on the value of each vector element in the business text feature vector, and generates the second parameter of the text feature vector to be matched based on the value of each vector element in the text feature vector to be matched. Among them, the first parameter and the second parameter are both Euclidean norms. The first parameter can be the modulus of the business text feature vector, and the second parameter can be the modulus of the text feature vector to be matched. The computer device can perform a dot product operation on the business text feature vector and the text feature vector to be matched to obtain the matching parameter. Based on the matching parameter, the first parameter and the second parameter, the text similarity between the business text feature vector and the text feature vector to be matched is generated. The process can be shown in formula (9):

[0156] Where cosine(A, B) is the text similarity, A can represent the business text feature vector, B can represent the text feature vector to be matched, ||A|| is the first parameter, and ||B|| is the second parameter.

[0157] Thus, generating the first and second parameters based on each element value of the business text feature vector and the text feature vector to be matched can fully utilize the overall distribution information of the vectors, providing a standardized benchmark for subsequent similarity calculations. The matching parameters obtained by performing a dot product operation on the two vectors can effectively measure the same direction between the vectors and accurately capture the degree of semantic association between them. The process of generating text similarity based on the matching parameters, the first parameter, and the second parameter essentially eliminates the interference of the absolute numerical size of the vectors through the standardization of the vector magnitude, focusing only on the consistency of the semantic direction represented by the vectors. This makes the similarity results more reflective of the true association between the two videos in the core content, improving the accuracy of similarity judgment and enhancing the model's adaptability to vectors of different lengths and numerical ranges. This lays a reliable foundation for subsequently selecting high-quality similar video retrieval results and improves the efficiency and accuracy of target video retrieval.

[0158] Step S207: The business video data associated with the business text feature vectors whose text similarity is greater than or equal to the text similarity threshold is determined as the text retrieval video set; the text retrieval video set is used to determine the similar video retrieval results for the video data to be matched.

[0159] Specifically, computer equipment can identify a set of business video data associated with business text feature vectors whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set. This text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0160] In some embodiments, the text similarity threshold is a pre-set critical value (e.g., 0.7, 0.8, etc.) used to determine whether the feature vectors of two videos are sufficiently similar. It acts like a passing grade—when the text similarity between two videos (e.g., 0.85 calculated using cosine similarity) is greater than or equal to this threshold, they are considered sufficiently related in content; otherwise, they are considered dissimilar. The threshold setting can be adjusted according to scenario requirements: a high threshold (e.g., 0.85) can be set for scenarios with high accuracy requirements (e.g., professional tutorial retrieval), and a low threshold (e.g., 0.6) can be set for scenarios with high recall requirements (e.g., general entertainment video recommendation). Each "business text feature vector" in the database corresponds one-to-one with specific "business video data" (i.e., existing video files in the database)—the feature vector is the "digital tag" of the video, and the video is the "entity content" corresponding to the tag. The computer device traverses all business text feature vectors in the database, calculates their similarity to the text feature vectors to be matched, then filters out feature vectors with "similarity ≥ text similarity threshold," and then finds the business video data associated with these vectors. These selected business videos together form the "text retrieval video set," which is a collection of all candidate results that are "sufficiently similar" to the video to be matched.

[0161] This application embodiment can sample the video data to be matched based on the context window length of the target video retrieval model to obtain unit sampled images. Feature extraction is then performed on these unit sampled images to obtain unit visual marker vectors. By rearranging the pixels of the image, the dimensionality of the vectors representing image height and width in the unit sampled images is reduced, thereby reducing the number of visual markers. By learning a fixed set of compressed query vectors and performing multi-head attention processing, the visual marker vectors to be matched and the compressed query vectors are concatenated together. This better captures multimodal information, helping the target video retrieval model understand and abstract image information at different levels and granularities, thus improving the processing performance of the target video retrieval model. Through a visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector. The compressed visual marker vector is obtained by performing multi-head attention processing on the compressed query vector and the visual marker vector sequence. The vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector. This effectively reduces the number of visual markers and significantly reduces memory load and computational resource consumption.

[0162] The process involves acquiring business text information and concatenating it with compressed visual marker vectors to obtain a fused feature vector. The business text information serves as a textual prompt indicating the content description text corresponding to the extracted video data, while the fused feature vector includes the semantics representing the content description text of the video data to be matched. The fused feature vector can ignore visual elements in the video data that are irrelevant to the video content, such as adversarial techniques like adding borders, texture effects, or occluded emojis, thus improving the model's robustness and resistance to perturbations. Since the fused feature vector only needs to retain the semantics representing the content description text, it effectively reduces the number of visual markers to be processed, decreases computational costs, and improves the efficiency of similar video retrieval. Alignment processing of the fused feature vector yields a matching text feature vector with a length equal to the alignment length. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Finally, the process involves acquiring the business text feature vector corresponding to the business video data, where the business text feature vector includes the semantics representing the content description text of the business video data. By calculating the text similarity between the feature vectors of the business text and the feature vectors of the text to be matched, the text similarity between them is obtained. Specifically, calculating the text similarity between business text feature vectors and the text feature vectors of the text to be matched with the same vector length avoids the padding problem caused by inconsistent vector lengths, reduces computational costs, and improves the efficiency of similar video retrieval. The business video data associated with business text feature vectors whose text similarity is greater than or equal to a text similarity threshold is defined as the text retrieval video set. Using this text retrieval video set to determine the similar video retrieval results for the video data to be matched avoids the problem of missed detections and false detections caused by misleading similarity scores due to visual elements, thus improving the accuracy and completeness of similar video retrieval.

[0163] Please refer to Figure 6, which is a flowchart illustrating a data processing method according to an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1 or any terminal device in a terminal device cluster, such as terminal device 10a. The following description will use the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S301-S311:

[0164] Step S301, data preprocessing;

[0165] Specifically, as shown in Figure 6, the processing flow of the business video data is represented by dashed lines, and the processing flow of the video data to be matched is represented by solid lines. The computer equipment can perform data preprocessing on the business video data and the video data to be matched. Data preprocessing may include sampling the video data based on the video duration and frame rate to obtain sampled video frames, performing image segmentation on the sampled video frames to obtain a first sampled image corresponding to the business video data and a second sampled image corresponding to the video data to be matched. The data preprocessing process can be referred to the specific description of step S201 in the embodiment corresponding to Figure 4 above, and will not be repeated here.

[0166] Step S302, visual feature extraction;

[0167] Specifically, the computer equipment can extract features from the first sampled image to obtain the business visual feature vector corresponding to the business video data, and extract features from the second sampled image to obtain the visual feature vector to be matched corresponding to the video data to be matched. The business visual feature vector can be the global features of the visual markers in the business video data, and the visual feature vector to be matched can be the global features of the visual markers in the video data to be matched. Both the business visual feature vector and the visual feature vector to be matched can be obtained using aggregation methods such as average pooling, max pooling, and a temporal shift module (TSM).

[0168] Step S303, visual recall matching;

[0169] Specifically, the computer equipment can establish a first index database based on business visual feature vectors, and then establish a video visual feature database connecting business video data and business visual feature vectors through this first index database. The first index database can be an IndexIVFFlat index, and the video visual feature database can be a FAISS library. The video visual feature database includes cluster centers generated based on the IndexIVFFlat index, and each business text feature vector can be assigned to a different cluster center. The computer equipment can use the visual feature vector to be matched as a query vector. The FAISS library can find the closest cluster center based on the query vector. Then, a precise search is performed within the closest cluster center to find the feature vector most similar to the query vector. By using the inverted file index (IndexIVFFlat) in FAISS and its search method, a set of potentially similar business visual feature vectors is quickly retrieved from the video visual feature database and identified as the recalled business visual feature vectors, thus narrowing the search scope.

[0170] Step S304, visual similarity calculation;

[0171] Specifically, the computer device can perform visual similarity calculation between the visual feature vector to be matched and the recalled business visual feature vector to generate a visual similarity between the business visual feature vector and the visual feature vector to be matched. The visual similarity calculation can be determined using methods such as cosine similarity, Jaccard similarity coefficient, and Hamming distance, and this application embodiment does not impose any limitations on this. For details on the visual similarity calculation, please refer to the specific description of step S206 in the embodiment corresponding to Figure 4 above, which will not be repeated here.

[0172] Step S305: Generate a visual retrieval video set;

[0173] Specifically, computer equipment can associate business video data with business visual feature vectors whose visual similarity is greater than or equal to a video similarity threshold to determine a visual retrieval video set. This visual retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0174] Step S306, text description processing;

[0175] Specifically, the computer equipment can extract features from the business video data to obtain a business visual marker vector, and the computer equipment can extract features from the video data to be matched to obtain a compressed visual marker vector. The business visual marker vector can be a vector generated by compressing the visual markers corresponding to the business video data, and the compressed visual marker vector can be a vector generated by compressing the visual markers corresponding to the video data to be matched. For details on generating the business visual marker vector and compressing the visual marker vector, please refer to the specific description of steps S201 to S202 in the embodiment corresponding to Figure 4 above; these details will not be repeated here.

[0176] The computer device can acquire business text information, which can be text prompts indicating the content description text corresponding to the extracted video data. For example, it could be a need to provide accurate and concise annotations for the input video data to be matched. The business text information can be used to instruct and guide the model to generate content description text that conforms to the text prompts. The computer device can perform feature extraction on the business text information to obtain the text prompt feature vector corresponding to the business text information. The alignment length is determined based on the vector length of the compressed visual marker vector and the vector length of the business visual marker vector. For example, the vector length of the business visual marker vector can be determined as the alignment length, or the median or average of the vector lengths of the business visual marker vectors can be determined as the alignment length, etc. The embodiments of this application are not limited here.

[0177] The computer device can concatenate the text prompt feature vector and the compressed visual mark vector to obtain a fused feature vector, and concatenate the text prompt feature vector and the business visual mark vector to obtain a business feature vector. For details on generating the fused feature vector and the business feature vector, please refer to the specific description of step S203 in the embodiment corresponding to Figure 4 above. The embodiments of this application will not be repeated here.

[0178] Step S307, Alignment Processing;

[0179] Specifically, the computer device aligns the fused feature vectors to obtain a text feature vector of length equal to the alignment length. The fused feature vector includes semantics representing the content description text of the video data to be matched, and can be a feature vector used to generate the content description text for the video data to be matched. The computer device can also align the business feature vectors to obtain a business text feature vector of length equal to the alignment length. The business text feature vector includes semantics representing the content description text of the business video data, and can be a feature vector used to generate the content description text for the business video data. For details on the alignment process, please refer to the specific description of step S204 in the embodiment corresponding to Figure 4 above; this embodiment will not repeat it here.

[0180] Step S308, text recall and matching;

[0181] Specifically, the computer equipment can establish a second index database based on business text feature vectors, and then use this second index database to create a video text feature database connecting business video data and business text feature vectors. The second index database can be an IndexIVFFlat index, the video text feature database can be a FAISS library, and the video visual feature database includes cluster centers generated based on the IndexIVFFlat index. Each business text feature vector can be assigned to a different cluster center. The computer equipment can use the text feature vector to be matched as a query vector. The FAISS library can find the closest cluster center based on the query vector. Then, it will perform a precise search within the closest cluster center to find the feature vector most similar to the query vector.

[0182] Step S309, text similarity calculation;

[0183] Specifically, the computer device can perform visual similarity calculation between the feature vector of the text to be matched and the feature vector of the recalled business text, generating a text similarity between the business text feature vector and the feature vector of the text to be matched. The text similarity calculation can be determined using methods such as cosine similarity, Jaccard similarity coefficient, and Hamming distance, and this application embodiment does not impose any limitations on this. For details on the visual similarity calculation, please refer to the specific description of step S206 in the embodiment corresponding to Figure 4 above, which will not be repeated here.

[0184] Step S310: Generate a text-retrieval video collection;

[0185] Specifically, computer equipment can associate business video data with business text feature vectors whose text similarity is greater than or equal to a video similarity threshold to determine a text retrieval video set. This text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0186] Step S311: Determine the similar video retrieval results.

[0187] Specifically, computer devices can determine the intersection of visually retrieved video sets and text-retrieved video sets as the similar video retrieval results for the video data to be matched. This avoids the model focusing too much on local features and ignoring the global contextual information of the video content, which would reduce the accuracy of video retrieval and improve the accuracy of similar video retrieval. It also avoids false detections (identifying different videos as similar videos) caused by adversarial techniques (such as cutting videos into multiple segments, adding borders, texture effects, or occluding emojis).

[0188] Alternatively, computer devices can determine the similar video retrieval results for the video data to be matched by taking the union of the visually retrieved video set and the text-retrieved video set. This avoids missed detections due to adversarial tactics (classifying similar videos as different videos), and improves the accuracy and robustness of similar video retrieval.

[0189] This application's embodiments can serve as a plugin for systems that use visual elements for similar video retrieval. By generating a fused feature vector containing semantic descriptions of the content of the video data to be matched, visual elements irrelevant to the video content, such as adversarial techniques like adding borders, texture effects, or occluded emojis, can be ignored, improving the model's robustness and resistance to perturbations. Using the text-based video set retrieved by the fused feature vector as a supplement to visual element-based similar video retrieval allows the computer to determine the intersection of the visual and text-based video sets as the similar video retrieval result for the video data to be matched. This avoids the model focusing too much on local features and neglecting the global contextual information of the video content, which could lead to a decrease in video retrieval accuracy and improve the accuracy of similar video retrieval. Alternatively, the computer can determine the union of the visual and text-based video sets as the similar video retrieval result for the video data to be matched. This avoids false positives or false negatives caused by adversarial techniques, improving the accuracy and resistance to perturbations in similar video retrieval. Integrating the embodiments of this application into a visual element similarity video retrieval system can significantly reduce the decrease in visual feature similarity caused by various attack methods, thereby effectively alleviating the problem of missed detection caused by the decrease in feature similarity during the retrieval process and improving the accuracy and completeness of the retrieval.

[0190] Please refer to Figure 7, which is a flowchart illustrating a data processing method according to an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1 or any terminal device in a terminal device cluster, such as terminal device 10a. The following description will use the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S401-S404:

[0191] Step S401: Obtain sample video data and labeled video data;

[0192] Specifically, the computer device can acquire sample video data and labeled video data. The sample video data and labeled video data can include a similarity score, which characterizes the degree of similarity between them. The computer device can then input the sample video data and labeled video data into an initial video retrieval model. The text description layer in the multimodal text description layer of the initial video retrieval model can be a Phi-3-mini-128k-instruct model (an open-source lightweight model), which can be a large language model.

[0193] Model training for computing devices can be achieved by fine-tuning multiple publicly accessible image-text pair datasets and video-text pair datasets. Since multimodal large language models are primarily used to generate accurate and concise descriptions for short videos input by users, computing devices can be specifically focused on fine-tuning using datasets relevant to video captioning tasks to help the model generate content description text in both English and Chinese environments.

[0194] Specifically, during the training phase, for image captioning tasks, Chinese and English image captioning datasets can be used, such as Laion-EN (English & Chinese), COYO (Chinese), GRIT (Chinese), COCO (English), TextCaps (English), and ShareGPT4o (English & Chinese). For video captioning tasks, Chinese and English video captioning datasets can be used, such as mPLUG-YOUKU (Chinese), ShareGPT4o (English & Chinese), Mementos (English), STAR (English), VideoChat2IT (English), and LSMDC-QA (English). In the general question answering (QA) domain, question-answering pair datasets can be used, such as VQAv2 (English), GQA (English), OKVQA (English), VSR (English), and VisualDialog (English), thereby improving the initial video retrieval model's ability to handle various question-answering scenarios.

[0195] Furthermore, since user-uploaded short videos often contain a large amount of text information, this text information is crucial for the model to correctly understand the video content. Therefore, computer devices can specifically enhance the model's Optical Character Recognition (OCR) capabilities. Open-source OCR models, such as the Paddle OCR model, can be used to perform OCR processing on Chinese images (such as the Wukong dataset) and English images (such as the Laion COCO dataset), and the SROIE, FUNSD, and POIE datasets have been incorporated to enhance the model's ability to recognize handwritten characters.

[0196] During the fine-tuning phase, the computing device can focus on using video and image captioning datasets, as well as video and image dialogue datasets, to fine-tune the multimodal large model, enhancing its ability to generate content description text for videos. For video and image captioning tasks, the COCO (English), mPLUG-YOUKU (Chinese), and ShareGPT4o (English & Chinese) datasets can be used. For video and image dialogue tasks, the LLaVA-150K (English & Chinese), LVIS-Instruct4V (English), ALLaVA (English & Chinese), Laion-GPT4V (English), and TextOCR-GPT4V (English) datasets can be used.

[0197] Because existing video datasets mainly originate from long-form video websites and undergo semantic or scene-based segmentation during data processing to become video clips, these datasets differ significantly from real user-uploaded short videos. To address this issue, 100,000 UGC short videos were collected, and ChatGPT4o (Generative Pre-trained Transformer 4omni) was used to generate a concise video description for each short video (each short video is uniformly sampled at 20 frames) using prompts such as business text information, thus constructing targeted short video-content description text pairs.

[0198] The computer device can use Paddle OCR to perform OCR on each frame of the image. Subsequently, the OCR results of multiple frames can be merged into a video-level OCR result. For the same text that appears repeatedly in multiple frames, it is recorded only once in the result, thus constructing a dataset of UGC short video-OCR result text pairs (also known as short video-content description text pairs). During the fine-tuning phase, the computer device can optimize the model using these two collected short video datasets.

[0199] Step S402: Using the initial video retrieval model, feature extraction is performed on the sample video data to obtain a first visual marker vector. Business text information is obtained, and the business text information is concatenated with the first visual marker vector to obtain a first fusion feature vector. Feature extraction is performed on the labeled video data to obtain a second visual marker vector. The business text information is concatenated with the second visual marker vector to obtain a second fusion feature vector. The business text information is a text prompt used to indicate the extraction of video content description text. The first fusion feature vector includes semantics used to characterize the content description text of the sample video data, and the second fusion feature vector includes semantics used to characterize the content description text of the labeled video data.

[0200] Specifically, the computer device can extract features from the sample video data through the initial video retrieval model to obtain a first visual marker vector, and extract features from the labeled video data to obtain a second visual marker vector. For details on generating the first and second visual marker vectors, please refer to the specific description of steps S201 to S202 in the embodiment corresponding to Figure 4 above. The embodiments of this application will not be repeated here.

[0201] In some embodiments, the initial video retrieval model includes an initial abstraction layer; the process of extracting features from sample video data to obtain a first visual marker vector may include: extracting features from sample video data to obtain a sample visual marker vector, obtaining an initial query vector randomly initialized in the initial abstraction layer, and concatenating the sample visual marker vectors based on the initial query vector to obtain a first visual marker vector; the initial query vector is a model parameter in the initial abstraction layer; the vector length of the initial query vector is the same as the vector length of the first visual marker vector;

[0202] Specifically, the computer device obtains an initial query vector randomly initialized in the initial abstraction layer. The initial query vector consists of model parameters in the initial abstraction layer. The computer device can then concatenate the sample visual label vectors based on the initial query vector to obtain the first visual label vector. The initial query vector can be a fixed query vector used to extract higher-level semantic features from the visual labels of multiple frames of images by learning a fixed set of queries.

[0203] In some embodiments, the initial video retrieval model includes an initial abstraction layer; the process of extracting features from sample video data to obtain a first visual marker vector may include: extracting features from the sample video data to obtain a sample visual marker vector; obtaining a randomly initialized initial query vector from the initial abstraction layer; and concatenating the sample visual marker vectors based on the initial query vector to obtain the first visual marker vector; the initial query vector is a model parameter in the initial abstraction layer; the vector length of the initial query vector is the same as the vector length of the first visual marker vector. Feature extraction is performed on the "sample video data" (videos used to train the model, such as categorized instructional videos, landscape videos, etc.) to obtain the "sample visual marker vector"—this is the raw visual feature encoding extracted from the video (such as a digital representation of information like objects, colors, and actions in the scene), but it has not yet undergone model abstraction. The "initial query vector" is obtained from the initial abstraction layer; this is a randomly initialized model parameter (like giving the model a random "initial problem"), and its vector length is preset and the same as the length of the final "first visual marker vector". Since it is the early stage of training, this vector does not yet have actual semantics and is just a random sequence of numbers. The sample visual label vector (original visual features) is concatenated with the initial query vector (random parameters) to form a longer vector, namely the first visual label vector. The purpose of this step is to establish a relationship between the original visual features and the initial parameters of the model, providing a basic input for subsequent model training—similar to combining the original data with the initial state of the model, allowing the model to gradually learn how to process features using these initial parameters during training.

[0204] As an example, suppose the sample video data used for training is a video of a golden retriever running on a lawn. First, feature extraction is performed on this sample video to obtain a sample visual label vector—this is the raw encoding of visual information in the video, such as a numerical vector containing features like "golden retriever's yellow fur (0.8), running posture on four legs (0.6), green lawn background (0.3), and sunlight (0.2)" (let's assume it's [0.8, 0.6, 0.3, 0.2], with a length of 4). In the initial abstraction layer of the initial video retrieval model, there is a randomly initialized initial query vector (as a model parameter). Since it is randomly set, it might be [0.1, 0.5, 0.9, 0.4] (with a length of 4, related to the length of the subsequent first visual label vector). This vector currently has no practical meaning; it's just the initial state parameter of the model. Following the rule of concatenating sample visual label vectors based on the initial query vector, the sample visual label vector [0.8, 0.6, 0.3, 0.2] is concatenated with the initial query vector [0.1, 0.5, 0.9, 0.4]. It's important to note that the fact that the initial query vector has the same length as the first visual label vector means that the length of the concatenated first visual label vector is twice that of the initial query vector (because it's a concatenation of two equal-length vectors), i.e., a length of 8. The resulting first visual label vector is [0.8, 0.6, 0.3, 0.2, 0.1, 0.5, 0.9, 0.4]. This first visual label vector will serve as the initial input to the model. In subsequent training, the model will continuously adjust the value of the initial query vector based on the annotation information of the "Golden Retriever video" (such as the tags "Golden Retriever" and "running"), gradually teaching it to focus on key information such as "canine characteristics" and "movement posture," rather than using random initial values. The initial stitching operation essentially establishes a connection between the original visual features and the initial parameters of the model, providing a foundation for model learning.

[0205] Thus, by extracting sample visual label vectors from sample video data, the original visual features of the video can be preserved, providing basic data for model learning. Obtaining the randomly initialized initial query vector in the initial abstraction layer and using it as model parameters provides the model with a learnable initial benchmark, and random initialization ensures the objectivity and diversity of model training, avoiding initial bias. Concatenating the sample visual label vector with the initial query vector yields the first visual label vector, establishing a connection between the original visual features and the model's initial parameters, creating conditions for subsequent model training to optimize parameters and learn feature associations. The setting that the initial query vector and the first visual label vector have the same length ensures the smooth concatenation operation and the consistency of the vector structure, providing stable input for subsequent model calculations and feature processing. This lays a solid foundation for the model to gradually learn effective feature extraction and retrieval capabilities, improving the effectiveness of model training and the final retrieval performance.

[0206] The computer device can acquire business text information, concatenate the business text information with a first visual marker vector to obtain a first fusion feature vector, and concatenate the business text information with a second visual marker vector to obtain a second fusion feature vector. The business text information is a text prompt used to indicate the extraction of video content description text; the first fusion feature vector includes semantics used to characterize the content description text of the sample video data, and the second fusion feature vector includes semantics used to characterize the content description text of the labeled video data. For details on generating the first and second fusion feature vectors, please refer to the specific description of step S203 in the embodiment corresponding to Figure 4 above; this application embodiment will not repeat it here.

[0207] Step S403: Determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector; perform alignment processing on the first fused feature vector to obtain a first text feature vector with a vector length equal to the alignment length; perform alignment processing on the second fused feature vector to obtain a second text feature vector with a vector length equal to the alignment length.

[0208] Specifically, the computer device can determine the alignment length based on the vector lengths of the first fused feature vector and the second fused feature vector. For example, the alignment length can be determined as the average or median of the vector lengths of the first and second fused feature vectors. The computer device can perform alignment processing on the first fused feature vector to obtain a first text feature vector with a vector length equal to the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector with a vector length equal to the alignment length. For details on the alignment processing, please refer to the specific description of step S204 in the embodiment corresponding to Figure 4 above; this application embodiment will not repeat it here.

[0209] In some embodiments, the initial video retrieval model includes a feature alignment layer. The training process of the feature alignment layer may be as follows: inputting sample feature vectors into an encoder-decoder network; the encoder-decoder network includes an initial encoder layer, an initial reconstruction layer, and an initial decoder layer; the length of the model parameters of the initial reconstruction layer is the alignment length; in the initial encoder layer, the feature vector to be aligned is encoded to obtain an encoded feature vector, and the encoded feature vector is downsampled based on the time step to obtain a downsampled feature vector; in the initial reconstruction layer, an aligned feature vector is generated based on the dependency of the downsampled feature vector at each time step; in the initial decoder layer, the aligned feature vector is upsampled to obtain an upsampled feature vector; an alignment loss value is generated based on the feature vector to be aligned and the upsampled feature vector, and the model parameters of the initial encoder layer and the initial reconstruction layer are adjusted based on the alignment loss value; when the initial encoder layer and the initial reconstruction layer meet the model training convergence condition, the converged initial encoder layer and the converged initial reconstruction layer are determined as the feature alignment layer of the initial video retrieval model.

[0210] Specifically, please also refer to Figure 8, which is a schematic diagram of the model structure of an encoder-decoder network proposed in an embodiment of this application. As shown in Figure 8, the encoder-decoder network may include an initial encoding layer, an initial reconstruction layer, and an initial decoding layer. The encoder-decoder network can be a Long Short-Term Memory (LSTM) autoencoder. The initial encoding layer and the initial decoding layer in the encoder-decoder network are both LSTM (Long Short-Term Memory) layers, and the initial reconstruction layer is a linear layer. The encoder-decoder network can be trained in a self-supervised manner without any explicit labels.

[0211] The computer device can extend the length of the model parameters in the initial reconstruction layer to the alignment length, so that the length of the vector output by the initial reconstruction layer is fixed to the alignment length. In the initial encoding layer, the computer device... 0 Encoding is performed to obtain encoded feature vectors. Based on the number of time steps, the same initial hidden state is provided for each time step. The encoded feature vectors are then downsampled to obtain downsampled feature vectors.

[0212] In some embodiments, the encoder-decoder network is the core framework for training the feature alignment processing layer, consisting of three parts: an initial encoder layer responsible for compressing and simplifying the input features; an initial reconstruction layer, the core layer, whose model parameter length is fixed at the "alignment length," responsible for generating aligned feature vectors of the same length; and an initial decoder layer responsible for restoring (upsampling) the aligned feature vectors to a vector with a length close to that of the input features, for subsequent loss calculation. In the initial reconstruction layer, the aligned feature vector x is generated based on the dependency relationship of the downsampled feature vectors at each time step. tAlign the feature vector x t The vector length is the alignment length. In the initial decoding layer, the computer device can upsample the alignment feature vector to reconstruct the feature vector to be aligned, x. 0 The corresponding upsampled feature vector x 1 .

[0213] In some embodiments, during training, the input is a "feature vector to be aligned" (which can be understood as the original feature vector of the sample video, the length of which may not be fixed and needs to be converted to the alignment length). The "feature vector to be aligned" undergoes "encoding processing" (similar to compression, extracting core information) to obtain a more compact "encoded feature vector"; then, based on the "number of time steps" (the length of the feature sequence, such as the number of video frames), "downsampling processing" (such as merging features from adjacent frames to reduce data volume) is performed to obtain a "downsampled feature vector"—this step aims to reduce the complexity of subsequent processing while retaining key information. Based on the "dependency between the features of each time step and the preceding and following steps (such as the order of actions in the video)" in the downsampled feature vector, an "aligned feature vector" is generated. Since the parameter length of the initial reconstruction layer is fixed to the alignment length, the length of the generated aligned feature vector is strictly equal to the alignment length (the core of achieving alignment). The "aligned feature vector" undergoes upsampling processing (the opposite of downsampling, restoring the feature length to be close to the original length of the feature vector to be aligned) to obtain an upsampled feature vector—this step is for comparison with the original input feature vector to be aligned. By comparing the difference between the feature vector to be aligned (the original input) and the upsampled feature vector (the decoded output), an alignment loss value is generated (the greater the difference, the higher the loss value). The model uses this loss value to adjust the parameters of the initial encoding layer and the initial reconstruction layer (to make the encoding and reconstruction process more accurate and reduce loss). The above process is repeated, continuously adjusting the parameters until the outputs of the "initial encoding layer" and the "initial reconstruction layer" stabilize (i.e., the alignment loss value drops below a preset threshold, satisfying the model training convergence condition). At this point, these two converged layers are determined as the "feature alignment processing layer" of the initial video retrieval model, used to subsequently transform the input features into vectors of fixed alignment length.

[0214] As an example, the 10-length feature vector to be aligned is first encoded (extracting core information such as dribbling force and wrist rotation angle) to obtain an 8-length encoded feature vector. Then, it is downsampled based on the number of time steps (10 frames) (e.g., merging every 2 frames into 1 frame feature) to obtain a 4-length downsampled feature vector (preserving the key temporal dependencies of dribbling → ball bounce → dribbling again). Utilizing the "dependency relationship between each frame feature and the preceding and following frames" in the downsampled feature vector (e.g., the ball bounces in the 2nd frame, depending on the 1st frame, dribbling, affecting the dribbling again in the 3rd frame), an alignment feature vector of length 5 is generated (because the initial reconstruction layer parameter length is fixed at alignment length 5). This vector condenses the core temporal features of dribbling. The 5-length alignment feature vector is then upsampled (expanding the feature length) to obtain an 10-length upsampled feature vector (attempting to restore it to a vector with the same length as the input feature vector). By comparing the differences between the "original feature vector to be aligned" (10 frames) and the "upsampled feature vector" (10 frames), an alignment loss value is generated (if the upsampled vector loses the key feature of "wrist rotation", the loss value will be very high). The model uses the loss value to adjust the parameters of the initial encoding layer (optimizing the downsampling strategy) and the initial reconstruction layer (optimizing the temporal dependency capture ability) to reduce feature restoration error. Training is repeated. When the alignment loss value drops to a preset threshold (e.g., decreases and stabilizes below 0.01), it indicates that the initial encoding layer and the initial reconstruction layer can stably transform 10 frames of features into 5 frames of aligned features while retaining key information. At this point, these two converged layers are determined as the feature alignment processing layers.

[0215] Computer devices can be based on the feature vector x to be aligned 0 and upsampled feature vector x 1 The alignment loss value is generated and calculated as shown in formula (10):

[0216] Where L is the alignment loss value and T is the number of time steps.

[0217] The computer equipment adjusts the model parameters of the initial encoding layer and the initial reconstruction layer using the alignment loss value. When the initial encoding layer and the initial reconstruction layer meet the model training convergence condition, the converged initial encoding layer and the converged initial reconstruction layer are determined as the feature alignment processing layer of the initial video retrieval model. The vector output by the feature alignment processing layer can be a vector with a fixed length equal to the alignment length.

[0218] Thus, through closed-loop training of the encoder-decoder network, the encoding and downsampling processing of the initial encoding layer can retain core information while compressing features. The initial reconstruction layer generates feature vectors of fixed alignment length based on temporal dependencies, ensuring the consistency of feature length and providing a unified standard for subsequent processing. The upsampling processing and alignment loss calculation of the initial decoding layer form an effective supervision mechanism, which can back-optimize the parameters of the initial encoding and initial reconstruction layers by the difference between the original features and the reconstructed features. This prompts the model to retain key information to the maximum extent when compressing features to the alignment length, especially capturing the sequential dependencies in video temporal data. The feature alignment processing layer, determined after convergence, has both stable feature compression and alignment capabilities and ensures the integrity and effectiveness of feature information. This provides a reliable feature preprocessing foundation for the initial video retrieval model, improving the model's adaptability to features of different lengths and the accuracy of subsequent retrieval tasks.

[0219] Step S404: Generate the text similarity between the first text feature vector and the second text feature vector; generate a model loss value based on the text similarity; adjust the model parameters of the initial video retrieval model using the model loss value; when the initial video retrieval model meets the model training convergence condition, the target video retrieval model is obtained; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0220] Specifically, the computer device can generate a text similarity between the first text feature vector and the second text feature vector. Based on the text similarity and the similarity score between the sample video data and the labeled video data, a model loss value is generated. The model parameters of the initial video retrieval model are adjusted using the model loss value. When the initial video retrieval model meets the model training convergence condition, the target video retrieval model is obtained.

[0221] The initial training process of the video retrieval model is divided into two stages: a pre-training stage and an instruction tuning stage. First, in the pre-training stage, the computer freezes the visual encoding layer and the text description layer, and initializes the initial abstraction layer using checkpoints from Video-LLaMA (a multimodal large language model for video and audio). The focus is on training the initial abstraction layer and the multilayer perceptrons (MLPs) connected to it. During this stage, the computer performs 80,000 steps of fine-tuning on the multimodal large model. The AdamW optimizer (a regularized optimizer that introduces weight decay to prevent overfitting) is used, with parameters set to β1 = 0.9, β2 = 0.98, and a weight decay of 0.05. Here, β1 and β2 are hyperparameters for the AdamW optimizer.

[0222] Furthermore, the learning rate can be linearly warmed up during the initial 2000 iterations (steps), starting from 10.-8 Increase to 5×10 -5 Then, a cosine annealing learning rate scheduler is used, with a peak learning rate of 5×10. -5 Next, the parameters of the entire model were comprehensively fine-tuned over five training epochs to enhance its responsiveness to short videos. During this phase, the learning rate was set to 5 × 10⁻⁶. -5 A layer-wise learning rate decay factor of 0.9 was applied to preserve low-level visual representations. During the instruction tuning phase, a 5×10⁻⁶ learning rate decay was used. -5 The entire model is trained using a learning rate of 256 and a batch size of 256.

[0223] It is understood that the target video retrieval model can be used to perform the data processing method described in any of the embodiments corresponding to Figures 3, 4 and 6 above. The visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer. The visual abstraction layer contains a compressed query vector obtained by training the initial query vector. The compressed query vector is used to perform attention processing on the video data to be matched, resulting in a visual label vector with the same vector length as the compressed query vector.

[0224] This application's embodiments can fine-tune the model by focusing on datasets relevant to video captioning tasks, helping the model generate content description text in both English and Chinese environments. Furthermore, by learning a fixed set of compressed query vectors and performing multi-head attention processing, the visual marker vectors to be matched and the compressed query vectors are concatenated together. This better captures multimodal information, helping the target video retrieval model understand and abstract image information at different levels and granularities, thus improving the processing performance of the target video retrieval model. Through a visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector. The compressed visual marker vector is obtained by performing multi-head attention processing on the compressed query vector and the visual marker vector sequence, and the vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector. This effectively reduces the number of visual markers, significantly reducing memory load and computational resource consumption.

[0225] Alignment processing using a Long Short-Term Memory (LSTM) autoencoder avoids padding due to inconsistent vector lengths. For example, padding the business text feature vector and the fused feature vector to the same length can lead to reduced computational efficiency, increased memory consumption, and the model's inability to capture real data due to redundant information, resulting in decreased generalization. This embodiment effectively generates a matching text feature vector with the same alignment length from the fused feature vector. Similarity calculation between the matching text feature vector and the business text feature vector reduces computational costs and improves accuracy. Business video data associated with business text feature vectors whose text similarity is greater than or equal to a text similarity threshold is defined as a text retrieval video set. Using this set to determine similar video retrieval results for the matching video data avoids false positives and false negatives caused by misleading similarity scores due to visual elements, improving the accuracy and completeness of similar video retrieval.

[0226] Please refer to Figure 9, which is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. As shown in Figure 9, the data processing device 1 includes a visual marker extraction module 610, a visual marker processing module 620, a text feature acquisition module 630, and a similarity calculation module 640.

[0227] The visual marker extraction module 610 is configured to extract features from the video data to be matched to obtain a compressed visual marker vector.

[0228] The visual marker processing module 620 is configured to acquire business text information, concatenate the business text information with compressed visual marker vectors to obtain a fused feature vector, and perform alignment processing on the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length. The business text information is a text prompt used to indicate the content description text corresponding to the extracted video data. The fused feature vector includes the semantics of the content description text used to characterize the video data to be matched, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0229] The text feature acquisition module 630 is configured to acquire the business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0230] The similarity calculation module 640 is configured to generate text similarity between business text feature vectors and text feature vectors to be matched, and to determine the business video data associated with business text feature vectors whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine the similar video retrieval results for the video data to be matched.

[0231] In one possible implementation, the similarity calculation module 640 is also configured to perform the following operations:

[0232] Feature extraction is performed on the business video data to obtain the business visual feature vector, and feature extraction is performed on the video data to be matched to obtain the visual feature vector to be matched.

[0233] Generate visual similarity between business visual feature vectors and visual feature vectors to be matched, associate business video data with business visual feature vectors whose visual similarity is greater than or equal to the video similarity threshold, and determine the visual retrieval video set.

[0234] The intersection of the visually retrieved video set and the text-retrieved video set is determined as the similar video retrieval results for the video data to be matched;

[0235] Alternatively, the union of the visually retrieved video set and the text-retrieved video set can be used as the similar video retrieval result for the video data to be matched.

[0236] In one possible implementation, the visual marker extraction module 610 is configured to perform the following operations when extracting features from the video data to be matched to obtain a compressed visual marker vector:

[0237] The video data to be matched is input into the target video retrieval model; the target video retrieval model includes a multimodal text description layer, which includes a preprocessing layer, a visual encoding layer, and a visual abstraction layer;

[0238] In the preprocessing layer, based on the video duration and video frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames. The T sampled video frames are then segmented to obtain S unit sampled images. T is a positive integer determined by the video duration and video frame rate, and S is a positive integer greater than T.

[0239] In the visual coding layer, features are extracted from S unit sampled images to obtain S unit visual label vectors. Based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual label vectors are remapped to the feature channels to obtain S visual label vectors to be matched. The rearrangement factor is determined based on the number S of unit visual label vectors, and the vector length of the unit visual label vector is greater than the vector length of the visual label vector to be matched.

[0240] In the visual abstraction layer, attention processing is performed on the visual tag vector to be matched based on the compressed query vector to obtain the compressed visual tag vector; the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector.

[0241] In one possible implementation, the visual marker extraction module 610 is configured to perform attention processing on the visual marker vector to be matched based on the compressed query vector in the visual abstraction layer. When obtaining the compressed visual marker vector, it is specifically used to perform the following operations:

[0242] In the visual abstraction layer, the compressed query vector is obtained, and the visual tag vector to be matched and the compressed query vector are concatenated to obtain a sequence of visual tag vectors; the compressed query vector is a learnable parameter.

[0243] The compressed query vector and the visual label vector sequence are assigned to N unit processing spaces. In each unit processing space, cross-attention processing is performed on the compressed query vector and the visual label vector sequence to obtain N unit attention result vectors. Based on the attention weights corresponding to the N unit attention result vectors and the N unit processing spaces, a compressed visual label vector is generated; N is a positive integer.

[0244] In one possible implementation, the N unit processing spaces include the target unit processing space; the visual marker extraction module 610 is configured to perform cross-attention processing on the compressed query vector and the visual marker vector sequence in each unit processing space to obtain N unit attention result vectors, specifically for performing the following operations:

[0245] In the target unit processing space, the compressed query vector is multiplied by the query parameter matrix to obtain the query vector, the visual marker vector sequence is multiplied by the key parameter matrix to obtain the key vector, and the visual marker vector sequence is multiplied by the value parameter matrix to obtain the value vector.

[0246] An attention score vector is generated based on the query vector and the key vector. The attention score vector is then reduced in dimensionality based on the adjacent lengths of the key vectors. The reduced attention score vector is then normalized to obtain the attention weights. Finally, the attention weights are multiplied by the value vector to obtain the unit attention result vector corresponding to the target unit processing space.

[0247] In one possible implementation, the visual marker processing module 620 is configured to acquire business text information, concatenate the business text information with the compressed visual marker vector to obtain a fused feature vector, and align the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length. Specifically, it is used to perform the following operations:

[0248] The compressed visual tag vector is input into the target video retrieval model; the target video retrieval model includes a multimodal text description layer and a feature alignment processing layer, and the multimodal text description layer includes a text description layer;

[0249] In the text description layer, business text information is obtained, features are extracted from the business text information to obtain text prompt feature vectors, and the text prompt feature vectors are concatenated with compressed visual marker vectors to obtain fused feature vectors.

[0250] In the feature alignment layer, the fused feature vectors are aligned to obtain a text feature vector of the alignment length.

[0251] In one possible implementation, the visual marker processing module 620 is configured to perform alignment processing on the fused feature vector in the feature alignment processing layer to obtain a text feature vector to be matched with a vector length equal to the alignment length, specifically for performing the following operations:

[0252] The fused feature vector is input into the feature alignment processing layer, which includes an autoencoder layer and a bottleneck reconstruction layer. The bottleneck reconstruction layer is trained based on the alignment length, and the length of the vector output by the bottleneck reconstruction layer is fixed to the alignment length.

[0253] In the autoencoder layer, the fused feature vector is encoded to obtain an encoded feature vector. Based on the number of time steps, the encoded feature vector is downsampled to obtain a downsampled feature vector.

[0254] In the bottleneck reconstruction layer, based on the dependency relationship of the downsampled feature vectors at each time step, a text feature vector of the alignment length is generated to be matched.

[0255] In one possible implementation, the visual marker processing module 620 is configured to perform alignment processing on the fused feature vector in the feature alignment processing layer to obtain a text feature vector to be matched with a vector length equal to the alignment length, specifically for performing the following operations:

[0256] The fused feature vector is input into the feature alignment processing layer, which includes a word frequency processing layer and a weighted alignment layer.

[0257] In the word frequency processing layer, based on the fused feature vector, the predicted probability corresponding to each word in the vocabulary is generated. Based on the predicted probability, the content description text of the video data to be matched is generated. Based on the alignment length, the content description text is segmented to obtain F text segments; the alignment length is F, where F is a positive integer.

[0258] Obtain the word frequencies of F text fragments in the content description text, and the inverse document frequencies of F text fragments in the content description text. Generate F text weights based on the F word frequencies and F inverse document frequencies.

[0259] In the weighted alignment layer, the F text weights are multiplied by the fused feature vector to obtain F weighted feature vectors. Based on the F weighted feature vectors, a text feature vector to be matched with a vector length equal to the alignment length is generated.

[0260] In one possible implementation, the fused feature vector includes G unit prediction vectors, each with an alignment length. The last unit prediction vector in the G unit prediction vectors includes the global semantics obtained by concatenating the business text information and the compressed visual marker vector. The visual marker processing module 620 is configured to perform alignment processing on the fused feature vector in the feature alignment processing layer to obtain a text feature vector to be matched with a vector length equal to the alignment length. Specifically, the configuration includes the following operations:

[0261] In the feature alignment processing layer, the last unit prediction vector among the G unit prediction vectors is determined as the text feature vector to be matched with a vector length equal to the alignment length.

[0262] In one possible implementation, when the similarity calculation module 640 is configured to generate the text similarity between the business text feature vector and the text feature vector to be matched, it is specifically used to perform the following operations:

[0263] The business text feature vector and the text feature vector to be matched are input into the target video retrieval model; the target video retrieval model includes a recall matching layer.

[0264] Based on the value of each vector element in the business text feature vector, the first parameter of the business text feature vector is generated; based on the value of each vector element in the text feature vector to be matched, the second parameter of the text feature vector to be matched is generated.

[0265] The dot product operation is performed between the business text feature vector and the text feature vector to be matched to obtain the matching parameters. Based on the matching parameters, the first parameter, and the second parameter, the text similarity between the business text feature vector and the text feature vector to be matched is generated.

[0266] This application embodiment can sample the video data to be matched based on the context window length of the target video retrieval model to obtain unit sampled images. Feature extraction is then performed on these unit sampled images to obtain unit visual marker vectors. By rearranging the pixels of the image, the dimensionality of the vectors representing image height and width in the unit sampled images is reduced, thereby reducing the number of visual markers. By learning a fixed set of compressed query vectors and performing multi-head attention processing, the visual marker vectors to be matched and the compressed query vectors are concatenated together. This better captures multimodal information, helping the target video retrieval model understand and abstract image information at different levels and granularities, thus improving the processing performance of the target video retrieval model. Through a visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector. The compressed visual marker vector is obtained by performing multi-head attention processing on the compressed query vector and the visual marker vector sequence. The vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector. This effectively reduces the number of visual markers and significantly reduces memory load and computational resource consumption.

[0267] The process involves acquiring business text information and concatenating it with compressed visual marker vectors to obtain a fused feature vector. The business text information serves as a textual prompt indicating the content description text corresponding to the extracted video data, while the fused feature vector includes the semantics representing the content description text of the video data to be matched. The fused feature vector can ignore visual elements in the video data that are irrelevant to the video content, such as adversarial techniques like adding borders, texture effects, or occluded emojis, thus improving the model's robustness and resistance to perturbations. Since the fused feature vector only needs to retain the semantics representing the content description text, it effectively reduces the number of visual markers to be processed, decreases computational costs, and improves the efficiency of similar video retrieval. Alignment processing of the fused feature vector yields a matching text feature vector with a length equal to the alignment length. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Finally, the process involves acquiring the business text feature vector corresponding to the business video data, where the business text feature vector includes the semantics representing the content description text of the business video data. By calculating the text similarity between the business text feature vector and the text feature vector to be matched, the text similarity between them is obtained. Specifically, calculating the text similarity between business text feature vectors and the text feature vector to be matched with the same vector length avoids the padding problem caused by inconsistent vector lengths, effectively reducing computational costs and improving the efficiency of similar video retrieval. The business video data associated with business text feature vectors whose text similarity is greater than or equal to the text similarity threshold is defined as the text retrieval video set. Using this text retrieval video set to determine the similar video retrieval results for the video data to be matched avoids the problem of missed detections and false detections caused by reduced similarity due to visual elements, thus improving the accuracy and completeness of similar video retrieval.

[0268] In this application embodiment, the term "module" or "unit" refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0269] Please refer to Figure 10, which is a second structural schematic diagram of a data processing device provided in an embodiment of this application. As shown in Figure 10, the data processing device 2 includes a sample acquisition module 710, a fusion vector generation module 720, a vector alignment module 730, and a model adjustment module 740.

[0270] The sample acquisition module 710 is configured to acquire sample video data and labeled video data, and input the sample video data and labeled video data into the initial video retrieval model.

[0271] The fusion vector generation module 720 is configured to extract features from sample video data using an initial video retrieval model to obtain a first visual marker vector, acquire business text information, concatenate the business text information with the first visual marker vector to obtain a first fusion feature vector, extract features from labeled video data to obtain a second visual marker vector, and concatenate the business text information with the second visual marker vector to obtain a second fusion feature vector; the business text information is a text prompt used to indicate the extraction of video content description text; the first fusion feature vector includes semantics used to characterize the content description text of the sample video data, and the second fusion feature vector includes semantics used to characterize the content description text of the labeled video data;

[0272] The vector alignment module 730 is configured to determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, perform alignment processing on the first fused feature vector to obtain a first text feature vector with a vector length equal to the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector with a vector length equal to the alignment length.

[0273] The model adjustment module 740 is configured to generate the text similarity between the first text feature vector and the second text feature vector, generate the model loss value based on the text similarity, and adjust the model parameters of the initial video retrieval model through the model loss value. When the initial video retrieval model meets the model training convergence condition, the target video retrieval model is obtained. The target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0274] In one possible implementation, the initial video retrieval model includes a feature alignment processing layer; the model adjustment module 740 is also configured to perform the following operations:

[0275] The sample feature vector is input into the encoder-decoder network; the encoder-decoder network includes an initial encoder layer, an initial reconstruction layer, and an initial decoder layer; the length of the model parameters in the initial reconstruction layer is the alignment length.

[0276] In the initial coding layer, the feature vector to be aligned is encoded to obtain the encoded feature vector. Based on the time step, the encoded feature vector is downsampled to obtain the downsampled feature vector.

[0277] In the initial reconstruction layer, an aligned feature vector is generated based on the dependency of the downsampled feature vector at each time step. In the initial decoding layer, the aligned feature vector is upsampled to obtain the upsampled feature vector.

[0278] Alignment loss values ​​are generated based on the feature vectors to be aligned and the upsampled feature vectors. The model parameters of the initial encoding layer and the initial reconstruction layer are adjusted using the alignment loss values. When the initial encoding layer and the initial reconstruction layer meet the model training convergence conditions, the converged initial encoding layer and the converged initial reconstruction layer are determined as the feature alignment processing layer of the initial video retrieval model.

[0279] In one possible implementation, the initial video retrieval model includes an initial abstraction layer; the fusion vector generation module 720 is configured to perform the following operations when extracting features from the sample video data to obtain the first visual label vector:

[0280] Feature extraction is performed on the sample video data to obtain sample visual label vectors. An initial query vector, randomly initialized in the initial abstraction layer, is obtained. The sample visual label vectors are concatenated based on the initial query vector to obtain the first visual label vector. The initial query vector is the model parameter in the initial abstraction layer. The vector length of the initial query vector is the same as the vector length of the first visual label vector.

[0281] The visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer. The visual abstraction layer contains a compressed query vector trained by the initial query vector. The compressed query vector is used to perform attention processing on the video data to be matched, resulting in a visual label vector with the same vector length as the compressed query vector.

[0282] This application's embodiments can fine-tune the model by focusing on datasets relevant to video captioning tasks, helping the model generate content description text in both English and Chinese environments. Furthermore, by learning a fixed set of compressed query vectors and performing multi-head attention processing, the visual marker vectors to be matched and the compressed query vectors are concatenated together. This better captures multimodal information, helping the target video retrieval model understand and abstract image information at different levels and granularities, thus improving the processing performance of the target video retrieval model. Through a visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector. The compressed visual marker vector is obtained by performing multi-head attention processing on the compressed query vector and the visual marker vector sequence, and the vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector. This effectively reduces the number of visual markers, significantly reducing memory load and computational resource consumption.

[0283] Alignment processing using a Long Short-Term Memory (LSTM) autoencoder avoids padding due to inconsistent vector lengths. For example, padding the business text feature vector and the fused feature vector to the same length can lead to reduced computational efficiency, increased memory consumption, and the model's inability to capture real data due to redundant information, resulting in decreased generalization. This embodiment effectively generates a matching text feature vector with the same alignment length from the fused feature vector. Similarity calculation between the matching text feature vector and the business text feature vector reduces computational costs and improves accuracy. Business video data associated with business text feature vectors whose text similarity is greater than or equal to a text similarity threshold is defined as a text retrieval video set. Using this set to determine similar video retrieval results for the matching video data avoids false positives and false negatives caused by misleading similarity scores due to visual elements, improving the accuracy and completeness of similar video retrieval.

[0284] In this application embodiment, the term "module" or "unit" refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0285] Please refer to Figure 11, which is a schematic diagram of the structure of a computer device provided in an embodiment of this application. As shown in Figure 11, the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is configured to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. As shown in Figure 11, the memory 1005, as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and computer programs.

[0286] In the computer device 1000 shown in Figure 11, the network interface 1004 provides network communication elements; the user interface 1003 is mainly configured to provide an input interface for users; and the processor 1001 can be used to call computer programs stored in the memory 1005.

[0287] When the computer device 1000 is configured to execute the data processing device 1, it achieves the following:

[0288] Feature extraction is performed on the video data to be matched to obtain a compressed visual label vector;

[0289] The process involves acquiring business text information, concatenating it with compressed visual marker vectors to obtain a fused feature vector, aligning the fused feature vector to obtain a text feature vector of the target text with a length equal to the alignment length. The business text information is a text prompt indicating the content description text corresponding to the extracted video data. The fused feature vector includes semantics representing the content description text of the target video data, and its alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0290] Obtain the business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0291] The text similarity between the business text feature vector and the text feature vector to be matched is generated. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as the text retrieval video set. The text retrieval video set is used to determine the similar video retrieval results for the video data to be matched.

[0292] When computer device 100 is configured to execute data processing device 2, it achieves the following:

[0293] Obtain sample video data and labeled video data, and input the sample video data and labeled video data into the initial video retrieval model;

[0294] The initial video retrieval model extracts features from sample video data to obtain a first visual marker vector. Business text information is then acquired, and the business text information is concatenated with the first visual marker vector to obtain a first fusion feature vector. Features are extracted from labeled video data to obtain a second visual marker vector, and the business text information is concatenated with the second visual marker vector to obtain a second fusion feature vector. The business text information is a text prompt indicating the extraction of video content description text. The first fusion feature vector includes semantics representing the content description text of the sample video data, and the second fusion feature vector includes semantics representing the content description text of the labeled video data.

[0295] The alignment length is determined based on the vector lengths of the first and second fused feature vectors. The first fused feature vector is aligned to obtain a first text feature vector with a vector length equal to the alignment length. The second fused feature vector is aligned to obtain a second text feature vector with a vector length equal to the alignment length.

[0296] The text similarity between the first and second text feature vectors is generated. A model loss value is generated based on the text similarity. The model parameters of the initial video retrieval model are adjusted based on the model loss value. When the initial video retrieval model meets the model training convergence condition, the target video retrieval model is obtained. The target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0297] It should be understood that the computer device 1000 described in the embodiments of this application can execute the data processing method described in any one of the embodiments corresponding to Figures 3, 4, 6 and 7 above, and will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here either.

[0298] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When the processor executes the computer program, it can perform the data processing method described in any one of the embodiments corresponding to Figures 3, 4, 6, and 7 above. Therefore, it will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0299] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. The computer-readable storage medium may include both internal and external storage units of the computer device. The computer-readable storage medium is configured to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be configured to temporarily store data that has been displayed or will be displayed.

[0300] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in any one of the embodiments corresponding to Figures 3, 4, 6, and 7 above.

[0301] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "including" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but also includes steps or modules not listed, or other step units inherent to these processes, methods, apparatuses, products, or devices.

[0302] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the foregoing description as a network element. Whether these network elements are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described network elements using different methods for each specific application, but such implementation should not be considered beyond the scope of this application.

[0303] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowcharts and / or structural diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, create means for implementing the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions can also be stored in a computer-readable storage medium capable of directing a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable device, provide steps for implementing the functions specified in one or more flowcharts and / or one or more structural diagrams in blocks.

[0304] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0305] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0306] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, comprising: Feature extraction is performed on the video data to be matched to obtain a compressed visual label vector; Obtain business text information, concatenate the business text information with the compressed visual marker vector to obtain a fused feature vector, and perform alignment processing on the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length; The business text information is a text prompt used to indicate the content description text corresponding to the extracted video data. The fused feature vector includes the semantics of the content description text used to characterize the video data to be matched. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Obtain the business text feature vector corresponding to the business video data; the business text feature vector includes semantics used to characterize the content description text of the business video data; A text similarity is generated between the business text feature vector and the text feature vector to be matched. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set. The text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

2. The method according to claim 1, wherein, After determining the business video data associated with the business text feature vectors whose text similarity is greater than or equal to the text similarity threshold as the text retrieval video set, the method further includes: Feature extraction is performed on the business video data to obtain a business visual feature vector, and feature extraction is performed on the video data to be matched to obtain a visual feature vector to be matched. Generate the visual similarity between the business visual feature vector and the visual feature vector to be matched, and determine the business video data associated with the business visual feature vector whose visual similarity is greater than or equal to the video similarity threshold as the visual retrieval video set; The intersection of the visual retrieval video set and the text retrieval video set is determined as the similar video retrieval result for the video data to be matched; Alternatively, the union of the visually retrieved video set and the text-retrieved video set can be used as the similar video retrieval result for the video data to be matched.

3. The method according to claim 1, wherein, The feature extraction is achieved through a multimodal text description layer in the target video retrieval model, which includes a preprocessing layer, a visual encoding layer, and a visual abstraction layer. The step of extracting features from the video data to be matched to obtain a compressed visual marker vector includes: In the preprocessing layer, based on the video duration and video frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames. The T sampled video frames are then segmented to obtain S unit sampled images. T is a positive integer determined by the video duration and the video frame rate, and S is a positive integer greater than T. In the visual coding layer, feature extraction is performed on the S unit sampled images to obtain S unit visual marker vectors. Based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual marker vectors are remapped to the feature channels to obtain S visual marker vectors to be matched. The rearrangement factor is determined based on the number S of the unit visual marker vectors, and the vector length of the unit visual marker vectors is greater than the vector length of the visual marker vectors to be matched. In the visual abstraction layer, attention processing is performed on the visual tag vector to be matched based on the compressed query vector to obtain a compressed visual tag vector; the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector.

4. The method according to claim 3, wherein, In the visual abstraction layer, attention processing is performed on the visual tag vector to be matched based on the compressed query vector to obtain the compressed visual tag vector, including: In the visual abstraction layer, a compressed query vector is obtained, and the visual tag vector to be matched and the compressed query vector are concatenated to obtain a visual tag vector sequence; the compressed query vector is a learnable parameter. The compressed query vector and the visual tag vector sequence are assigned to N unit processing spaces. In each unit processing space, the compressed query vector and the visual tag vector sequence are subjected to cross-attention processing to obtain N unit attention result vectors. Based on the N unit attention result vectors and the attention weights corresponding to the N unit processing spaces, a compressed visual tag vector is generated; N is a positive integer.

5. The method according to claim 4, wherein, The N unit processing spaces include the target unit processing space; The cross-attention processing is performed on the compressed query vector and the visual tag vector sequence in each unit processing space to obtain N unit attention result vectors, including: In the target unit processing space, the compressed query vector is multiplied by the query parameter matrix to obtain the query vector; the visual marker vector sequence is multiplied by the key parameter matrix to obtain the key vector; and the visual marker vector sequence is multiplied by the value parameter matrix to obtain the value vector. An attention score vector is generated based on the query vector and the key vector. The attention score vector is then reduced in dimensionality based on the adjacent lengths of the key vector. The reduced attention score vector is then normalized to obtain the attention weights corresponding to the target unit processing space. Finally, the attention weights corresponding to the target unit processing space are multiplied by the value vector to obtain the unit attention result vector corresponding to the target unit processing space.

6. The method according to claim 1, wherein, The concatenation is achieved through the text description layer in the target video retrieval model. The concatenation of the business text information with the compressed visual marker vector to obtain a fused feature vector includes: In the text description layer, features are extracted from the business text information to obtain a text prompt feature vector. The text prompt feature vector is then concatenated with the compressed visual marker vector to obtain a fused feature vector. The alignment process is implemented through the feature alignment layer in the target video retrieval model. The alignment process on the fused feature vector to obtain a text feature vector to be matched with a length equal to the alignment length includes: In the feature alignment processing layer, the fused feature vector is aligned to obtain a text feature vector to be matched with a vector length equal to the alignment length.

7. The method according to claim 6, wherein, The feature alignment processing layer includes an autoencoder layer and a bottleneck reconstruction layer. The bottleneck reconstruction layer is trained based on the alignment length, and the length of the vector output by the bottleneck reconstruction layer is fixed to the alignment length. In the feature alignment processing layer, the fused feature vector is aligned to obtain a text feature vector to be matched with a vector length equal to the alignment length, including: In the autoencoding layer, the fused feature vector is encoded to obtain an encoded feature vector. Based on the number of time steps, the encoded feature vector is downsampled to obtain a downsampled feature vector. In the bottleneck reconstruction layer, based on the dependency of the downsampled feature vector at each time step, a text feature vector to be matched with a vector length equal to the alignment length is generated.

8. The method according to claim 6, wherein, The feature alignment processing layer includes a word frequency processing layer and a weighted alignment layer; in the feature alignment processing layer, the fused feature vector is aligned to obtain a text feature vector to be matched with a vector length equal to the alignment length, including: In the word frequency processing layer, based on the fused feature vector, a predicted probability corresponding to each word in the vocabulary is generated. Based on the predicted probability, a content description text for the video data to be matched is generated. Based on the alignment length, the content description text is segmented to obtain F text segments; where the alignment length is F, and F is a positive integer. Obtain the word frequencies of the F text segments in the content description text, and determine the inverse document frequencies of the F text segments in the content description text. Generate F text weights based on the F word frequencies and the F inverse document frequencies. In the weighted alignment layer, the F text weights are multiplied by the fused feature vector to obtain F weighted feature vectors. Based on the F weighted feature vectors, a text feature vector to be matched with a vector length equal to the alignment length is generated.

9. The method according to claim 8, wherein, Determining the inverse document frequencies of the F text fragments in the content description text includes: Obtain a collection of documents including the content description text, and perform the following processing on each text fragment: Determine the total number of documents in the document set, identify the documents in the document set that include the text fragment as target documents, and determine the number of target documents; The inverse document probability of the text fragment is determined based on the ratio of the total number of documents to the number of target documents.

10. The method according to claim 6, wherein, The fused feature vector includes G unit prediction vectors, each with an alignment length. The last unit prediction vector in the G unit prediction vectors includes the global semantics obtained by concatenating the business text information and the compressed visual marker vector. In the feature alignment processing layer, the fused feature vector is aligned to obtain a text feature vector to be matched with an alignment length, including: In the feature alignment processing layer, the last unit prediction vector among the G unit prediction vectors is determined as the text feature vector to be matched with a vector length equal to the alignment length.

11. The method according to claim 1, wherein the text similarity is achieved through a recall matching layer in the target video retrieval model, wherein, The process of generating the text similarity between the business text feature vector and the text feature vector to be matched includes: In the recall matching layer, a first parameter of the business text feature vector is generated based on the value of each vector element in the business text feature vector, and a second parameter of the text feature vector to be matched is generated based on the value of each vector element in the text feature vector to be matched. The business text feature vector and the text feature vector to be matched are multiplied by a dot product to obtain matching parameters. Based on the matching parameters, the first parameter, and the second parameter, the text similarity between the business text feature vector and the text feature vector to be matched is generated.

12. A data processing method, comprising: Acquire sample video data and labeled video data; Using an initial video retrieval model, feature extraction is performed on the sample video data to obtain a first visual marker vector. Business text information is then acquired, and the business text information is concatenated with the first visual marker vector to obtain a first fusion feature vector. Feature extraction is performed on the labeled video data to obtain a second visual marker vector, and the business text information is concatenated with the second visual marker vector to obtain a second fusion feature vector. The business text information is a text prompt indicating the extraction of video content description text. The first fusion feature vector includes semantics characterizing the content description text of the sample video data, and the second fusion feature vector includes semantics characterizing the content description text of the labeled video data. The alignment length is determined based on the vector length of the first fused feature vector and the vector length of the second fused feature vector. The first fused feature vector is aligned to obtain a first text feature vector with a vector length equal to the alignment length. The second fused feature vector is aligned to obtain a second text feature vector with a vector length equal to the alignment length. A text similarity is generated between the first text feature vector and the second text feature vector. A model loss value is generated based on the text similarity. The model parameters of the initial video retrieval model are adjusted using the model loss value. When the initial video retrieval model meets the model training convergence condition, a target video retrieval model is obtained. The target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

13. The method according to claim 12, wherein, The initial video retrieval model includes a feature alignment layer, which is obtained by training an initial encoding layer and an initial reconstruction layer of an encoder-decoder network. The encoder-decoder network includes the initial encoding layer, the initial reconstruction layer, and the initial decoding layer. The length of the model parameters of the initial reconstruction layer is the alignment length. The method further includes: In the initial coding layer, the feature vector to be aligned is encoded to obtain a coded feature vector. Based on the number of time steps, the coded feature vector is downsampled to obtain a downsampled feature vector. In the initial reconstruction layer, an aligned feature vector is generated based on the dependency of the downsampled feature vector at each time step. In the initial decoding layer, the aligned feature vector is upsampled to obtain an upsampled feature vector. An alignment loss value is generated based on the feature vector to be aligned and the upsampled feature vector. The model parameters of the initial encoding layer and the initial reconstruction layer are adjusted using the alignment loss value. When the initial encoding layer and the initial reconstruction layer meet the model training convergence condition, the converged initial encoding layer and the converged initial reconstruction layer are determined as the feature alignment processing layer of the initial video retrieval model.

14. The method according to claim 13, wherein, The step of generating an alignment loss value based on the feature vector to be aligned and the upsampled feature vector includes: Determine the vector difference between the feature vector to be aligned and the upsampled feature vector at each time step, and sum the vector differences at each time step to obtain the overall difference; The ratio of the total difference to the total number of time steps is determined as the alignment loss value.

15. The method according to claim 12, wherein, The initial video retrieval model includes an initial abstraction layer; the feature extraction of the sample video data to obtain a first visual label vector includes: Feature extraction is performed on the sample video data to obtain sample visual marker vectors. An initial query vector, randomly initialized in the initial abstraction layer, is obtained. The sample visual marker vectors are concatenated based on the initial query vectors to obtain a first visual marker vector. The initial query vectors are model parameters in the initial abstraction layer. The vector length of the initial query vector is the same as the vector length of the first visual marker vector. The visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer. The visual abstraction layer includes a compressed query vector trained by the initial query vector. The compressed query vector is used to perform attention processing on the video data to be matched, so as to obtain a visual tag vector with the same vector length as the compressed query vector.

16. A data processing apparatus, comprising: The visual marker extraction module is configured to extract features from the video data to be matched, and obtain a compressed visual marker vector. The visual marker processing module is configured to acquire business text information, concatenate the business text information with the compressed visual marker vector to obtain a fused feature vector, and align the fused feature vector to obtain a text feature vector to be matched with a vector length equal to the alignment length. The business text information is a text prompt indicating the extraction of content description text corresponding to the video data. The fused feature vector includes semantics representing the content description text of the video data to be matched. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. The text feature acquisition module is configured to acquire the business text feature vector corresponding to the business video data; the business text feature vector includes semantics used to characterize the content description text of the business video data; The similarity calculation module is configured to generate a text similarity between the business text feature vector and the text feature vector to be matched, and to determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

17. The apparatus according to claim 16, wherein, The visual marker extraction module is further configured to, in the preprocessing layer, sample the video data to be matched based on the video duration and video frame rate of the video data to be matched to obtain T sampled video frames, and perform image segmentation on the T sampled video frames respectively to obtain S unit sampled images; T is a positive integer determined by the video duration and the video frame rate, and S is a positive integer greater than T. In the visual coding layer, feature extraction is performed on the S unit sampled images to obtain S unit visual marker vectors. Based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual marker vectors are remapped to the feature channels to obtain S visual marker vectors to be matched. The rearrangement factor is determined based on the number S of the unit visual marker vectors, and the vector length of the unit visual marker vectors is greater than the vector length of the visual marker vectors to be matched. In the visual abstraction layer, attention processing is performed on the visual marker vectors to be matched based on the compressed query vector to obtain compressed visual marker vectors. The vector length of the compressed visual marker vectors is the same as the vector length of the compressed query vector.

18. A data processing apparatus, comprising: The sample acquisition module is configured to acquire sample video data and labeled video data. The fusion vector generation module is configured to extract features from the sample video data using an initial video retrieval model to obtain a first visual marker vector, acquire business text information, concatenate the business text information with the first visual marker vector to obtain a first fusion feature vector, extract features from the labeled video data to obtain a second visual marker vector, and concatenate the business text information with the second visual marker vector to obtain a second fusion feature vector; the business text information is a text prompt indicating the extraction of video content description text; the first fusion feature vector includes semantics characterizing the content description text of the sample video data, and the second fusion feature vector includes semantics characterizing the content description text of the labeled video data; The vector alignment module is configured to determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, perform alignment processing on the first fused feature vector to obtain a first text feature vector with a vector length equal to the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector with a vector length equal to the alignment length. The model adjustment module is configured to generate a text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, adjust the model parameters of the initial video retrieval model using the model loss value, and obtain a target video retrieval model when the initial video retrieval model meets the model training convergence condition; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

19. The apparatus according to claim 18, wherein, The fusion vector generation module is further configured to extract features from the sample video data to obtain sample visual marker vectors, acquire an initial query vector randomly initialized in the initial abstraction layer, and concatenate the sample visual marker vectors based on the initial query vectors to obtain a first visual marker vector; the initial query vector is a model parameter in the initial abstraction layer; the vector length of the initial query vector is the same as the vector length of the first visual marker vector; wherein, the visual abstraction layer of the target video retrieval model is trained through the initial abstraction layer, the visual abstraction layer includes a compressed query vector trained through the initial query vector, and the compressed query vector is used to perform attention processing on the video data to be matched to obtain a visual marker vector with the same vector length as the compressed query vector.

20. A computer device, comprising: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the data processing method according to any one of claims 1-15.

21. A computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the data processing method according to any one of claims 1-15.

22. A computer program product comprising a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform the data processing method of any one of claims 1-15.

Images