Data processing method, device, equipment and readable storage medium

Abstract

The present application discloses a data processing method, device, equipment and readable storage medium, the method comprising: extracting features from video data to be matched to obtain a compressed visual tag vector; generating a text feature vector to be matched whose vector length is the alignment length based on the compressed visual tag vector; obtaining a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data; generating text similarity between the business text feature vector and the text feature vector to be matched, and determining the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched. The present application can improve the accuracy and completeness of video retrieval.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment and readable storage medium. Background Art

[0002] Existing visual similarity retrieval technology is mainly based on visual content video retrieval (CBVR). This technology uses the user to input the content to be queried (such as images or videos), uses the visual feature extractor to extract visual features, or performs structured analysis on the content to be queried to obtain the visual features of the content to be queried, and then compares the visual features of the content to be queried with the visual features in the video library to determine some videos that are similar to the content to be queried.

[0003] For video retrieval based on visual content, the image content in the video frame is complex, and the model may focus too much on local features and ignore the global contextual information of the video content, thus reducing the accuracy of video retrieval.

[0004] In video review applications, users may use adversarial attacks to change the visual representation of a video through tiny perturbations, such as cutting the video into multiple segments, adding borders, texture effects, or blocking emoticons. The model cannot correctly recognize its visual features, resulting in false detections (different videos are judged to be similar) or missed detections (similar videos are judged to be different videos), resulting in low video retrieval accuracy, incomplete retrieval, and low anti-disturbance performance. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, apparatus, device and readable storage medium, which can improve the accuracy and completeness of video retrieval.

[0006] On the one hand, an embodiment of the present application provides a data processing method, including:

[0007] Perform feature extraction on the video data to be matched to obtain a compressed visual tag vector;

[0008] Acquire business text information, perform vector concatenation of the business text information and the compressed visual mark vector to obtain a fused feature vector, perform alignment processing on the fused feature vector, and obtain a to-be-matched text feature vector whose vector length is the alignment length; the business text information is a text prompt for indicating the extraction of content description text corresponding to the video data, the fused feature vector includes semantics for characterizing the content description text of the to-be-matched video data, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database;

[0009] Acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0010] Generate text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0011] On the one hand, an embodiment of the present application provides another data processing method, including:

[0012] Obtaining sample video data and annotated video data, and inputting the sample video data and the annotated video data into an initial video retrieval model;

[0013] Through the initial video retrieval model, feature extraction is performed on the sample video data to obtain a first visual tag vector, business text information is obtained, the business text information and the first visual tag vector are vector-joined to obtain a first fused feature vector, feature extraction is performed on the annotated video data to obtain a second visual tag vector, and the business text information and the second visual tag vector are vector-joined to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data;

[0014] Determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, perform alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length;

[0015] Generate text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, adjust the model parameters of the initial video retrieval model according to the model loss value, and obtain a target video retrieval model when the initial video retrieval model meets the model training convergence condition; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0016] An embodiment of the present application provides a data processing device, including:

[0017] A visual marker extraction module is used to extract features from the video data to be matched and obtain a compressed visual marker vector;

[0018] A visual tag processing module is used to obtain business text information, perform vector concatenation of the business text information and the compressed visual tag vector to obtain a fused feature vector, and perform alignment processing on the fused feature vector to obtain a to-be-matched text feature vector whose vector length is the alignment length; the business text information is a text prompt for indicating the extraction of the content description text corresponding to the video data, the fused feature vector includes the semantics of the content description text for characterizing the to-be-matched video data, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database;

[0019] A text feature acquisition module, used to acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0020] The similarity calculation module is used to generate text similarity between the business text feature vector and the text feature vector to be matched, and the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0021] In a possible implementation, the similarity calculation module is further configured to perform the following operations:

[0022] Extract features from the business video data to obtain a business visual feature vector, and extract features from the video data to be matched to obtain a visual feature vector to be matched;

[0023] Generate visual similarity between the business visual feature vector and the visual feature vector to be matched, associate the business video data with the business visual feature vector whose visual similarity is greater than or equal to the video similarity threshold, and determine the visual retrieval video set;

[0024] Determine the intersection of the visual retrieval video set and the text retrieval video set as a similar video retrieval result for the video data to be matched;

[0025] Alternatively, a union of the visual retrieval video set and the text retrieval video set is determined as a similar video retrieval result for the video data to be matched.

[0026] In a possible implementation, the visual marker extraction module is used to extract features from the video data to be matched, and when obtaining the compressed visual marker vector, it is specifically used to perform the following operations:

[0027] Inputting the to-be-matched video data into a target video retrieval model; the target video retrieval model includes a multimodal text description layer, and the multimodal text description layer includes a preprocessing layer, a visual encoding layer, and a visual abstraction layer;

[0028] In the preprocessing layer, based on the video duration and video frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames, and the T sampled video frames are respectively segmented to obtain S unit sampled images; T is a positive integer determined based on the video duration and the video frame rate, and S is a positive integer greater than T;

[0029] In the visual encoding layer, feature extraction is performed on S unit sampled images to obtain S unit visual tag vectors. Based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual tag vectors are remapped to the feature channels to obtain S visual tag vectors to be matched; the rearrangement factor is determined based on the number S of unit visual tag vectors, and the vector length of the unit visual tag vector is greater than the vector length of the visual tag vector to be matched;

[0030] In the visual abstraction layer, attention processing is performed on the matching visual tag vector based on the compressed query vector to obtain a compressed visual tag vector; the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector.

[0031] In a possible implementation, the visual tag extraction module is used to perform attention processing on the matching visual tag vector based on the compressed query vector in the visual abstraction layer, and when the compressed visual tag vector is obtained, specifically perform the following operations:

[0032] In the visual abstraction layer, a compressed query vector is obtained, and the visual tag vector to be matched and the compressed query vector are concatenated to obtain a visual tag vector sequence; the compressed query vector is a learnable parameter;

[0033] The compressed query vector and the visual tag vector sequence are assigned to N unit processing spaces, and cross-attention processing is performed on the compressed query vector and the visual tag vector sequence in each unit processing space to obtain N unit attention result vectors. Based on the attention weights corresponding to the N unit attention result vectors and the N unit processing spaces, a compressed visual tag vector is generated; N is a positive integer.

[0034] In a possible implementation, the N unit processing spaces include a target unit processing space; the visual tag extraction module is used to perform cross-attention processing on the compressed query vector and the visual tag vector sequence in each unit processing space, and when N unit attention result vectors are obtained, specifically to perform the following operations:

[0035] In the target unit processing space, the compressed query vector is multiplied by the query parameter matrix to obtain the query vector, the visual tag vector sequence is multiplied by the key parameter matrix to obtain the key vector, and the visual tag vector sequence is multiplied by the value parameter matrix to obtain the value vector;

[0036] An attention score vector is generated based on the query vector and the key vector. The attention score vector is reduced in dimension based on the adjacent length of the key vector. The reduced-dimensional attention score vector is normalized to obtain the attention weight. The attention weight is point-multiplied with the value vector to obtain the unit attention result vector corresponding to the target unit processing space.

[0037] In a possible implementation, the visual tag processing module is used to obtain business text information, perform vector concatenation on the business text information and the compressed visual tag vector to obtain a fused feature vector, perform alignment processing on the fused feature vector, and obtain a to-be-matched text feature vector whose vector length is the alignment length, and specifically perform the following operations:

[0038] inputting the compressed visual tag vector into a target video retrieval model; the target video retrieval model comprises a multimodal text description layer and a feature alignment processing layer, and the multimodal text description layer comprises a text description layer;

[0039] In the text description layer, business text information is obtained, features are extracted from the business text information to obtain a text prompt feature vector, and the text prompt feature vector is concatenated with the compressed visual mark vector to obtain a fused feature vector;

[0040] In the feature alignment processing layer, the fused feature vector is aligned to obtain a feature vector of the text to be matched whose vector length is the alignment length.

[0041] In a possible implementation, the visual tag processing module is used to perform alignment processing on the fused feature vector in the feature alignment processing layer, and when obtaining the to-be-matched text feature vector whose vector length is the alignment length, specifically to perform the following operations:

[0042] The fused feature vector is input into the feature alignment processing layer, which includes an automatic encoding layer and a bottleneck reconstruction layer; the bottleneck reconstruction layer is obtained based on the alignment length training, and the vector length output by the bottleneck reconstruction layer is fixed to the alignment length;

[0043] In the automatic encoding layer, the fused feature vector is encoded to obtain an encoded feature vector, and the encoded feature vector is downsampled based on the time step to obtain a downsampled feature vector;

[0044] In the bottleneck reconstruction layer, based on the dependency of the downsampled feature vectors in each time step, a feature vector of the text to be matched with a vector length equal to the alignment length is generated.

[0045] In a possible implementation, the visual tag processing module is used to perform alignment processing on the fused feature vector in the feature alignment processing layer, and when obtaining the to-be-matched text feature vector whose vector length is the alignment length, specifically to perform the following operations:

[0046] Input the fused feature vector into the feature alignment processing layer, which includes a word frequency processing layer and a weighted alignment layer;

[0047] In the word frequency processing layer, based on the fused feature vector, the prediction probability corresponding to each word in the vocabulary is generated, and the content description text of the video data to be matched is generated based on the prediction probability. Based on the alignment length, the content description text is segmented to obtain F text segments; the alignment length is F, and F is a positive integer;

[0048] Obtain the word frequencies corresponding to the F text segments in the content description text, and the inverse document frequencies corresponding to the F text segments in the content description text, and generate F text weights based on the F word frequencies and the F inverse document frequencies;

[0049] In the weighted alignment layer, the F text weights are respectively multiplied with the fused feature vector to obtain F weighted feature vectors, and a text feature vector to be matched whose vector length is the alignment length is generated based on the F weighted feature vectors.

[0050] In a possible implementation, the fused feature vector includes G unit prediction vectors, the vector lengths of the G unit prediction vectors are all alignment lengths, and the last unit prediction vector of the G unit prediction vectors includes the global semantics obtained by vector concatenation of the business text information and the compressed visual tag vector; the visual tag processing module is used to align the fused feature vector in the feature alignment processing layer, and when the to-be-matched text feature vector having a vector length of the alignment length is obtained, specifically, to perform the following operations:

[0051] In the feature alignment processing layer, the last unit prediction vector among the G unit prediction vectors is determined as the feature vector of the text to be matched whose vector length is the alignment length.

[0052] In a possible implementation, when the similarity calculation module is used to generate the text similarity between the service text feature vector and the to-be-matched text feature vector, it is specifically used to perform the following operations:

[0053] Inputting the business text feature vector and the to-be-matched text feature vector into a target video retrieval model; the target video retrieval model includes a recall matching layer;

[0054] Based on each vector element value in the business text feature vector, a first parameter of the business text feature vector is generated; based on each vector element value in the to-be-matched text feature vector, a second parameter of the to-be-matched text feature vector is generated;

[0055] A dot multiplication operation is performed on the business text feature vector and the feature vector of the text to be matched to obtain a matching parameter, and a text similarity between the business text feature vector and the feature vector of the text to be matched is generated based on the matching parameter, the first parameter and the second parameter.

[0056] On the one hand, an embodiment of the present application provides another data processing device, including:

[0057] A sample acquisition module is used to acquire sample video data and annotated video data, and input the sample video data and the annotated video data into an initial video retrieval model;

[0058] A fusion vector generation module is used to perform feature extraction on sample video data through an initial video retrieval model to obtain a first visual tag vector, obtain business text information, perform vector splicing on the business text information and the first visual tag vector to obtain a first fused feature vector, perform feature extraction on the annotated video data to obtain a second visual tag vector, and perform vector splicing on the business text information and the second visual tag vector to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data;

[0059] A vector alignment module, used for determining an alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, performing alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and performing alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length;

[0060] A model adjustment module is used to generate text similarity between a first text feature vector and a second text feature vector, generate a model loss value based on the text similarity, and adjust the model parameters of the initial video retrieval model through the model loss value. When the initial video retrieval model meets the model training convergence conditions, a target video retrieval model is obtained; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0061] In a possible implementation, the initial video retrieval model includes a feature alignment processing layer; and the model adjustment module is further used to perform the following operations:

[0062] The sample feature vector is input into the encoding and decoding network; the encoding and decoding network includes an initial encoding layer, an initial reconstruction layer and an initial decoding layer; the length of the model parameter of the initial reconstruction layer is the alignment length;

[0063] In the initial coding layer, the feature vector to be aligned is encoded to obtain an encoded feature vector, and the encoded feature vector is downsampled based on the number of time steps to obtain a downsampled feature vector;

[0064] In the initial reconstruction layer, an aligned feature vector is generated based on the dependency of the downsampled feature vector in each time step. In the initial decoding layer, the aligned feature vector is upsampled to obtain an upsampled feature vector.

[0065] An alignment loss value is generated based on the feature vector to be aligned and the upsampled feature vector, and the model parameters of the initial coding layer and the initial reconstruction layer are adjusted according to the alignment loss value. When the initial coding layer and the initial reconstruction layer meet the model training convergence conditions, the converged initial coding layer and the converged initial reconstruction layer are determined as the feature alignment processing layers of the initial video retrieval model.

[0066] In a possible implementation, the initial video retrieval model includes an initial abstraction layer; the fusion vector generation module is used to extract features from the sample video data, and when the first visual tag vector is obtained, it is specifically used to perform the following operations:

[0067] Extract features from sample video data to obtain a sample visual tag vector, obtain an initial query vector randomly initialized in the initial abstract layer, and perform vector concatenation on the sample visual tag vector based on the initial query vector to obtain a first visual tag vector; the initial query vector is a model parameter in the initial abstract layer; the vector length of the initial query vector is the same as the vector length of the first visual tag vector;

[0068] Among them, the visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer. The visual abstraction layer contains a compressed query vector obtained by training the initial query vector. The compressed query vector is used to perform attention processing on the matching video data to obtain a visual label vector with the same vector length as the compressed query vector.

[0069] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface;

[0070] The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, and the memory is used to store a computer program. When the computer program is executed by the processor, the computer device executes the method provided in the embodiment of the present application.

[0071] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0072] In one aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in the embodiment of the present application.

[0073] The embodiment of the present application extracts features from the video data to be matched, obtains a compressed visual marker vector, obtains business text information, and performs vector splicing on the business text information and the compressed visual marker vector to obtain a fused feature vector, wherein the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics of the content description text for characterizing the video data to be matched. The fused feature vector can ignore visual elements in the video data to be matched that are irrelevant to the video content, such as lace, texture effects, or occluded emoticons added by adversarial means, which can improve the robustness and anti-disturbance of the model. By aligning the fused feature vector, a text feature vector to be matched whose vector length is the alignment length is obtained. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Obtain the business text feature vector corresponding to the business video data, wherein the business text feature vector includes the semantics of the content description text for characterizing the business video data. By calculating the text similarity between the business text feature vector and the text feature vector to be matched, the text similarity between the business text feature vector and the text feature vector to be matched is obtained. The text similarity between the business text feature vector and the text feature vector to be matched with the same vector length is calculated, which can avoid the problem of filling in due to the inconsistency of the vector length, further reducing the calculation cost and improving the efficiency of similar video retrieval. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set. The similar video retrieval results for the video data to be matched are determined by the text retrieval video set, which can avoid the problem of missed detection and false detection caused by misleading similarity caused by visual elements, and improve the accuracy and completeness of similar video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 This is a schematic diagram of a network architecture provided by an embodiment of the present application;

[0076] Figure 2 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0077] Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 ;

[0078] Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 ;

[0079] Figure 5 is a schematic diagram of a model structure of a target video retrieval model provided in an embodiment of the present application;

[0080] Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 ;

[0081] Figure 7 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 ;

[0082] Figure 8 It is a schematic diagram of a model structure of a coding and decoding network provided in an embodiment of the present application;

[0083] Fig. 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 1 ;

[0084] Fig.10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 2 ;

[0085] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0087] It can be understood that in the specific implementation of this application, the user (object or player) data involved, when the above and following embodiments of this application are applied to specific products or technologies, needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions.

[0088] Among them, if it is necessary to collect object (such as user, etc.) data in this application, a prompt interface or pop-up window will be displayed before and during the collection. The prompt interface or pop-up window is used to prompt the user that certain data is currently being collected. Only after the user confirms the prompt interface or pop-up window, the relevant steps of data acquisition will be started, otherwise it will end. Moreover, the acquired user data will be used in reasonable and legal scenarios or purposes. Optionally, in some scenarios where user data needs to be used but has not been authorized by the user, authorization can be requested from the user, and the user data can be used when the authorization is passed.

[0089] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a business server 100 and a terminal device cluster, and the terminal device cluster may include terminal device 10a, terminal device 10b, ..., terminal device 10n, wherein any terminal device in the terminal device cluster may have a communication connection with the business server 100, for example, there is a communication connection between the terminal device 10a and the business server 100, and there is a communication connection between the terminal device 10b and the business server 100, wherein the above-mentioned communication connection does not limit the connection method, and may be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and the present application does not make any restrictions here.

[0090] Each terminal device in the terminal device cluster may include: smart phones, tablet computers, laptop computers, desktop computers, intelligent voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, vehicle terminals, aircraft, and other smart terminals with data processing functions. The vehicle terminal may be a terminal device in smart transportation scenarios and assisted driving scenarios. It should be understood that Figure 1Each terminal device in the terminal device cluster shown in the figure can be installed with an application client having a data processing function. When the application client runs in each terminal device, it can be respectively connected to the above Figure 1 The business servers 100 shown perform data interaction.

[0091] The application client may specifically include: an in-vehicle client, a smart home client, an entertainment client (e.g., a game client), a multimedia client (e.g., a video client), a social client, and an information client (e.g., a news client). The application client in the embodiment of the present application may be integrated in a client (e.g., a social client) or may be an independent client (e.g., a news client). The embodiment of the present application does not limit the type of the application client.

[0092] Among them, the business server 100 can be a server corresponding to the application client. The business server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0093] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support to be realized through cloud computing.

[0094] like Figure 1 As shown, for ease of understanding, the terminal device 10a is taken as an example for description. A user can upload the video data to be matched through the application client in the terminal device 10a, and send the video data to be matched to the service server 100.

[0095] The service server 100 may obtain the video data to be matched sent by the terminal device 10a, perform feature extraction on the video data to be matched, and obtain a fused feature vector. The fused feature vector includes the semantics of the content description text (Caption) used to characterize the video data to be matched, and the fused feature vector may be a feature vector used to generate the content description text for the video data to be matched. The fused feature vector may be extracted through a multimodal large language model (LLM).

[0096] The business server 100 can align the fused feature vector to obtain a text feature vector to be matched whose vector length is the alignment length, compare the text feature vector to be matched with the business text feature vector corresponding to the business video data in the video text feature database to obtain the text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set.

[0097] Among them, the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database, the business text feature vector includes the semantics of the content description text used to characterize the business video data, and the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0098] It can be understood that the terminal device 10a and the service server 100 can provide users with a more personalized and consistent viewing experience through similar video retrieval results. The recommendation system can recommend content similar to videos previously watched by users, thereby improving user satisfaction and content discovery rate. During the video review process, similar video retrieval results can help identify and filter out duplicate or illegal content, ensuring the diversity and compliance of platform content.

[0099] The embodiment of the present application obtains a fused feature vector by extracting features from the video data to be matched, wherein the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics of the content description text for characterizing the video data to be matched. The fused feature vector can ignore visual elements in the video data to be matched that are irrelevant to the video content, such as lace, texture effects or occluded emoticons added by adversarial means, which can improve the robustness and anti-disturbance of the model. Generating a fused feature vector through a multimodal large language model can more effectively resist interference and maintain the accuracy and stability of the description. When describing a pair of videos of the same source but processed by different enhancement means, the multimodal large model tends to ignore elements such as emoticons and borders in the video that are not highly related to the video content, and outputs a more semantically consistent text feature vector for homologous videos with consistent content. Therefore, the similarity between them can be more accurately calculated through the text feature vector, thereby improving the accuracy of similar video retrieval.

[0100] See also Figure 2 , Figure 2 Schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 2 As shown, the user can upload the video to be matched through the terminal device, and the business server searches for similar videos to the acquired video to be matched through the video text database. The video to be matched can be shown as video frame sequence 110, and video frame sequence 110 can include video frame 111, video frame 112 and video frame 113. Taking the video text database including business video data 1 and business video data 2 as an example, business video data 1 can be shown as video frame sequence 120, and video frame sequence 120 can include video frame 121, video frame 122 and video frame 123. Business video data 2 can be shown as video frame sequence 130, and video frame sequence 130 can include video frame 131, video frame 132 and video frame 133. Among them, the terminal device can be the above-mentioned Figure 1 Corresponding to any terminal device in the embodiment, for example, the terminal device 10a, the service server may be the above-mentioned Figure 1 The business server 100 in the corresponding embodiment. The video data to be matched may be user-generated content (UGC), that is, interactive media in the form of short videos, long videos, etc. created by users.

[0101] It can be understood that the video data to be matched can be obtained by the user modifying the business video data 1 by means of confrontation, and the confrontation means may include editing video frames, deleting text, adding mosaics, etc. For example, video frame 111 can be the picture body A1 in video frame 111 obtained by modifying the video title 2 of video frame 121 to video title 1 and adding mosaics to the picture body A2 in video frame 121; video frame 112 can be the picture body B1 in video frame 112 obtained by adding mosaics to the picture body B2 in video frame 122, and adding a texture D; video frame 113 can be the picture body C1 in video frame 113 obtained by adding mosaics to the picture body C2 in video frame 123.

[0102] The business server can perform feature extraction on the video data to be matched to obtain a compressed visual mark vector. The business server can obtain business text information, which can be a text prompt (prompt) used to indicate the content description text corresponding to the extracted video data, for example, "You need to accurately and concisely annotate the input video data to be matched". The business server can perform feature extraction on the business text information to obtain a text prompt feature vector corresponding to the business text information, perform vector splicing on the text prompt feature vector and the compressed visual mark vector to obtain a fused feature vector, align the fused feature vector, and obtain a text feature vector to be matched whose vector length is the alignment length. Among them, the fused feature vector includes the semantics of the content description text used to characterize the video data to be matched, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0103] The business server can perform text similarity calculation on the text feature vector to be matched and the feature vector in the video text feature database. For example, the text feature vector to be matched can be calculated with text feature vector 1, and the text feature vector to be matched can be calculated with text feature vector 2. Among them, text feature vector 1 can be obtained by extracting features from business video data 1, and text feature vector 2 can be obtained by extracting features from business video data 2. The generation method of text feature vector 1 and text feature vector 2 can be the same as the generation method of the text feature vector to be matched, so it can be ensured that the vector length of the text feature vector to be matched is the same as the vector length of the text feature vector.

[0104] The business server can generate a text similarity 1 between a text feature vector 1 and a text feature vector to be matched, and generate a text similarity 2 between a text feature vector 2 and a text feature vector to be matched. For example, if the text similarity 1 is greater than the text similarity threshold, the business server can determine the business video data 1 associated with the text similarity 1 as a text retrieval video set, and determine the similar video retrieval result for the video data to be matched through the text retrieval video set. Since the fused feature vector can only include the semantics of the text describing the content, various deformations of the visual elements in the video frame sequence 110 can be ignored, and the fused feature vector mainly includes the semantics of the plot content of the video data to be matched, so it can be more accurately detected that the video data to be matched is similar to the business video data 1. The embodiment of the present application aims to improve the accuracy of similar video retrieval, and can be applied to a copyright detection system, a video recommendation system, and a video repetition removal system. By improving the accuracy of similar video detection, the embodiment of the present application can effectively resist various audit attack methods, such as adding texture effects, emoticons, lace occlusion, etc. to the video. Even after the video is processed by these attack methods, it can still be well matched to the same source video in the video library, thereby improving the retrieval accuracy and reducing the situation of false detection and missed detection. In a copyright detection system, the embodiment of the present application can more accurately search whether there are similar videos to the input video in the copyright video library, so as to better intercept the upload and recommendation of uncopyrighted videos to prevent users from uploading uncopyrighted content; or search whether there are too many videos that are consistent with the current video in the recent video library uploaded by other users, so as to reduce the upload and recommendation of too much homogeneous content. In a video recommendation system, the embodiment of the present application can reduce the situation where exactly the same video is repeatedly recommended in a short period of time due to the missed detection of similar videos, and reduce the number of recommendations to users for videos that are exactly the same in content in a short period of time, thereby improving the user experience. In a video deduplication system, the embodiment of the present application can more accurately search whether there are homologous videos in the video library for the user's newly uploaded video, improve the review, and reduce the excessive appearance of the same video in the video library.

[0105] See also Figure 3 , Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The data processing method can be executed by a computer device, which can be Figure 1 The service server 100 shown or any terminal device in the terminal device cluster is, for example, the terminal device 10a. The following will take the data processing method executed by a computer device as an example for description. The data processing method may at least include the following steps S101 to S104:

[0106] Step S101, extracting features from the video data to be matched to obtain a compressed visual marker vector;

[0107] Specifically, the computer device may sample the video data to be matched to obtain sampled video frames. For example, if the video length of the video data to be matched is 60 seconds and the frame rate is 24 frames per second, the computer device may sample one video frame per second as a sampled video frame to obtain 60 sampled video frames. The computer device may perform feature extraction on the sampled video frames to obtain a compressed visual tag vector.

[0108] It can be understood that the computer device can extract the visual tag vector to be matched from the video data to be matched, and then perform attention processing on the visual tag vector to be matched based on the compressed query vector to obtain a compressed visual tag vector. The compressed visual tag vector can be a vector generated by performing attention processing on the visual token corresponding to the sampled video frame through a fixed set of query vectors, such as the compressed query vector. The compressed visual tag vector can be used to represent higher-level semantic features in the visual tag.

[0109] Step S102, obtain business text information, perform vector splicing on the business text information and the compressed visual mark vector to obtain a fused feature vector, perform alignment processing on the fused feature vector, and obtain a to-be-matched text feature vector whose vector length is the alignment length; the business text information is a text prompt for indicating the extraction of the content description text corresponding to the video data, the fused feature vector includes the semantics for characterizing the content description text of the to-be-matched video data, and the fused feature vector can be a feature vector for generating the content description text for the to-be-matched video data. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database.

[0110] Specifically, the business server can obtain business text information, which can be a text prompt used to indicate the extraction of content description text corresponding to video data. For example, it can be "You need to accurately and concisely annotate the input video data to be matched." The business text information can be used to instruct and guide the model to generate content description text that conforms to the text prompt.

[0111] The computer device can split the business text information into tokens through a Tokenizer. For example, if the business text information is "Let's do tokenization!", the token sequence corresponding to the business text information can be expressed as [Let,'s,do,token,ization]. Among them, [Let,'s], [do], [token] and [ization] are all tokens. The text splitting method can be word-based, character-based, or subword-based, which is not limited in the embodiments of the present application. The tokens of the business text information may include [CLS], [Let,'s], [do], [token], [ization] and [SEP]. A token in the token sequence refers to the smallest basic unit obtained after the business text information is split. [CLS] can be added to the beginning of the Tokenizer segmentation result, which is a special token used to serve the classification task. [SEP] can be added to the end of the Tokenizer segmentation result. It is a special token used to serve the division of sentence pairs. All tokens in the business text information can be embedded into vectors of the same size to obtain a text prompt feature vector corresponding to the business text information. The text prompt feature vector and the compressed visual tag vector are vector-concatenated to obtain a fused feature vector. The fused feature vector is aligned to obtain a text feature vector to be matched whose vector length is the alignment length. Among them, the fused feature vector includes the semantics of the content description text used to characterize the video data to be matched. The fused feature vector can be a feature vector used to generate the content description text for the video data to be matched. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Since the fused feature vector only needs to contain the semantics of the content description text, the fused feature vector does not need to retain all visual tag-related features, so the length of the fused feature vector can be reduced, reducing the computational cost.

[0112] It can be understood that the video length and video content of the video data to be matched are not fixed, and the fused feature vector generated by the video data to be matched is also not fixed. Directly performing similarity calculation with the business text feature vector of the business video data will require padding due to the inconsistency of the vector lengths, such as padding the business text feature vector and the fused feature vector to the same length. This may lead to reduced computing efficiency, increased video memory consumption, filling of some redundant information affecting the model's capture of real data, reduced generalization, and other issues. Through the above-mentioned alignment processing, the embodiment of the present application can effectively generate a text feature vector to be matched whose vector length is the alignment length by fusing the feature vector, and perform similarity calculation on the text feature vector to be matched and the business text feature vector, which can reduce the computational cost of the similarity calculation.

[0113] Step S103, obtaining a service text feature vector corresponding to the service video data; the service text feature vector includes the semantics of the content description text used to characterize the service video data;

[0114] Specifically, the computer device can obtain the business text feature vector corresponding to the business video data in the video text feature database. The business text feature vector can be obtained by extracting features from the business video data, and the business text feature vector includes the semantics of the content description text used to characterize the business video data. The business text feature vector can be a feature vector used to generate the content description text for the video data to be matched. The business text feature vector can be generated in the same way as the text feature vector to be matched, so it can be ensured that the vector length of the text feature vector to be matched is the same as the vector length of the business text feature vector.

[0115] Step S104, generating text similarity between the business text feature vector and the text feature vector to be matched, and determining the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0116] Specifically, the business server can calculate the text similarity between the feature vector of the text to be matched and the feature vector in the video text feature database to obtain the text similarity corresponding to each business video data, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as the text retrieval video set. The text retrieval video set can be used to determine the similar video retrieval results for the video data to be matched.

[0117] Computer devices can provide users with a more personalized and consistent viewing experience through similar video retrieval results. Recommendation systems can recommend content similar to videos that users have watched before, improving user satisfaction and content discovery. In the video review process, similar video retrieval results can help identify and filter out duplicate or illegal content to ensure the diversity and compliance of platform content. In the copyright detection system, computer devices can use similar video retrieval results to more accurately retrieve whether there are similar videos to the input video in the copyright video library, thereby better intercepting the upload and recommendation of uncopyrighted videos to prevent users from uploading uncopyrighted content.

[0118] Optionally, the computer device can calculate the visual similarity between the video data to be matched and the business video data through visual elements, extract features from the business video data to obtain a business visual feature vector, and extract features from the video data to be matched to obtain a visual feature vector to be matched. The business visual feature vector includes the semantics of the visual elements represented in the business video data, and the visual feature vector to be matched includes the semantics of the visual elements represented in the video data to be matched. The computer device can generate the visual similarity between the business visual feature vector and the visual feature vector to be matched, associate the business video data with the business visual feature vector whose visual similarity is greater than or equal to the video similarity threshold, and determine the visual retrieval video set.

[0119] It can be understood that the computer device uses the text retrieval video set retrieved by the fusion feature vector as a supplement to the visual element similar video retrieval, and the computer device can determine the intersection of the visual retrieval video set and the text retrieval video set as the similar video retrieval result for the video data to be matched, so as to avoid the model from paying too much attention to local features and ignoring the global context information of the video content, resulting in a reduced accuracy of video retrieval, thereby improving the accuracy of similar video retrieval. Or the computer device can determine the union of the visual retrieval video set and the text retrieval video set as the similar video retrieval result for the video data to be matched. Avoid false detection or missed detection caused by adversarial means, and improve the accuracy and anti-disturbance performance of similar video retrieval. Integrating the embodiment of the present application into the similar video retrieval system of visual elements can significantly reduce the decrease in visual feature similarity caused by various attack means, thereby effectively alleviating the missed detection problem caused by the reduced feature similarity in the system during the retrieval process, and improving the accuracy of retrieval, because the problem of missed detection and false detection in the retrieval process is solved, similar videos can be retrieved more comprehensively, thereby ensuring the integrity of similar video retrieval. The embodiment of the present application extracts features from the video data to be matched, obtains a compressed visual marker vector, obtains business text information, and performs vector splicing on the business text information and the compressed visual marker vector to obtain a fused feature vector, wherein the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics of the content description text for characterizing the video data to be matched. The fused feature vector can ignore visual elements in the video data to be matched that are irrelevant to the video content, such as lace, texture effects, or occluded emoticons added by adversarial means, which can improve the robustness and anti-disturbance of the model. By aligning the fused feature vector, a text feature vector to be matched whose vector length is the alignment length is obtained. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. Obtain the business text feature vector corresponding to the business video data, wherein the business text feature vector includes the semantics of the content description text for characterizing the business video data. By calculating the text similarity between the business text feature vector and the text feature vector to be matched, the text similarity between the business text feature vector and the text feature vector to be matched is obtained. The text similarity between the business text feature vector and the text feature vector to be matched with the same vector length is calculated, which can avoid the problem of filling in due to the inconsistency of the vector length, further reducing the calculation cost and improving the efficiency of similar video retrieval. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set. The similar video retrieval results for the video data to be matched are determined by the text retrieval video set, which can avoid the problem of missed detection and false detection caused by misleading similarity caused by visual elements, and improve the accuracy and completeness of similar video retrieval.

[0120] See also Figure 4 , Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 The data processing method can be executed by a computer device, which can be Figure 1 The service server 100 or any terminal device in the terminal device cluster is shown, for example, the terminal device 10a. The following will take the data processing method executed by a computer device as an example for description. The data processing method may at least include the following steps S201 to S207:

[0121] Step S201, input the video data to be matched into the target video retrieval model; the target video retrieval model includes a multimodal text description layer, and the multimodal text description layer includes a preprocessing layer, a visual coding layer and a visual abstraction layer; in the preprocessing layer, based on the video duration and video frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames, and the T sampled video frames are respectively image segmented to obtain S unit sampled images; T is a positive integer determined based on the video duration and the video frame rate, and S is a positive integer greater than T; in the visual coding layer, feature extraction is performed on the S unit sampled images to obtain S unit visual tag vectors, and based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual tag vectors are remapped to the feature channels to obtain S visual tag vectors to be matched; the rearrangement factor is determined based on the number S of unit visual tag vectors, and the vector length of the unit visual tag vector is greater than the vector length of the visual tag vector to be matched;

[0122] Specifically, the computer device can input the to-be-matched video data into the target video retrieval model, and then refer to Figure 5 , Figure 5 is a schematic diagram of a model structure of a target video retrieval model proposed in an embodiment of the present application, such as Figure 5 As shown in FIG. 1 , the target video retrieval model includes a multimodal text description layer, a feature alignment layer, and a recall matching layer. The multimodal text description layer includes a preprocessing layer, a visual encoding layer, a visual abstraction layer, and a text description layer.

[0123] The computer device can input the video data to be matched into the preprocessing layer of the target video retrieval model. In the preprocessing layer, the computer device can uniformly sample the video data to be matched based on the video length and video frame rate of the video data to be matched to obtain T sampled video frames. For example, if the video length of the video data to be matched is 60 seconds and the frame rate is 24 frames per second, the computer device can sample one video frame per second as a sampled video frame to obtain 60 sampled video frames. When the video length is 30 seconds and the frame rate is 50 per second, the computer device can sample two video frames per second as sampled video frames to obtain 60 sampled video frames. Wherein, T can be a positive integer jointly determined based on the video length, the video frame rate, and the context window length of the target video retrieval model. Wherein, the context window length refers to the maximum data length that the target video retrieval model can input when processing input information.

[0124] The computer device can perform uniform preprocessing on the sampled video frames, and the preprocessing can include mean normalization and bicubic interpolation. Among them, the normalization process can adjust the pixel value of the image data to a specific range, usually [0, 1] or [-1, 1]. Bicubic interpolation can scale the size of the image to a specific range determined by the context window length of the target video retrieval model, for example, 448×448 pixels. The computer device can perform image segmentation on the preprocessed T sampled video frames respectively, and divide the preprocessed T sampled video frames into a number of fixed-size unit sampled images, each of which can be flattened and mapped to a vector of a fixed dimension (for example, 14×14 pixels), and add position encoding to retain the relative position information of the unit sampled image, thereby obtaining S unit sampled images. S is a positive integer greater than T.

[0125] In the visual encoding layer, the computer device can perform feature extraction on the S unit sampled images to obtain S unit visual tag vectors. The computer device can perform pixel shuffle on the S unit visual tag vectors, rearrange the elements of the unit visual tag vectors, and redistribute the high-dimensional spatial channel information to the feature channel dimension. For example, the shape of the S unit sampled images can be a tensor of B×T×H×W×C. Among them, T×H×W is S, B (Batchsize, which refers to the number of data processed by the model at one time), H (Height, the height of the image) and W (Width, the width of the image) are used to represent the spatial channel, and C (Channels, the number of feature channels) is used to represent the feature channel. The visual encoding layer can be trained by the InternViT-300M (ViT (Vision Transformer, Visual Transformation Encoder)) model with a parameter count of 300M.

[0126] Taking the rearrangement factor r as an example, the rearrangement factor is determined based on the number S of unit visual marker vectors, where r is a positive integer less than S. Specifically, the computer device may rearrange the S unit visual marker vectors according to the rearrangement factor r, and remap the vector elements representing the spatial channels to the feature channels. Specifically, the computer device may rearrange the unit sampled image according to the rearrangement factor r, reduce the width and height of the image to 1 / r of the original, and then rearrange the reduced image blocks into new feature channels, so that the number of feature channels becomes For example, the shape of the S visual marker vectors to be matched can be B×T×(H / r)×(W / r)×(C×r 2 ), where the vector length of the unit visual marker vector is greater than the vector length of the visual marker vector to be matched. For example, if the shuffle factor is 2 and a video A has 10 frames, each frame will be resized to 448×448, and each frame will be divided into 1024 visual tokens. Each frame can be processed by pixel shuffle to obtain 256 visual tokens. Then, the video A can include 10×256 visual tokens.

[0127] It can be understood that Pixel Shuffle can reduce the dimension of the vector representing the image height and image width in the unit sampling image by rearranging the pixels of the image, thereby reducing the number of visual tags.

[0128] Step S202, in the visual abstraction layer, obtain a compressed query vector, concatenate the visual tag vector to be matched and the compressed query vector to obtain a visual tag vector sequence; the compressed query vector is a learnable parameter; the compressed query vector and the visual tag vector sequence are assigned to N unit processing spaces, and cross-attention processing is performed on the compressed query vector and the visual tag vector sequence in each unit processing space to obtain N unit attention result vectors, and a compressed visual tag vector is generated based on the attention weights corresponding to the N unit attention result vectors and the N unit processing spaces respectively; N is a positive integer.

[0129] Specifically, the computer device may input S to-be-matched visual tag vectors into the visual abstraction layer, obtain a compressed query vector in the visual abstraction layer, and perform vector concatenation of the S to-be-matched visual tag vectors and the compressed query vector to obtain a visual tag vector sequence. The compressed query vector is a learnable parameter obtained by training, and the vector length of the compressed query vector may be R, where R is a positive integer less than S. The compressed query vector may be a fixed query vector, which is used to extract higher-level semantic features from visual tags of multiple frames of images by learning a set of fixed query vectors.

[0130] The computer device can perform attention processing on the S to-be-matched visual tag vectors based on the compressed query vector. For example, the computer device can perform multi-head attention processing on the compressed query vector and the visual tag vector sequence, and distribute the compressed query vector and the visual tag vector sequence to N unit processing spaces. , in each unit processing space In the process, the compressed query vector and the visual tag vector sequence are cross-attended to obtain N unit attention result vectors. Multi-head attention processing can include N unit processing spaces , each unit processes space Can include query parameter matrix , key parameter matrix Parameter matrix with values , the learnable parameters in different unit processing spaces are independent of each other, and each unit processing space Each unit can perform attention calculation independently, and each unit processes space The compressed query vector and the query parameter matrix Perform a dot multiplication operation to obtain the query vector Q, and combine the visual tag vector sequence with the key parameter matrix Perform a dot multiplication operation to obtain the key vector K, and combine the visual tag vector sequence with the value parameter The matrix is ​​multiplied to obtain the value vector V. Among them, the query parameter matrix , key parameter matrix Parameter matrix with values They are all matrices composed of learnable parameters in each unit processing space.

[0131] The computing device can distribute the compressed query vector and the sequence of visual tag vectors to N unit processing spaces , in each unit processing space In the cross-attention process, the compressed query vector and the visual tag vector sequence are processed to obtain N unit attention result vectors , based on N unit attention result vectors and N units of processing space The corresponding attention weights are used to generate compressed visual tag vectors. The process of multi-head attention processing can be shown as formula (1):

[0132] Formula (1)

[0133] in, To compress the visual label vector, Functions can be used to process spaces with different units Output unit attention result vector Concatenated together, we get a higher dimensional vector representation. is the multi-head attention weight matrix, including the attention weights corresponding to the N unit processing spaces, Used to process each unit space The output unit attention result vector is weighted and the attention vector output by the multi-head attention layer is obtained.

[0134] It can be understood that the attention mechanism can be divided into multiple "heads" through multi-head attention calculation. Each head can learn the representation of the input data in different subspaces, which enables the model to capture richer feature information in different subspaces and allows the model to process information in multiple subspaces in parallel, thereby improving the calculation efficiency of the model.

[0135] Among them, N units process space Including a target unit processing space; the process of performing cross-attention processing in the target unit processing space can be: in the target unit processing space, performing a point multiplication operation on the compressed query vector and the query parameter matrix to obtain a query vector, performing a point multiplication operation on the visual tag vector sequence and the key parameter matrix to obtain a key vector, performing a point multiplication operation on the visual tag vector sequence and the value parameter matrix to obtain a value vector; generating an attention score vector based on the query vector and the key vector, performing dimensionality reduction processing on the attention score vector based on the adjacent length of the key vector, normalizing the reduced dimensionality attention score vector to obtain an attention weight, and performing a point multiplication operation on the attention weight and the value vector to obtain a unit attention result vector corresponding to the target unit processing space.

[0136] Specifically, the computer device can query the matrix The result of the transpose operation with the key matrix Perform a dot multiplication operation to obtain the attention score matrix. Based on the dimension value of the key matrix K The square root of Perform dimensionality reduction on the attention score vector, normalize the reduced dimensionality attention score vector to obtain the attention weight, perform dot multiplication on the attention weight and the value vector to obtain the unit attention result vector corresponding to the target unit processing space , the process can be shown as formula (2):

[0137] Formula (2)

[0138] in, The function is normalized. The computer device can be based on N unit attention result vectors Generate Compressed Visual Markup Vector .

[0139] Since the fused feature vector only needs to contain the semantics of the text representing the content description, the fused feature vector does not need to retain all visual tag related features, so the length of the fused feature vector can be reduced and the computational cost can be reduced. The embodiment of the present application splices the visual tag vector to be matched and the compressed query vector together, which can better capture multimodal information, help the target video retrieval model understand and abstract image information at different levels and granularities, and improve the processing performance of the target video retrieval model. Through the visual abstraction layer, the S visual tag vectors to be matched are compressed to obtain a compressed visual tag vector. The compressed visual tag vector is obtained by multi-head attention processing of the compressed query vector and the visual tag vector sequence. The vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector, and the vector length is R. The number of visual tags can be effectively reduced, and the video memory load and computing resource usage can be significantly reduced.

[0140] The computer can compress the visual label vector through a multi-layer perceptron (MLP). After performing linear transformation and nonlinear activation function processing, MLP can first perform layer normalization on the compressed visual tag vector output by the visual layer, first through the linear layer The compressed visual marker vector is processed linearly, then processed by the SwiGLU activation function, and finally passed through the linear layer Linear processing is performed to obtain the compressed visual tag vector after MLP processing. The process can be shown as formula (3):

[0141] Formula (3)

[0142] Among them, σ function represents the SwiGLU activation function, and Represents a linear layer operation in a multilayer perceptron.

[0143] The embodiment of the present application proposes a method of connecting a visual model InternViT-300M and a language model Phi-3 using a multi-layer perceptron. The method converts visual markers into feature representations suitable for language model processing, ensuring that visual information can be effectively used by the language model, thereby realizing multimodal information processing. Through the above method, the present invention realizes the efficient fusion of visual information and language model, and improves the performance of multimodal information processing.

[0144] Step S203, inputting the compressed visual tag vector into a target video retrieval model; the target video retrieval model includes a multimodal text description layer and a feature alignment processing layer, and the multimodal text description layer includes a text description layer; in the text description layer, obtaining business text information, performing feature extraction on the business text information, obtaining a text prompt feature vector, and performing vector splicing on the text prompt feature vector and the compressed visual tag vector to obtain a fused feature vector;

[0145] Specifically, the computer device may input the compressed visual tag vector into the text description layer of the multimodal text description layer, obtain the business text information in the text description layer, perform feature extraction on the business text information, and obtain a text prompt feature vector. The business text information may be a text prompt for indicating the extraction of content description text corresponding to the video data, for example, "You need to accurately and concisely annotate the input video data to be matched", and the business text information may be used to instruct and guide the model to generate content description text that conforms to the text prompt.

[0146] The computer device can split the business text information into tokens through a Tokenizer. For example, if the business text information is "Let's do tokenization!", the token sequence corresponding to the business text information can be expressed as [Let,'s,do,token,ization]. Among them, [Let,'s], [do], [token] and [ization] are all tokens. The text splitting method can be word-based, character-based, or subword-based, which is not limited in the embodiments of the present application. The tokens of the business text information may include [CLS], [Let,'s], [do], [token], [ization] and [SEP]. A token in the token sequence refers to the smallest basic unit obtained after the business text information is split. [CLS] can be added to the beginning of the Tokenizer segmentation result, which is a special token used to serve the classification task. [SEP] can be added to the end of the Tokenizer segmentation result. It is a special token used to divide sentence pairs. All tokens in the business text information can be embedded into vectors of the same size to obtain the text prompt feature vector corresponding to the business text information.

[0147] It can be understood that the target video retrieval model can obtain business text information and context information of business text information. If the context information is empty, that is, the model does not carry information of previous rounds of conversations, an empty list will be initialized and an image tag will be generated, which may include an image start tag, an image context tag, and an image end tag. Then the business text information is added after the image tag. If the context information is not empty, the historical conversation will be merged as context information and business text information. Then the merged context information and business text information are added after the image tag. This enables the target video retrieval model to have multi-round conversation capabilities, capture the long-distance dependency between context information and business text information, and generate content description text that is more in line with text prompts.

[0148] The computer device may perform vector concatenation on the text prompt feature vector and the compressed visual marker vector to obtain a fused feature vector. The fused feature vector includes the semantics of the content description text for characterizing the video data to be matched, and the fused feature vector may be a feature vector for generating the content description text for the video data to be matched. The process of cross attention processing may refer to the specific description of the above step S202.

[0149] The text description layer can be a large language model, which can be a Phi-3-mini-128k-instruct model (an open source lightweight model), which is designed to generate high-quality text and perform various natural language processing tasks. The multimodal large language model can more effectively resist interference and maintain the accuracy and stability of the description when generating fusion feature frames. When describing a pair of videos of the same source but processed by different enhancement methods, the multimodal large model tends to ignore elements in the video that are not highly relevant to the video content, such as emoticons and borders, and outputs more semantically consistent content description text for homologous videos with consistent content, thereby improving the accuracy of similar video retrieval.

[0150] Step S204, in the feature alignment processing layer, the fused feature vector is aligned to obtain a feature vector of the text to be matched whose vector length is the alignment length.

[0151] Specifically, the computer device can input the fused feature vector into the feature alignment processing layer, and in the feature alignment processing layer, the fused feature vector is aligned to obtain the feature vector of the text to be matched whose vector length is the alignment length. The feature alignment processing layer can use three different alignment methods for processing. The feature alignment processing layer can use the first alignment method to perform alignment processing by training the Long Short-Term Memory Autoencoder (LSTMAutoencoder); the feature alignment processing layer can use the second alignment method to perform alignment processing by word frequency and inverse document frequency; the feature alignment processing layer can use the third alignment method to determine the last unit prediction vector in the fused feature vector as the feature vector of the text to be matched whose vector length is the alignment length.

[0152] Optionally, if the feature alignment processing layer adopts the first alignment method, that is, the feature alignment processing layer can be a long short-term memory network autoencoder, then the computer device can perform alignment processing through the long short-term memory network autoencoder, and the process may include: inputting the fused feature vector into the feature alignment processing layer, the feature alignment processing layer includes an automatic encoding layer and a bottleneck reconstruction layer; the bottleneck reconstruction layer is obtained based on alignment length training, and the vector length output by the bottleneck reconstruction layer is fixed to the alignment length; in the automatic encoding layer, encoding processing is performed on the fused feature vector to obtain an encoded feature vector, and based on the time step, the encoded feature vector is downsampled to obtain a downsampled feature vector; in the bottleneck reconstruction layer, based on the dependency of the downsampled feature vector in each time step, a text feature vector to be matched with a vector length of the alignment length is generated.

[0153] Specifically, the computer device can input the fused feature vector into the feature alignment processing layer, the feature alignment processing layer includes an automatic encoding layer and a bottleneck reconstruction layer, the feature alignment processing layer can be trained by LSTMAutoencoder, the feature alignment processing layer includes an automatic encoding layer and a bottleneck reconstruction layer, and the vector length output by the bottleneck reconstruction layer is fixed to the alignment length.

[0154] LSTMAutoencoder includes a pair of symmetrical encoders and decoders and a reconstruction layer. The goal is to reconstruct the original input as much as possible after the input data passes through the encoder and decoder. By minimizing the error between the input and the reconstructed output, LSTMAutoencoder can learn the feature representation of the input data. Among them, the autoencoder layer is trained by the encoding layer of the long short-term memory network autoencoder, and the bottleneck reconstruction layer is trained by the initial reconstruction layer of the long short-term memory network autoencoder. The vector length of the vector output by the initial reconstruction layer and the bottleneck reconstruction layer is fixed to the alignment length.

[0155] In the auto-encoding layer, the computer device encodes the fused feature vector to obtain an encoded feature vector, and downsamples the encoded feature vector in a number of time steps to obtain a downsampled feature vector. In the bottleneck reconstruction layer, based on the dependency of the downsampled feature vector in each time step, a feature vector of the text to be matched with a vector length of the alignment length is generated. That is, the computer device can select the feature output of the LSTMAutoencoder reconstruction layer as the feature of the unified dimension after dimensionality reduction, that is, the feature vector of the text to be matched with a vector length of the alignment length.

[0156] Optionally, if the feature alignment processing layer adopts the second alignment method, the computer device can perform alignment processing through word frequency and inverse document frequency, and the process can be: inputting the fused feature vector into the feature alignment processing layer, the feature alignment processing layer includes a word frequency processing layer and a weighted alignment layer; in the word frequency processing layer, based on the fused feature vector, generating the predicted probability corresponding to each word in the vocabulary, generating the content description text of the video data to be matched based on the predicted probability, and segmenting the content description text based on the alignment length to obtain F text segments; the alignment length is F, and F is a positive integer; obtaining the word frequencies corresponding to the F text segments in the content description text, and the inverse document frequencies corresponding to the F text segments in the content description text, and generating F text weights based on the F word frequencies and the F inverse document frequencies; in the weighted alignment layer, performing dot multiplication operations on the F text weights and the fused feature vector respectively to obtain F weighted feature vectors, and generating a feature vector of the text to be matched whose vector length is the alignment length based on the F weighted feature vectors.

[0157] Specifically, the computer device can input the fused feature vector into the word frequency processing layer in the feature alignment processing layer. In the word frequency processing layer, based on the fused feature vector, the prediction probability corresponding to each word in the vocabulary is generated, and the content description text of the video data to be matched is generated based on the prediction probability.

[0158] The computer device can segment the content description text based on the alignment length to obtain F text segments (terms). The segmentation can be performed by rule matching or probability matching. For example, the text segmentation of the content description text C is performed by the text set A (the text set A can be a predefined text library) in the probability matching method. For the text segmentation of the content description text C, there will be a segmentation scheme S. The task of text segmentation is to find the scheme in the segmentation scheme S that has the largest probability P (S|C) of the segmented words appearing in the text set A. The probability of P (S|C) can be calculated according to the Bayesian formula as shown in formula (4):

[0159] Formula (4)

[0160] Among them, P(C) is the probability of the content description text C appearing in the text set A, and P(C|S) is the probability of the word segmentation scheme restoring the content description text C. The word segmentation scheme S divides the content description text set into M text segments (text segments , text snippets , ..., text snippet ), then P(S) = , For text snippets The ratio of the number of times a word appears in text set A to the total number of words in text set A.

[0161] The computer device can obtain the term frequencies (TermFrequency) corresponding to the F text segments in the content description text. The term frequency TF of the target text segment i is calculated as shown in formula (5):

[0162] TF(i,C)= Formula (5)

[0163] The target text segment i can be any one of the F text segments, TF is the word frequency, C is the content description text, is the number of times the target text segment i appears in the content description text.

[0164] The computer device can obtain the inverse document frequencies corresponding to the F text segments in the content description text. The calculation of the inverse document frequency of the target text segment i is shown in formula (6):

[0165] IDF(i,D)=log Formula (6)

[0166] Wherein, D represents a document set corresponding to the content description text, and the document set may also include only one content description text C, which is not limited in the embodiment of the present application. Represents the total number of documents in the document collection. Represents the number of documents containing the target text segment i.

[0167] The computer device can generate F text weights based on F word frequencies and F inverse document frequencies. The calculation of the text weights is shown in formula (7):

[0168] TF-IDF(i,C,D)=TF(i,C)×IDF(i,D) formula (7)

[0169] In the weighted alignment layer, the computer device can perform point multiplication operations on the F text weights and the fused feature vectors to obtain F weighted feature vectors, determine the sum of the F weighted feature vectors as the text weighted feature vector, and divide each vector element of the text weighted feature vector by the sum of the text weights corresponding to the F weighted feature vectors to obtain a to-be-matched text feature vector with a vector length of the alignment length. The to-be-matched text feature vector with a vector length of the alignment length is generated based on the F weighted feature vectors, and the calculation is shown in formula (8):

[0170] Formula (8)

[0171] in, To fuse the feature vectors, WeightedAverage is the feature vector of the text to be matched whose vector length is the alignment length. is the weighted feature vector.

[0172] Optionally, if the feature alignment processing layer adopts the third alignment method, the computer device can obtain the last unit prediction vector in the fused feature vector through the alignment processing layer, and determine the last unit prediction vector in the fused feature vector as the text feature vector to be matched whose vector length is the alignment length. The fused feature vector can be used to generate content description text for the video data to be matched, the fused feature vector includes G unit prediction vectors, one unit prediction vector can correspond to a text segment or word item of the content description text, the vector lengths of the G unit prediction vectors are all the alignment length, the last unit prediction vector in the G unit prediction vectors includes the global semantics obtained by vector concatenation of the business text information and the compressed visual mark vector, and in the feature alignment processing layer, the computer device can determine the last unit prediction vector in the G unit prediction vectors as the text feature vector to be matched whose vector length is the alignment length.

[0173] It can be understood that the large language model (LLM) generates video descriptions word by word. When generating each word, the model will infer based on the previously generated content and the input image information. Therefore, when the model generates the last word, the last unit prediction vector can capture the semantic information of all the previously generated texts. Therefore, the last unit prediction vector among the G unit prediction vectors can be determined as the feature vector of the text to be matched with a vector length of the alignment length, and input into the feature vector recall and precise matching module for similar video retrieval. That is, the alignment length is the vector length of the last unit prediction vector.

[0174] It can be understood that the video length and video content of the video data to be matched are not fixed, and the fused feature vector generated by the video data to be matched is also not fixed. Directly performing similarity calculation with the business text feature vector of the business video data will require padding due to the inconsistency of the vector lengths, such as padding the business text feature vector and the fused feature vector to the same length. This may lead to reduced computing efficiency, increased video memory consumption, filling of some redundant information affecting the model's capture of real data, reduced generalization, and other issues. Through the above-mentioned alignment processing, the embodiment of the present application can effectively generate a text feature vector to be matched whose vector length is the alignment length by fusing the feature vector, and perform similarity calculation on the text feature vector to be matched and the business text feature vector, which can reduce the computational cost of the similarity calculation.

[0175] Step S205, obtaining a service text feature vector corresponding to the service video data; the service text feature vector includes the semantics of the content description text used to characterize the service video data;

[0176] Specifically, the computer device may obtain a business text feature vector corresponding to the business video data, wherein the business text feature vector includes the semantics of the content description text used to characterize the business video data, and the business text feature vector may be a feature vector used to generate the content description text for the business video data. The business text feature vector is generated in the same manner as the feature vector of the text to be matched, so the vector length of the feature vector of the text to be matched is guaranteed to be the same as the vector length of the text feature vector.

[0177] The computer device can complete the retrieval of the video text feature database through the IndexIVFFlat index in the FAISS library. Among them, FAISS (Facebook AI Similarity Search) is an efficient similarity search library. The embodiment of the present application can use the IndexIVFFlat index type in the FAISS library to initialize the business text feature vector of each business video data in the video text feature database, and add the business text feature vector to the index.

[0178] The video text feature database may include cluster centers generated based on the IndexIVFFlat index, and each business text feature vector may be assigned to a different cluster center. When performing a recall operation, the computer device may use the text feature vector to be matched as a query vector. The FAISS library will first find the closest cluster center based on the query vector. Secondly, a precise search will be performed within the closest cluster center to find the feature vector that is most similar to the query vector. This step ensures the efficiency and accuracy of the recall, because the search is only performed within the most relevant cluster center, reducing the amount of calculation.

[0179] Step S206, inputting the business text feature vector and the to-be-matched text feature vector into the target video retrieval model; the target video retrieval model includes a recall matching layer; based on each vector element value in the business text feature vector, generating a first parameter of the business text feature vector, based on each vector element value in the to-be-matched text feature vector, generating a second parameter of the to-be-matched text feature vector; performing a dot multiplication operation on the business text feature vector and the to-be-matched text feature vector to obtain a matching parameter, and generating a text similarity between the business text feature vector and the to-be-matched text feature vector based on the matching parameter, the first parameter and the second parameter.

[0180] Specifically, the computer device may input the business text feature vector and the to-be-matched text feature vector into the recall matching layer in the target video retrieval model. The computer device may use cosine similarity, Jaccard Similarity Coefficient, Hamming Distance, etc. to determine the similarity between the business text feature vector and the to-be-matched text feature vector. Taking cosine similarity as an example, the computer device generates a first parameter of the business text feature vector based on each vector element value in the business text feature vector, and generates a second parameter of the to-be-matched text feature vector based on each vector element value in the to-be-matched text feature vector. Wherein, the first parameter and the second parameter are both Euclidean norms, the first parameter may be the modulus of the business text feature vector, and the second parameter may be the modulus of the to-be-matched text feature vector. The computer device may perform a dot multiplication operation on the business text feature vector and the to-be-matched text feature vector to obtain a matching parameter, and generate the text similarity between the business text feature vector and the to-be-matched text feature vector based on the matching parameter, the first parameter, and the second parameter. The process may be shown in formula (9):

[0181] cosine(A,B)= Formula (9)

[0182] Among them, cosine(A,B) is the text similarity, A can represent the business text feature vector, and B can represent the feature vector of the text to be matched. is the first parameter, is the second parameter.

[0183] Step S207, determining the business video data associated with the business text feature vectors whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0184] Specifically, the computer device may determine the business video data associated with the business text feature vectors whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set, wherein the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0185] The embodiment of the present application can sample the video data to be matched based on the context window length of the target video retrieval model to obtain a unit sampled image, perform feature extraction on the sampled unit sampled image, obtain a unit visual marker vector, and reduce the vector dimension representing the image height and image width in the unit sampled image by rearranging the pixels of the image, thereby reducing the number of visual markers. By learning a set of fixed compressed query vectors for multi-head attention processing, splicing the visual marker vector to be matched and the compressed query vector together, multimodal information can be better captured, helping the target video retrieval model to understand and abstract image information at different levels and different granularities, and improving the processing performance of the target video retrieval model. Through the visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector, which is obtained by multi-head attention processing of the compressed query vector and the visual marker vector sequence, and the vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector. The number of visual markers can be effectively reduced, and the video memory load and computing resource occupancy can be significantly reduced.

[0186] The business text information is obtained, and the business text information and the compressed visual mark vector are vector-concatenated to obtain a fused feature vector, wherein the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics of the content description text for characterizing the video data to be matched. The fused feature vector can ignore visual elements in the video data to be matched that are irrelevant to the video content, such as lace, texture effects, or occluded emoticons added by adversarial means, etc., which can improve the robustness and anti-disturbance of the model. Since the fused feature vector only needs to retain the semantics of the content description text, it can effectively reduce the number of visual marks to be processed, reduce the computational cost, and improve the efficiency of similar video retrieval. By aligning the fused feature vector, a text feature vector to be matched with a vector length of the alignment length is obtained. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. The business text feature vector corresponding to the business video data is obtained, wherein the business text feature vector includes the semantics of the content description text for characterizing the business video data. By calculating the text similarity between the business text feature vector and the text feature vector to be matched, the text similarity between the business text feature vector and the text feature vector to be matched is obtained. The text similarity between the business text feature vector and the text feature vector to be matched with the same vector length is calculated, which can avoid the problem of filling in due to the inconsistency of the vector length, further reducing the calculation cost and improving the efficiency of similar video retrieval. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set. The similar video retrieval results for the video data to be matched are determined by the text retrieval video set, which can avoid the problem of missed detection and false detection caused by misleading similarity caused by visual elements, and improve the accuracy and completeness of similar video retrieval.

[0187] See also Figure 6 , Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 The data processing method can be executed by a computer device, which can be Figure 1 The service server 100 shown or any terminal device in the terminal device cluster is, for example, the terminal device 10a. The following will take the data processing method executed by a computer device as an example for description. The data processing method may at least include the following steps S301-S311:

[0188] Step S301, data preprocessing;

[0189] Specifically, Figure 6As shown, the processing flow of the business video data is represented by a dotted line, and the processing flow of the video data to be matched is represented by a solid line. The computer device can perform data preprocessing on the business video data and the video data to be matched. The data preprocessing may include sampling the video data based on the video duration and video frame rate of the video data to obtain sampled video frames, performing image segmentation on the sampled video frames, and obtaining a first sampled image corresponding to the business video data and a second sampled image corresponding to the video data to be matched. The process of data preprocessing can be referred to above Figure 4 The specific description of step S201 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0190] Step S302, visual feature extraction;

[0191] Specifically, the computer device may perform feature extraction on the first sampled image to obtain a business visual feature vector corresponding to the business video data, and perform feature extraction on the second sampled image to obtain a to-be-matched visual feature vector corresponding to the to-be-matched video data. The business visual feature vector may be a global feature of a visual tag of the business video data, the to-be-matched visual feature vector may be a global feature of a visual tag of the to-be-matched video data, and the business visual feature vector and the to-be-matched visual feature vector may be obtained by using aggregation methods such as average pooling, maximum pooling, and a temporal shift module (TSM).

[0192] Step S303, visual recall matching;

[0193] Specifically, the computer device can establish a first index database based on the business visual feature vector, and establish a video visual feature database between the business video data and the business visual feature vector through the first index database. Among them, the first index database can be an IndexIVFFlat index, the video visual feature database can be a FAISS library, and the video visual feature database includes cluster centers generated based on the IndexIVFFlat index, and each business text feature vector can be assigned to a different cluster center. The computer device can use the visual feature vector to be matched as a query vector. The FAISS library can find the closest cluster center according to the query vector. Secondly, a precise search will be performed in the closest cluster center to find the feature vector most similar to the query vector. By using the inverted file index (IndexIVFFlat) in FAISS and the search method of the index, a group of possibly similar business visual feature vectors can be quickly retrieved from the video visual feature database and determined as the recalled business visual feature vector to narrow the search scope.

[0194] Step S304, visual similarity calculation;

[0195] Specifically, the computer device can perform visual similarity calculation on the visual feature vector to be matched and the recalled business visual feature vector to generate the visual similarity between the business visual feature vector and the visual feature vector to be matched. The visual similarity calculation can be determined by using cosine similarity, Jaccard similarity coefficient, Hamming distance, etc., which is not limited in the present embodiment of the application. For details on visual similarity calculation, please refer to the above Figure 4 The specific description of step S206 of the corresponding embodiment will not be repeated here in this embodiment of the present application.

[0196] Step S305, generating a visual retrieval video set;

[0197] Specifically, the computer device may determine a visual retrieval video set by associating the business video data with the business visual feature vectors whose visual similarity is greater than or equal to the video similarity threshold, wherein the visual retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0198] Step S306, text description processing;

[0199] Specifically, the computer device can extract features from the business video data to obtain a business visual tag vector, and the computer device can extract features from the video data to be matched to obtain a compressed visual tag vector. The business visual tag vector can be a vector compressed and generated by the visual tag corresponding to the business video data, and the compressed visual tag vector can be a vector compressed and generated by the visual tag corresponding to the video data to be matched. For the content of generating the business visual tag vector and the compressed visual tag vector, please refer to the above Figure 4 The specific description of step S201 to step S202 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0200] The computer device can obtain business text information, which can be a text prompt for indicating the extraction of content description text corresponding to the video data, for example, "You need to accurately and concisely annotate the input video data to be matched", and the business text information can be used to instruct and guide the model to generate content description text that conforms to the text prompt. The computer device can perform feature extraction on the business text information to obtain a text prompt feature vector corresponding to the business text information, and determine the alignment length based on the vector length of the compressed visual marker vector and the vector length of the business visual marker vector. For example, the vector length of the business visual marker vector can be determined as the alignment length, or the median or average of the vector length of the business visual marker vector can be determined as the alignment length, etc. The embodiments of the present application are not limited here.

[0201] The computer device can perform vector concatenation on the text prompt feature vector and the compressed visual mark vector to obtain a fused feature vector, and perform vector concatenation on the text prompt feature vector and the business visual mark vector to obtain a business feature vector. For the content of generating the fused feature vector and the business feature vector, please refer to the above Figure 4 The specific description of step S203 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0202] Step S307, alignment processing;

[0203] Specifically, the computer device performs alignment processing on the fused feature vector to obtain a text feature vector to be matched whose vector length is the alignment length. The fused feature vector includes the semantics of the content description text for characterizing the video data to be matched, and the fused feature vector can be a feature vector for generating the content description text for the video data to be matched. The computer device can perform alignment processing on the business feature vector to obtain a business text feature vector whose vector length is the alignment length. The business text feature vector includes the semantics of the content description text for characterizing the business video data, and the business text feature vector can be a feature vector for generating the content description text for the business video data. For details about the alignment processing, please refer to the above Figure 4 The specific description of step S204 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0204] Step S308, text recall matching;

[0205] Specifically, the computer device can establish a second index database based on the business text feature vector, and establish a video text feature database between the business video data and the business text feature vector through the second index database. Among them, the second index database can be an IndexIVFFlat index, the video text feature database can be a FAISS library, and the video visual feature database includes cluster centers generated based on the IndexIVFFlat index, and each business text feature vector can be assigned to a different cluster center. The computer device can use the text feature vector to be matched as a query vector. The FAISS library can find the closest cluster center based on the query vector. Secondly, an accurate search will be performed in the closest cluster center to find the feature vector most similar to the query vector.

[0206] Step S309, text similarity calculation;

[0207] Specifically, the computer device can perform visual similarity calculation on the feature vector of the text to be matched and the feature vector of the recalled business text to generate text similarity between the feature vector of the business text and the feature vector of the text to be matched. The text similarity calculation can be determined by using cosine similarity, Jaccard similarity coefficient, Hamming distance, etc., which is not limited in the embodiment of the present application. For details on visual similarity calculation, please refer to the above Figure 4 The specific description of step S206 of the corresponding embodiment will not be repeated here in this embodiment of the present application.

[0208] Step S310, generating a text retrieval video set;

[0209] Specifically, the computer device may associate the business video data with the business text feature vectors whose text similarity is greater than or equal to the video similarity threshold to determine the text retrieval video set, wherein the text retrieval video set is used to determine the similar video retrieval result for the video data to be matched.

[0210] Step S311, determining similar video retrieval results.

[0211] Specifically, the computer device can determine the intersection of the visual retrieval video set and the text retrieval video set as the similar video retrieval result for the video data to be matched, so as to avoid the model focusing too much on local features and ignoring the global context information of the video content, resulting in a decrease in the accuracy of video retrieval, and improve the accuracy of similar video retrieval. Avoid false detection (determining different videos as similar videos) caused by adversarial means (such as cutting the video into multiple segments, adding lace, texture effects, or blocking emoticons, etc.).

[0212] Alternatively, the computer device may determine the union of the visual retrieval video set and the text retrieval video set as the similar video retrieval result for the video data to be matched, thereby avoiding missed detection (determining similar videos as different videos) due to adversarial means and improving the accuracy and anti-disturbance performance of similar video retrieval.

[0213] The embodiment of the present application can be used as a plug-in for a system that uses visual elements to retrieve similar videos. By generating a fused feature vector containing the semantics of the content description text of the video data to be matched, visual elements in the video data to be matched that are irrelevant to the video content, such as lace, texture effects, or occluded emoticons added by adversarial means, can be ignored, and the robustness and anti-disturbance of the model can be improved. The text retrieval video set retrieved by the fused feature vector is used as a supplement to the visual element similar video retrieval, and the computer device can determine the intersection of the visual retrieval video set and the text retrieval video set as the similar video retrieval result for the video data to be matched, avoiding the model from paying too much attention to local features and ignoring the global context information of the video content, resulting in a reduced accuracy of video retrieval, thereby improving the accuracy of similar video retrieval. Or the computer device can determine the union of the visual retrieval video set and the text retrieval video set as the similar video retrieval result for the video data to be matched. Avoid false detection or missed detection caused by adversarial means, and improve the accuracy and anti-disturbance performance of similar video retrieval. Integrating the embodiments of the present application into a similar video retrieval system of visual elements can significantly reduce the decrease in visual feature similarity caused by various attack methods, thereby effectively alleviating the problem of missed detection caused by the decrease in feature similarity during the retrieval process of the system, and improving the accuracy and completeness of the retrieval.

[0214] See also Figure 7 , Figure 7 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 The data processing method can be executed by a computer device, which can be Figure 1 The service server 100 shown or any terminal device in the terminal device cluster is, for example, the terminal device 10a. The following will take the data processing method executed by a computer device as an example for description. The data processing method may at least include the following steps S401-S404:

[0215] Step S401, obtaining sample video data and annotated video data, and inputting the sample video data and the annotated video data into an initial video retrieval model;

[0216] Specifically, the computer device may obtain sample video data and annotated video data, and the sample video data and the annotated video data may include an annotated similarity score, and the similarity score is used to characterize the degree of similarity between the sample video data and the annotated video data. The computer device may input the sample video data and the annotated video data into an initial video retrieval model. Among them, the text description layer in the multimodal text description layer of the initial video retrieval model may be a Phi-3-mini-128k-instruct model (an open source lightweight model) or a large language model.

[0217] The model training of the computer device can be achieved by fine-tuning multiple publicly accessible image-text pair datasets and video-text pair datasets. Since the multimodal large language model is mainly used to generate accurate and concise descriptions for short videos input by users, the computer device can specifically focus on fine-tuning using datasets related to video captioning tasks to help the model generate content description text in English and Chinese environments.

[0218] Specifically, in the training phase, for image captioning tasks, Chinese and English image captioning datasets can be used, such as Laion-EN (English & Chinese), COYO (Chinese), GRIT (Chinese), COCO (English), TextCaps (English), and ShareGPT4o (English & Chinese). For video captioning tasks, Chinese and English video captioning datasets can be used, such as mPLUG-YOUKU (Chinese), ShareGPT4o (English & Chinese), Mementos (English), STAR (English), VideoChat2IT (English), and LSMDC-QA (English). In the field of general question answering (QA), question-answering pair datasets can be used, such as VQAv2 (English), GQA (English), OKVQA (English), VSR (English), and VisualDialog (English), so as to improve the processing capabilities of the initial video retrieval model in handling various question-answering scenarios.

[0219] In addition, since short videos uploaded by users often contain a large amount of text information, this text information is crucial for the model to correctly understand the video content. Therefore, the computer device can especially enhance the optical character recognition (OCR) capability of the model. You can use open source OCR models, such as the Paddle OCR model, to perform OCR processing on Chinese images (which can be the Wukong dataset) and English images (which can be the Laion COCO dataset), and incorporate the SROIE dataset, FUNSD dataset, and POIE dataset to further enhance the model's ability to recognize handwritten fonts.

[0220] In the fine-tuning stage, the computer device can focus on using video and image captioning datasets, as well as video and image dialogue datasets to fine-tune the multimodal large model to enhance its ability to generate content description text for videos. For video and image captioning tasks, the COCO (English) dataset, mPLUG-YOUKU (Chinese) dataset, and ShareGPT4o (English & Chinese) dataset can be used. For video and image dialogue tasks, the LLaVA-150K (English & Chinese) dataset, LVIS-Instruct4V (English) dataset, ALLaVA (English & Chinese) dataset, Laion-GPT4V (English) dataset, and TextOCR-GPT4V (English) dataset can be used.

[0221] Since existing video datasets mainly come from long video websites, and they are semantically or scene-segmented during data processing to turn them into video clips, these data are quite different from short videos uploaded by real users. To solve this problem, we collected 100,000 UGC short videos and used ChatGPT4o (Generative Pre-trainedTransformer 4 omni) to generate a concise video description for each short video (each short video uniformly samples 20 frames) through prompts such as business text information to construct targeted short video-content description text pairs.

[0222] The computer device can use Paddle OCR to perform OCR on each frame of its image. Subsequently, the OCR results of multiple frames can be merged into video-level OCR results. For the same text that appears repeatedly in multiple frames, it is only recorded once in the result, thereby constructing a dataset of UGC short video-OCR result text pairs (also known as short video-content description text pairs). In the fine-tuning stage, the computer device can use these two collected short video datasets to further optimize the model.

[0223] Step S402, extracting features from sample video data through an initial video retrieval model to obtain a first visual tag vector, acquiring business text information, performing vector concatenation of the business text information and the first visual tag vector to obtain a first fused feature vector, extracting features from the annotated video data to obtain a second visual tag vector, and concatenating the business text information and the second visual tag vector to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data;

[0224] Specifically, the computer device can extract features from the sample video data through the initial video retrieval model to obtain a first visual tag vector, and extract features from the annotated video data to obtain a second visual tag vector. For the content of generating the first visual tag vector and the second visual tag vector, please refer to the above Figure 4 The specific description of step S201 to step S202 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0225] The initial video retrieval model includes an initial abstract layer; the process of extracting features from sample video data to obtain a first visual tag vector may include: extracting features from sample video data to obtain a sample visual tag vector, obtaining an initial query vector randomly initialized in the initial abstract layer, and performing vector concatenation on the sample visual tag vector based on the initial query vector to obtain a first visual tag vector; the initial query vector is a model parameter in the initial abstract layer; the vector length of the initial query vector is the same as the vector length of the first visual tag vector;

[0226] Specifically, the computer device obtains an initial query vector randomly initialized in the initial abstract layer, the initial query vector is a model parameter in the initial abstract layer, and the computer device can perform vector concatenation on the sample visual tag vector based on the initial query vector to obtain a first visual tag vector. The initial query vector can be a fixed query vector used to extract higher-level semantic features from visual tags of multiple frames of images by learning a set of fixed queries.

[0227] The computer device can obtain business text information, perform vector splicing on the business text information and the first visual mark vector to obtain a first fused feature vector, and perform vector splicing on the business text information and the second visual mark vector to obtain a second fused feature vector. The business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes the semantics of the content description text for characterizing the sample video data, and the second fused feature vector includes the semantics of the content description text for characterizing the annotated video data. For information on generating the first fused feature vector and the second fused vector, please refer to the above Figure 4 The specific description of step S203 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0228] Step S403, determining an alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, performing alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and performing alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length;

[0229] Specifically, the computer device may determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, for example, the average or median of the vector length of the first fused feature vector and the vector length of the second fused feature vector is determined as the alignment length. The computer device may perform alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length. For details about the alignment processing, please refer to the above Figure 4 The specific description of step S204 of the corresponding embodiment will not be repeated here in the embodiment of the present application.

[0230] The initial video retrieval model includes a feature alignment processing layer; the training process of the feature alignment processing layer can be: inputting a sample feature vector into a coding and decoding network; the coding and decoding network includes an initial coding layer, an initial reconstruction layer and an initial decoding layer; the length of the model parameter of the initial reconstruction layer is the alignment length; in the initial coding layer, encoding processing is performed on the feature vector to be aligned to obtain an encoding feature vector, and based on the time step, the encoding feature vector is downsampled to obtain a downsampled feature vector; in the initial reconstruction layer, an alignment feature vector is generated based on the dependency of the downsampled feature vector in each time step, and in the initial decoding layer, the alignment feature vector is upsampled to obtain an upsampled feature vector; an alignment loss value is generated based on the feature vector to be aligned and the upsampled feature vector, and the model parameters of the initial coding layer and the initial reconstruction layer are adjusted according to the alignment loss value. When the initial coding layer and the initial reconstruction layer meet the model training convergence condition, the converged initial coding layer and the converged initial reconstruction layer are determined as the feature alignment processing layer of the initial video retrieval model.

[0231] For details, please refer to Figure 8 , Figure 8 is a schematic diagram of a model structure of a coding and decoding network proposed in an embodiment of the present application, such as Figure 8 As shown, the encoding and decoding network may include an initial encoding layer, an initial reconstruction layer, and an initial decoding layer. The encoding and decoding network may be a long short-term memory network autoencoder (LSTM Autoencoder), in which the initial encoding layer and the initial decoding layer are both LSTM (Long Short-Term Memory) layers, and the initial reconstruction layer is a linear layer. The encoding and decoding network may be trained in a self-supervised manner without any explicit labels.

[0232] The computer device may extend the length of the model parameter of the initial reconstruction layer to the alignment length, so that the length of the vector output by the initial reconstruction layer is fixed to the alignment length. In the initial encoding layer, the computer device treats the alignment feature vector The encoding process is performed to obtain an encoded feature vector, and based on the number of time steps, the same initial hidden state is provided for each time step, and the encoded feature vector is downsampled to obtain a downsampled feature vector.

[0233] In the initial reconstruction layer, an aligned feature vector is generated based on the dependencies of the downsampled feature vectors at each time step. , align the feature vectors The vector length of is the alignment length. In the initial decoding layer, the computer device can upsample the aligned feature vector to reconstruct the feature vector to be aligned. The corresponding upsampled feature vector .

[0234] The computer device can be based on the feature vector to be aligned and the upsampled feature vector Generate the alignment loss value, which is calculated as shown in formula (10):

[0235] Formula (10)

[0236] Among them, L is the alignment loss value and T is the number of time steps.

[0237] The computer device adjusts the model parameters of the initial coding layer and the initial reconstruction layer by using the alignment loss value. When the initial coding layer and the initial reconstruction layer meet the model training convergence condition, the converged initial coding layer and the converged initial reconstruction layer are determined as the feature alignment processing layer of the initial video retrieval model. The vector output by the feature alignment processing layer can be a vector whose vector length is fixed to the alignment length.

[0238] Step S404, generate text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, adjust the model parameters of the initial video retrieval model according to the model loss value, and obtain the target video retrieval model when the initial video retrieval model meets the model training convergence conditions; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0239] Specifically, the computer device can generate a text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity and the similarity score between the sample video data and the annotated video data, adjust the model parameters of the initial video retrieval model through the model loss value, and obtain the target video retrieval model when the initial video retrieval model meets the model training convergence conditions.

[0240] The training process of the initial video retrieval model is divided into two stages: the pre-training stage and the instruction adjustment stage. First, in the pre-training stage, the computer device can freeze the visual encoding layer and the text description layer, and use the checkpoint of Video-LLaMA (a multimodal large language model for video and audio) to initialize the initial abstract layer, focusing on training the initial abstract layer and the multi-layer perceptron (MLP) connected to the initial abstract layer. In this stage, the computer device can fine-tune the multimodal large model for 80,000 steps. The AdamW optimizer (a regularized optimizer that introduces weight decay to prevent model overfitting) is used, and the parameters are set to =0.9, =0.98, and the weight decay is 0.05. Among them, and These are hyperparameters for the AdamW optimizer.

[0241] In addition, in the first 2000 iterations (steps), the learning rate can be linearly warmed up from Increase to , and then the cosine annealing learning rate scheduler is used, with a peak learning rate of Next, the parameters of the entire model were fully fine-tuned for 5 training cycles (epochs) to enhance the model's responsiveness to short videos. At this stage, the learning rate was set to , and a layer-by-layer learning rate decay of factor 0.9 is applied to preserve the low-level visual representation. The entire model is trained with a learning rate of 0 and a batch size of 256.

[0242] It can be understood that the target video retrieval model can be used to perform the above Figure 3 , Figure 4 and Figure 6 In the description of the data processing method in any one of the corresponding embodiments, the visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer, and the visual abstraction layer includes a compressed query vector obtained by training the initial query vector. The compressed query vector is used to perform attention processing on the matching video data to obtain a visual tag vector whose vector length is the same as the vector length of the compressed query vector.

[0243] The embodiments of the present application can be fine-tuned by focusing on using data sets related to video subtitling tasks to help the model generate content description text in English and Chinese environments. And by learning a fixed set of compressed query vectors for multi-head attention processing, splicing the visual tag vector to be matched and the compressed query vector together, it is possible to better capture multimodal information, help the target video retrieval model understand and abstract image information at different levels and different granularities, and improve the processing performance of the target video retrieval model. Through the visual abstraction layer, S visual tag vectors to be matched are compressed to obtain a compressed visual tag vector. The compressed visual tag vector is obtained by multi-head attention processing of the compressed query vector and the visual tag vector sequence, and the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector. The number of visual tags can be effectively reduced, and the video memory load and computing resource usage can be significantly reduced.

[0244] By performing alignment processing through the long short-term memory network autoencoder, it is possible to avoid padding processing due to inconsistent vector lengths, such as filling the business text feature vector and the fused feature vector to the same length, which may lead to reduced computing efficiency, increased video memory consumption, filling some redundant information affecting the model's capture of real data, reduced generalization, and other problems. Through the above-mentioned alignment processing, the embodiment of the present application can effectively generate a to-be-matched text feature vector with a vector length of the alignment length by fusing the feature vector, and perform similarity calculation on the to-be-matched text feature vector and the business text feature vector, which can reduce the computational cost of the similarity calculation and improve the accuracy of the similarity calculation. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set, and the similar video retrieval results for the to-be-matched video data are determined through the text retrieval video set, which can avoid the problem of missed detection and false detection caused by misleading similarity caused by visual elements, and improve the accuracy and completeness of similar video retrieval.

[0245] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 1 .like Fig. 9 As shown, the data processing device 1 includes a visual marker extraction module 610 , a visual marker processing module 620 , a text feature acquisition module 630 and a similarity calculation module 640 .

[0246] A visual marker extraction module 610 is used to extract features from the video data to be matched to obtain a compressed visual marker vector;

[0247] The visual tag processing module 620 is used to obtain business text information, perform vector concatenation of the business text information and the compressed visual tag vector to obtain a fused feature vector, and perform alignment processing on the fused feature vector to obtain a to-be-matched text feature vector whose vector length is the alignment length; the business text information is a text prompt for indicating the extraction of the content description text corresponding to the video data, the fused feature vector includes the semantics of the content description text for characterizing the to-be-matched video data, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database;

[0248] The text feature acquisition module 630 is used to acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0249] The similarity calculation module 640 is used to generate text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0250] In a possible implementation, the similarity calculation module 640 is further configured to perform the following operations:

[0251] Extract features from the business video data to obtain a business visual feature vector, and extract features from the video data to be matched to obtain a visual feature vector to be matched;

[0252] Generate visual similarity between the business visual feature vector and the visual feature vector to be matched, associate the business video data with the business visual feature vector whose visual similarity is greater than or equal to the video similarity threshold, and determine the visual retrieval video set;

[0253] Determine the intersection of the visual retrieval video set and the text retrieval video set as a similar video retrieval result for the video data to be matched;

[0254] Alternatively, a union of the visual retrieval video set and the text retrieval video set is determined as a similar video retrieval result for the video data to be matched.

[0255] In a possible implementation, the visual marker extraction module 610 is used to extract features from the video data to be matched, and when obtaining the compressed visual marker vector, specifically performs the following operations:

[0256] Inputting the to-be-matched video data into a target video retrieval model; the target video retrieval model includes a multimodal text description layer, and the multimodal text description layer includes a preprocessing layer, a visual encoding layer, and a visual abstraction layer;

[0257] In the preprocessing layer, based on the video duration and video frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames, and the T sampled video frames are respectively segmented to obtain S unit sampled images; T is a positive integer determined based on the video duration and the video frame rate, and S is a positive integer greater than T;

[0258] In the visual encoding layer, feature extraction is performed on S unit sampled images to obtain S unit visual tag vectors. Based on the rearrangement factor, the vector elements representing the spatial channels in the S unit visual tag vectors are remapped to the feature channels to obtain S visual tag vectors to be matched; the rearrangement factor is determined based on the number S of unit visual tag vectors, and the vector length of the unit visual tag vector is greater than the vector length of the visual tag vector to be matched;

[0259] In the visual abstraction layer, attention processing is performed on the matching visual tag vector based on the compressed query vector to obtain a compressed visual tag vector; the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector.

[0260] In a possible implementation, the visual marker extraction module 610 is used to perform attention processing on the matching visual marker vector based on the compressed query vector in the visual abstraction layer, and when the compressed visual marker vector is obtained, specifically perform the following operations:

[0261] In the visual abstraction layer, a compressed query vector is obtained, and the visual tag vector to be matched and the compressed query vector are concatenated to obtain a visual tag vector sequence; the compressed query vector is a learnable parameter;

[0262] The compressed query vector and the visual tag vector sequence are assigned to N unit processing spaces, and cross-attention processing is performed on the compressed query vector and the visual tag vector sequence in each unit processing space to obtain N unit attention result vectors. Based on the attention weights corresponding to the N unit attention result vectors and the N unit processing spaces, a compressed visual tag vector is generated; N is a positive integer.

[0263] In a possible implementation, the N unit processing spaces include a target unit processing space; the visual marker extraction module 610 is used to perform cross-attention processing on the compressed query vector and the visual marker vector sequence in each unit processing space, and when N unit attention result vectors are obtained, specifically to perform the following operations:

[0264] In the target unit processing space, the compressed query vector is multiplied by the query parameter matrix to obtain the query vector, the visual tag vector sequence is multiplied by the key parameter matrix to obtain the key vector, and the visual tag vector sequence is multiplied by the value parameter matrix to obtain the value vector;

[0265] An attention score vector is generated based on the query vector and the key vector. The attention score vector is reduced in dimension based on the adjacent length of the key vector. The reduced-dimensional attention score vector is normalized to obtain the attention weight. The attention weight is point-multiplied with the value vector to obtain the unit attention result vector corresponding to the target unit processing space.

[0266] In a possible implementation, the visual tag processing module 620 is used to obtain business text information, perform vector concatenation on the business text information and the compressed visual tag vector to obtain a fused feature vector, perform alignment processing on the fused feature vector, and obtain a to-be-matched text feature vector whose vector length is the alignment length, and specifically perform the following operations:

[0267] inputting the compressed visual tag vector into a target video retrieval model; the target video retrieval model comprises a multimodal text description layer and a feature alignment processing layer, and the multimodal text description layer comprises a text description layer;

[0268] In the text description layer, business text information is obtained, features are extracted from the business text information to obtain a text prompt feature vector, and the text prompt feature vector is concatenated with the compressed visual mark vector to obtain a fused feature vector;

[0269] In the feature alignment processing layer, the fused feature vector is aligned to obtain a feature vector of the text to be matched whose vector length is the alignment length.

[0270] In a possible implementation, the visual marker processing module 620 is used to perform alignment processing on the fused feature vector in the feature alignment processing layer, and when obtaining the to-be-matched text feature vector whose vector length is the alignment length, specifically to perform the following operations:

[0271] The fused feature vector is input into the feature alignment processing layer, which includes an automatic encoding layer and a bottleneck reconstruction layer; the bottleneck reconstruction layer is obtained based on the alignment length training, and the vector length output by the bottleneck reconstruction layer is fixed to the alignment length;

[0272] In the automatic encoding layer, the fused feature vector is encoded to obtain an encoded feature vector, and the encoded feature vector is downsampled based on the time step to obtain a downsampled feature vector;

[0273] In the bottleneck reconstruction layer, based on the dependency of the downsampled feature vectors in each time step, a feature vector of the text to be matched with a vector length equal to the alignment length is generated.

[0274] In a possible implementation, the visual marker processing module 620 is used to perform alignment processing on the fused feature vector in the feature alignment processing layer, and when obtaining the to-be-matched text feature vector whose vector length is the alignment length, specifically to perform the following operations:

[0275] Input the fused feature vector into the feature alignment processing layer, which includes a word frequency processing layer and a weighted alignment layer;

[0276] In the word frequency processing layer, based on the fused feature vector, the prediction probability corresponding to each word in the vocabulary is generated, and the content description text of the video data to be matched is generated based on the prediction probability. Based on the alignment length, the content description text is segmented to obtain F text segments; the alignment length is F, and F is a positive integer;

[0277] Obtain the word frequencies corresponding to the F text segments in the content description text, and the inverse document frequencies corresponding to the F text segments in the content description text, and generate F text weights based on the F word frequencies and the F inverse document frequencies;

[0278] In the weighted alignment layer, the F text weights are respectively multiplied with the fused feature vector to obtain F weighted feature vectors, and a text feature vector to be matched whose vector length is the alignment length is generated based on the F weighted feature vectors.

[0279] In a possible implementation, the fused feature vector includes G unit prediction vectors, the vector lengths of the G unit prediction vectors are all alignment lengths, and the last unit prediction vector of the G unit prediction vectors includes the global semantics obtained by vector concatenation of the business text information and the compressed visual tag vector; the visual tag processing module 620 is used to align the fused feature vector in the feature alignment processing layer, and when the feature vector of the text to be matched is obtained and the vector length is the alignment length, it is specifically used to perform the following operations:

[0280] In the feature alignment processing layer, the last unit prediction vector among the G unit prediction vectors is determined as the feature vector of the text to be matched whose vector length is the alignment length.

[0281] In a possible implementation, when the similarity calculation module 640 is used to generate the text similarity between the service text feature vector and the to-be-matched text feature vector, it is specifically used to perform the following operations:

[0282] Inputting the business text feature vector and the to-be-matched text feature vector into a target video retrieval model; the target video retrieval model includes a recall matching layer;

[0283] Based on each vector element value in the business text feature vector, a first parameter of the business text feature vector is generated; based on each vector element value in the to-be-matched text feature vector, a second parameter of the to-be-matched text feature vector is generated;

[0284] A dot multiplication operation is performed on the business text feature vector and the feature vector of the text to be matched to obtain a matching parameter, and a text similarity between the business text feature vector and the feature vector of the text to be matched is generated based on the matching parameter, the first parameter and the second parameter.

[0285] The embodiment of the present application can sample the video data to be matched based on the context window length of the target video retrieval model to obtain a unit sampled image, perform feature extraction on the sampled unit sampled image, obtain a unit visual marker vector, and reduce the vector dimension representing the image height and image width in the unit sampled image by rearranging the pixels of the image, thereby reducing the number of visual markers. By learning a set of fixed compressed query vectors for multi-head attention processing, splicing the visual marker vector to be matched and the compressed query vector together, multimodal information can be better captured, helping the target video retrieval model to understand and abstract image information at different levels and different granularities, and improving the processing performance of the target video retrieval model. Through the visual abstraction layer, S visual marker vectors to be matched are compressed to obtain a compressed visual marker vector, which is obtained by multi-head attention processing of the compressed query vector and the visual marker vector sequence, and the vector length of the compressed visual marker vector is the same as the vector length of the compressed query vector. The number of visual markers can be effectively reduced, and the video memory load and computing resource occupancy can be significantly reduced.

[0286] The business text information is obtained, and the business text information and the compressed visual mark vector are vector-concatenated to obtain a fused feature vector, wherein the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics of the content description text for characterizing the video data to be matched. The fused feature vector can ignore visual elements in the video data to be matched that are irrelevant to the video content, such as lace, texture effects, or occluded emoticons added by adversarial means, etc., which can improve the robustness and anti-disturbance of the model. Since the fused feature vector only needs to retain the semantics of the content description text, it can effectively reduce the number of visual marks to be processed, reduce the computational cost, and improve the efficiency of similar video retrieval. By aligning the fused feature vector, a text feature vector to be matched with a vector length of the alignment length is obtained. The alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database. The business text feature vector corresponding to the business video data is obtained, wherein the business text feature vector includes the semantics of the content description text for characterizing the business video data. By calculating the text similarity between the business text feature vector and the text feature vector to be matched, the text similarity between the business text feature vector and the text feature vector to be matched is obtained. The text similarity between the business text feature vector and the text feature vector to be matched with the same vector length is calculated, which can avoid the problem of filling in due to the inconsistency of the vector length, further reducing the calculation cost and improving the efficiency of similar video retrieval. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set. The similar video retrieval results for the video data to be matched are determined by the text retrieval video set, which can avoid the problem of missed detection and false detection caused by the reduction of similarity due to visual elements, and improve the accuracy and completeness of similar video retrieval.

[0287] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0288] See also Fig.10 , Fig.10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 2 .like Fig.10 As shown, the data processing device 2 includes a sample acquisition module 710 , a fusion vector generation module 720 , a vector alignment module 730 and a model adjustment module 740 .

[0289] A sample acquisition module 710 is used to acquire sample video data and annotated video data, and input the sample video data and the annotated video data into an initial video retrieval model;

[0290] The fusion vector generation module 720 is used to extract features from the sample video data through the initial video retrieval model to obtain a first visual tag vector, obtain business text information, perform vector splicing on the business text information and the first visual tag vector to obtain a first fused feature vector, perform feature extraction on the annotated video data to obtain a second visual tag vector, and perform vector splicing on the business text information and the second visual tag vector to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data;

[0291] A vector alignment module 730 is used to determine an alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, perform alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length;

[0292] The model adjustment module 740 is used to generate text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, and adjust the model parameters of the initial video retrieval model according to the model loss value. When the initial video retrieval model meets the model training convergence conditions, a target video retrieval model is obtained; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0293] In a possible implementation, the initial video retrieval model includes a feature alignment processing layer; the model adjustment module 740 is further configured to perform the following operations:

[0294] The sample feature vector is input into the encoding and decoding network; the encoding and decoding network includes an initial encoding layer, an initial reconstruction layer and an initial decoding layer; the length of the model parameter of the initial reconstruction layer is the alignment length;

[0295] In the initial coding layer, the feature vector to be aligned is encoded to obtain an encoded feature vector, and the encoded feature vector is downsampled based on the number of time steps to obtain a downsampled feature vector;

[0296] In the initial reconstruction layer, an aligned feature vector is generated based on the dependency of the downsampled feature vector in each time step. In the initial decoding layer, the aligned feature vector is upsampled to obtain an upsampled feature vector.

[0297] An alignment loss value is generated based on the feature vector to be aligned and the upsampled feature vector, and the model parameters of the initial coding layer and the initial reconstruction layer are adjusted according to the alignment loss value. When the initial coding layer and the initial reconstruction layer meet the model training convergence conditions, the converged initial coding layer and the converged initial reconstruction layer are determined as the feature alignment processing layers of the initial video retrieval model.

[0298] In a possible implementation, the initial video retrieval model includes an initial abstraction layer; the fusion vector generation module 720 is used to extract features from the sample video data, and when the first visual marker vector is obtained, it is specifically used to perform the following operations:

[0299] Extract features from sample video data to obtain a sample visual tag vector, obtain an initial query vector randomly initialized in the initial abstract layer, and perform vector concatenation on the sample visual tag vector based on the initial query vector to obtain a first visual tag vector; the initial query vector is a model parameter in the initial abstract layer; the vector length of the initial query vector is the same as the vector length of the first visual tag vector;

[0300] Among them, the visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer. The visual abstraction layer contains a compressed query vector obtained by training the initial query vector. The compressed query vector is used to perform attention processing on the matching video data to obtain a visual label vector with the same vector length as the compressed query vector.

[0301] The embodiments of the present application can be fine-tuned by focusing on using data sets related to video subtitling tasks to help the model generate content description text in English and Chinese environments. And by learning a fixed set of compressed query vectors for multi-head attention processing, splicing the visual tag vector to be matched and the compressed query vector together, it is possible to better capture multimodal information, help the target video retrieval model understand and abstract image information at different levels and different granularities, and improve the processing performance of the target video retrieval model. Through the visual abstraction layer, S visual tag vectors to be matched are compressed to obtain a compressed visual tag vector. The compressed visual tag vector is obtained by multi-head attention processing of the compressed query vector and the visual tag vector sequence, and the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector. The number of visual tags can be effectively reduced, and the video memory load and computing resource usage can be significantly reduced.

[0302] By performing alignment processing through the long short-term memory network autoencoder, it is possible to avoid padding processing due to inconsistent vector lengths, such as filling the business text feature vector and the fused feature vector to the same length, which may lead to reduced computing efficiency, increased video memory consumption, filling some redundant information affecting the model's capture of real data, reduced generalization, and other problems. Through the above-mentioned alignment processing, the embodiment of the present application can effectively generate a to-be-matched text feature vector with a vector length of the alignment length by fusing the feature vector, and perform similarity calculation on the to-be-matched text feature vector and the business text feature vector, which can reduce the computational cost of the similarity calculation and improve the accuracy of the similarity calculation. The business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold is determined as a text retrieval video set, and the similar video retrieval results for the to-be-matched video data are determined through the text retrieval video set, which can avoid the problem of missed detection and false detection caused by misleading similarity caused by visual elements, and improve the accuracy and completeness of similar video retrieval.

[0303] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0304] See also Fig.11 , Fig.11 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.11 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.11As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.

[0305] In such Fig.11 In the computer device 1000 shown, the network interface 1004 can provide a network communication element; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call a computer program stored in the memory 1005.

[0306] When the computer device 1000 is used to execute the data processing apparatus 1, the following steps are implemented:

[0307] Perform feature extraction on the video data to be matched to obtain a compressed visual tag vector;

[0308] Acquire business text information, perform vector concatenation of the business text information and the compressed visual mark vector to obtain a fused feature vector, perform alignment processing on the fused feature vector, and obtain a to-be-matched text feature vector whose vector length is the alignment length; the business text information is a text prompt for indicating the extraction of content description text corresponding to the video data, the fused feature vector includes semantics for characterizing the content description text of the to-be-matched video data, and the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database;

[0309] Acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data;

[0310] Generate text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to the text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

[0311] When the computer device 100 is used to execute the data processing apparatus 2, the following can be achieved:

[0312] Obtaining sample video data and annotated video data, and inputting the sample video data and the annotated video data into an initial video retrieval model;

[0313] Through the initial video retrieval model, feature extraction is performed on the sample video data to obtain a first visual tag vector, business text information is obtained, the business text information and the first visual tag vector are vector-joined to obtain a first fused feature vector, feature extraction is performed on the annotated video data to obtain a second visual tag vector, and the business text information and the second visual tag vector are vector-joined to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data;

[0314] Determine the alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, perform alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and perform alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length;

[0315] Generate text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, adjust the model parameters of the initial video retrieval model according to the model loss value, and obtain a target video retrieval model when the initial video retrieval model meets the model training convergence condition; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

[0316] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 , Figure 4 , Figure 6 and Figure 7 The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0317] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the processor executes the computer program, it can execute the above Figure 3 , Figure 4 , Figure 6 and Figure 7 The description of the above data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0318] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been displayed or is to be displayed.

[0319] In addition, it should be noted that: the embodiment of the present application also provides a computer program product, the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above Figure 3 , Figure 4 , Figure 6 and Figure 7 A method provided by any corresponding embodiment.

[0320] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0321] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of network elements in the above description. Whether these network elements are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described network elements for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0322] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0323] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0324] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0325] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Perform feature extraction on the video data to be matched to obtain a compressed visual tag vector; Acquire business text information, and perform vector concatenation on the business text information and the compressed visual mark vector to obtain a fused feature vector; the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes semantics for characterizing the content description text of the video data to be matched; Inputting the fused feature vector into a feature alignment processing layer of a target video retrieval model; The feature alignment processing layer includes an automatic encoding layer and a bottleneck reconstruction layer; the bottleneck reconstruction layer is obtained based on alignment length training, and the vector length output by the bottleneck reconstruction layer is fixed to the alignment length; In the automatic encoding layer, the fused feature vector is encoded to obtain an encoded feature vector, and based on the time step, the encoded feature vector is downsampled to obtain a downsampled feature vector, and in the bottleneck reconstruction layer, based on the dependency of the downsampled feature vector in each time step, a to-be-matched text feature vector having a vector length of the alignment length is generated; the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database; Acquire a service text feature vector corresponding to the service video data; the service text feature vector includes the semantics of the content description text used to characterize the service video data; Generate text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

2. The method according to claim 1, characterized in that Also includes: Extracting features from the business video data to obtain a business visual feature vector, and extracting features from the to-be-matched video data to obtain a to-be-matched visual feature vector; Generate visual similarity between the business visual feature vector and the visual feature vector to be matched, associate the business video data with the business visual feature vector whose visual similarity is greater than or equal to a video similarity threshold, and determine a visual retrieval video set; Determine the intersection of the visual retrieval video set and the text retrieval video set as a similar video retrieval result for the video data to be matched; Alternatively, a union of the visual retrieval video set and the text retrieval video set is determined as a similar video retrieval result for the video data to be matched.

3. The method according to claim 1, characterized in that The step of extracting features from the video data to be matched to obtain a compressed visual marker vector includes: Inputting the to-be-matched video data into a target video retrieval model; the target video retrieval model comprises a multimodal text description layer, and the multimodal text description layer comprises a preprocessing layer, a visual encoding layer, and a visual abstraction layer; In the preprocessing layer, based on the video duration and the video frame rate of the video data to be matched, the video data to be matched is sampled to obtain T sampled video frames, and the T sampled video frames are respectively image segmented to obtain S unit sampled images; T is a positive integer jointly determined based on the video duration and the video frame rate, and S is a positive integer greater than T; In the visual coding layer, feature extraction is performed on the S unit sampling images to obtain S unit visual mark vectors, and based on a rearrangement factor, vector elements representing spatial channels in the S unit visual mark vectors are remapped to feature channels to obtain S visual mark vectors to be matched; the rearrangement factor is determined based on the number S of the unit visual mark vectors, and the vector length of the unit visual mark vector is greater than the vector length of the visual mark vector to be matched; In the visual abstraction layer, attention processing is performed on the visual tag vector to be matched based on the compressed query vector to obtain a compressed visual tag vector; the vector length of the compressed visual tag vector is the same as the vector length of the compressed query vector.

4. The method according to claim 3, characterized in that In the visual abstraction layer, performing attention processing on the visual tag vector to be matched based on the compressed query vector to obtain the compressed visual tag vector includes: In the visual abstraction layer, a compressed query vector is obtained, and the visual tag vector to be matched and the compressed query vector are vector-concatenated to obtain a visual tag vector sequence; the compressed query vector is a learnable parameter; The compressed query vector and the visual label vector sequence are allocated to N unit processing spaces, and cross-attention processing is performed on the compressed query vector and the visual label vector sequence in each unit processing space to obtain N unit attention result vectors, and a compressed visual label vector is generated based on the attention weights corresponding to the N unit attention result vectors and the N unit processing spaces respectively; N is a positive integer.

5. The method according to claim 4, characterized in that The N unit processing spaces include a target unit processing space; The cross-attention processing is performed on the compressed query vector and the visual tag vector sequence in each unit processing space to obtain N unit attention result vectors, including: In the target unit processing space, performing a dot multiplication operation on the compressed query vector and a query parameter matrix to obtain a query vector, performing a dot multiplication operation on the visual tag vector sequence and a key parameter matrix to obtain a key vector, and performing a dot multiplication operation on the visual tag vector sequence and a value parameter matrix to obtain a value vector; An attention score vector is generated based on the query vector and the key vector, the attention score vector is dimensionally reduced based on the adjacent length of the key vector, the reduced attention score vector is normalized to obtain an attention weight, and a dot multiplication operation is performed on the attention weight and the value vector to obtain a unit attention result vector corresponding to the target unit processing space.

6. The method according to claim 1, characterized in that The acquiring of the business text information and performing vector concatenation of the business text information and the compressed visual mark vector to obtain a fused feature vector includes: Inputting the compressed visual tag vector into a target video retrieval model; the target video retrieval model comprises a multimodal text description layer and a feature alignment processing layer, and the multimodal text description layer comprises a text description layer; In the text description layer, business text information is obtained, and features are extracted from the business text information to obtain a text prompt feature vector. The text prompt feature vector and the compressed visual mark vector are vector-joined to obtain a fused feature vector.

7. The method according to claim 1, characterized in that The generating of the text similarity between the service text feature vector and the to-be-matched text feature vector comprises: Inputting the business text feature vector and the to-be-matched text feature vector into a target video retrieval model; the target video retrieval model includes a recall matching layer; Based on each vector element value in the business text feature vector, a first parameter of the business text feature vector is generated; based on each vector element value in the to-be-matched text feature vector, a second parameter of the to-be-matched text feature vector is generated; Performing a dot multiplication operation on the business text feature vector and the to-be-matched text feature vector to obtain a matching parameter, and generating a text similarity between the business text feature vector and the to-be-matched text feature vector based on the matching parameter, the first parameter and the second parameter.

8. A data processing method, characterized in that: include: Perform feature extraction on the video data to be matched to obtain a compressed visual tag vector; Acquire business text information, and perform vector concatenation on the business text information and the compressed visual mark vector to obtain a fused feature vector; the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes semantics for characterizing the content description text of the video data to be matched; Inputting the fused feature vector into a feature alignment processing layer of a target video retrieval model; The feature alignment processing layer includes a word frequency processing layer and a weighted alignment layer; In the word frequency processing layer, based on the fused feature vector, a prediction probability corresponding to each word in the vocabulary is generated, based on the prediction probability, a content description text of the to-be-matched video data is generated, based on the alignment length, the content description text is segmented to obtain F text segments, the word frequencies corresponding to the F text segments in the content description text are obtained, and the inverse document frequencies corresponding to the F text segments in the content description text are obtained, and F text weights are generated based on the F word frequencies and the F inverse document frequencies; the alignment length is F, and F is a positive integer; In the weighted alignment layer, the F text weights are respectively subjected to a dot multiplication operation with the fused feature vector to obtain F weighted feature vectors, and a text feature vector to be matched whose vector length is the alignment length is generated based on the F weighted feature vectors; the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database; Acquire a service text feature vector corresponding to the service video data; the service text feature vector includes the semantics of the content description text used to characterize the service video data; Generate text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

9. A data processing method, characterized in that: include: Perform feature extraction on the video data to be matched to obtain a compressed visual tag vector; Acquire business text information, perform vector splicing on the business text information and the compressed visual mark vector, and obtain a fused feature vector; the business text information is a text prompt for indicating the extraction of content description text corresponding to the video data, and the fused feature vector includes semantics for characterizing the content description text of the video data to be matched; the fused feature vector includes G unit prediction vectors, where G is a positive integer; the vector lengths of the G unit prediction vectors are all alignment lengths, and the last unit prediction vector of the G unit prediction vectors includes global semantics obtained by vector splicing of the business text information and the compressed visual mark vector; Input the fused feature vector into the feature alignment processing layer of the target video retrieval model, and determine the last unit prediction vector among the G unit prediction vectors as the text feature vector to be matched whose vector length is the alignment length; the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database; Acquire a service text feature vector corresponding to the service video data; the service text feature vector includes the semantics of the content description text used to characterize the service video data; Generate text similarity between the business text feature vector and the text feature vector to be matched, and determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

10. A data processing method, characterized in that: include: Acquire sample video data and annotated video data, and input the sample video data and the annotated video data into an initial video retrieval model; Through the initial video retrieval model, feature extraction is performed on the sample video data to obtain a first visual tag vector, business text information is obtained, vector splicing is performed on the business text information and the first visual tag vector to obtain a first fused feature vector, feature extraction is performed on the annotated video data to obtain a second visual tag vector, and vector splicing is performed on the business text information and the second visual tag vector to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data; Determine an alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, and input the sample feature vector into a coding and decoding network; the coding and decoding network includes an initial coding layer, an initial reconstruction layer and an initial decoding layer; the length of the model parameter of the initial reconstruction layer is the alignment length; In the initial coding layer, encoding is performed on the feature vector to be aligned to obtain an encoded feature vector, and based on the number of time steps, downsampling is performed on the encoded feature vector to obtain a downsampled feature vector; In the initial reconstruction layer, an aligned feature vector is generated based on the dependency of the down-sampled feature vector in each time step, and in the initial decoding layer, the aligned feature vector is up-sampled to obtain an up-sampled feature vector; Generate an alignment loss value based on the feature vector to be aligned and the upsampled feature vector, adjust the model parameters of the initial coding layer and the initial reconstruction layer according to the alignment loss value, and when the initial coding layer and the initial reconstruction layer meet the model training convergence condition, determine the converged initial coding layer and the converged initial reconstruction layer as the feature alignment processing layer of the initial video retrieval model; Performing alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and performing alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length; the feature alignment processing layer is used to perform alignment processing on the first fused feature vector and the second fused feature vector; Generate text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, adjust the model parameters of the initial video retrieval model according to the model loss value, and obtain a target video retrieval model when the initial video retrieval model meets the model training convergence condition; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

11. The method according to claim 10, characterized in that The initial video retrieval model includes an initial abstraction layer; extracting features from the sample video data to obtain a first visual tag vector includes: Perform feature extraction on the sample video data to obtain a sample visual tag vector, obtain an initial query vector randomly initialized in the initial abstract layer, and perform vector concatenation on the sample visual tag vector based on the initial query vector to obtain a first visual tag vector; the initial query vector is a model parameter in the initial abstract layer; the vector length of the initial query vector is the same as the vector length of the first visual tag vector; Among them, the visual abstraction layer of the target video retrieval model is obtained by training the initial abstraction layer, and the visual abstraction layer contains a compressed query vector obtained by training the initial query vector. The compressed query vector is used to perform attention processing on the video data to be matched to obtain a visual label vector with the same vector length as the compressed query vector.

12. A data processing device, characterized in that: include: A visual marker extraction module is used to extract features from the video data to be matched and obtain a compressed visual marker vector; A visual tag processing module is used to obtain business text information, and perform vector splicing on the business text information and the compressed visual tag vector to obtain a fused feature vector; the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics for characterizing the content description text of the video data to be matched; The visual mark processing module is further used to input the fused feature vector into a feature alignment processing layer of a target video retrieval model; The feature alignment processing layer includes an automatic encoding layer and a bottleneck reconstruction layer; the bottleneck reconstruction layer is obtained based on alignment length training, and the vector length output by the bottleneck reconstruction layer is fixed to the alignment length; The visual tag processing module is further used to encode the fused feature vector in the automatic encoding layer to obtain an encoded feature vector, downsample the encoded feature vector based on the time step to obtain a downsampled feature vector, and generate a to-be-matched text feature vector with a vector length of the alignment length based on the dependency of the downsampled feature vector in each time step in the bottleneck reconstruction layer; the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database; A text feature acquisition module, used to acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data; A similarity calculation module is used to generate text similarity between the business text feature vector and the text feature vector to be matched, and to determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

13. A data processing device, characterized in that: include: A visual marker extraction module is used to extract features from the video data to be matched and obtain a compressed visual marker vector; A visual tag processing module is used to obtain business text information, and perform vector splicing on the business text information and the compressed visual tag vector to obtain a fused feature vector; the business text information is a text prompt for indicating the content description text corresponding to the extracted video data, and the fused feature vector includes the semantics for characterizing the content description text of the video data to be matched; The visual mark processing module is further used to input the fused feature vector into a feature alignment processing layer of a target video retrieval model; The feature alignment processing layer includes a word frequency processing layer and a weighted alignment layer; The visual tag processing module is further used to generate, in the word frequency processing layer, a prediction probability corresponding to each word in the vocabulary based on the fused feature vector, generate a content description text of the to-be-matched video data based on the prediction probability, segment the content description text based on the alignment length to obtain F text segments, obtain the word frequencies corresponding to the F text segments in the content description text, and the inverse document frequencies corresponding to the F text segments in the content description text, and generate F text weights based on the F word frequencies and the F inverse document frequencies; the alignment length is F, and F is a positive integer; The visual mark processing module is further used to perform a dot multiplication operation on the F text weights and the fused feature vector in the weighted alignment layer to obtain F weighted feature vectors, and generate a to-be-matched text feature vector with a vector length of an alignment length based on the F weighted feature vectors; the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database; A text feature acquisition module, used to acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data; A similarity calculation module is used to generate text similarity between the business text feature vector and the text feature vector to be matched, and to determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

14. A data processing device, characterized in that: include: A visual marker extraction module is used to extract features from the video data to be matched and obtain a compressed visual marker vector; A visual tag processing module is used to obtain business text information, perform vector splicing on the business text information and the compressed visual tag vector to obtain a fused feature vector; the business text information is a text prompt for indicating the extraction of content description text corresponding to the video data, and the fused feature vector includes semantics for characterizing the content description text of the video data to be matched; the fused feature vector includes G unit prediction vectors, G is a positive integer; the vector lengths of the G unit prediction vectors are all alignment lengths, and the last unit prediction vector of the G unit prediction vectors includes the global semantics obtained by vector splicing of the business text information and the compressed visual tag vector; The visual tag processing module is further used to input the fused feature vector into the feature alignment processing layer of the target video retrieval model, and determine the last unit prediction vector of the G unit prediction vectors as a text feature vector to be matched whose vector length is the alignment length; the alignment length is equal to the vector length of the business text feature vector corresponding to the business video data in the video text feature database; A text feature acquisition module, used to acquire a business text feature vector corresponding to the business video data; the business text feature vector includes the semantics of the content description text used to characterize the business video data; A similarity calculation module is used to generate text similarity between the business text feature vector and the text feature vector to be matched, and to determine the business video data associated with the business text feature vector whose text similarity is greater than or equal to a text similarity threshold as a text retrieval video set; the text retrieval video set is used to determine similar video retrieval results for the video data to be matched.

15. A data processing device, characterized in that: include: A sample acquisition module, used to acquire sample video data and annotated video data, and input the sample video data and the annotated video data into an initial video retrieval model; A fusion vector generation module is used to perform feature extraction on the sample video data through the initial video retrieval model to obtain a first visual tag vector, obtain business text information, perform vector splicing on the business text information and the first visual tag vector to obtain a first fused feature vector, perform feature extraction on the annotated video data to obtain a second visual tag vector, and perform vector splicing on the business text information and the second visual tag vector to obtain a second fused feature vector; the business text information is a text prompt for indicating the extraction of video content description text; the first fused feature vector includes semantics for characterizing the content description text of the sample video data, and the second fused feature vector includes semantics for characterizing the content description text of the annotated video data; A vector alignment module, used to determine an alignment length based on the vector length of the first fused feature vector and the vector length of the second fused feature vector, and input the sample feature vector into a coding and decoding network; the coding and decoding network includes an initial coding layer, an initial reconstruction layer and an initial decoding layer; the length of the model parameter of the initial reconstruction layer is the alignment length; The vector alignment module is further used to encode the feature vector to be aligned in the initial coding layer to obtain an encoded feature vector, and downsample the encoded feature vector based on the time step to obtain a downsampled feature vector; The vector alignment module is further used to generate an alignment feature vector in the initial reconstruction layer based on the dependency relationship of the down-sampled feature vector in each time step, and to perform up-sampling processing on the alignment feature vector in the initial decoding layer to obtain an up-sampled feature vector; The vector alignment module is further used to generate an alignment loss value based on the feature vector to be aligned and the upsampled feature vector, adjust the model parameters of the initial coding layer and the initial reconstruction layer according to the alignment loss value, and when the initial coding layer and the initial reconstruction layer meet the model training convergence condition, determine the converged initial coding layer and the converged initial reconstruction layer as the feature alignment processing layer of the initial video retrieval model; The vector alignment module is further used to perform alignment processing on the first fused feature vector to obtain a first text feature vector whose vector length is the alignment length, and to perform alignment processing on the second fused feature vector to obtain a second text feature vector whose vector length is the alignment length; the feature alignment processing layer is used to perform alignment processing on the first fused feature vector and the second fused feature vector; A model adjustment module is used to generate text similarity between the first text feature vector and the second text feature vector, generate a model loss value based on the text similarity, and adjust the model parameters of the initial video retrieval model according to the model loss value. When the initial video retrieval model meets the model training convergence conditions, a target video retrieval model is obtained; the target video retrieval model is used to generate similar video retrieval results for the video data to be matched.

16. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-11.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.

18. A computer program product, characterized in that The computer program product comprises a computer program, which is stored in a computer-readable storage medium and is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.

Images