Video clip retrieval model generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a video clip retrieval model generation method and device, equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: carrying out the segmentation processing of a video clip, and obtaining a plurality of video blocks, video text information corresponding to the video blocks, and a priori box; performing feature coding processing on the plurality of video blocks and the corresponding video text information to generate video coding information and text coding information corresponding to the plurality of video blocks; constructing a first graph network structure; updating nodes in the first graph network structure according to the video coding information and the text coding information to obtain a second graph network structure; determining a loss function of the video clip retrieval model according to a label corresponding to the prior frame; and training the video clip retrieval model based on the second graph network structure and the loss function to obtain a target video clip retrieval model. The invention aims to solve the problem that a complete video cannot be accurately retrieved through a video clip in the prior art.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for generating a video clip retrieval model, a device for generating a video clip retrieval model, a computer device, and a computer-readable storage medium. Background Art

[0002] At present, the task of video clip retrieval generally refers to retrieving video clips related to the query video clip from a large-scale video database. It has a wide range of applications. For example, insurance companies can use text-based video clips to retrieve complete videos, thereby accelerating the insurance claim review process. For example, by inputting relevant video clips of the claim case, the system can automatically retrieve the complete video to verify the authenticity and legality of the claim. However, in the related technology, the complete video retrieved through video clips has poor reliability, low efficiency, and limited application scenarios. Summary of the invention

[0003] The present application provides a method for generating a video clip retrieval model, a device for generating a video clip retrieval model, a computer device and a computer-readable storage medium, aiming to solve the problem in related technologies that a complete video cannot be accurately retrieved through a video clip.

[0004] To achieve the above object, the present application provides a method for generating a video clip retrieval model, the method comprising:

[0005] Obtaining a video clip, and segmenting the video clip to obtain a plurality of video blocks and their corresponding video text information and a priori frames;

[0006] Performing feature coding processing on the multiple video blocks and the corresponding video text information to generate video coding information and text coding information corresponding to the multiple video blocks;

[0007] Using the priori frame as a first graph network node, using the video text information as a second graph network node, and constructing a first graph network structure according to the first graph network node and the second graph network node;

[0008] According to the video encoding information and the text encoding information, updating the nodes in the first graph network structure to obtain a second graph network structure;

[0009] Determining a loss function of the video clip retrieval model according to a label corresponding to the priori frame;

[0010] Based on the second graph network structure and the loss function, the video clip retrieval model is trained to obtain a target video clip retrieval model.

[0011] To achieve the above-mentioned purpose, the present application provides a device for generating a video clip retrieval model, and the device for generating a video clip retrieval model comprises:

[0012] A video segmentation module is used to obtain a video clip and perform segmentation processing on the video clip to obtain a plurality of video blocks and their corresponding video text information and a priori frames;

[0013] A feature coding module, used to perform feature coding processing on the plurality of video blocks and the corresponding video text information, and generate video coding information and text coding information corresponding to the plurality of video blocks;

[0014] A network structure construction module, used to use the priori frame as a first graph network node, use the video text information as a second graph network node, and construct a first graph network structure according to the first graph network node and the second graph network node;

[0015] A network structure updating module, used for updating the nodes in the first graph network structure according to the video encoding information and the text encoding information to obtain a second graph network structure;

[0016] A loss function determination module, used to determine the loss function of the video clip retrieval model according to the label corresponding to the priori frame;

[0017] A model training module is used to train the video clip retrieval model based on the second graph network structure and the loss function to obtain a target video clip retrieval model.

[0018] To achieve the above-mentioned purpose, the present application provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the method for generating a video clip retrieval model as described above when executing the computer program.

[0019] To achieve the above objectives, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for generating a video clip retrieval model as described above.

[0020] The generation method of the video clip retrieval model disclosed in the embodiment of the present application, the generation device of the video clip retrieval model, the computer device and the computer-readable storage medium, by obtaining the video clip, and segmenting the video clip to obtain multiple video blocks and their corresponding video text information and prior frames; feature encoding processing is performed on the multiple video blocks and their corresponding video text information to generate video encoding information and text encoding information corresponding to the multiple video blocks; the prior frame is used as the first graph network node, the video text information is used as the second graph network node, and the first graph network structure is constructed according to the first graph network node and the second graph network node; according to the video encoding information and the text encoding information, the nodes in the first graph network structure are updated to obtain the second graph network structure; according to the label corresponding to the prior frame, the loss function of the video clip retrieval model is determined; based on the second graph network structure and the loss function, the video clip retrieval model is trained to obtain the target video clip retrieval model. In this way, the relationship between the video blocks and the interactive relationship between the video and the text can be effectively captured, so that the complete video can be accurately retrieved through the video clip, the reliability and retrieval efficiency of the video retrieval are improved, and the application scenario is flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a scene schematic diagram of a method for generating a video clip retrieval model provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a method for generating a video clip retrieval model provided in an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of a scene for segmenting a video clip provided in an embodiment of the present application;

[0025] Figure 4 It is a structural schematic diagram of a first network structure provided in an embodiment of the present application;

[0026] Figure 5 is a schematic block diagram of a device for generating a video clip retrieval model provided in one embodiment of the present application;

[0027] Figure 6 It is a schematic block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0030] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] The present application proposes a method for generating a video clip retrieval model, a device for generating a video clip retrieval model, a computer device and a computer-readable storage medium, which can effectively capture the relationship between video blocks and the interactive relationship between video and text, so that the complete video can be accurately retrieved through the video clip, thereby improving the reliability and efficiency of video retrieval, and the application scenarios are flexible.

[0032] The method can be applied to a server, and of course can also be applied to a terminal device, to generate a corresponding image classification model, wherein the terminal device may include a fixed terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA), etc. The server can be, for example, a separate server or a server cluster. However, for ease of understanding, the following embodiment will be described in detail with a method for generating a video clip retrieval model applied to a server.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] like Figure 1 As shown, the method for generating a video clip retrieval model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. The application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 obtains a video clip, and performs segmentation processing on the video clip to obtain multiple video blocks and their corresponding video text information and a priori boxes; the a priori boxes are used as first graph network nodes, the video text information is used as second graph network nodes, and a first graph network structure is constructed according to the first graph network nodes and the second graph network nodes; feature encoding processing is performed on the multiple video blocks and their corresponding video text information to generate multiple video encoding information and text encoding information corresponding to the video blocks; according to the video encoding information and the text encoding information, the nodes in the first graph network structure are updated to obtain a second graph network structure; according to the label corresponding to the a priori box, the loss function of the video clip retrieval model is determined; based on the second graph network structure and the loss function, the video clip retrieval model is trained to obtain a target video clip retrieval model, and the target video clip retrieval model is sent to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 110 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0035] See also Figure 2 , Figure 2 This is a schematic flow chart of a method for generating a video clip retrieval model provided in an embodiment of the present application. The method for generating a video clip retrieval model can effectively capture the relationship between video blocks and the interactive relationship between video and text, so that the complete video can be accurately retrieved through the video clip, improving the reliability and efficiency of video retrieval, and the application scenario is flexible.

[0036] like Figure 2 As shown, the method for generating the video clip retrieval model includes steps S101 to S106.

[0037] S101: Obtain a video clip, and segment the video clip to obtain a plurality of video blocks and their corresponding video text information and a priori frames.

[0038] The video clip may be a video clip to be retrieved, a video clip may be a video clip corresponding to a person or animal to be claimed, or a video clip of other scenes. A video block may be one or more video frames in a video clip. Video text information may be text information contained in each video block. The prior frame may be an Anchor frame, which is a frame of several sizes and aspect ratios created based on a center point.

[0039] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0040] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0041] In some embodiments, according to the number of frames of the video segment, the video segment is segmented to obtain multiple video blocks; the multiple video blocks are subjected to text extraction to obtain video text information corresponding to the multiple video blocks; the multiple video blocks are clustered to obtain prior frames corresponding to the multiple video blocks. In this way, multiple video blocks and their corresponding video text information and prior frames can be accurately extracted from the video segment.

[0042] The video clip may be 30 frames or 60 frames per second. The video block may be one or more video frames in the video clip.

[0043] Specifically, the video segments can be segmented according to the number of frames in the video segments to obtain video images of each frame, and the video images are used as video blocks; text extraction is performed on multiple video blocks to extract text information contained in the multiple video blocks; and the multiple video blocks are clustered using the Kmeans clustering algorithm to obtain prior frames corresponding to the multiple video blocks.

[0044] like Figure 3 As shown, for example, the video V of N frames can be divided into N equally spaced b Based on all the grounding truth (GT) in the dataset, we use the Kmeans clustering algorithm to cluster N aAnchor frames of different lengths are preset for each video block, centered at the midpoint of the video block. a Anchor boxes, that is, there are N in total b ×N a Anchor box.

[0045] S102: Perform feature coding processing on the multiple video blocks and the corresponding video text information to generate video coding information and text coding information corresponding to the multiple video blocks.

[0046] The video encoding information may be the video representation information corresponding to the anchor box, that is, the node representation information corresponding to the first graph network node. The text encoding information may be the text representation information corresponding to the video text information, that is, the node representation information corresponding to the second graph network node.

[0047] In some embodiments, a vector conversion process is performed on a video block to obtain an embedding vector corresponding to the video block; the embedding vector is encoded to obtain video encoding information corresponding to the video block; the video text information is input into a pre-trained encoder to obtain text encoding information, and the feature dimensions of the video encoding information and the text encoding information are the same.

[0048] The embedding vector corresponding to the video block may be an embedding representation corresponding to the video block.

[0049] Specifically, each frame of the video V can be fed into the pre-trained ViT model to obtain the embedding vector corresponding to the video block, and the representation of the entire video clip can be obtained by splicing, and then further input into the stacked multi-layer convolutional neural network and Transformer encoder to obtain N b ×N a The video of the anchor box is represented as That is, video encoding information.

[0050] The ViT model divides the image into a series of patches, and then linearly embeds these patches into a high-dimensional space to generate a series of sequence embeddings. These sequence embeddings are added with position embeddings (used to provide spatial position information for each patch) and then sent to the standard Transformer encoder for processing. Finally, the output of the Transformer encoder is decoded through a classification head (fully connected layer) to obtain video encoding information.

[0051] Specifically, a text Q containing multiple words can be given and input into the Sentence-BERT pre-trained sentence encoder (PSE) to obtain the text representation X of the video text information.Q ∈R d , that is, text encoding information.

[0052] It should be noted that during the encoding process, the feature dimensions of the video representation and the text representation can be controlled to be the same so as to facilitate subsequent dot product operations.

[0053] S103: Use the priori frame as a first graph network node, use the video text information as a second graph network node, and construct a first graph network structure according to the first graph network node and the second graph network node.

[0054] Among them, the generated N b ×N a Anchor boxes are used as the first graph network nodes, the second graph network nodes can be global nodes, and the first graph network nodes and the second graph network nodes are used to construct the first graph network structure. The first graph network structure is an unupdated graph network structure.

[0055] In some embodiments, node information of the first graph network node and the second graph network node is obtained; based on the node information of the first graph network node, the position association information of each first graph network node is determined; based on the node information of the second graph network node, the interactive association information of the first graph network node and the second graph network node is determined; based on the position association information and the interactive association information, the first graph network node and the second graph network node are network constructed to generate a first graph network structure. Thus, the first graph network structure can be created by the first graph network node and the second graph network node.

[0056] Among them, the node information of the first graph network node may include the node representation, length and other information of the prior frame. The node information of the second graph network node may include the node representation of the video text information and other information. The position association information can be used to represent the position relationship of each first graph network node, such as adjacent, same row or same column position relationship. The interactive association information can be used to represent the information transfer relationship between the first graph network node and the second graph network node.

[0057] Specifically, the node representation and length information of the first graph network node are obtained; according to the node representation and length information of the first graph network node, the same row node, the same column node and the adjacent node corresponding to each first graph network node are determined, so that the position association information of each first graph network node can be determined. In this way, the first graph network node corresponding to the anchor box of different lengths can be adaptively selected, so as to better adapt to the characteristics of different video blocks.

[0058] like Figure 4 As shown, for example, since the graph network structure is regular N a ×N bThe nodes in each column are N nodes in the same video block. a The anchor boxes are sorted by length, and each row of nodes is N. b The Anchor boxes of the same length in the video blocks. Secondly, the edge E of the network needs to be defined, so the length relationship of the Anchor boxes and their corresponding position relationship at different times in the video block need to be considered. For each first-graph network node, the four first-graph network nodes above, below, left and right of the node can be regarded as adjacent nodes, that is, for a certain Anchor box, it will communicate with other Anchor boxes in the same video block and Anchor boxes of the same length in the previous and next video blocks.

[0059] Specifically, according to the node information of the second graph network node, the interactive association information between the first graph network node and the second graph network node is determined; according to the position association information and the interactive association information, the first graph network node and the second graph network node are network constructed to generate the first graph network structure. At this time, the second graph network node can be used as a global node, thereby effectively capturing the interaction between the two modalities, enabling each anchor box to integrate video text information, helping to solve the problem of information transmission between long-distance nodes, and improving the performance of video clip retrieval.

[0060] At the same time, the second graph network node can also facilitate multiple transmissions of node information. In the video clip retrieval task, there may be situations where it depends on nodes that are far away. For example, if a word like "the second time" appears in the text, the previous "first time" information is required. The second graph network node, that is, the global node, can act as a bridge to transmit information between nodes that are far away, so that richer and more complex graph representations can be learned, thereby generating interactive correlation information between the first graph network node and the second graph network node, so as to facilitate the construction of the first graph network structure.

[0061] S104. Update the nodes in the first graph network structure according to the video coding information and the text coding information to obtain a second graph network structure.

[0062] Among them, the second graph network structure is a graph network structure after updating the first graph network structure.

[0063] In some embodiments, according to the video coding information, the weight ratio of the first graph network node is determined; according to the weight ratio of the first graph network node, the first graph network node is updated; according to the text coding information, the updated first graph network node and the second graph network node are subjected to dot product processing to obtain the updated second graph network node; according to the updated first graph network node and the updated second graph network node, the second graph network structure is generated. Thus, the second graph network structure can be generated by updating the first graph network node and the second graph network node.

[0064] Specifically, the message passing in the network adopts GAT, which introduces the attention mechanism, allowing each node to dynamically assign weights according to the importance of its neighbors, so that the model can capture the complex relationships between different nodes. Therefore, the weight ratio of the first graph network nodes can be determined by video encoding information.

[0065] Exemplarily, GAT can be used to update the node representation:

[0066]

[0067] in, is the node representation, X V is the video representation of the Anchor box corresponding to the network node v in the first figure, as the initial representation of the node, for k = 1, 2, ..., K,

[0068]

[0069] Among them, W (k ) is the weight constant, α (k) By the attention mechanism A (k ), normalized so that the sum of all neighbors of each node is 1:

[0070]

[0071] It should be noted that the information transmission with the adjacent nodes here is only in the Anchor box node set V, that is, the first graph network node, and does not include the second graph network node, that is, the global node O.

[0072] Specifically, in order to interact with the text, each node representation must be compared with the representation of the second graph network node O Perform a dot product operation to update the node representation again:

[0073]

[0074] Here, β is a hyperparameter that controls the influence of the second graph network node on the update of each first graph network node.

[0075] The representation update of the second graph network node O is also based on the attention GAT. Since it connects all other nodes in the network, the adjacent nodes are all v∈V. The formula is as follows:

[0076]

[0077] Among them, X Q is the text representation of the video text information, as the initial representation of the second graph network node O, for k = 12 ... K

[0078]

[0079] Among them, W (k) is the weight constant, α (k) By the attention mechanism A (k) For generation, please refer to the above embodiment.

[0080] Through the above formula, the weight ratio of the first graph network node can be allocated through video coding information to update the first graph network node; then, according to the text coding information, the dot product processing is performed on the updated first graph network node and the second graph network node to obtain the updated second graph network node; finally, the updated first graph network node and the updated second graph network node are used to generate the second graph network structure.

[0081] S105. Determine the loss function of the video clip retrieval model according to the label corresponding to the priori box.

[0082] The label corresponding to the prior frame may include a confidence label, a midpoint position label, and a length label corresponding to the prior frame. The video clip retrieval model is an untrained video clip retrieval model, and the effect may not be optimal. The loss function of the video clip retrieval model can be used to train the video clip retrieval model to obtain the target video clip retrieval model with the best effect.

[0083] In some embodiments, before determining the loss function of the video clip retrieval model based on the label corresponding to the priori box, the node representation information, midpoint position information and length information corresponding to each priori box are obtained; the confidence label is constructed based on the node representation information corresponding to each priori box; the midpoint position label is constructed based on the midpoint position information corresponding to each priori box; and the length label is constructed based on the length information corresponding to each priori box. In this way, the confidence label, midpoint position label and length label can be accurately constructed.

[0084] Among them, the node representation information can be the video representation corresponding to the priori box, the midpoint position information can be the midpoint position coordinates corresponding to the priori box, and the length information can be the border length corresponding to the priori box.

[0085] Specifically, the nodes of the last layer of the first graph network nodes can be represented by v∈V is concatenated to get the final multimodal representation X, and the confidence C′ is generated through the convolutional layer:

[0086] C′=σ(Conv(X))∈(0,1)

[0087] Among them, σ is the sigmoid function.

[0088] Specifically, the midpoint position and length prediction are deviations, which are based on the difference between the ground truth and the anchor box, rather than directly using the ground truth position, making subsequent regression training easier. The deviation is predicted by another convolutional layer:

[0089] (t′ M ,t′ L )=Conv(X)

[0090] Therefore, for N b ×N a Anchor boxes can be used to construct confidence labels C through node representation information and midpoint position labels t through midpoint position information. M , construct the length label t through the length information L .

[0091] In some embodiments, the categories corresponding to the confidence label, the midpoint position label, and the length label are determined; and the loss function of the video segment retrieval model is determined based on the confidence label, the midpoint position label, the length label, and their corresponding categories. Thus, the loss function of the video segment retrieval model can be accurately determined.

[0092] Among them, the categories corresponding to the confidence label, midpoint position label and length label can be divided into positive example labels and negative example labels respectively.

[0093] Specifically, the confidence label C can be determined as a positive label or a negative label based on the interaction ratio IOU between the anchor box and the GT. If the interaction ratio IOU is greater than a preset threshold, the confidence label C is regarded as a positive label, that is, P = 1. If the interaction ratio IOU is not greater than the preset threshold, the confidence label C is regarded as a negative label, that is, P = 0. The midpoint position label t M and length label and t L The difference calculation can be used to determine whether the corresponding label is a positive label or a negative label.

[0094] Specifically, the confidence uses cross entropy as the loss function, t M ,t L The mean square error (MSE) is used as the loss function. The loss function formula of the algorithm is as follows:

[0095]

[0096] Among them, λ box , obj , noobj Is a weight constant, if it is a positive label, Then output 1, if it is a negative example label, Then output 0; if it is a positive label, Then output 0; if it is a negative example label, Then output 1.

[0097] S106. Based on the second graph network structure and the loss function, the video clip retrieval model is trained to obtain a target video clip retrieval model.

[0098] Among them, the target video clip retrieval model is a trained video clip retrieval model, which has the best effect in video clip retrieval.

[0099] Specifically, the video clip retrieval model can be trained and automatically machine learned through the second graph network structure and loss function to obtain a target video clip retrieval model.

[0100] The method for generating a video clip retrieval model disclosed in the embodiment of the present application obtains a video clip, and performs segmentation processing on the video clip to obtain multiple video blocks and their corresponding video text information and prior frames; performs feature encoding processing on multiple video blocks and their corresponding video text information to generate video encoding information and text encoding information corresponding to multiple video blocks; uses the prior frame as a first graph network node, uses the video text information as a second graph network node, and constructs a first graph network structure based on the first graph network node and the second graph network node; updates the nodes in the first graph network structure based on the video encoding information and the text encoding information to obtain the second graph network structure; determines the loss function of the video clip retrieval model based on the label corresponding to the prior frame; and trains the video clip retrieval model based on the second graph network structure and the loss function to obtain the target video clip retrieval model. In this way, the relationship between video blocks and the interactive relationship between video and text can be effectively captured, so that the complete video can be accurately retrieved through the video clip, the reliability and efficiency of video retrieval can be improved, and the application scenario is flexible.

[0101] See also Figure 5 , Figure 5 It is a schematic block diagram of a device for generating a video clip retrieval model provided in one embodiment of the present application. The device for generating a video clip retrieval model can be configured in a server to execute the aforementioned method for generating a video clip retrieval model.

[0102] like Figure 5 As shown, the device 200 for generating the video clip retrieval model includes: a video segmentation module 201, a feature encoding module 202, a network structure construction module 203, a network structure update module 204, a loss function determination 205 and a module model training module 206.

[0103] The video segmentation module 201 is used to obtain a video segment and perform segmentation processing on the video segment to obtain a plurality of video blocks and their corresponding video text information and prior frames;

[0104] A feature coding module 202, configured to perform feature coding processing on the plurality of video blocks and the corresponding video text information, and generate video coding information and text coding information corresponding to the plurality of video blocks;

[0105] A network structure construction module 203, used to use the priori frame as a first graph network node, use the video text information as a second graph network node, and construct a first graph network structure according to the first graph network node and the second graph network node;

[0106] A network structure updating module 204, configured to update the nodes in the first graph network structure according to the video encoding information and the text encoding information to obtain a second graph network structure;

[0107] A loss function determination module 205, configured to determine a loss function of the video clip retrieval model according to a label corresponding to the priori frame;

[0108] The model training module 206 is used to train the video clip retrieval model based on the second graph network structure and the loss function to obtain a target video clip retrieval model.

[0109] In some embodiments, the video segmentation module 201 is also used to segment the video segment according to the frame number of the video segment to obtain multiple video blocks; perform text extraction on the multiple video blocks to obtain video text information corresponding to the multiple video blocks; perform clustering on the multiple video blocks to obtain prior frames corresponding to the multiple video blocks.

[0110] In some embodiments, the feature encoding module 202 is also used to perform vector conversion processing on the video block to obtain an embedding vector corresponding to the video block; perform encoding processing on the embedding vector to obtain video encoding information corresponding to the video block; input the video text information into a pre-trained encoder to obtain text encoding information, and the feature dimensions of the video encoding information and the text encoding information are the same.

[0111] In some embodiments, the network structure construction module 203 is also used to obtain the node information of the first graph network node and the second graph network node; determine the position association information of each of the first graph network nodes based on the node information of the first graph network node; determine the interaction association information between the first graph network node and the second graph network node based on the node information of the second graph network node; and perform network construction on the first graph network node and the second graph network node based on the position association information and the interaction association information to generate a first graph network structure.

[0112] In some embodiments, the network structure update module 204 is also used to determine the weight ratio of the first graph network node according to the video encoding information; update the first graph network node according to the weight ratio of the first graph network node; perform dot product processing on the updated first graph network node and the second graph network node according to the text encoding information to obtain an updated second graph network node; generate a second graph network structure based on the updated first graph network node and the updated second graph network node.

[0113] In some embodiments, the generation device 200 of the video clip retrieval model also includes a label determination module 207, which is used to obtain the node representation information, midpoint position information and length information corresponding to each of the prior boxes; construct a confidence label according to the node representation information corresponding to each of the prior boxes; construct a midpoint position label according to the midpoint position information corresponding to each of the prior boxes; and construct a length label according to the length information corresponding to each of the prior boxes.

[0114] In some embodiments, the loss function determination module 205 is also used to determine the categories corresponding to the confidence label, midpoint position label and length label; and determine the loss function of the video clip retrieval model based on the confidence label, midpoint position label and length label and their corresponding categories.

[0115] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0117] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.

[0118] See also Figure 6 , Figure 6 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0119] like Figure 6 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0120] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for generating a video segment retrieval model.

[0121] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0122] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for generating a video clip retrieval model.

[0123] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0125] Among them, in some embodiments, the processor is used to run a computer program stored in the memory to implement the following steps: obtain a video clip, and segment the video clip to obtain multiple video blocks and their corresponding video text information and prior frames; perform feature encoding processing on the multiple video blocks and their corresponding video text information to generate multiple video encoding information and text encoding information corresponding to the video blocks; use the prior frame as a first graph network node, use the video text information as a second graph network node, and construct a first graph network structure based on the first graph network node and the second graph network node; update the nodes in the first graph network structure according to the video encoding information and the text encoding information to obtain a second graph network structure; determine the loss function of the video clip retrieval model according to the label corresponding to the prior frame; train the video clip retrieval model based on the second graph network structure and the loss function to obtain a target video clip retrieval model.

[0126] In some embodiments, the processor is further used to segment the video segment according to the number of frames of the video segment to obtain multiple video blocks; perform text extraction on the multiple video blocks to obtain video text information corresponding to the multiple video blocks; and perform clustering on the multiple video blocks to obtain prior frames corresponding to the multiple video blocks.

[0127] In some embodiments, the processor is further used to perform vector conversion processing on the video block to obtain an embedding vector corresponding to the video block; perform encoding processing on the embedding vector to obtain video encoding information corresponding to the video block; input the video text information into a pre-trained encoder to obtain text encoding information, and the video encoding information and the text encoding information have the same feature dimension.

[0128] In some embodiments, the processor is also used to obtain node information of the first graph network node and the second graph network node; determine position association information of each of the first graph network nodes based on the node information of the first graph network node; determine interaction association information between the first graph network node and the second graph network node based on the node information of the second graph network node; and perform network construction on the first graph network node and the second graph network node based on the position association information and the interaction association information to generate a first graph network structure.

[0129] In some embodiments, the processor is also used to determine the weight ratio of the first graph network node based on the video encoding information; update the first graph network node according to the weight ratio of the first graph network node; perform dot product processing on the updated first graph network node and the second graph network node according to the text encoding information to obtain an updated second graph network node; generate a second graph network structure based on the updated first graph network node and the updated second graph network node.

[0130] In some embodiments, the processor is also used to obtain node representation information, midpoint position information and length information corresponding to each of the prior boxes; construct a confidence label based on the node representation information corresponding to each of the prior boxes; construct a midpoint position label based on the midpoint position information corresponding to each of the prior boxes; and construct a length label based on the length information corresponding to each of the prior boxes.

[0131] In some embodiments, the processor is also used to determine the categories corresponding to the confidence label, midpoint position label and length label; and determine the loss function of the video clip retrieval model based on the confidence label, midpoint position label and length label and their corresponding categories.

[0132] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any method for generating a video clip retrieval model provided in the embodiment of the present application is implemented.

[0133] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0134] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0135] The present invention refers to a new application model of computer technologies such as storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. of the blockchain language model. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a video clip retrieval model, characterized in that: The method comprises: Obtaining a video clip, and segmenting the video clip to obtain a plurality of video blocks and their corresponding video text information and a priori frames; Performing feature coding processing on the multiple video blocks and the corresponding video text information to generate video coding information and text coding information corresponding to the multiple video blocks; Using the priori frame as a first graph network node, using the video text information as a second graph network node, and constructing a first graph network structure according to the first graph network node and the second graph network node; According to the video encoding information and the text encoding information, updating the nodes in the first graph network structure to obtain a second graph network structure; Determining a loss function of the video clip retrieval model according to a label corresponding to the priori frame; Based on the second graph network structure and the loss function, the video clip retrieval model is trained to obtain a target video clip retrieval model.

2. The method according to claim 1, characterized in that The segmenting process of the video clip to obtain a plurality of video blocks and their corresponding video text information and prior frames includes: Segmenting the video segment according to the number of frames of the video segment to obtain a plurality of video blocks; Performing text extraction processing on the multiple video blocks to obtain video text information corresponding to the multiple video blocks; Clustering is performed on the multiple video blocks to obtain prior frames corresponding to the multiple video blocks.

3. The method according to claim 1, characterized in that The performing feature coding processing on the plurality of video blocks and the corresponding video text information to generate the video coding information and text coding information corresponding to the plurality of video blocks includes: Performing vector conversion processing on the video block to obtain an embedding vector corresponding to the video block; Encoding the embedded vector to obtain video encoding information corresponding to the video block; The video text information is input into a pre-trained encoder to obtain text encoding information, and the feature dimensions of the video encoding information and the text encoding information are the same.

4. The method according to claim 1, characterized in that: The constructing a first graph network structure according to the first graph network node and the second graph network node includes: Obtaining node information of the first graph network node and the second graph network node; Determining location association information of each of the first graph network nodes according to the node information of the first graph network nodes; Determining, according to the node information of the second graph network node, interactive association information between the first graph network node and the second graph network node; According to the position association information and the interaction association information, a network is constructed for the first graph network node and the second graph network node to generate a first graph network structure.

5. The method according to claim 1, characterized in that The updating of nodes in the first graph network structure according to the video encoding information and the text encoding information to obtain a second graph network structure includes: Determining weight ratios of the first graph network nodes according to the video encoding information; Updating the first graph network nodes according to the weight ratios of the first graph network nodes; According to the text encoding information, performing dot product processing on the updated first graph network node and the second graph network node to obtain an updated second graph network node; A second graph network structure is generated according to the updated first graph network nodes and the updated second graph network nodes.

6. The method according to claim 1, characterized in that Before determining the loss function of the video clip retrieval model according to the label corresponding to the priori frame, the method further includes: Obtaining node representation information, midpoint position information and length information corresponding to each of the priori boxes; Constructing confidence labels according to the node representation information corresponding to each of the prior boxes; Constructing a midpoint position label according to the midpoint position information corresponding to each of the prior frames; A length label is constructed according to the length information corresponding to each of the prior frames.

7. The method according to claim 1, characterized in that The label includes a confidence label, a midpoint position label and a length label, and determining the loss function of the video clip retrieval model according to the label corresponding to the priori box includes: Determine the categories corresponding to the confidence label, midpoint position label, and length label; A loss function of the video clip retrieval model is determined based on the confidence label, the midpoint position label and the length label and their corresponding categories.

8. A device for generating a video clip retrieval model, characterized in that: The generating device of the video clip retrieval model comprises: A video segmentation module is used to obtain a video clip and perform segmentation processing on the video clip to obtain a plurality of video blocks and their corresponding video text information and a priori frames; A feature coding module, used to perform feature coding processing on the plurality of video blocks and the corresponding video text information, and generate video coding information and text coding information corresponding to the plurality of video blocks; A network structure construction module, used to use the priori frame as a first graph network node, use the video text information as a second graph network node, and construct a first graph network structure according to the first graph network node and the second graph network node; A network structure updating module, used for updating the nodes in the first graph network structure according to the video encoding information and the text encoding information to obtain a second graph network structure; A loss function determination module, used to determine the loss function of the video clip retrieval model according to the label corresponding to the priori frame; A model training module is used to train the video clip retrieval model based on the second graph network structure and the loss function to obtain a target video clip retrieval model.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement, when executing the computer program: A method for generating a video clip retrieval model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for generating a video clip retrieval model according to any one of claims 1 to 7.

Images