Video clip processing method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a video clip processing method and device, electronic equipment and a medium. The method specifically comprises the following steps: receiving query information input by a user; querying a target video clip matched with the query information in a database according to the tag content of a preset dimension tag corresponding to a video clip in the database; and outputting the target video clip. Wherein the label content of the preset dimension label is the content output by a large language model according to the video description information, the picture description information and the text data corresponding to the video clip; the video description information is information obtained according to overall video data corresponding to a video clip; the picture description information is information obtained according to frame picture data and face picture data corresponding to the video clip. According to the embodiment of the invention, the information coverage degree and the precision degree of the preset dimension label content can be improved, so that the accuracy of a retrieval result can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video technology, and in particular to a method, device, electronic device and medium for processing video clips. Background Art

[0002] With the development of video technology, the number of videos has exploded. In order to quickly retrieve the target video clips of interest from the video, video clip retrieval technology has been widely used. The current video clip retrieval method usually obtains the target video clips matching the query string from the video resource library as the retrieval result based on the query string input by the user and the video tags of the video clips in the video resource library.

[0003] In the related art, the process of determining a video tag specifically includes: first, extracting frames from a video clip; then performing dialogue extraction and character recognition on the extracted video frames; and finally, obtaining a video tag for the video clip based on the extracted dialogue and recognized characters.

[0004] In practical applications, multiple video frames in a video clip are coherent and together present complete video content. However, in the process of determining the video tags of video clips, related technologies only focus on local information such as lines and characters contained in discrete video frames; therefore, the video tags generated by related technologies lack an overall understanding of the video clips, resulting in incomplete information coverage and inaccurate characterization. When the above-mentioned video tags are applied to video clip retrieval, it is difficult for the video clip retrieval system to accurately locate the target video clip that meets the user's needs based on these video tags, which affects the accuracy of the retrieval results. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and medium for processing video clips, which can improve the information coverage and accuracy of preset dimensional tag content, thereby improving the accuracy of retrieval results.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect of the present invention, a method for processing a video clip is provided, the method comprising:

[0008] Receive query information input by the user;

[0009] According to the tag content of the preset dimension tag corresponding to the video clip in the database, searching the database for a target video clip matching the query information;

[0010] Outputting the target video segment;

[0011] Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

[0012] In a second aspect of the present invention, a video clip processing device is provided, the device comprising:

[0013] A receiving module, used for receiving query information input by a user;

[0014] A query module, used to query a target video segment matching the query information in the database according to the tag content of the preset dimension tag corresponding to the video segment in the database;

[0015] An output module, used for outputting the target video segment;

[0016] Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

[0017] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0018] Memory, used to store computer programs;

[0019] The processor is used to implement the aforementioned method steps when executing the program stored in the memory.

[0020] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above method.

[0021] In a fifth aspect of the implementation of the present invention, a computer program product is also provided, including a computer program / instruction, which implements the above method when executed by a processor.

[0022] The video clip processing method, device, electronic device and medium provided by the embodiment of the present invention provide the video description information corresponding to the overall video data, the picture description information and text data corresponding to the frame picture data and the face picture data to the large language model, and the large language model outputs the label content of the preset dimension label. In addition, the embodiment of the present invention applies the label content of the preset dimension label to the retrieval process of the video clip. Specifically, according to the label content of the preset dimension label corresponding to the video clip in the database, the target video clip matching the query information is queried in the database, and the target video clip is output.

[0023] Since the video description information is obtained based on the overall video data corresponding to the video clip, it can cover the overall characteristics of the video clip. Since the image description information is obtained based on the frame image data and the face image data, it can cover the picture elements and character features of the frame image. On this basis, the large language model can fuse and refine the video description information, the image description information, and the semantic information in the text data, so that the output preset dimensional label content can reflect the overall characteristics of the video clip. Therefore, the embodiments of the present invention can improve the information coverage and accuracy of the preset dimensional label content.

[0024] While being able to improve the information coverage and accuracy of the preset dimensional label content, the embodiments of the present invention can effectively exclude video clips that are irrelevant or have low relevance to the query information based on more comprehensive and accurate label content, thereby improving the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0026] Figure 1 A flowchart of a method for processing video clips according to an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a flow chart of a method for obtaining video clips according to an embodiment of the present invention;

[0028] Figure 3 A flowchart of a method for obtaining overall video data, frame image data, face image data and text data of a video clip according to an embodiment of the present invention;

[0029] Figure 4 A schematic flow chart of a method for processing video clips according to an embodiment of the present invention;

[0030] Figure 5 A flowchart of a method for processing video clips according to an embodiment of the present invention;

[0031] Figure 6 A schematic diagram of the structure of a video clip processing device according to an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the structure of a video clip processing device according to an embodiment of the present invention;

[0033] Figure 8 The structure block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0035] The video clip processing method according to the embodiment of the present invention can be used to generate comprehensive and accurate video tags. The video tags can be used to search for video clips to improve the accuracy of the search results of the video clips.

[0036] The embodiments of the present invention can be applied to application scenarios such as film and television production, intelligent video surveillance analysis, and personalized recommendations on video platforms. For example, in the film and television production scenario, the target video clips obtained based on the video clip retrieval technology can serve as a guide for shooting. In the intelligent video surveillance analysis scenario, the video clip retrieval technology can be used to quickly locate the target video clip where a specific event occurs, such as accurately finding the video clip where a suspicious person appears in security monitoring, providing strong support for security protection. In the personalized recommendation scenario of the video platform, through the precise analysis and label generation of video clips, a collection of video clips that better suits the user's interests and preferences is pushed to the user, thereby improving the user experience. It can be understood that the embodiments of the present invention do not limit the specific application scenarios corresponding to the video clip processing method.

[0037] In the related art, the process of determining a video tag specifically includes: first, extracting frames from a video clip; then performing dialogue extraction and character recognition on the extracted video frames; and finally, obtaining a video tag for the video clip based on the extracted dialogue and recognized characters.

[0038] However, the video tags generated by related technologies lack an overall understanding of the video clips, resulting in incomplete information coverage and inaccurate characterization.

[0039] In view of the technical problems that the information coverage of video tags in the related art is not comprehensive and the description is not accurate enough, an embodiment of the present invention provides a method for processing video clips, which specifically includes:

[0040] Receive query information input by the user;

[0041] According to the tag content of the preset dimension tag corresponding to the video clip in the database, searching the database for a target video clip matching the query information;

[0042] Outputting the target video segment;

[0043] Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

[0044] The embodiment of the present invention provides the video description information corresponding to the overall video data, the picture description information corresponding to the frame picture data and the face picture data, and the text data to the large language model, and the large language model outputs the label content of the preset dimension label. In addition, the embodiment of the present invention applies the label content of the preset dimension label to the retrieval process of the video clip. Specifically, according to the label content of the preset dimension label corresponding to the video clip in the database, the target video clip matching the query information is searched in the database, and the target video clip is output.

[0045] Since the video description information is obtained based on the complete video data corresponding to the video clip, it can cover the overall characteristics of the video clip. Since the picture description information is obtained based on the frame picture data and the face picture data, it can cover the picture elements and character features of the frame picture. On this basis, the large language model can fuse and refine the video description information, picture description information and semantic information in the text data, so that the output preset dimensional label content can reflect the overall characteristics of the video clip. Therefore, the embodiments of the present invention can improve the information coverage and accuracy of the preset dimensional label content.

[0046] While being able to improve the information coverage and accuracy of the preset dimensional label content, the embodiments of the present invention can effectively exclude video clips that are irrelevant or have low relevance to the query information based on more comprehensive and accurate label content, thereby improving the accuracy of the retrieval results.

[0047] The embodiments of the present invention are described below through specific examples.

[0048] Reference Figure 1 , shows a flowchart of a method for processing a video clip according to an embodiment of the present invention. The method may specifically include the following steps:

[0049] Step 101, obtaining complete video data, frame image data, face image data and text data corresponding to the video clip;

[0050] Step 102: Determine the video description information corresponding to the complete video data using a video understanding model;

[0051] Step 103: using the image understanding model, determine the image description information corresponding to the frame image data and the face image data;

[0052] Step 104: input the video description information, the image description information and the text data into a large language model, and the large language model outputs label content of a preset dimensional label.

[0053] Figure 1 The illustrated method embodiment can be used to generate comprehensive and accurate video tags for video clips. Figure 1 The method embodiment shown can be executed by a client or a server. It can be understood that the embodiment of the present invention is for Figure 1 The specific execution subject of the illustrated method embodiment is not limited.

[0054] In step 101, the video clip refers to a portion of content cut from the original video with a clear start time and end time.

[0055] The embodiment of the present invention can use the video stripping technology to extract video segments from the original video. Video stripping refers to the process of dividing the original video according to the time axis to divide the video into multiple video segments.

[0056] In one implementation, the original video may be subjected to transition point analysis to obtain transition point information corresponding to the original video, and the original video may be split into a plurality of video segments according to the transition point information.

[0057] Reference Figure 2 , showing a flow chart of a method for obtaining video clips according to an embodiment of the present invention, wherein the original video can firstly be subjected to transition point analysis to obtain transition point information corresponding to the original video; then, according to the transition point information, the original video can be stripped to split the original video into multiple video clips: video clip 1, video clip 2 and video clip 3.

[0058] Transition is the transition between different scenes and sections in a video. Common transition methods include fade in and fade out, flash to white, and wipe. By analyzing the transition points, we can clearly distinguish the boundaries between different plots and different time and space scenes in the video. For example, in a TV series, the switch from an indoor dialogue scene to an outdoor chase scene is a transition point. Accurately identifying such transition points is crucial for accurately splitting video clips.

[0059] The embodiment of the present invention may adopt the following transition point analysis method:

[0060] The first is the transition point analysis method based on pixel difference analysis. The principle is that when the video is playing normally, the pixel difference between adjacent frames is relatively stable, but when the transition occurs, the pixel difference will suddenly increase, or in the gradual transition, the pixels will gradually change according to a certain rule. The transition point is determined by calculating the pixel difference between adjacent frames and setting a threshold.

[0061] The second is the transition point analysis method based on feature vector analysis. The principle is to first extract the feature vectors of the video frame (such as color features, texture features, etc.). Under normal circumstances, the difference between the feature vectors of adjacent frames is small, and the feature vectors will change significantly during the transition. The transition point is determined by comparing the difference between the feature vectors of adjacent frames and combining the threshold.

[0062] The third method is based on deep learning. The principle is to use a large amount of video frame data with good transition and non-transition labels to train the neural network model. The neural network model learns the characteristic pattern of transition, extracts and judges the features of new video frames, and determines the transition point based on the output probability value.

[0063] The above implementation method of splitting the original video into multiple video segments according to the transition point information can improve the plot integrity and scene coherence of the video segments, and can also make the video segments better focus on the subject.

[0064] Of course, the above implementation of splitting the original video into multiple video segments according to the transition point information is only an example of the video stripping technology. In fact, those skilled in the art can adopt other video stripping technologies according to actual application requirements.

[0065] For example, other video stripping technologies may include: video stripping technology based on time intervals and video stripping technology based on semantic analysis.

[0066] Among them, the time interval-based video stripping technology pre-sets fixed time intervals, such as every 5 minutes or 10 minutes, and cuts them sequentially along the timeline of the original video.

[0067] Video segmentation technology based on semantic analysis uses natural language processing related tools and algorithms to first perform semantic understanding of the text information involved in the original video, analyze the semantic units such as different themes and events reflected in the text information, and then divide the video clips according to the time intervals when these semantic units appear in the original video.

[0068] The embodiment of the present invention does not limit the specific video stripping technology. The above text carries the description information of the target image (image description information), and its role is to provide clear guidance and reference content for the generation of the original composition. Therefore, the above text can be called image description text.

[0069] Those skilled in the art can determine the duration of the video clip according to actual application requirements. For example, the duration of the video clip can be N seconds, where N is a positive integer, and its value can be determined by those skilled in the art. For example, N can be 1, 5, and the like.

[0070] The complete video data corresponding to the video clip may refer to all the information covered by the video clip. The complete video may be understood as the dynamic content presented by playing consecutive frame images in sequence at a certain frame rate and combining other elements such as audio.

[0071] In a specific implementation, the process of obtaining the frame image data and face image data corresponding to the video clip includes:

[0072] Step A1, extracting frames from the video clip to obtain frame image data contained in the video clip;

[0073] Step A2: performing face recognition on the frame image data to obtain face image data contained in the frame image data.

[0074] In step A1, frame image data can be extracted from the video clip at a preset time interval. For example, when the preset time interval is set to 0.2 seconds, within the duration of the video clip, a frame image is extracted from the picture sequence of the video clip at an accurate location every 0.2 seconds, thereby gradually gathering into the required frame image data set.

[0075] In step A2, the role of face recognition technology is to locate the area where the face is located in the frame image data, distinguish it from other elements such as the frame image background by analyzing the facial features, and then extract the images of these face areas as face image data.

[0076] The process of obtaining text data corresponding to the video clip specifically includes:

[0077] Step B1: performing speech recognition on the audio in the video clip to obtain text data; and / or

[0078] Step B2: Perform text recognition on the frame image data in the video clip to obtain text data.

[0079] In step B1, speech recognition technology is used to convert the audio in the video clip into text. In the process of converting the audio in the video clip into text, the speech recognition technology can analyze the acoustic features of the voice in the audio, such as phonemes, syllables, intonation, and speech speed, and convert these acoustic features into corresponding text using a pre-trained model and algorithm.

[0080] The text data corresponding to the audio may specifically include: dialogue text, ambient sound effect text or narration text, etc. The dialogue text may include: speaker information. It is understood that the embodiment of the present invention does not limit the specific text data corresponding to the audio.

[0081] In step B2, text recognition technology is used to convert frame image data in the video clip into text.

[0082] In one example, the process of converting frame image data in a video clip into text specifically includes: first, the frame image is preprocessed, such as adjusting the contrast, brightness, clarity, etc. of the image to make the text in the image clearer and more recognizable; then, the image segmentation algorithm is used to locate the text area and extract the features of these text areas, such as the stroke structure, font shape, character spacing, etc. of the text; and then, based on a pre-trained character recognition model, these text features are converted into corresponding text.

[0083] The text data corresponding to the frame image may specifically include: dialogue text, character appearance information (such as character name, identity identification, etc.), identification text in the scene (such as place name identification, product name, store sign, etc.), etc. It can be understood that the embodiment of the present invention does not limit the specific text data corresponding to the frame image.

[0084] Reference Figure 3 , shows a flow chart of a method for acquiring complete video data, frame image data, face image data and text data of a video clip according to an embodiment of the present invention.

[0085] The embodiment of the present invention can obtain video data, which specifically includes: frame images arranged in time, and audio matching the frame images, etc.

[0086] The embodiment of the present invention can also extract frames from the video clip to obtain frame image data contained in the video clip; and perform face recognition on the frame image data to obtain face image data contained in the frame image data.

[0087] The embodiment of the present invention can also perform speech recognition on the audio in the video clip to obtain the second text data, and perform text recognition on the frame image data in the video clip to obtain the first text data.

[0088] In step 102, the video understanding model may be used to analyze the complete video data corresponding to the entire video clip, and the obtained video description information may cover the overall features of the video clip.

[0089] The video understanding model is an artificial intelligence model that can process and deeply analyze video data. It can receive the complete video data corresponding to the video clip, including continuous frame images and audio information, and automatically extract various video features in the video through a series of complex algorithms and neural network architectures; and the video understanding model can understand the content of the video based on the extracted video features, thereby generating description information for the overall features of the video clip as video description information. Therefore, the video description information of the present invention specifically represents: description information for the overall features of the video clip.

[0090] Examples of video understanding models may include: GPT-V (Generative Pretrained Transformer-for Videos), etc. It can be understood that the embodiments of the present invention are not limited to specific video understanding models.

[0091] In one example, the video description information of the present invention specifically includes: theme-related features of the video clip, style-related features of the video clip, and plot-related features of the video clip, etc. It can be understood that the embodiment of the present invention does not limit the specific video description information.

[0092] In step 103, the image understanding model may be used to analyze the frame image data and the face image data, and the obtained image description information may cover the picture features and character features of the frame image.

[0093] The picture understanding model is an artificial intelligence model that can process and deeply analyze picture data (including frame picture data and face picture data). After receiving these picture data, it first extracts the picture features of the picture data, which specifically include: the picture features corresponding to the frame picture data, and the character features such as age, gender, clothing, and action corresponding to the character picture data; then, based on the above picture features and character features, it generates picture description information for the frame picture and face picture as picture description information.

[0094] Therefore, the picture description information of the present invention specifically includes: picture content description and character action description and other information.

[0095] In one example, assuming that frame image A is used to describe "two people fighting on the street", frame image A contains: character image data of character A, and character image data of character B; then the picture content description may include: scene environment, related objects, character appearance and position and other information; the character action description may include: character's body movements or visual movements, etc.

[0096] An example of a screen content description is as follows:

[0097] Scene environment: street road with shops and street lights on both sides.

[0098] Related objects: There are scattered debris on the ground and a bicycle parked on the side of the road.

[0099] Appearance and Position: Two adult males, wearing ordinary T-shirts and trousers, standing opposite each other, relatively close to each other.

[0100] Examples of character action descriptions are as follows:

[0101] Person 1: Throws a punch at the opponent, leans forward, and puts his other hand in a defensive posture.

[0102] Character 2: He moves sideways to avoid the punch, raises his leg to counterattack, and keeps his eyes on his opponent.

[0103] Examples of image understanding models may include: GPT-4O (Omnipotent Generative Pretrained Transformer 4), etc. It is understood that the embodiments of the present invention do not limit the specific image understanding model.

[0104] In step 104, the video description information, the image description information and the text data are input into a large language model. The large language model can fuse and refine the semantic information in the video description information, the image description information and the text data, so that the output preset dimension label content can reflect the overall characteristics of the video clip.

[0105] Those skilled in the art can determine the type of preset dimension tags according to actual application requirements. For example, the preset dimension tags specifically include: video content tags, gender tags, character number tags, age tags, and scene tags.

[0106] The video content tag is used to represent the main content presented in the video clip.

[0107] Gender tags are used to characterize the gender of people in video clips.

[0108] The number of people label is used to characterize the number of people appearing in a video clip.

[0109] Age tags are used to characterize the age range of people in a video clip.

[0110] The scene tag is used to characterize the scene type shown in the video clip. The scene type may include: indoor scene and outdoor scene, etc. Indoor scene and outdoor scene may also be further divided according to actual application requirements.

[0111] It can be understood that in addition to the video content tag, gender tag, character number tag, age tag and scene tag, the above-mentioned preset dimension tags can also include: time tag, action tag, expression tag and other tags.

[0112] Reference Figure 4 , shows a flow chart of a method for processing a video clip according to an embodiment of the present invention, wherein the complete video data of the video clip can be input into a video understanding model, and the video understanding model outputs video description information. The frame image data and face image data of the video clip are input into an image understanding model, and the image understanding model outputs image description information. The video description information, image description information and text data are input into a large language model, and the large language model performs fusion processing on the video description information, image description information and text data. The video description information and image description information can be in text form.

[0113] A large language model is an artificial intelligence model designed to understand and generate human language. Large language models can perform a wide range of tasks by training on large amounts of text data. Large language models are usually based on deep learning architectures, such as transformer structures. The embodiments of the present invention do not limit the specific types of large language models. For example, the types of large language models may include: GPT (Generative Pre-Trained Transformer) model and GLM (Generative Language Model, General Language Model), etc.

[0114] In the field of natural language processing, a large amount of text data can be used to pre-train a large language model. After the large language model is pre-trained, it can be fine-tuned to adapt to various downstream text generation tasks. Specifically in the embodiment of the present invention, the downstream text generation task specifically includes the generation task of preset dimension label content.

[0115] Therefore, the embodiment of the present invention can perform fine-tuning training on the large language model so that the large language model has the ability to generate label content of preset dimensions.

[0116] Accordingly, the process of fine-tuning the large language model specifically includes:

[0117] Step C1, obtaining training data; the training data includes: training samples and annotation results; the training samples include: video description information text, picture description information text, and original text data corresponding to preset video clips; the annotation results include: label content of preset dimension labels annotated for the training samples;

[0118] Step C2: input the training sample into a large language model, and the large language model outputs a prediction result of the label content of a preset dimensional label;

[0119] Step C3: determining loss information according to the prediction result and the marking result;

[0120] Step C4: Update the model parameters of the large language model according to the loss information.

[0121] In step C1, the collection of training data is the training basis of the large language model. For the present invention, specific training samples are required, namely, the video description information text corresponding to the preset video clip (which can be generated by the video understanding model, or, can be generated by the user), the picture description information text (which can be generated by the picture understanding model, or, can be generated by the user) and the original text data (which can be obtained by processing the audio and frame pictures of the video clip).

[0122] The annotated text can be the label content of the preset dimension labels manually annotated for the training samples. These annotated results are used as the target values ​​of the output results of the large language model for comparison and learning in the subsequent training process.

[0123] In step C2, the large language model comprehensively analyzes and processes various information in the input training samples based on the language knowledge and pattern recognition capabilities learned in the pre-training stage. It mines and understands the semantics, grammar, logical relationships, and potential feature information in the video description information text, picture description information text, and original text data, and uses its internal neural network structure and parameters to try to generate prediction results corresponding to the preset dimension label content.

[0124] In step C3, the loss information is used to measure the difference between the prediction result of the large language model and the actual annotation result. A common method is to use a set loss function, such as the cross entropy loss function.

[0125] In step C4, based on the loss information calculated in step C3, an optimization algorithm (such as stochastic gradient descent and its variants) is used to update the model parameters of the large language model. The goal of the optimization algorithm is to gradually reduce the value of the loss information by adjusting the model parameters, that is, the prediction result of the large language model is getting closer and closer to the actual annotation result.

[0126] Steps C1 to C4 constitute a complete training process. The above training process can enable the large language model to have the ability to generate label content of preset dimensions, and can achieve the goal of allowing the large language model to accurately output label content that meets the requirements based on the relevant text information of the video clip.

[0127] Therefore, after receiving the video description information, image description information and text data corresponding to the video clip, the large language model can fuse and refine the semantic information in the video description information, image description information and text data, so that the output preset dimensional label content can reflect the overall characteristics of the video clip.

[0128] The embodiment of the present invention can store the mapping relationship between the video clip and the tag content of the preset dimension tag in the database. The tag content in the above mapping relationship can be used to match the query information during the retrieval process of the video clip; thus, the target video clip matching the query information can be located in the database according to the query information input by the user.

[0129] The above database specifically includes: a text database and a vector database; the text database is used to record label content in text form, and the vector database is used to record label content in vector form.

[0130] After receiving query information in text form input by a user (hereinafter referred to as query text), an embodiment of the present invention can match the query text with text-form tag content stored in a text database, find target tag content that matches the query text, and then locate the corresponding target video clip, thereby completing the retrieval of the video clip.

[0131] Here, the matching between the tag content and the query text may be: string matching. As a commonly used matching method between the tag content and the query text, string matching may include two forms: exact matching or fuzzy matching.

[0132] Exact matching means that the string of the tag content must be exactly the same as the string of the query text to be considered a successful match. Fuzzy matching is relatively more flexible and allows a certain degree of difference. For example, matching is determined based on rules such as semantic similarity and partial character inclusion.

[0133] The embodiment of the present invention can also convert the query text into a query vector, and then calculate the similarity between the query vector and the label content in vector form stored in the vector database to find the target label content whose similarity meets the preset conditions (for example, the similarity is greater than the similarity threshold), and then locate the corresponding target video clip, thereby completing the retrieval of the video clip.

[0134] It should be noted that the preset dimension label can be one or more.

[0135] In the case of multiple preset dimension labels, one method is to combine the vectors corresponding to the multiple preset dimension labels into a new comprehensive vector and store the comprehensive vector. Common combination methods include splicing, for example, forming a new comprehensive vector by sequentially splicing the vector of the first preset dimension label and the vector of the second preset dimension label; weighted splicing and other methods can also be used to assign different weights to each dimension according to its importance before splicing. In the retrieval process, the query information is combined according to the same rules, and then the similarity between the combined vector and the stored comprehensive vector is calculated.

[0136] Another way is to store vectors of multiple preset dimension labels separately. During the retrieval process, the similarity between the vector of a single preset dimension label and a single query vector is calculated separately. For example, the first similarity corresponding to the first preset dimension label is calculated first, and then the second similarity corresponding to the second preset dimension label is calculated, and the first similarity and the second similarity are fused according to certain rules (such as weighted average, etc.). The setting of the weight can be comprehensively determined based on factors such as the importance of each preset dimension label in the feature description of the video clip, the user's attention to each dimension during retrieval, and finally, whether the video clip meets the retrieval requirements is judged based on the fusion result.

[0137] In summary, the video clip processing method of the embodiment of the present invention obtains multi-dimensional video clip data: video data, frame image data, face image data and text data, and provides the video description information corresponding to the video data, the image description information and text data corresponding to the frame image data and the face image data to a large language model, and the large language model outputs the label content of the preset dimensional label.

[0138] Since the video description information is obtained based on the overall video data corresponding to the video clip, it can cover the overall characteristics of the video clip. Since the image description information is obtained based on the frame image data and the face image data, it can cover the picture elements and character features of the frame image. On this basis, the large language model can fuse and refine the video description information, the image description information, and the semantic information in the text data, so that the output preset dimensional label content can reflect the overall characteristics of the video clip. Therefore, the embodiments of the present invention can improve the information coverage and accuracy of the preset dimensional label content.

[0139] Reference Figure 5 , shows a flowchart of a method for processing a video clip according to an embodiment of the present invention. The method may specifically include the following steps:

[0140] Step 501: receiving query information input by a user;

[0141] Step 502: According to the tag content of the preset dimension tag corresponding to the video clip in the database, search the database for a target video clip that matches the query information;

[0142] Step 503: output the target video clip;

[0143] Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

[0144] The embodiment of the present invention can be used to search for video segments according to the generated video tags, so as to improve the accuracy of the search results of the video segments.

[0145] During the video tag generation stage, the embodiment of the present invention obtains multi-dimensional video clip data: video data, frame image data, face image data and text data, and provides the video description information corresponding to the video data, the image description information and text data corresponding to the frame image data and the face image data to a large language model, which outputs the label content of the preset dimensional label.

[0146] Since the large language model can integrate and understand the video description information, image description information and text data, and generate label content of the preset dimension label closely related thereto, the embodiment of the present invention can improve the information coverage and accuracy of the preset dimension label content.

[0147] While the embodiments of the present invention can improve the information coverage and accuracy of preset dimensional label content, the embodiments of the present invention can effectively exclude video clips that are irrelevant or have low relevance to the query information based on more comprehensive and accurate label content, thereby improving the accuracy of retrieval results.

[0148] In actual applications, the query information may specifically include: first query information entered by the user in the input box, and / or second query information entered by the user for a preset dimension label.

[0149] The first query information may be text information or voice information.

[0150] The second query information may include: the target tag content selected by the user for the preset dimension tag. For example, the preset dimension tag is an age tag, and the tag content of the age tag includes: male and female. In an embodiment of the present invention, a check box for the tag content of the preset dimension tag may be provided, and the tag content corresponding to the check box selected by the user may be used as the target tag content.

[0151] The second query information may include: tag description content input by the user for the preset dimension tag.

[0152] The embodiment of the present invention can provide the following technical solution for searching a database for a target video segment that matches the query information:

[0153] Technical Solution 1: If the query information includes: target tag content of N preset dimensional tags, the target tag content of the N preset dimensional tags is matched with the tag content in text form in the text database to obtain a target video segment that matches the target tag content; wherein the tag content in text form in the target video segment hits the target tag content of N preset dimensional tags, and N can be a positive integer.

[0154] The text-based label content in the target video clip hits the target label content of a preset dimension label, which specifically means: there is a text-based label content matching the target label content in the target video clip. The text-based label content in the target video clip hits the target label content of N preset dimension labels, which specifically means: there are N different text-based label contents in the target video clip, and these N text-based label contents have a successful matching relationship with the target label contents corresponding to the N preset dimension labels.

[0155] It should be noted that if there is no target video clip with target tag content that matches N preset dimension tags in the text database, a corresponding prompt message can be output. The above prompt message can let the user know clearly at the first time that there are no video resources that meet the requirements under the current search conditions, avoiding the user from waiting for a long time or constantly wondering why no search results appear, thereby improving the user experience.

[0156] Technical Solution 2: If the query information includes: target tag content of N preset dimensional tags and first query information, a first match is performed on the target tag content of the N preset dimensional tags with the tag content in text form in a text database to obtain a candidate video segment; and a second match is performed on the first query vector corresponding to the first query information with the tag content in vector form corresponding to the candidate video segment in the vector database to obtain a target video segment that matches the query information.

[0157] Technical Solution 2 combines the target tag content of the preset dimensional tag and the first query information to achieve multi-dimensional retrieval.

[0158] Among them, for the target tag content of the preset dimension tag, with the help of the precise matching ability of the text database, the candidate video clips that meet the target tag content can be efficiently screened from the text database. Since the dimension tags corresponding to the target tag content are clear and targeted, the candidate video clips can quickly narrow the search scope.

[0159] The first query information input by the user is converted into a first query vector, and the first query vector is similarly matched with the vector-based label content corresponding to the candidate video clips in the vector database. This gives full play to the advantages of the vector database in semantic understanding and fuzzy matching, and can mine target video clips that are highly relevant to the query intent of the query information at the semantic level.

[0160] The multi-dimensional retrieval of Technical Solution 2 can make the retrieval process both accurate and comprehensive, effectively improving the accuracy of video clip retrieval.

[0161] In one example, assuming that the query information input by the user includes: a character quantity tag "two people", an age tag "middle-aged", a gender tag "male", and the first query information "a busy street, two people fighting fiercely", the processing process of the embodiment of the present invention may include:

[0162] First, the target tag content of the preset dimension tags such as the number of people tag, age tag, gender tag, etc. is matched with the text tag content in the text database. The first match can filter out candidate video clips whose text tag content meets the requirements of "two people", "middle-aged", and "male" from a large number of video clips, and quickly narrow the search scope by relying on the precise matching characteristics of the text database;

[0163] Next, the first query information "Two people fighting fiercely on a busy street" is converted into a first query vector, and a second match is performed between the first query vector and the vector-form label content corresponding to the candidate video clips in the vector database, making full use of the advantages of the vector database in semantic understanding and fuzzy matching to mine video clips that are highly relevant to the query intent at the semantic level, and finally determining the target video clip that meets the clear requirements of the preset dimensional labels and is consistent with the semantics of the first query information, thereby achieving efficient, accurate and comprehensive video clip retrieval, and accurately locating the video content that meets the needs of users.

[0164] Technical Solution 3: If the query information includes: the label description content entered by the user for the preset dimensional label, a third match is performed on the second query vector corresponding to the label description content and the label content in vector form corresponding to the preset dimensional label in the vector database to obtain the target video clip that matches the query information.

[0165] When the user inputs a label description for a preset dimension label, it means that the user expresses his / her desired features of the video clip in a specific dimension in a more flexible and customized way.

[0166] In this case, Technical Solution 3 will first convert the label description content input by the user into a second query vector. This process uses natural language processing technology and vector representation methods to enable the computer to understand and process the semantic meaning of these text information and map it to the vector space. Then, in the vector database, there are label contents in vector form corresponding to each preset dimensional label. These contents are also pre-processed and vectorized, and have a unified vector representation form, which is convenient for comparison with the second query vector input by the user. By calculating the similarity between the second query vector and the label content in vector form, for example, using the cosine similarity algorithm or other appropriate vector similarity measurement methods, the degree of matching between them in semantics and features is determined, and then the target vector corresponding to the preset dimensional label that best matches the label description content input by the user is screened out, and the video clip associated with the target vector is the target video clip that matches the query information obtained after the third match.

[0167] In a specific implementation, a user can input a tag description for a video content tag; the tag description supports the user to describe the content of the video with a custom description based on their own unique cognition and specific search needs. On this basis, the embodiment of the present invention determines the target video segment based on the vector similarity between the custom description in the tag description and the tag content in vector form, which can improve the accuracy and personalization of the search and meet the diversified and refined needs of users.

[0168] In a specific implementation, the process of outputting the target video segment may specifically include: sorting the target video segment according to the matching degree between the query information and the tag content of the target video segment to obtain a sorting result; and outputting the target video segment according to the sorting result.

[0169] According to the matching degree between the query information and the target video clip label content, the target video clips are sorted and output, so that the output results have a priority order, which can give priority to the display of target video clips that are more consistent with the user's query information, optimize the user's experience of obtaining satisfactory results, and improve the practicality and pertinence of the retrieval results.

[0170] In a specific implementation, the higher the matching degree is, the higher the position of the target video segment is in the sorting result. Therefore, the target video segment can be output according to the order of the sorting result from front to back.

[0171] The matching degree may specifically include: text matching degree or string matching degree between the query text and the tag content in text form, or vector similarity between the query vector and the tag content in vector form.

[0172] In technical solution 1, the matching degree between the query information and the tag content of the target video clip specifically includes: the text matching degree between the corresponding tag content of the target video clip and the target tag content of the N preset dimensional tags. For example, the text matching degree corresponding to the exact match is 100%, and the text matching degree corresponding to the fuzzy match is less than 100%.

[0173] When the text matching degrees corresponding to the target video clips are different, they can be sorted from high to low according to the text matching degrees, and the video clips that accurately match the preset dimension labels of the query information in more dimensions are placed in the front.

[0174] For target video clips with similar or identical text matching degrees, the sorting can be further refined by combining factors such as their corresponding popularity (such as number of views, likes, comments, etc.), timeliness of release time (newer releases are prioritized), and video quality (including resolution, picture clarity, audio effects, etc.). This allows the sorting results to highlight the text relevance to the query information while taking into account other key factors that affect user experience and video value.

[0175] In technical solution 2, the matching degree between the query information and the label content of the target video clip specifically includes: specifically including: the text matching degree between the corresponding label content of the target video clip and the target label content of the N preset dimensional labels, and / or, the vector similarity between the first query vector and the label content in vector form of the target video clip. The text matching degree and the vector similarity can be fused by using a fusion method such as weighted average to obtain a fused matching degree.

[0176] When sorting the target video clips according to the fusion matching degree, if the fusion matching degrees of each target video clip are different, they are directly sorted from high to low according to the fusion matching degree values. The target video clips with higher fusion matching degrees are ranked at the top, which means that these target video clips have the best fit with the query information under the comprehensive consideration of text matching degree and vector similarity.

[0177] For target video clips with similar or identical fusion matching degrees, other factors are further referenced to refine the sorting. For example, popularity-related indicators can be considered, such as target video clips with more playback volume, likes, and comments are prioritized; or based on the timeliness of release time, the more recently released videos are prioritized; and in terms of video quality, target video clips with high resolution, clear picture quality, and good audio effects are ranked better.

[0178] Of course, the comprehensive score of the target video clips can be calculated based on the fusion matching degree and heat sorting indicators, and the video clips can be sorted according to the comprehensive score.

[0179] In technical solution 3, the matching degree between the query information and the label content of the target video segment specifically includes: specifically includes: the vector similarity between the second query vector and the label content in vector form of the target video segment.

[0180] If the second query vector corresponds to multiple preset dimension labels, in order to more accurately measure the matching degree, corresponding weights can be set for these preset dimension labels, and the vector similarities corresponding to the multiple preset dimension labels can be comprehensively calculated by weighted averaging to determine the final comprehensive vector similarity.

[0181] For the sorting process of comprehensive vector similarity, please refer to Technical Solution 2, which will not be described in detail here.

[0182] In an optional embodiment of the present invention, the target video segment corresponds to time information in the original video, and the process of outputting the above-mentioned target video segment may specifically include: when two target video segments correspond to the same original video, if the interval between the time information corresponding to the two target video segments is less than a time threshold, the two target video segments are merged to obtain a merged video segment; and the merged video segment is output.

[0183] When two target video clips correspond to the same original video and the time interval is less than a time threshold, the embodiment of the present invention merges and outputs the two target video clips, which can effectively reduce the presentation of fragmented video clips, integrate related video clips, avoid users frequently switching between multiple adjacent video clips, enhance the coherence and completeness of the retrieval results, and improve the user viewing experience.

[0184] This method of merging and outputting the target video clips based on time information optimizes the display of the retrieval results, allowing users to more smoothly obtain related continuous content in the original video, improving the practicality and convenience of video clip retrieval and output, and better meeting users' needs for obtaining video information.

[0185] The time threshold may be determined by those skilled in the art according to actual application requirements. For example, the time threshold may be 0.1 seconds.

[0186] It should be noted that the merging process of the target video segments in the embodiment of the present invention may be an iterative execution process. Here, an example of the merging process is provided. The merging process specifically includes:

[0187] Step E1, sorting multiple target video segments of the same original video according to the order of time information from front to back, to obtain a first sorting result;

[0188] Step E2, obtaining the adjacent i-th target video segment and i+1-th target video segment from the sorting results in order from front to back; i may be a positive integer;

[0189] Step E3, determining whether the interval between the i-th target video segment and the i+1-th target video segment is less than a time threshold, if so, executing step E4, otherwise executing step E8;

[0190] Step E4: merge the i-th target video segment and the i+1-th target video segment to obtain a merged video segment;

[0191] Step E5, determining whether the interval between the i+1th target video segment and the i+2th target video segment is less than a time threshold, if so, executing step E6, otherwise executing step E7;

[0192] Step E6, merging the merged video segment with the i+2th target video segment, i=i+1, and executing step E5;

[0193] Step E7, ending the merging process and outputting the merged video clip;

[0194] Step E8, i=i+1, and execute step E2.

[0195] In actual applications, a corresponding original video identifier can be set for a video clip, and then it can be determined whether two target video clips belong to the same original video based on the original video identifier. The original video identifier can be a video number or a work name + work episode.

[0196] After merging multiple target video segments that meet the merging conditions (two target video segments correspond to the same original video and the time interval is less than the time threshold), the embodiment of the present invention can determine the similarity of the merged video segments according to a preset rule. For example, the preset rule can be to use the maximum similarity of multiple target video segments in the merged video segment as the similarity of the merged video segment. For another example, the average value of multiple target video segments in the merged video segment can be used as the similarity of the merged video segment.

[0197] The similarity of the merged video clips can be used to sort the search results. Specifically, the merged video clips and the target video clips that do not meet the merge conditions can be sorted in descending order of similarity. The ones with high similarity are ranked first, and the ones with low similarity are ranked last, so as to present the search results to the user in a more likely order to meet their query needs, so that the user can preferentially obtain the video clips with higher relevance to the query information.

[0198] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0199] Reference Figure 6 , shows a structural schematic diagram of a video clip processing device according to an embodiment of the present invention. The video clip processing device may specifically include the following modules: a multi-dimensional data acquisition module 601, a video understanding module 602, an image understanding module 603 and a large model processing module 604.

[0200] The multi-dimensional data acquisition module 601 is used to acquire the overall video data, frame image data, face image data and text data corresponding to the video clip;

[0201] A video understanding module 602, configured to determine video description information corresponding to the overall video data;

[0202] The picture understanding module 603 is used to determine the picture description information corresponding to the frame picture data and the face picture data;

[0203] The large model processing module 604 is used to input the video description information, the image description information and the text data into a large language model, and the large language model outputs label content of a preset dimensional label.

[0204] Optionally, the device further comprises:

[0205] A transition point analysis module is used to analyze the transition points of the original video to obtain the transition point information corresponding to the original video;

[0206] The video splitting module is used to split the original video into multiple video segments according to the transition point information.

[0207] Reference Figure 7, shows a schematic structural diagram of a video clip processing device according to an embodiment of the present invention. The video clip processing device may specifically include the following modules: a receiving module 701, a query module 702 and an output module 703.

[0208] The receiving module 701 is used to receive query information input by the user;

[0209] A query module 702, configured to query a target video segment matching the query information in the database according to the tag content of the preset dimension tag corresponding to the video segment in the database;

[0210] An output module 703, configured to output the target video segment;

[0211] Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

[0212] Optionally, the device further comprises:

[0213] A multi-dimensional data acquisition module, used to acquire overall video data, frame image data, face image data and text data corresponding to the video clip;

[0214] A video understanding module, used to determine video description information corresponding to the overall video data;

[0215] A picture understanding module, used to determine picture description information corresponding to the frame picture data and the face picture data;

[0216] The large model processing module is used to input the video description information, the image description information and the text data into a large language model, and the large language model outputs label content of a preset dimension label.

[0217] Optionally, the preset dimension tags include: video content tag, gender tag, character quantity tag, age tag and scene tag.

[0218] Optionally, the device further comprises:

[0219] A training data acquisition module is used to acquire training data; the training data includes: training samples and annotation results; the training samples include: video description information text, picture description information text, and original text data corresponding to a preset video clip; the annotation results include: label content of preset dimension labels annotated for the training samples;

[0220] A prediction module, used for inputting the training sample into a large language model, and having the large language model output a prediction result of a label content of a preset dimensional label;

[0221] A loss determination module, used to determine loss information according to the prediction result and the annotation result;

[0222] The parameter updating module is used to update the model parameters of the large language model according to the loss information.

[0223] Optionally, the multi-dimensional data acquisition module includes:

[0224] A frame extraction module is used to extract frames from a video clip to obtain frame image data contained in the video clip;

[0225] A face recognition module, used to perform face recognition on the frame image data to obtain face image data contained in the frame image data;

[0226] A speech recognition module is used to perform speech recognition on the audio in the video clip to obtain text data;

[0227] The text recognition module is used to perform text recognition on the frame image data in the video clip to obtain text data.

[0228] Optionally, the device further comprises:

[0229] A storage module, used for storing in a database a mapping relationship between video clips and tag contents of preset dimension tags;

[0230] The database includes: a text database and a vector database; the text database is used to record label contents in text form, and the vector database is used to record label contents in vector form.

[0231] Optionally, the query information specifically includes: first query information input by the user in the input box, and / or, second query information input by the user for a preset dimension tag; the second query information includes: target tag content selected by the user for the preset dimension tag, and / or, tag description content input by the user for the preset dimension tag;

[0232] The query module 702 specifically includes:

[0233] A first query module is used for, if the query information includes: target tag contents of N preset dimensional tags, matching the target tag contents of the N preset dimensional tags with tag contents in text form in a text database to obtain a target video segment that matches the target tag contents; wherein the tag contents in text form in the target video segment hit the target tag contents of the N preset dimensional tags, and N is a positive integer; or

[0234] A second query module is used for, if the query information includes: target tag contents of N preset dimensional tags and the first query information, first matching the target tag contents of the N preset dimensional tags with tag contents in text form in a text database to obtain candidate video clips; and second matching the first query vector corresponding to the first query information with tag contents in vector form corresponding to the candidate video clip in the vector database to obtain a target video clip matching the query information; or

[0235] The third query module is used to perform a third match between the second query vector corresponding to the tag description content and the tag content in vector form corresponding to the preset dimensional tag in the vector database if the query information includes: the tag description content input by the user for the preset dimensional tag, so as to obtain the target video clip matching the query information.

[0236] Optionally, the output module 703 specifically includes:

[0237] A sorting module, used to sort the target video segments according to the matching degree between the query information and the tag content of the target video segments to obtain a sorting result;

[0238] The sorting output module is used to output the target video segment according to the sorting result.

[0239] Optionally, the target video segment corresponds to time information in the original video, and the output module 703 specifically includes:

[0240] A merging module, configured to merge the two target video segments to obtain a merged video segment if the interval between the time information corresponding to the two target video segments is less than a time threshold when the two target video segments correspond to the same original video;

[0241] The merging output module is used to output the merged video segment.

[0242] An embodiment of the present invention further provides an electronic device that can implement the functions of the aforementioned storage node or scheduling node or ledger node or management node.

[0243] like Figure 8 As shown, the electronic device may include a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other via the communication bus 1004.

[0244] Memory 1003, used for storing computer programs;

[0245] The processor 1001 is used to execute the program stored in the memory 1003, and implements the following steps:

[0246] Storing the storage object;

[0247] Sending the storage object to a coupled second storage node, so that the second storage node stores and / or forwards the storage object;

[0248] After the storage object is successfully stored, the object information of the storage object is sent to the ledger node, so that the ledger node saves the ledger information of the first storage node; the ledger information includes: the object information stored in the corresponding storage node.

[0249] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0250] The communication interface is used for communication between the above electronic device and other devices.

[0251] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one video segment processing device located away from the aforementioned processor.

[0252] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0253] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes the method for processing video clips described in any of the above embodiments.

[0254] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for processing a video clip described in any one of the above embodiments.

[0255] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc. Examples of optical media may include DVD (Digital Video Disc) and the like.

[0256] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0257] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0258] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for processing a video clip, characterized in that: The method comprises: Receive query information input by the user; According to the tag content of the preset dimension tag corresponding to the video clip in the database, searching the database for a target video clip matching the query information; Outputting the target video segment; Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

2. The method according to claim 1, characterized in that: The process of determining the label content of the preset dimension label includes: Obtaining overall video data, frame image data, face image data and text data corresponding to the video clip; Determine video description information corresponding to the overall video data using a video understanding model; Determine the picture description information corresponding to the frame picture data and the face picture data by using the picture understanding model; The video description information, the picture description information and the text data are input into a large language model, and the large language model outputs label content of a preset dimensional label.

3. The method according to claim 1 or 2, characterized in that: The preset dimension tags include: video content tag, gender tag, character quantity tag, age tag and scene tag.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire training data; the training data includes: training samples and annotation results; the training samples include: video description information text, picture description information text, and original text data corresponding to preset video clips; the annotation results include: label content of preset dimension labels annotated for the training samples; The training sample is input into a large language model, and the large language model outputs a prediction result of a label content of a preset dimension label; Determining loss information according to the prediction result and the marking result; Based on the loss information, the model parameters of the large language model are updated.

5. The method according to claim 2, characterized in that: The obtaining of the overall video data, frame image data, face image data and text data corresponding to the video clip includes: Extract frames from the video clip to obtain frame image data contained in the video clip; Performing face recognition on the frame image data to obtain face image data contained in the frame image data; Perform speech recognition on the audio in the video clip to obtain text data; Perform text recognition on the frame image data in the video clip to obtain text data.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Storing in a database a mapping relationship between the video clips and the tag contents of the preset dimension tags; The database includes: a text database and a vector database; the text database is used to record label contents in text form, and the vector database is used to record label contents in vector form.

7. The method according to claim 1 or 2, characterized in that: The query information specifically includes: the first query information input by the user in the input box, and / or, the second query information input by the user for the preset dimension tag; the second query information includes: the target tag content selected by the user for the preset dimension tag, and / or, the tag description content input by the user for the preset dimension tag; The step of searching the database for a target video segment that matches the query information according to the tag content of the preset dimension tag corresponding to the video segment in the database includes: If the query information includes: target tag contents of N preset dimensional tags, the target tag contents of the N preset dimensional tags are matched with tag contents in text form in a text database to obtain a target video segment that matches the target tag contents; wherein the tag contents in text form in the target video segment hit the target tag contents of the N preset dimensional tags, and N is a positive integer; or If the query information includes: target tag contents of N preset dimensional tags, and first query information, a first match is performed on the target tag contents of the N preset dimensional tags with tag contents in text form in a text database to obtain candidate video clips; and a second match is performed on a first query vector corresponding to the first query information with tag contents in vector form corresponding to the candidate video clip in a vector database to obtain a target video clip matching the query information; or If the query information includes: the label description content input by the user for the preset dimensional label, a third match is performed on the second query vector corresponding to the label description content and the label content in vector form corresponding to the preset dimensional label in the vector database to obtain the target video clip matching the query information.

8. The method according to claim 1 or 2, characterized in that: The outputting the target video segment includes: Sorting the target video segments according to the matching degree between the query information and the tag content of the target video segments to obtain a sorting result; The target video segment is output according to the sorting result.

9. The method according to claim 1 or 2, characterized in that: The target video segment has time information corresponding to the original video, and the outputting of the target video segment includes: In the case where the two target video segments correspond to the same original video, if the interval between the time information corresponding to the two target video segments is less than the time threshold, the two target video segments are merged to obtain a merged video segment; The merged video segment is output.

10. A video clip processing device, characterized in that: The device comprises: A receiving module, used for receiving query information input by a user; A query module, used to query a target video segment matching the query information in the database according to the tag content of the preset dimension tag corresponding to the video segment in the database; An output module, used for outputting the target video segment; Among them, the label content of the preset dimensional label is the content output by the large language model based on the video description information, picture description information and text data corresponding to the video clip; the video description information is the information obtained based on the overall video data corresponding to the video clip; the picture description information is the information obtained based on the frame picture data and face picture data corresponding to the video clip.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method described in claims 1-9 is implemented.

Images