Video content processing method and device, server and storage medium

Abstract

The invention discloses a video content processing method and device, a server and a storage medium, and the method comprises the steps: obtaining an input target video, and carrying out the extraction of a target video frame and video text information according to the target video; obtaining visual features of the target video frame, constructing an abstract generation prompt text according to the visual features of the target video frame and the video text information, and inputting the abstract generation prompt text into a visual language model for abstract extraction to obtain a video abstract; acquiring a predefined attribute extraction prompt text, and inputting the attribute extraction prompt text and the target video frame into a visual language model to perform attribute extraction on the target video frame to obtain first video attribute information of the target video frame; and inputting the video abstract and the first video attribute information into a large language model to generate second video attribute information of the target video, thereby realizing extraction of video attributes.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video content processing method, device, server and storage medium. Background Art

[0002] With the development of short video platforms, short videos play an increasingly important role in marketing. As a high-dimensional data form, videos contain a wealth of information. At present, by identifying and analyzing the attributes of popular videos to obtain current trends, user preferences, and viewing habits, marketing effects can be achieved more accurately. However, the information contained in videos is extremely complex. In order to make full use of the information in the videos, the content of the videos must be structured. Summary of the invention

[0003] The embodiments of the present application provide a video content processing method, device, server and storage medium, which can realize the extraction of video attributes.

[0004] In a first aspect, an embodiment of the present application provides a video content processing method, the method comprising:

[0005] Obtaining an input target video, and extracting a target video frame and video text information according to the target video;

[0006] Acquire visual features of the target video frame, construct summary generation prompt text according to the visual features of the target video frame and the video text information, input the summary generation prompt text into a visual language model for summary extraction, and obtain a video summary;

[0007] Obtaining a predefined attribute extraction prompt text, inputting the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtaining first video attribute information of the target video frame;

[0008] The video summary and the first video attribute information are input into a large language model, and the large language model is used to generate second video attribute information of the target video.

[0009] In a second aspect, an embodiment of the present application provides a video content processing device, the device comprising:

[0010] A video acquisition unit, used to acquire an input target video, and extract a target video frame and video text information according to the target video;

[0011] A summary generation unit, used for acquiring visual features of the target video frame, constructing summary generation prompt text according to the visual features of the target video frame and the video text information, and inputting the summary generation prompt text into a visual language model for summary extraction to obtain a video summary;

[0012] an attribute extraction unit, configured to obtain a predefined attribute extraction prompt text, input the attribute extraction prompt text and the target video frame into the visual language model to extract the attribute of the target video frame, and obtain first video attribute information of the target video frame;

[0013] The information generating unit is used to input the video summary and the first video attribute information into a large language model, and use the large language model to generate second video attribute information of the target video.

[0014] In a third aspect, an embodiment of the present application provides a server, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above-mentioned video content processing method.

[0015] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the steps of the above-mentioned video content processing method.

[0016] In the embodiment of the present application, by extracting the target video frame and video text information according to the input target video, the key content in the video can be efficiently obtained; by obtaining the visual features of the target video frame, and combining the video text information to construct a summary to generate a prompt text, the visual and text features in the target video can be comprehensively utilized for multimodal fusion; by taking the predefined attribute extraction prompt text and the target video frame as input, the visual language model is used for attribute extraction, so that the attribute extraction is more flexible, and the attribute extraction prompt text can be adjusted as needed to extract different video attribute information. By inputting the video summary and the first video attribute information into the large language model, the second video attribute information of the target video can be generated, and the structured extraction of the video attribute can be realized. The method based on the embodiment of the present application makes full use of the capabilities of large pre-trained models, such as visual language models and large language models. Since these models have been pre-trained on a large amount of data, they contain rich and in-depth information. By using these pre-trained models, various different attributes of the video from text to vision can still be extracted without preparing a large amount of sample data for the relevant video attributes. Since this method does not rely on domain-specific models or labels for transfer learning, it has potential cross-domain applicability and can be applied on video data from different domains without requiring extensive domain adaptation work. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a system architecture diagram of a video content processing method provided by an embodiment of the present application;

[0019] Figure 2 It is a flowchart of a video content processing method provided by an embodiment of the present application;

[0020] Figure 3 It is a flowchart of a video content processing method provided by an embodiment of the present application;

[0021] Figure 4 It is a flowchart of a video content processing method provided by an embodiment of the present application;

[0022] Figure 5 is an example schematic diagram involved in a video content processing method provided in an embodiment of the present application;

[0023] Figure 6 is an example schematic diagram involved in a video content processing method provided in an embodiment of the present application;

[0024] Figure 7 It is a flowchart of a video content processing method provided by an embodiment of the present application;

[0025] Figure 8 is an example schematic diagram involved in a video content processing method provided in an embodiment of the present application;

[0026] Fig. 9 is an example schematic diagram involved in a video content processing method provided in an embodiment of the present application;

[0027] Fig.10 is a structural diagram of a video content processing device provided in an embodiment of the present application;

[0028] Fig.11 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0031] The technical terms that may be involved in the embodiments of this application are:

[0032] 1. Zero-shot learning:

[0033] Zero-shot learning is a machine learning paradigm that aims to solve the problem of learning without any labeled samples. In traditional supervised learning, the model needs to be trained with a large number of labeled samples in order to be able to accurately predict new unseen data. However, in real life, obtaining labeled samples may be expensive and time-consuming, or for some tasks, labeled samples may not be available at all. The goal of zero-shot learning is to enable the model to infer new knowledge from limited known information and to be able to perform accurate reasoning or generalization when encountering new categories.

[0034] 2. Large-Language Model (LLM):

[0035] A large language model is a language model that can consist of an artificial neural network with many parameters (typically billions of weights or more), trained on large amounts of unlabeled text using self-supervised learning or semi-supervised learning.

[0036] 3. Vision-and-Large-Language Model (VLLM):

[0037] The visual language model is an algorithm model that uses deep learning technology to process images or videos. Its basic principle is based on neural networks, which can achieve efficient processing and understanding of image information through large-scale training data sets and high-performance computing hardware.

[0038] 4. Video content structuring:

[0039] Video content structuring refers to ordering and organizing the information in a video so that it can be understood, processed and analyzed by a computer system. Typically, the process of video content structuring involves extracting key information from the video and converting it into a computer-readable and processable form.

[0040] In related technologies, it is usually necessary to collect a large amount of sample data in advance for model training, and use the trained model to structure the video content. However, this method faces difficulties such as no matching model for long-tail tags, high cost of tag acquisition and different standards, large modeling workload and high maintenance cost, which seriously limits the full utilization of video data. For example:

[0041] 1. The problem of no matching model for long-tail tags: Currently, there are many types of attributes or tags used for market transaction analysis, covering a wide range and a large number. Many tags that need to be processed do not have corresponding recognition models or pre-established data sets, and there is a serious lack of sample data.

[0042] 2. The cost of obtaining labels is high and the standards are inconsistent: Labeling video data often requires a lot of manpower. Not only is the labeling speed slow, but there is also the problem of inconsistent standards, making the process of obtaining data labels expensive and time-consuming.

[0043] 3. The modeling workload is large and the maintenance cost is high: Building a model for each label requires a lot of manpower and material resources. Once the label standard needs to be changed or new attributes need to be added, it is usually necessary to re-label the data, train the model, and update the online system. This process is not flexible and has high maintenance costs.

[0044] In addition, in the case of insufficient sample data, the existing video attribute recognition tasks mainly use the zero-sample learning method in machine learning. The core idea of ​​this method is to establish a correspondence between visible labels and unknown labels that are difficult to obtain. Usually, this method assumes that in a certain feature space, the unknown label features that need to be identified can be mapped from the features of visible labels. By constructing a set of classifiers for training visible labels, training consistency functions between visible labels and unknown labels, etc., a mapping relationship from visible labels to unknown labels is established, and it is hoped that this relationship has a certain generalization performance, thereby achieving the effect of zero-sample learning. However, the existing video attribute recognition methods have some limitations when performing video attribute analysis, such as:

[0045] 1. Limited information extraction dimensions: Existing methods mainly focus on extracting action information from video images, while ignoring other equally important information such as text and sound. This neglected information is also crucial for video attribute analysis and structured analysis.

[0046] 2. Semantic drift and domain transfer: Zero-shot learning is essentially the transfer of knowledge from other models or attributes, which will inevitably encounter problems such as semantic gaps and domain drift. For example, when the content to be recognized cannot find a trained model or similar samples in a similar domain, the recognition performance may be limited.

[0047] In order to completely or partially solve the above problems, the present application embodiment proposes a video content processing system. Figure 1 , is a system architecture diagram of a video content processing method provided in an embodiment of the present application. Figure 1 As shown, the video content processing system is configured with a visual language model and a large language model. The video content processing system can obtain an input target video, extract a target video frame and video text information based on the target video; obtain visual features of the target video frame, construct a summary generation prompt text based on the visual features of the target video frame and the video text information, input the summary generation prompt text into the visual language model for summary extraction, and obtain a video summary; obtain a predefined attribute extraction prompt text, input the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtain first video attribute information of the target video frame; input the video summary and the first video attribute information into the large language model to generate second video attribute information of the target video, so as to realize structured extraction of video attributes.

[0048] The video content processing system in the embodiment of the present application can efficiently obtain the key content in the video by extracting the target video frame and video text information according to the input target video; by obtaining the visual features of the target video frame and combining the video text information to construct a summary to generate a prompt text, the visual and text features in the target video can be comprehensively utilized for multimodal fusion; by taking the predefined attribute extraction prompt text and the target video frame as input, the visual language model is used for attribute extraction, so that the attribute extraction is more flexible, and this method can adjust the attribute extraction prompt text as needed to extract different video attribute information. By inputting the video summary and the first video attribute information into the large language model, the second video attribute information of the target video can be generated, and the structured extraction of the video attribute can be realized. The method based on the embodiment of the present application makes full use of the capabilities of large pre-trained models, such as visual language models and large language models. Since these models have been pre-trained on a large amount of data, they contain rich and in-depth information. By using these pre-trained models, various different attributes of the video from text to vision can still be extracted without preparing a large amount of sample data for the relevant video attributes. Since this method does not rely on domain-specific models or labels for transfer learning, it has potential cross-domain applicability and can be applied on video data from different domains without requiring extensive domain adaptation work.

[0049] The following is a detailed description of the video content processing method provided by the present application based on the above system architecture but not limited to the above architecture in combination with specific embodiments.

[0050] See also Figure 2 , is a flow chart of a video content processing method provided in an embodiment of the present application. Figure 2 As shown, the video content processing method of the embodiment of the present application may include the following steps S101-S104.

[0051] S101, obtaining an input target video, and extracting a target video frame and video text information according to the target video;

[0052] The video content processing method provided in the embodiment of the present application can be implemented by a computer program and can be run on a video content processing device. The computer program can be integrated in an application or run as an independent tool application. The video content processing device in the embodiment of the present application includes, but is not limited to, a terminal device such as a mobile phone, a personal computer, a tablet computer, a handheld device, a wearable device, a computing device, or other processing devices connected to a wireless modem, and the video content processing device can also be a module on the terminal device that can implement the video content processing method.

[0053] In some embodiments, an input target video is obtained, and the target video is a video used for extracting video attribute information. The content of the target video may include, but is not limited to, text, sound, characters, scenes, and other information. The input method of the target video may be to obtain a target video pre-stored in a storage module to which the video content processing device belongs, or to receive a target video transmitted by an external device that has a communication relationship with the video content processing device, or to control a camera module configured by the video content processing device to shoot a target video in real time, etc.

[0054] According to some embodiments, for the acquired target video, a target video frame and video text information are extracted from the target video. The target video frame can be extracted by, for example, extracting frames from the target video based on a preset rate, for example, extracting frames from the target video at a rate of 1 frame per second, and using the extracted video frames as the target video frames.

[0055] In some embodiments, the method of extracting the target video frame may also be, for example, extracting frames from the target video to obtain a video frame set, wherein the video frame set includes at least two video frames; performing visual feature extraction on each video frame in the video frame set to obtain a visual feature set; clustering each visual feature in the video feature set to obtain cluster centers, and the number of cluster centers is one or more; for each obtained cluster center, obtaining the video frame corresponding to each cluster center from the video frame set, and determining the video frame corresponding to each cluster center as the target video frame.

[0056] In some embodiments, the method of extracting the target video frame may also be, for example, extracting frames from the target video to obtain a video frame set, wherein the video frame set includes at least two video frames; performing visual feature extraction on each video frame in the video frame set to obtain a visual feature set; determining the similarity between each visual feature in the visual feature set to obtain a feature similarity result; determining repeated visual features from the visual feature set based on the feature similarity result, and filtering the repeated visual features to obtain a filtered visual feature set; clustering each visual feature in the filtered visual feature set to obtain cluster centers, the number of cluster centers is one or more; for each obtained cluster center, obtaining the video frame corresponding to each cluster center from the video frame set, and determining the video frame corresponding to each cluster center as the target video frame.

[0057] In some embodiments, the video text information may include at least one of video subtitle information, voice recognition information, and screen text information. The method of extracting the video text information may be, for example, using voice recognition technology to perform voice recognition on the sound audio in the target video to obtain voice recognition information; using video screen recognition technology to perform text recognition on the video screen in the target video to obtain screen text information; if the target video has subtitles, extracting subtitles from the target video to obtain video subtitle information; integrating the obtained voice recognition information, screen text information, and / or video subtitle information into the video text information of the target video.

[0058] In some embodiments, after the video text information is extracted, predefined keywords for target business attributes may also be obtained. The target business attribute refers to the video attribute set for the video content that the user needs to extract, such as but not limited to the video attributes such as the scene, subject object, character, behavior and text information involved in the video content. For the keywords of the target video attributes obtained, the keywords are matched with the video text information to obtain the target text information matching the keywords, and the target text information is determined as the target business attribute information of the target video.

[0059] S102, obtaining visual features of the target video frame, constructing summary generation prompt text according to the visual features of the target video frame and the video text information, and inputting the summary generation prompt text into a visual language model for summary extraction to obtain a video summary;

[0060] In some embodiments, for the extracted target video frame, the visual features of the target video frame are obtained; based on the visual features of the target video frame and the video text information of the target video, a summary generation prompt text is constructed, and the summary generation prompt text is used to instruct the model to output a video summary according to the video material and summary generation requirements provided in the prompt text. The way to construct the summary generation prompt text may be, for example, to obtain a predefined summary generation prompt template, and the summary generation prompt template may include a summary generation requirement placeholder, a visual feature placeholder of the target video frame, and a video text information placeholder; the summary generation requirements are input into the summary generation requirement placeholder of the summary generation prompt template, and the obtained visual features and video text information are correspondingly filled into the visual feature placeholder and the video text information placeholder to construct the summary generation prompt text.

[0061] According to some embodiments, for the constructed summary generation prompt text, the summary generation prompt text is input into a visual language model, the visual language model is used to perform summary extraction, and the video summary is output.

[0062] In some embodiments, a method of generating a video summary using a visual language model may, for example, include inputting summary generation prompt text into a visual language model, wherein the summary generation prompt text is constructed by visual features of a target video frame and video text information of a target video; in order to align the visual features of the target video frame with the text information to ensure the model's accurate understanding of the video content, the text features of the target video frame are obtained from the video text information contained in the summary generation prompt text through the visual language model; and using the visual language model, a summary is extracted based on the visual features and text features of the target video frame to obtain a video summary of the target video frame.

[0063] S103, obtaining a predefined attribute extraction prompt text, inputting the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtaining first video attribute information of the target video frame;

[0064] In some embodiments, a predefined attribute extraction prompt text is obtained, and the attribute extraction prompt text is constructed according to the video attribute extraction requirements of the target video. For example, the attribute extraction prompt text may be "Which [video attribute placeholder to be extracted: food] appear in the input target video frame?".

[0065] According to some embodiments, the acquired attribute extraction prompt text and the target video frame are input into a visual language model, the visual language model is used to extract attributes of the target video frame, and the first video attribute information of the target video frame is output.

[0066] In some embodiments, the method of extracting the first video attribute information using a visual language model may be, for example, obtaining a question element of an attribute extraction prompt text, the question element refers to key information or a question object identified and extracted from the attribute extraction prompt text. Based on the question element, multiple sub-attribute extraction prompt texts related to the attribute extraction prompt text are generated. For example, for the attribute extraction prompt text "Which [video attribute placeholder to be extracted: food] appear in the input target video frame?", the question element "which food" of the attribute extraction prompt text is obtained, and multiple sub-attribute extraction prompt texts related to the attribute extraction prompt text are generated from multiple questioning angles for the question element. For example, the sub-attribute extraction prompt text may include "Is the scene in the target video frame a kitchen or a restaurant?", "Do the items appearing in the target video frame include vegetables and fruits?" and "Do the fruits appearing in the target video frame include apples?", etc. Afterwards, the sub-attribute extraction prompt text and the target video frame are input into the visual language model, and the target video frame is subjected to attribute extraction through the visual language model to obtain the third video attribute information corresponding to each sub-attribute extraction prompt text, and the third video attribute information is output as the first video attribute information corresponding to the target video frame.

[0067] S104: Input the video summary and the first video attribute information into a large language model, and use the large language model to generate second video attribute information of the target video.

[0068] In some embodiments, the video summary and the first video attribute information output by the visual language model are input into a large language model, and the large language model is used to integrate the information to generate the second video attribute information of the target video.

[0069] In some embodiments, when the visual language model outputs the third video attribute information corresponding to each extracted sub-attribute extraction prompt text as the first video attribute information corresponding to the target video frame, the large language model is used to generate the second video attribute information. For example, the attribute detection standard corresponding to each sub-attribute extraction prompt text can be obtained; that is, for each sub-attribute extraction prompt text, an attribute detection standard is set respectively, and the attribute detection standard is used to detect whether the third video attribute information extracted by the model according to the sub-attribute extraction prompt text meets the requirements of the video attribute. Afterwards, the large language model is used to perform information verification on the third video attribute information according to the attribute detection standard and the video summary. If the information verification passes, the fourth video attribute information corresponding to the target video frame is obtained; based on the fourth video attribute information, the second video attribute information of the target video is generated and output.

[0070] In the embodiment of the present application, by extracting the target video frame and video text information according to the input target video, the key content in the video can be efficiently obtained; by obtaining the visual features of the target video frame, and combining the video text information to construct a summary to generate a prompt text, the visual and text features in the target video can be comprehensively utilized for multimodal fusion; by taking the predefined attribute extraction prompt text and the target video frame as input, the visual language model is used for attribute extraction, so that the attribute extraction is more flexible, and the attribute extraction prompt text can be adjusted as needed to extract different video attribute information. By inputting the video summary and the first video attribute information into the large language model, the second video attribute information of the target video can be generated, and the structured extraction of the video attribute can be realized. The method based on the embodiment of the present application makes full use of the capabilities of large pre-trained models, such as visual language models and large language models. Since these models have been pre-trained on a large amount of data, they contain rich and in-depth information. By using these pre-trained models, various different attributes of the video from text to vision can still be extracted without preparing a large amount of sample data for the relevant video attributes. Since this method does not rely on domain-specific models or labels for transfer learning, it has potential cross-domain applicability and can be applied on video data from different domains without requiring extensive domain adaptation work.

[0071] See also Figure 3 , is a flow chart of a video content processing method provided in an embodiment of the present application. Figure 3 As shown, the method described in the embodiment of the present application may include the following steps S201-S212.

[0072] S201, obtaining an input target video, and extracting video text information according to the target video;

[0073] In some embodiments, an input target video is obtained, and video text information is extracted based on the target video. The video text information may include but is not limited to voice recognition information, video subtitle information, and screen text information.

[0074] According to some embodiments, the method of extracting video text information may be, for example, using speech recognition technology to perform speech recognition on the sound audio in the target video to obtain speech recognition information; using video screen recognition technology to perform text recognition on the video screen in the target video to obtain screen text information; if the target video has subtitles, then extracting subtitles from the target video to obtain video subtitle information; integrating the obtained speech recognition information, screen text information and / or video subtitle information into the video text information of the target video.

[0075] S202, extracting frames from the target video to obtain a video frame set, where the video frame set includes at least two video frames;

[0076] In some embodiments, the acquired target video is subjected to frame extraction to obtain a video frame set, wherein the video frame set includes at least two video frames. Specifically, the target video is subjected to frame extraction based on a preset rate, for example, the target video is subjected to frame extraction at a rate of 1 frame per second, and the video frame set is constructed based on the extracted video frames.

[0077] S203, extracting visual features from each video frame in the video frame set to obtain a visual feature set;

[0078] In some embodiments, visual features are extracted from each video frame in the video frame set to obtain a visual feature set. Specifically, for each video frame in the video frame set, an image feature extraction algorithm is used to extract visual features in the video frame. For example, the image feature extraction algorithm used may be the DinoV2 algorithm. Based on the DinoV2 algorithm, each video frame is converted into dimensional visual features F v , where N is the number of video frames and D is the dimension of visual features, which is generally 1024. dimensional visual features F v A set of visual features is constructed.

[0079] S204, determining the similarity between each visual feature in the visual feature set to obtain a feature similarity result;

[0080] In some embodiments, in order to filter repeated visual features in a visual feature set, a feature similarity result is obtained by determining the similarity between each visual feature in the visual feature set. For example, each visual feature in the visual feature set is represented as a vector or matrix, and the similarity is calculated for each pair of visual features in the visual feature set using methods such as Euclidean distance, cosine similarity, or Hamming distance to obtain a similarity value for each pair of visual features; and the similarity value for each pair of visual features is used as the feature similarity result.

[0081] S205, determining repeated visual features from the visual feature set according to the feature similarity result, filtering the repeated visual features to obtain a filtered visual feature set;

[0082] In some embodiments, repeated visual features are determined from the visual feature set based on the feature similarity result, and the repeated visual features are filtered to obtain a filtered visual feature set. For example, a pre-set threshold is obtained, and the threshold is matched with the similarity value of each pair of visual features. When the similarity value exceeds the threshold, the pair of visual features corresponding to the similarity value is determined as a highly similar feature pair, and one of the visual features is arbitrarily selected from the highly similar feature pair as a repeated visual feature. The repeated visual features are filtered, and the remaining visual features are aggregated to obtain a filtered visual feature set.

[0083] In the embodiment of the present application, filtering each visual feature in the visual feature set based on the similarity between the visual features helps to avoid excessive concentration or overlap of subsequent clustering results due to too many highly similar visual features.

[0084] S206, clustering each visual feature in the visual feature set to obtain a cluster center;

[0085] Optionally, each visual feature in the filtered visual feature set is clustered to obtain a cluster center.

[0086] S207, acquiring the video frame corresponding to the cluster center from the video frame set, and determining the video frame corresponding to the cluster center as the target video frame;

[0087] In some embodiments, each visual feature in the filtered visual feature set is clustered to obtain cluster centers, and the number of cluster centers is one or more; for each obtained cluster center, the video frame corresponding to each cluster center is obtained from the video frame set, and the video frame corresponding to each cluster center is determined as the target video frame.

[0088] According to some embodiments, the method of clustering each visual feature may be, for example, using a kernel density clustering method to cluster the visual features F v Clustering is performed to obtain cluster centers; the video frames corresponding to each cluster center are extracted from the video frame set, and the video frames corresponding to each cluster center are determined as target video frames, which are recorded as Among them, N k is the number of target video frames (i.e., the number of cluster centers), D is the dimension of visual features, and F kv It can be regarded as a concise representation of the visual information in the target video.

[0089] In the embodiment of the present application, by extracting frames from the target video to obtain a video frame set, the key frames in the video can be efficiently extracted; by extracting visual features from each frame in the video frame set, a visual feature set containing rich information is obtained, which can ensure that the key information of each frame is captured; by clustering the visual feature set, similar video frames can be classified into one category, and the cluster center of each category is found, which helps to remove redundant frames and retain representative video frames; by obtaining the video frame corresponding to the cluster center from the video frame set and determining it as the target video frame, it is ensured that the selected target video frame is representative and informative, which can effectively reduce the number of video frames that need to be processed, thereby improving the efficiency of subsequent video content processing. At the same time, since representative video frames are selected, the accuracy of the processing results is also guaranteed.

[0090] S208, obtaining visual features of the target video frame, constructing a summary and generating a prompt text according to the visual features of the target video frame and the video text information;

[0091] In some embodiments, for the extracted target video frame, the visual features of the target video frame are obtained; based on the visual features of the target video frame and the video text information of the target video, a summary generation prompt text is constructed, and the summary generation prompt text is used to instruct the model to output a video summary according to the video material and summary generation requirements provided in the prompt text.

[0092] According to some embodiments, a method for constructing summary generation prompt text may be, for example, obtaining a predefined summary generation prompt template, which may include a summary generation requirement placeholder, a visual feature placeholder of a target video frame, and a video text information placeholder; inputting summary generation requirements into the summary generation requirement placeholder of the summary generation prompt template, and filling the obtained visual features and video text information into the visual feature placeholder and the video text information placeholder accordingly, so as to construct the summary generation prompt text. For example, the constructed summary generation prompt text may be "Given a target video, wherein the target video has [visual feature 1 placeholder of target video frame 1], [visual feature 2 placeholder of target video frame 1], [visual feature 1 placeholder of target video frame 2], ..., the target video has voice information [extracted voice recognition information placeholder], and the subtitles in the target video are [extracted video subtitle information placeholder], and combining the above information, a set of [summary generation requirement placeholder: within 200 words] video summaries are generated."

[0093] S209, inputting the summary generation prompt text into a visual language model, and acquiring text features of the target video frame from the video text information based on the visual language model;

[0094] S210, using the visual language model to extract a summary based on the visual features and the text features of the target video frame to obtain a video summary of the target video frame;

[0095] In some embodiments, for the constructed summary generation prompt text, the summary generation prompt text is input into a visual language model, the visual language model is used to perform summary extraction, and a video summary is output. Specifically, the summary generation prompt text is input into a visual language model, wherein the summary generation prompt text is constructed by visual features of a target video frame and video text information of a target video. In order to align the visual features of a target video frame with the text information to ensure the model's accurate understanding of the video content, the text features of the target video frame are obtained from the video text information contained in the summary generation prompt text through the visual language model. Using the visual language model, a summary is extracted based on the visual features and text features of the target video frame to obtain a video summary of the target video frame.

[0096] S211, obtaining a predefined attribute extraction prompt text, inputting the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtaining first video attribute information of the target video frame;

[0097] S212: Input the video summary and the first video attribute information into a large language model, and use the large language model to generate second video attribute information of the target video.

[0098] Specifically, for the specific process of steps S211-S212, please refer to the description of steps S103-S104 in the above embodiment, which will not be repeated here.

[0099] In an embodiment of the present application, a summary generation prompt text is constructed according to the visual features and video text information of a target video frame; the summary generation prompt text is input into a visual language model, and the text features of the target video frame are obtained from the video text information based on the visual language model, so that the visual features and text features of the same target video frame can be associated and aligned, thereby improving the model's ability to understand the target video frame; using the visual language model, a summary is extracted according to the visual features and text features of the same target video frame, so that a video summary of the target video frame can be generated more accurately.

[0100] See also Figure 4 , is a flow chart of a video content processing method provided in an embodiment of the present application. Figure 4 As shown, the method described in the embodiment of the present application may include the following steps S301-S314.

[0101] S301, obtaining an input target video, and extracting video text information according to the target video;

[0102] S302, extracting frames from the target video to obtain a video frame set, where the video frame set includes at least two video frames;

[0103] S303, extracting visual features from each video frame in the video frame set to obtain a visual feature set;

[0104] S304, determining the similarity between each visual feature in the visual feature set to obtain a feature similarity result;

[0105] S305, determining repeated visual features from the visual feature set according to the feature similarity result, filtering the repeated visual features to obtain a filtered visual feature set;

[0106] S306, clustering each visual feature in the visual feature set to obtain a cluster center;

[0107] Optionally, each visual feature in the filtered visual feature set is clustered to obtain a cluster center.

[0108] S307, obtaining the video frame corresponding to the cluster center from the video frame set, and determining the video frame corresponding to the cluster center as the target video frame;

[0109] S308, obtaining visual features of the target video frame, constructing summary generation prompt text according to the visual features of the target video frame and the video text information, and inputting the summary generation prompt text into a visual language model for summary extraction to obtain a video summary;

[0110] Specifically, for the specific process of steps S301-S308, please refer to the description of steps S201-S210 in the above embodiment, which will not be repeated here.

[0111] S309, obtaining a predefined attribute extraction prompt text, and obtaining question elements of the attribute extraction prompt text;

[0112] In some embodiments, a predefined attribute extraction prompt text is obtained, and the attribute extraction prompt text is constructed according to the video attribute extraction requirements of the target video. For example, the attribute extraction prompt text may be "Which [video attribute placeholder to be extracted: food] appear in the input target video frame?".

[0113] According to some embodiments, for the attribute extraction prompt text, the question element of the attribute extraction prompt text is obtained. The question element refers to the key information or question object identified and extracted from the attribute extraction prompt text. For example, for the attribute extraction prompt text "Which [video attribute placeholder to be extracted: food] appear in the input target video frame?", the question element "which food" in the attribute extraction prompt text can be obtained.

[0114] S310, generating a plurality of sub-attribute extraction prompt texts related to the attribute extraction prompt text based on the question element;

[0115] In some embodiments, for the obtained question elements, multiple sub-attribute extraction prompt texts related to the attribute extraction prompt text are generated from multiple questioning angles. For example, for the question element "which food" in the above attribute extraction prompt text, the generated sub-attribute extraction prompt text may include "Is the scene in the target video frame a restaurant, dining table or food outlet?", "Does the main object appearing in the target video frame include food or beverages?", "Do the characters appearing in the target video frame include chefs, waiters or deliverymen?" and "Do the behaviors appearing in the target video frame include cooking, promoting or introducing dishes?" etc.

[0116] S311, inputting the sub-attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtaining third video attribute information corresponding to each sub-attribute extraction prompt text as first video attribute information corresponding to the target video frame;

[0117] In some embodiments, the sub-attribute extraction prompt text and the target video frame are input into a visual language model, and the target video frame is subjected to attribute extraction through the visual language model to obtain third video attribute information corresponding to each sub-attribute extraction prompt text, and the third video attribute information is output as the first video attribute information corresponding to the target video frame.

[0118] In an embodiment of the present application, by obtaining the question elements of the attribute extraction prompt text, the attribute type and specific content that needs to be extracted can be accurately understood; multiple sub-attribute extraction prompt texts are generated based on the question elements, which increases the diversity and comprehensiveness of attribute extraction, and helps to extract the attribute information of the target video frame from different angles and levels, and achieve higher-precision attribute recognition. By combining the sub-attribute extraction prompt text and the target video frame, the attribute of the target video frame is extracted using a visual language model, and a flexible attribute extraction mechanism is implemented. This mechanism enables the system to adapt to changes in different attributes and questioning methods, and improves the scalability and adaptability of the system. Through the attribute extraction of the visual language model, the third video attribute information corresponding to each sub-attribute extraction prompt text is obtained as the first video attribute information corresponding to the target video frame, ensuring the accuracy and completeness of the attribute information.

[0119] S312, obtaining the attribute detection standard corresponding to the sub-attribute extraction prompt text;

[0120] S313, based on the large language model, performing information verification on the third video attribute information according to the attribute detection standard and the video summary to obtain fourth video attribute information corresponding to the target video frame;

[0121] S314: Generate second video attribute information of the target video based on the fourth video attribute information.

[0122] In some embodiments, the attribute detection standard corresponding to each sub-attribute extraction prompt text is obtained; that is, for each sub-attribute extraction prompt text, an attribute detection standard is set respectively, and the attribute detection standard is used to detect whether the third video attribute information extracted by the model according to the sub-attribute extraction prompt text meets the requirements of the video attribute. Afterwards, the third video attribute information is verified using the large language model according to the attribute detection standard and the video summary. If the information verification passes, the fourth video attribute information corresponding to the target video frame is obtained; based on the fourth video attribute information, the second video attribute information of the target video is generated and output.

[0123] For example, refer to Figure 5In order to meet the analysis requirements of short videos of business types, the video attribute extraction requirements are determined from five different dimensions: scene, subject object, person, behavior and text information. The attribute extraction prompt text is defined according to the video attribute extraction requirements. For example, the defined attribute extraction prompt text can be "Does [video attribute placeholder to be extracted: scene / subject object / person / behavior / text information] appear in the input target video frame?", where the scene can include a dining table, a restaurant, a food outlet or others; the subject object can include food, beverage or other specific dish names; the person can include a chef, a service staff, a delivery man or others; the behavior can include cooking, promotion, and introduction of dishes; the text can include automatic speech recognition (ASR) text, optical character recognition (OCR) text, and subtitle files made based on video subtitle editing software (SubRip Subtitles, SRT).

[0124] Assume that, for the attribute extraction prompt text, one of the question elements is obtained, "Does the scene [restaurant] appear?", refer to Figure 6 For the obtained question elements, multiple sub-attribute extraction prompt texts related to the attribute extraction prompt text are generated from multiple question angles, such as sub-attribute extraction prompt text 1 generated from angle 1 "Does the scene in the target video frame show a dining table?", sub-attribute extraction prompt text 2 generated from angle 2 "Does the main object appearing in the target video frame include food?" and sub-attribute extraction prompt text 3 generated from angle 3 "Does the person appearing in the target video frame include a chef?", etc.

[0125] Multiple sub-attribute extraction prompt texts and target video frames generated from different angles are input into a visual language model (VLLM), and attribute extraction is performed on the target video frame to obtain video attribute information corresponding to each sub-attribute extraction prompt text.

[0126] Get the attribute detection standard corresponding to each sub-attribute extraction prompt text. For example, the attribute detection standard for sub-attribute extraction prompt text 1 is "The definition of a dining table refers to a table used for eating, which can be divided into solid wood dining tables, steel-wood dining tables, marble dining tables, marble dining tables, marble coffee tables, jade dining tables, jade dining tables, jade coffee tables, marble dining tables, etc. according to the material." The attribute detection standard for sub-attribute extraction prompt text 2 is "The definition of food refers to various finished products and raw materials for human consumption or drinking, as well as items that are traditionally both food and medicine, but does not include items for therapeutic purposes." The attribute detection standard for sub-attribute extraction prompt text 3 is "The definition of a chef is a person who takes cooking as his profession and cooking dishes as his main job content."

[0127] The large language model (LLM) is used to verify the video attribute information corresponding to each sub-attribute extraction prompt text output by the VLLM model according to the video summary of the target video and the above-mentioned attribute detection standard, and then the video attribute information that passes the information verification is output. The video attribute information that passes the information verification is structured and output.

[0128] In the embodiment of the present application, by using the sub-attribute to extract the attribute detection standard corresponding to the prompt text, a basis is provided for the subsequent attribute information verification, which helps to ensure that the generated second video attribute information meets the business requirements and standards. By using the large language model to verify the third video attribute information according to the attribute detection standard and the video summary, the fourth video attribute information corresponding to the target video frame is obtained, thereby improving the accuracy and reliability of the video attribute information. The second video attribute information of the target video is generated based on the fourth video attribute information, further optimizing the structure and expression of the attribute information, making it more in line with business requirements and application scenarios.

[0129] Based on the embodiment of the present application, when new attribute requirements are added, there is no need to retrain the existing model, but support for new attributes can be achieved by constructing attribute extraction prompt text. This flexible attribute expansion mechanism enables the system to quickly adapt to changing transaction requirements, providing greater flexibility and scalability for video attribute recognition.

[0130] See also Figure 7 , is a flow chart of a video content processing method provided in an embodiment of the present application. Figure 7 As shown, the method described in the embodiment of the present application may include the following steps S401-S407.

[0131] S401, obtaining an input target video, and extracting a target video frame according to the target video;

[0132] In some embodiments, an input target video is obtained, and the target video is a video used for extracting video attribute information. The content of the target video may include, but is not limited to, text, sound, characters, scenes, and other information. A target video frame is extracted from the target video, wherein the target video frame is extracted by, for example, extracting frames from the target video based on a preset rate, for example, extracting frames from the target video at a rate of 1 frame per second, and using the extracted video frames as the target video frames.

[0133] In some embodiments, the method of extracting the target video frame may also be, for example, extracting frames from the target video to obtain a video frame set, wherein the video frame set includes at least two video frames; performing visual feature extraction on each video frame in the video frame set to obtain a visual feature set; clustering each visual feature in the video feature set to obtain cluster centers, and the number of cluster centers is one or more; for each obtained cluster center, obtaining the video frame corresponding to each cluster center from the video frame set, and determining the video frame corresponding to each cluster center as the target video frame.

[0134] S402, extracting video text information according to the target video, wherein the video text information includes at least one of video subtitle information, voice recognition information, and screen text information, including:

[0135] Extracting subtitles from the target video to obtain video subtitle information; and / or,

[0136] Performing speech recognition on the target video to obtain speech recognition information; and / or,

[0137] Perform text recognition on the video screen in the target video to obtain screen text information.

[0138] In some embodiments, video text information is extracted from the target video, and the video text information may include but is not limited to voice recognition information, video subtitle information, and screen text information. The method of extracting video text information may be, for example, to perform voice recognition on the sound and audio in the target video using voice recognition technology to obtain voice recognition information; to perform text recognition on the video screen in the target video using video screen recognition technology to obtain screen text information; if the target video has subtitles, extract subtitles from the target video to obtain video subtitle information; and integrate one or more of the obtained voice recognition information, screen text information, and video subtitle information into the video text information of the target video.

[0139] In some embodiments, the method of performing speech recognition on the target video can be, for example, using automatic speech recognition technology (ASR, Automatic Speech Recognition) to extract each video frame from the target video, separating the audio stream in the video frame through a video codec, using an audio processing tool to extract the speech signal from the audio stream, performing feature extraction on the speech information to convert the speech information into speech features that can be processed by a computer, using a speech recognition algorithm to recognize the speech features, and converting the recognition results into text form to obtain speech recognition information.

[0140] In some embodiments, the method of performing text recognition on the video screen of the target video may be, for example, to use optical character recognition technology (OCR) to decompose the target video into continuous video frames, determine the text area that needs to be recognized in the video screen of the video frame, perform image enhancement processing on the determined text area, perform text recognition on the enhanced text area through OCR software, and integrate the text recognition results to form screen text information.

[0141] In some embodiments, the method of extracting subtitles from a target video may be, for example, for a target video that uses video subtitle editing software (SubRip Subtitles, SRT) to create a subtitle file, parsing and processing the SRT file of the target video to obtain video subtitle information of the target video; or, for a target video that does not rely on video subtitle editing software to create a subtitle file, using OCR technology to obtain a text detection box that appears in each video frame of the target video, counting the probability of the text detection box appearing at the height of the video screen in the entire target video, so as to locate the position where the subtitle appears in each video frame, and extracting the video subtitle information based on the position.

[0142] S403, obtaining predefined keywords for target transaction attributes;

[0143] S404, matching the keyword with the video text information to obtain target text information matching the keyword, and determining the target text information as target transaction attribute information of the target video;

[0144] In some embodiments, predefined keywords for target business attributes are obtained. The target business attribute refers to the video attribute set for the video content that the user needs to extract, such as but not limited to the video attributes such as scenes, main objects, characters, behaviors and text information involved in the video content. For each target video attribute, keywords related to the target video attribute are pre-collected, and the keywords refer to specific words used to describe or identify the target video attribute. For example, keywords related to the scene may include kitchen, restaurant, dining table, food outlet, etc.; keywords related to the main object may include food, beverage, mobile phone, electric car, etc.; keywords related to the character may include chef, service staff, delivery man, customer, etc.; keywords related to behavior may include cooking, promotion, accepting dishes, eating, etc.; keywords related to text information may include subtitles, screen text, voice content, etc. For the keywords of the target video attribute obtained, the keyword is matched with the video text information to obtain the target text information matching the keyword, and the target text information is determined as the target business attribute information of the target video.

[0145] S405, obtaining visual features of the target video frame, constructing summary generation prompt text according to the visual features of the target video frame and the video text information, and inputting the summary generation prompt text into a visual language model for summary extraction to obtain a video summary;

[0146] S406, obtaining a predefined attribute extraction prompt text, inputting the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtaining first video attribute information of the target video frame;

[0147] S407: Input the video summary and the first video attribute information into a large language model, and use the large language model to generate second video attribute information of the target video.

[0148] Specifically, for the specific process of steps S405-S407, please refer to the description of steps S102-S104 in the above embodiment, which will not be repeated here.

[0149] For example, refer to Figure 8 , obtain the input target video (i.e., original video), use ASR technology to perform speech recognition on the target video to obtain speech recognition information; extract the subtitles of the target video to obtain video subtitle information. Decompose the target video into continuous video frames (i.e., original frames), use OCR technology to perform text recognition on the video screen of the video frame to obtain screen text information. Summarize the speech recognition information, video subtitle information and screen text information to obtain the global video text information of the target video. Obtain predefined keywords for target transaction attributes, match the keywords with video text information, obtain target text information matching the keywords, and determine the target text information as the target transaction attribute information of the target video to realize the detection of relevant target transaction attributes in the video.

[0150] For the original frame of the target video, an image feature extraction algorithm is used to extract the visual features in the original frame. For example, the image feature extraction algorithm used can be the DinoV2 algorithm. Based on the DinoV2 algorithm, each original frame is converted into dimensional visual features F v , where N is the number of original frames and D is the dimension of visual features, which is generally 1024. dimensional visual features F v A set of visual features is constructed.

[0151] Determine the similarity between each visual feature in the visual feature set to obtain a feature similarity result. According to the feature similarity result, determine the repeated visual features from the visual feature set, and filter the repeated visual features; at the same time, determine the APP screenshots and non-real shots in the visual feature set, filter the repeated visual features, APP screenshots and non-real shots, and obtain a filtered visual feature set.

[0152] Cluster each visual feature in the filtered visual feature set to obtain cluster centers, where the number of cluster centers is one or more; for each obtained cluster center, obtain the video frame corresponding to each cluster center from the video frame set, and determine the video frame corresponding to each cluster center as the target video frame (i.e., key frame).

[0153] For the extracted target video frame, the visual features of the target video frame are obtained; based on the visual features of the target video frame and the video text information of the target video, a summary generation prompt text (prompt) is constructed, such as "There are the following key images [visual features of the target video frame], and [video text information], combined with the above images, a 200-word summary is generated". The summary generation prompt text is input into the visual language model (VLLM), and the visual language model is used to extract the summary, and the overall video summary is output.

[0154] Obtain a predefined attribute extraction prompt text (prompt), such as "What food / person / scene / subject appears in the picture?", input the attribute extraction prompt text and the target video frame into the visual language model, extract the attributes of the target video frame, and obtain the first video attribute information of the target video frame; input the overall video summary of the target video frame and the first video attribute information into the large language model to generate the second video attribute information of the target video. The second video attribute information may include the food / person / scene / subject in the video clip represented by the key frame.

[0155] like Fig. 9 As shown, multiple predefined attribute extraction prompt texts are obtained, such as "Is the picture a real scene?", "Is the subject in the picture a person?", "Is the picture indoors?", "Is the scene in the picture in a restaurant?", "Does the picture contain food?", and "What food is in the picture?", combined with the target video frame (refer to Fig. 9 ), extract attributes through the visual language model to obtain the first video attribute information of the target video frame. Obtain the video summary of the target video frame, combine the first video attribute information, and generate the second video attribute information of the target video through the large language model. Fig. 9on the right side) can include: belonging to a real scene (is_realscences: 1), the subject is not a person (is_mainobject_people: 0), indoors (indoor: 1), not in a restaurant (in_restaurant: 0), food appears (has_food: 1), and the food is grilled lamb chops (foods: grilled lamb chops).

[0156] In an embodiment of the present application, by extracting the video subtitle information, voice recognition information and screen text information of the target video to obtain the video text information, the text, sound and video screen information contained in the target video can be obtained more comprehensively to avoid missing important attribute content in the target video.

[0157] The following will be combined with the attached Fig.10 In the embodiment of the present application, the video content processing device provided by the embodiment of the present application is described in detail. Fig.10 The video content processing device in the embodiment may be the terminal in the above embodiment, which is used to execute the present application Figure 2-Figure 9 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2-Figure 9 The embodiment shown.

[0158] See also Fig.10 , which shows a schematic diagram of the structure of a video content processing device provided by an exemplary embodiment of the present application. The video content processing device 1 can be implemented as all or part of the device through software, hardware or a combination of both. The device 1 includes a video acquisition unit 10, a summary generation unit 20, an attribute extraction unit 30 and an information generation unit 40.

[0159] The video acquisition unit 10 is used to acquire an input target video and extract a target video frame and video text information according to the target video;

[0160] A summary generation unit 20 is used to obtain visual features of the target video frame, construct summary generation prompt text according to the visual features of the target video frame and the video text information, and input the summary generation prompt text into a visual language model for summary extraction to obtain a video summary;

[0161] The attribute extraction unit 30 is used to obtain a predefined attribute extraction prompt text, input the attribute extraction prompt text and the target video frame into the visual language model to extract the attribute of the target video frame, and obtain the first video attribute information of the target video frame;

[0162] The information generating unit 40 is configured to input the video summary and the first video attribute information into a large language model, and use the large language model to generate second video attribute information of the target video.

[0163] According to some embodiments, the video acquisition unit 10 includes a video frame extraction subunit 101, a feature extraction subunit 102, a feature clustering subunit 103 and a video frame determination subunit 104, wherein the video acquisition unit 10 is used to extract and obtain a target video frame according to the target video:

[0164] The video frame extraction subunit 101 is used to extract frames from the target video to obtain a video frame set, where the video frame set includes at least two video frames;

[0165] The feature extraction subunit 102 is used to extract visual features from each video frame in the video frame set to obtain a visual feature set;

[0166] A feature clustering subunit 103, configured to cluster each visual feature in the visual feature set to obtain a cluster center;

[0167] The video frame determination subunit 104 is configured to obtain the video frame corresponding to the cluster center from the video frame set, and determine the video frame corresponding to the cluster center as a target video frame.

[0168] According to some embodiments, the video acquisition unit 10 further includes a similarity determination subunit 105 and a feature filtering subunit 106, wherein the feature extraction subunit 102 is configured to extract visual features from each video frame in the video frame set, and after obtaining the visual feature set:

[0169] A similarity determination subunit 105 is used to determine the similarity between the visual features in the visual feature set to obtain a feature similarity result;

[0170] A feature filtering subunit 106, configured to determine repeated visual features from the visual feature set according to the feature similarity result, and filter the repeated visual features to obtain a filtered visual feature set;

[0171] The feature clustering subunit 103 is further configured to cluster the visual features in the filtered visual feature set to obtain a cluster center.

[0172] According to some embodiments, the video acquisition unit 10 includes a subtitle extraction subunit 107, a speech recognition subunit 108, and a picture recognition subunit 109, wherein the video acquisition unit 10 is used to extract video text information according to the target video:

[0173] The subtitle extraction subunit 107 is used to extract subtitles from the target video to obtain video subtitle information; and / or,

[0174] The speech recognition subunit 108 is used to perform speech recognition on the target video to obtain speech recognition information; and / or,

[0175] The picture recognition subunit 109 is used to perform text recognition on the video picture in the target video to obtain picture text information.

[0176] According to some embodiments, the apparatus 1 further includes a keyword acquisition unit 50 and an attribute matching unit 60 .

[0177] A keyword acquisition unit 50, used to acquire predefined keywords for target transaction attributes;

[0178] The attribute matching unit 60 is used to match the keyword with the video text information to obtain target text information matching the keyword, and determine the target text information as the target transaction attribute information of the target video.

[0179] According to some embodiments, the summary generation unit 20 includes a text feature acquisition subunit 201 and a summary extraction subunit 202, wherein the summary generation unit 20 is used to input the summary generation prompt text into the visual language model for summary extraction, and when the video summary is obtained:

[0180] A text feature acquisition subunit 201 is used to input the summary generation prompt text into a visual language model, and acquire text features of the target video frame from the video text information based on the visual language model;

[0181] The summary extraction subunit 202 is used to use the visual language model to extract a summary according to the visual features and the text features of the target video frame to obtain a video summary of the target video frame.

[0182] According to some embodiments, the attribute extraction unit 30 includes an element acquisition subunit 301, a text generation subunit 302 and an attribute extraction subunit 303, wherein the attribute extraction unit 30 is used to input the attribute extraction prompt text and the target video frame into the visual language model to extract the attributes of the target video frame, and obtain the first video attribute information of the target video frame:

[0183] An element acquisition subunit 301 is used to acquire question elements of the attribute extraction prompt text;

[0184] A text generation subunit 302, for generating a plurality of sub-attribute extraction prompt texts related to the attribute extraction prompt text based on the question elements;

[0185] The attribute extraction subunit 303 is used to input the sub-attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtain the third video attribute information corresponding to each sub-attribute extraction prompt text as the first video attribute information corresponding to the target video frame.

[0186] According to some embodiments, the information generating unit 40 includes a standard obtaining subunit 401, an information checking subunit 402 and an information generating subunit 403, wherein the information generating unit 40 is used to input the video summary and the first video attribute information into a large language model, and use the large language model to generate the second video attribute information of the target video:

[0187] The standard acquisition subunit 401 is used to acquire the attribute detection standard corresponding to the sub-attribute extraction prompt text;

[0188] An information verification subunit 402 is used to perform information verification on the third video attribute information based on the large language model, according to the attribute detection standard and the video summary, so as to obtain fourth video attribute information corresponding to the target video frame;

[0189] The information generating subunit 403 is configured to generate second video attribute information of the target video based on the fourth video attribute information.

[0190] In the embodiment of the present application, by extracting the target video frame and video text information according to the input target video, the key content in the video can be efficiently obtained; by obtaining the visual features of the target video frame, and combining the video text information to construct a summary to generate a prompt text, the visual and text features in the target video can be comprehensively utilized for multimodal fusion; by taking the predefined attribute extraction prompt text and the target video frame as input, the visual language model is used for attribute extraction, so that the attribute extraction is more flexible, and the attribute extraction prompt text can be adjusted as needed to extract different video attribute information. By inputting the video summary and the first video attribute information into the large language model, the second video attribute information of the target video can be generated, and the structured extraction of the video attribute can be realized. The method based on the embodiment of the present application makes full use of the capabilities of large pre-trained models, such as visual language models and large language models. Since these models have been pre-trained on a large amount of data, they contain rich and in-depth information. By using these pre-trained models, various different attributes of the video from text to vision can still be extracted without preparing a large amount of sample data for the relevant video attributes. Since this method does not rely on domain-specific models or labels for transfer learning, it has potential cross-domain applicability and can be applied on video data from different domains without requiring extensive domain adaptation work.

[0191] Please refer to Fig.11 , which shows a schematic diagram of the structure of the server provided in the embodiment of the present application. The server in the embodiment of the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0192] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire server, and executes various functions and processes data of the server 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of CPU, graphics processing unit (GPU), modem, etc. Among them, the CPU mainly processes the operating system, user pages and applications, etc.; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but implemented separately through a communication chip.

[0193] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable medium (Non-Transitory Computer-Readable Storage Medium). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems.

[0194] The memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, the requirements for system resources in different application scenarios in the same third-party application are also different. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0195] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0196] The input device 130 is used to receive input commands or data, and includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is used to output commands or data, and includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays.

[0197] The touch display screen can be designed as a full screen, a curved screen or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of the present application.

[0198] In addition, those skilled in the art will appreciate that the structure of the terminal shown in the above drawings does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the terminal also includes a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, and other components, which will not be described in detail here.

[0199] exist Fig.11 In the server shown in FIG. 1 , the processor 110 may be used to call the video content processing application stored in the memory 120 and specifically perform the above-mentioned Figure 2-Figure 9 The video content processing method of the embodiment shown in the figure can be found in the specific execution process. Figure 2-Figure 9The specific description of the illustrated embodiment will not be repeated here.

[0200] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figure 2-Figure 9 The video content processing method of the embodiment shown in the figure can be found in the specific execution process. Figure 2-Figure 9 The specific description of the illustrated embodiment will not be repeated here.

[0201] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor as described above. Figure 2-Figure 9 The video content processing method of the embodiment shown in the figure can be found in the specific execution process. Figure 2-Figure 9 The specific description of the illustrated embodiment will not be repeated here.

[0202] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0203] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

[0204] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A video content processing method, characterized in that: The method comprises: Obtaining an input target video, and extracting a target video frame and video text information according to the target video; Acquire visual features of the target video frame, construct summary generation prompt text according to the visual features of the target video frame and the video text information, input the summary generation prompt text into a visual language model for summary extraction, and obtain a video summary; Obtaining a predefined attribute extraction prompt text, inputting the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame, and obtaining first video attribute information of the target video frame; The video summary and the first video attribute information are input into a large language model, and the large language model is used to generate second video attribute information of the target video.

2. The method according to claim 1, characterized in that Extracting a target video frame according to the target video includes: Extracting frames from the target video to obtain a video frame set, wherein the video frame set includes at least two video frames; Extracting visual features from each video frame in the video frame set to obtain a visual feature set; Clustering each visual feature in the visual feature set to obtain a cluster center; The video frame corresponding to the cluster center is obtained from the video frame set, and the video frame corresponding to the cluster center is determined as the target video frame.

3. The method according to claim 2, characterized in that After extracting visual features from each video frame in the video frame set to obtain a visual feature set, the method further includes: Determining the similarity between each visual feature in the visual feature set to obtain a feature similarity result; Determining repeated visual features from the visual feature set according to the feature similarity result, filtering the repeated visual features to obtain a filtered visual feature set; The clustering of each visual feature in the visual feature set to obtain a cluster center includes: Clustering is performed on each visual feature in the filtered visual feature set to obtain a cluster center.

4. The method according to claim 1, characterized in that The video text information includes at least one of video subtitle information, voice recognition information and screen text information; The step of extracting video text information according to the target video includes: Extracting subtitles from the target video to obtain video subtitle information; and / or, Performing speech recognition on the target video to obtain speech recognition information; and / or, Perform text recognition on the video screen in the target video to obtain screen text information.

5. The method according to any one of claims 1 to 4, characterized in that After extracting the video text information according to the target video, the method further includes: Get predefined keywords for target transaction attributes; The keyword is matched with the video text information to obtain target text information matching the keyword, and the target text information is determined as the target transaction attribute information of the target video.

6. The method according to claim 1, characterized in that The step of inputting the summary generation prompt text into a visual language model for summary extraction to obtain a video summary includes: Inputting the summary generation prompt text into a visual language model, and acquiring text features of the target video frame from the video text information based on the visual language model; The visual language model is used to extract a summary based on the visual features and the text features of the target video frame to obtain a video summary of the target video frame.

7. The method according to claim 1, characterized in that The step of inputting the attribute extraction prompt text and the target video frame into the visual language model to extract attributes of the target video frame to obtain first video attribute information of the target video frame includes: Obtaining question elements of the attribute extraction prompt text; Based on the question elements, generating a plurality of sub-attribute extraction prompt texts related to the attribute extraction prompt text; The sub-attribute extraction prompt text and the target video frame are input into the visual language model to extract attributes of the target video frame, and the third video attribute information corresponding to each sub-attribute extraction prompt text is obtained as the first video attribute information corresponding to the target video frame.

8. A video content processing device, characterized in that: The device comprises: A video acquisition unit, used to acquire an input target video, and extract a target video frame and video text information according to the target video; A summary generation unit, used to obtain visual features of the target video frame, construct summary generation prompt text according to the visual features of the target video frame and the video text information, and input the summary generation prompt text into a visual language model for summary extraction to obtain a video summary; an attribute extraction unit, configured to obtain a predefined attribute extraction prompt text, input the attribute extraction prompt text and the target video frame into the visual language model to extract the attribute of the target video frame, and obtain first video attribute information of the target video frame; The information generating unit is used to input the video summary and the first video attribute information into a large language model, and use the large language model to generate second video attribute information of the target video.

9. A server, characterized in that: The server comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the server executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images