Video content label determination method and device, electronic equipment and product

Abstract

The invention provides a video content label determination method and device, electronic equipment and a product, and the method comprises the steps: detecting a video shot boundary in video data, cutting the video data, and obtaining a video segment corresponding to at least one video shot; extracting a key image frame from the video clip based on the difference between adjacent image frames in the video clip; and identifying the key image frame, generating a description text of the key image frame, and extracting a video content tag corresponding to the video data based on the description text of the key image frame. By adopting the technical scheme of the invention, the description text of the image can be extracted from the key image frame in the video data, the description text can cover details of the video, so that the accuracy of the video content label extracted according to the description text is higher, and the video content label of the video data is automatically extracted, so that the user experience is improved. And the efficiency of determining the video content label is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, storage medium and product for determining a video content label. Background Art

[0002] With the popularization of the Internet and the rise of social media and short video platforms, the creation and dissemination of multimedia content have been greatly promoted. People are more and more inclined to quickly discover video content that matches their interests. Therefore, generating video content tags for video data is a necessary condition for users to search for videos that match their interests.

[0003] The existing technical solutions for determining video content labels are mainly to manually add labels by manually reviewing the content, or to extract keywords as labels by analyzing text information such as video titles and introductions using natural language processing technology. However, the manual method of determining labels mainly relies on people's subjective judgment and professional knowledge, and the efficiency and accuracy of label determination are low. The method of extracting keywords from text information such as video titles and introductions cannot cover the details of videos with less text information, and the accuracy of label determination is low. Summary of the invention

[0004] Based on the above requirements, the present application proposes a method, device, electronic device and product for determining video content tags, which can improve the accuracy and efficiency of determining video content tags.

[0005] To achieve the above objectives, this application proposes the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a method for determining a video content tag is provided, comprising:

[0007] By detecting a video shot boundary in the video data, the video data is cut to obtain a video segment corresponding to at least one video shot;

[0008] extracting key image frames from the video clip based on differences between adjacent image frames in the video clip;

[0009] The key image frame is identified, a description text of the key image frame is generated, and based on the description text of the key image frame, a video content tag corresponding to the video data is extracted.

[0010] Optionally, detecting a video shot boundary in the video data and cutting the video data to obtain a video segment corresponding to at least one video shot includes:

[0011] Determining a boundary image frame of a video shot in the video data based on a difference between adjacent image frames in the video data;

[0012] The video data is cut according to the boundary image frame to obtain a video segment corresponding to at least one video shot.

[0013] Optionally, determining a boundary image frame of a video shot in the video data based on a difference between adjacent image frames in the video data includes:

[0014] Determine a first difference image between the first image frame and the second image frame based on a difference between a pixel grayscale value of a first image frame and a pixel grayscale value of a second image frame in the video data; wherein the first image frame is any image frame other than the first image frame in the video data, and the second image frame is a previous image frame adjacent to the first image frame;

[0015] Based on a preset grayscale threshold, binarization is performed on each pixel in the first difference image to obtain a binarized image corresponding to the first difference image;

[0016] According to the preset connected grayscale value, a connected area in the binary image is determined, and based on the connected area in the binary image, it is judged whether the first image frame is a boundary image frame of a video shot in the video data.

[0017] Optionally, extracting a key image frame from the video segment based on a difference between adjacent image frames in the video segment includes:

[0018] Determining a difference value corresponding to the third image frame based on a difference between a pixel value of a pixel point of a third image frame and a pixel value of a pixel point of a fourth image frame in the video clip; wherein the third image frame is any image frame in the video clip except the first image frame, and the fourth image frame is a previous image frame adjacent to the third image frame;

[0019] A local maximum value is searched from a difference value sequence, and an image frame corresponding to the local maximum value is used as a key image frame in the video clip; the difference value sequence includes difference values ​​corresponding to each image frame in the video clip except the first image frame.

[0020] Optionally, determining the difference value corresponding to the third image frame based on the difference between the pixel value of a pixel point of the third image frame and the pixel value of a pixel point of the fourth image frame in the video clip includes:

[0021] Determining a second difference image between the third image frame and the fourth image frame based on a difference between pixel values ​​of pixels of a third image frame and pixel values ​​of pixels of a fourth image frame in the video clip;

[0022] An average value of pixel values ​​of all pixels in the second difference image is used as a difference value corresponding to the third image frame.

[0023] Optionally, identifying the key image frame and generating a description text of the key image frame includes:

[0024] The key image frame and the pre-built description text generation instruction are input into a pre-trained multimodal large model to obtain the description text of the key image frame.

[0025] Optionally, before extracting the video content tag corresponding to the video data based on the description text of the key image frame, the method further includes:

[0026] Performing text region detection on at least one image frame in the video data, and performing character recognition on the detected text region to obtain image text corresponding to the video data;

[0027] and / or, performing speech recognition on the audio corresponding to the video data to obtain a speech text corresponding to the video data;

[0028] Extracting a video content tag corresponding to the video data based on the description text of the key image frame includes:

[0029] Based on the description text of the key image frame, and the image text corresponding to the video data and / or the voice text corresponding to the video data, the video content tag corresponding to the video data is extracted.

[0030] Optionally, extracting a video content tag corresponding to the video data based on the description text of the key image frame, and the image text corresponding to the video data and / or the voice text corresponding to the video data includes:

[0031] The pre-built label extraction instructions, the description text of the key image frame, and the image text corresponding to the video data and / or the speech text corresponding to the video data are input into the pre-trained large language model to obtain the video content label corresponding to the video data.

[0032] According to a second aspect of an embodiment of the present application, a video content tag determination device is provided, including:

[0033] A video cutting module, configured to cut the video data by detecting a video shot boundary in the video data, and obtain a video segment corresponding to at least one video shot;

[0034] An image frame extraction module, configured to extract key image frames from the video segment based on differences between adjacent image frames in the video segment;

[0035] The tag determination module is used to identify the key image frame, generate a description text of the key image frame, and extract a video content tag corresponding to the video data based on the description text of the key image frame.

[0036] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: a memory and a processor;

[0037] The memory is connected to the processor and is used to store programs;

[0038] The processor is used to implement the above-mentioned video content label determination method by running the program in the memory.

[0039] According to a fourth aspect of an embodiment of the present application, a computer program product is provided, including computer program instructions, which, when executed by a processor, enable the processor to implement the above-mentioned video content tag determination method.

[0040] The video content label determination method proposed in the present application detects the video shot boundaries in the video data, cuts the video data, and obtains a video segment corresponding to at least one video shot; extracts key image frames from the video segment based on the differences between adjacent image frames in the video segment; identifies the key image frames, generates description texts of the key image frames, and extracts the video content labels corresponding to the video data based on the description texts of the key image frames. The technical solution of the present application can extract the description text of the image from the key image frames in the video data, and the description text can cover the details of the video, so that the video content labels extracted according to the description text are more accurate, and the video content labels of the video data are automatically extracted, which improves the efficiency of determining the video content labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A flowchart of a method for determining a video content tag provided in an embodiment of the present application;

[0043] Figure 2A schematic diagram of a process flow for determining a boundary image frame provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of another method for determining a video content tag provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a video content tag determination device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solution of the embodiment of the present application is applicable to the application scenario of video processing, and is specifically used in the application scenario of determining the video content label. The technical solution of the embodiment of the present application can improve the accuracy and efficiency of determining the video content label.

[0048] With the popularization of the Internet and the rise of social media and short video platforms, the creation and dissemination of multimedia content such as pictures, videos, and audio have been greatly promoted. People are increasingly inclined to quickly discover video content that matches their interests. Therefore, generating video content tags for video data is a necessary condition for users to search for videos that match their interests. For example, for a furniture advertising video, artificial intelligence technology can analyze the visual and auditory features of video data, and even emotions and context, and automatically generate accurate and multi-dimensional tags. Generating accurate tags not only helps users save time and improve search efficiency, but also promotes the exposure and conversion rate of the brand's videos.

[0049] The existing technical solutions for determining video content labels mainly fall into the following categories: The first is to manually add labels by manually reviewing the content, but the manual labeling method relies on human subjective judgment and professional knowledge, which affects the efficiency and accuracy of label determination. When labeling large-scale video content, the efficiency and accuracy are even lower. The second is to use natural language processing technology to extract keywords as labels by analyzing text information such as video titles and introductions, but the text information in video labels and introductions is not rich enough to cover all the details of the video, so that the keywords extracted from video labels and introductions cannot accurately describe the video, thereby affecting the accuracy of video content labels.

[0050] Based on this, the present application proposes a method for determining video content tags. This technical solution can realize automatic extraction of video content tags of video data, and enable the video content tags to cover the details of the video, thereby solving the problem of low efficiency and accuracy in determining video content tags in the prior art.

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] Exemplary Methods

[0053] See also Figure 1 As shown, the embodiment of the present application proposes a method for determining a video content tag. The method includes:

[0054] S101. Cut the video data by detecting video shot boundaries in the video data to obtain a video segment corresponding to at least one video shot.

[0055] In this embodiment, the video data for which the video content tag determination is required is first obtained, wherein the obtained video data may be a single video or multiple videos directly uploaded by the user, or may be video data automatically downloaded according to the path of the video file uploaded by the user (such as the video link address). In this embodiment, a video manager is constructed, and the video manager can load and manage videos according to the path of the video file uploaded by the user, wherein the video manager can be constructed using a python construction method.

[0056] In a video data, when the video content changes, it means that the video lens has changed, that is, the video scene has changed. Therefore, in order to extract the text content in the video, it is first necessary to detect the change of the video content, that is, to detect the change of the video lens, and detect the video lens contained in the video data. This embodiment cuts the video data by detecting the video lens boundary in the video data, so as to obtain a video segment corresponding to at least one video lens. Among them, the detection of the video lens boundary in the video data can be detected according to the pixel change between adjacent image frames. When the pixel change between adjacent image frames is large, it means that the video lens may have changed, and the two adjacent image frames are video lens boundaries. When the pixel change between adjacent image frames is not large, it means that the video lens may not have changed, and the two adjacent image frames are not video lens boundaries. After detecting the video lens boundary in the video data, the video data is directly cut according to the video lens boundary, so as to obtain the video segment corresponding to each video lens.

[0057] Specifically, by detecting the video shot boundary in the video data, the video data is cut to obtain a video segment corresponding to at least one video shot, including the following steps:

[0058] First, based on the differences between adjacent image frames in the video data, determine the boundary image frames of video shots in the video data.

[0059] In this embodiment, any two adjacent image frames in the video data are compared to detect the differences between the two adjacent image frames. Based on the differences between the adjacent image frames, it is determined whether a shot transition occurs between the two adjacent image frames. That is, when the differences between the adjacent image frames are large, it is determined that a shot transition occurs between the two adjacent image frames; when the differences between the adjacent image frames are small, it is determined that no shot transition occurs between the two adjacent image frames. Among them, the differences between adjacent image frames can be determined by the differences between the gray values of pixel points in the images. That is, the pixel points with the gray value difference between two adjacent image frames greater than a preset gray threshold can be used as shot transition points, and the pixel points with the gray value difference between two adjacent image frames not greater than the preset gray threshold can be used as non-shot transition points. Based on the number and area of shot transition points and the number and area of non-shot transition points, it is determined whether a shot transition occurs between the two adjacent image frames. Through the above determination, if it is determined that a shot transition occurs between two adjacent image frames, one of the two adjacent image frames is used as the boundary image frame of the video shot in the video data. If it is determined that no shot transition occurs between the two adjacent image frames, it means that there is no boundary image frame among the two adjacent image frames.

[0060] Through the above method, it is possible to determine whether there is a boundary image frame in any two adjacent image frames in the video data, so that all the boundary image frames existing in the video data can be determined.

[0061] Second, cut the video data according to the boundary image frames to obtain video segments corresponding to at least one video shot.

[0062] After determining the boundary image frames in the video data through the above steps, cut the video data according to the boundary image frames, so that video segments corresponding to at least one video shot can be obtained. When it is determined that there is a boundary image frame among the two adjacent image frames based on the differences between adjacent image frames in the video data, the previous image frame of the two adjacent image frames can be used as the boundary image frame. At this time, the boundary image frame is used as the last frame image of a video shot to cut the video data, so as to obtain video segments corresponding to at least one video shot. For example, if it is determined that the a-th frame image and the b-th frame image in the video data are boundary image frames (where a < b), then the first frame image to the a-th frame image are used as the video segment corresponding to the first video shot, the (a + 1)-th frame image to the b-th frame image are used as the video segment corresponding to the second video shot, and the (b + 1)-th frame image to the last frame image are used as the video segment corresponding to the third video shot.

[0063] When it is determined that there is a boundary image frame among two adjacent image frames according to the difference between adjacent image frames in video data, the latter image frame of the two adjacent image frames can also be used as the boundary image frame. At this time, the boundary image frame is used as the first image frame of a video shot to cut the video data, so as to obtain video segments corresponding to at least one video shot. For example, if it is determined that the m-th image frame and the n-th image frame in the video data are boundary image frames (where m < n), then the first image frame to the (m - 1)-th image frame are used as the video segment corresponding to the first video shot, the m-th image frame to the (n - 1)-th image frame are used as the video segment corresponding to the second video shot, and the n-th image frame to the last image frame are used as the video segment corresponding to the third video shot.

[0064] S102. Extract key image frames from the video segment based on the difference between adjacent image frames in the video segment.

[0065] Through the above steps, the video data can be cut into video segments corresponding to at least one video shot. For each video segment, key image frames need to be extracted. Among them, for the extraction of key image frames of any video segment, it is realized according to the difference between adjacent image frames in the video segment. In this embodiment, any two adjacent image frames in the video segment corresponding to the video shot are compared to detect the difference between any two adjacent image frames in the video segment corresponding to the video shot. When the difference between the current adjacent image frames is larger than the difference between its nearby adjacent image frames, it means that the content in the two adjacent image frames has changed, that is, new image content may appear in the changed image frame (i.e., the latter image frame of the adjacent image frames). At this time, the changed image frame (i.e., the latter image frame of the adjacent image frames) needs to be used as the key image frame so as to extract the text content in the key image frame later. Among them, the difference between adjacent image frames can be determined according to the difference between the pixel values of each pixel point in the adjacent image frames.

[0066] Specifically, extracting key image frames from the video segment based on the difference between adjacent image frames in the video segment includes the following steps:

[0067] First, determine the difference value corresponding to the third image frame based on the difference between the pixel values of the pixel points of the third image frame and the fourth image frame in the video segment.

[0068] In this embodiment, two adjacent image frames in a video clip are respectively used as the third image frame and the fourth image frame, wherein the third image frame is any image frame in the video clip except the first image frame, and the fourth image frame is the previous image frame adjacent to the third image frame. In this embodiment, it is first necessary to calculate the difference between the pixel values ​​of the pixels of the third image frame and the pixel values ​​of the pixels of the fourth image frame in the video clip, that is, to calculate the difference between the pixel values ​​of each pixel in the third image frame and the pixel values ​​of each corresponding pixel in the fourth image frame. Then, according to the difference between the pixel values ​​of each pixel in the third image frame and the fourth image frame, the difference value corresponding to the third image frame is determined.

[0069] Specifically, firstly, based on the difference between the pixel values ​​of the pixels of the third image frame and the pixel values ​​of the pixels of the fourth image frame in the video clip, a second difference image between the third image frame and the fourth image frame is determined.

[0070] This embodiment calculates the difference between the pixel values ​​of the pixels of the third image frame and the pixel values ​​of the pixels of the fourth image frame in the video clip, thereby obtaining the pixel value difference corresponding to each pixel in the third image frame and the fourth image frame, and constructs a difference image between the third image frame and the fourth image frame based on the pixel value difference corresponding to each pixel, and uses the difference image as the second difference image. Among them, calculating the difference between the pixel values ​​of the pixels of the third image frame and the pixel values ​​of the fourth image frame can directly calculate the difference between the pixel values ​​of each pixel in the third image frame and the fourth image frame, and can also convert the third image frame and the fourth image frame from the BGR color space to the LUV color space, and then calculate the difference between the pixel values ​​of each pixel in the third image frame after the color space conversion and the fourth image frame after the color space conversion.

[0071] In this embodiment, the third image frame and the fourth image frame are converted from the BGR color space to the LUV color space mainly for the following reasons: First, the LUV color space is designed with the characteristics of the human visual system in mind, especially in terms of uniformity of color perception. In the LUV space, the numerical difference of color is more consistent with the human eye's perception of color difference, which makes it more natural and accurate when performing color difference analysis; second, the LUV space separates brightness (L) from color information (U and V), which allows brightness information to be processed separately without being interfered by color components, and can more accurately analyze and compare changes in image content; third, because the LUV space separates brightness and chromaticity, it is more robust to changes in ambient light when performing difference analysis. For example, in outdoor surveillance video analysis, lighting conditions may change due to factors such as cloud movement and day and night changes. Using the LUV space can reduce the impact of these factors on difference analysis.

[0072] Secondly, the average value of the pixel values ​​of all the pixels in the second difference image is taken as the difference value corresponding to the third image frame.

[0073] After constructing the difference image (i.e., the second difference image) between the third image frame and the fourth image frame through the above steps, the sum of the pixel values ​​of all pixels in the second difference image is calculated, and then the average value of the sum of the pixel values ​​of all pixels in the second difference image is calculated. The average value is the difference between the third image frame and the image frame before the third image frame (i.e., the fourth image frame). Therefore, the average value is used as the difference value corresponding to the third image frame.

[0074] This embodiment can determine the difference value corresponding to the latter image frame of two adjacent image frames according to the difference between the pixel values ​​of the pixels of any two adjacent image frames in the video clip, that is, it can calculate the difference values ​​corresponding to each other image frame except the first image frame in the video clip. This embodiment combines the difference values ​​corresponding to the other image frames except the first image frame in the video clip into a difference value sequence according to the arrangement order of the image frames, thereby obtaining the difference value sequence corresponding to the video clip.

[0075] Second, the local maximum is searched from the difference value sequence, and the image frame corresponding to the local maximum is used as the key image frame in the video clip.

[0076] After the present embodiment determines the difference value sequence corresponding to the video clip, the local maximum is queried from the difference value sequence, and the image frame corresponding to the local maximum is used as the key image frame in the video clip. If multiple local maximum values ​​are queried from the difference value sequence, the image frames corresponding to the multiple local maximum values ​​are all used as the key image frames in the video clip, thereby obtaining multiple key image frames. Among them, the extraction of local maximum values ​​can be implemented by a traversal method, that is, the local maximum value is found by comparing the size of each element in the sequence with its adjacent elements. In addition, the local maximum value can also be extracted by other methods in the prior art, and the present embodiment does not limit the method for determining the local maximum value.

[0077] This embodiment selects the image frame corresponding to the local maximum as the key image frame because the local maximum means that the image frame corresponding to the local maximum has changed greatly compared with the previous image frame, usually because a major visual event has occurred, such as a scene change, an important action, etc. The image frame corresponding to the local maximum often better represents the key content and turning points in the video clip because of the significant changes, which is very useful for video summarization, indexing and retrieval. By selecting the image frame corresponding to the local maximum as the key image frame, it is not necessary to process all the image frames in the video data, effectively reducing the amount of data that needs to be processed and stored, while retaining the main information of the video clip.

[0078] Furthermore, in order to improve the accuracy of querying the local maximum value from the difference value sequence, after determining the difference value sequence corresponding to the video clip, the difference value sequence can be smoothed to reduce noise and make the local maximum value more obvious. The difference value sequence can be smoothed by using an existing smoothing method, such as a moving average method, which will not be described in detail in this embodiment.

[0079] This embodiment extracts key image frames from the video segments corresponding to each video shot through the above steps, thereby obtaining key image frames in each video segment. In addition, since the first image frame in the video segment corresponding to each video shot does not have an adjacent previous image frame in the video segment, this embodiment can directly determine the first image frame in the video segment corresponding to each video shot as a key image frame.

[0080] S103: Identify key image frames, generate description texts of the key image frames, and extract video content tags corresponding to the video data based on the description texts of the key image frames.

[0081] After extracting the key image frames in each video clip, the embodiment needs to perform reasoning and recognition on each key image frame, identify the description information of the key image frame such as the object, scene, etc. in the key image frame, and use the recognized description information as the description text of the key image frame. Specifically, the key image frame is recognized and the description text of the key image frame is generated in the following ways: First, a template-based method uses a predefined text template and combines the key information extracted from the image frame (such as object, scene, action, etc.) to generate a natural language sentence describing the image content. This method usually relies on a fixed sentence structure or pattern, and the blank part is filled by the entity, attribute or action recognized from the image. Second, a retrieval-based method uses a large-scale image-description pair data set to generate a natural language sentence describing the input image content by retrieving pre-stored image descriptions similar to or related to the image frame. Third, a rule engine-based method is a rule-based image description generation method that uses a set of rules to describe the feature information of the image and generates a description based on these rules. These rules can be a simple mapping of image features to text descriptions or a more complex logical combination. The fourth is a method based on a neural network model. After the neural network model is trained to perform image reasoning and recognition using pre-collected samples, the model can understand and analyze the image content and generate a natural language text to describe the key information in the image.

[0082] In this embodiment, the key image frames are identified and the description text of the key image frames is preferably generated in the following manner:

[0083] The key image frames and pre-built description text generation instructions are input into a pre-trained multimodal large model to obtain the description text of the key image frames.

[0084] A multimodal large model refers to a model that can process and understand a variety of different types of data (such as images, videos, audio, text, etc.). This type of model improves the ability to understand complex scenes by integrating and analyzing information from different data sources. This embodiment pre-constructs a description text generation instruction, which is a prompt, including the problem of reasoning the description information of the key image frame. Using the pre-constructed description text generation instruction and the pre-collected training samples (sample images and the real description text of the sample images), the multimodal large model is trained for image reasoning recognition, that is, the sample image and the pre-constructed description text generation instruction are input into the multimodal large model, and the multimodal large model outputs the predicted description text of the sample image. Based on the loss function between the predicted description text of the sample image and the real description text of the sample image, the model parameters of the multimodal large model are adjusted until the loss function meets the preset standard.

[0085] After the training of the multimodal large model is completed, the key image frames and the pre-built description text generation instructions are input into the trained multimodal large model. The multimodal large model can infer and recognize the key image frames according to the description text generation instructions and output the description text of the key image frames.

[0086] After each key image frame is identified, the description text of each key image frame can be obtained. This embodiment also needs to extract the video content label corresponding to the video data based on the description text of the key image frame, which specifically includes the following methods: First, a rule-based method is used to match the vocabulary in the text through a preset keyword list. If the text contains a keyword, the corresponding label can be assigned to the text, or a regular expression is used to define a specific pattern, and a string that meets these patterns is searched in the description text as a label. Second, a statistical method is used to classify the description text into predefined categories or labels. This usually involves the application of text preprocessing (such as word segmentation, stop word removal, etc.), feature extraction (such as bag of words model, TF-IDF, etc.) and classification algorithms (such as naive Bayes, support vector machine, neural network, etc.). The text classification method can automatically learn the relationship between labels and text content from a large amount of text data, and has high flexibility and accuracy. The third is a deep learning method, which pre-trains pre-trained language models (such as BERT, GPT, etc.) on large-scale text data to capture rich language knowledge and context information. By fine-tuning these models, they can be applied to specific label extraction tasks and achieve high-precision label generation.

[0087] This embodiment extracts the video content label corresponding to the video data based on the description text of the key image frame, preferably in the following manner:

[0088] The description text of the key image frames and the pre-built label extraction instructions are input into the pre-trained large language model to obtain the video content labels corresponding to the video data.

[0089] This embodiment pre-constructs a label extraction instruction, which is a prompt statement, including the problem of extracting labels from text. The pre-constructed label extraction instruction and the pre-collected training samples (sample text and the real label of the sample text) are used to perform label extraction training on the large language model, that is, the sample text and the pre-constructed label extraction instruction are input into the large language model, and the large language model outputs the predicted label of the sample text. Based on the loss function between the predicted label of the sample text and the real label of the sample text, the model parameters of the large language model are adjusted until the loss function meets the preset standard.

[0090] After the training of the large language model is completed, the description text of the key image frame and the pre-built label extraction instructions are input into the trained large language model. The large language model can extract labels from the description text of the key image frame according to the label extraction instructions and output video content labels.

[0091] From the above introduction, it can be seen that the video content label determination method proposed in the embodiment of the present application detects the boundaries of video shots in the video data, cuts the video data, and obtains a video segment corresponding to at least one video shot; based on the difference between adjacent image frames in the video segment, extracts key image frames from the video segment; identifies the key image frames, generates description texts of the key image frames, and extracts the video content labels corresponding to the video data based on the description texts of the key image frames. By adopting the technical solution of the present application, the description text of the image can be extracted from the key image frames in the video data, and the description text can cover the details of the video, so that the video content labels extracted according to the description text are more accurate, and the video content labels of the video data are automatically extracted, which improves the efficiency of determining the video content labels.

[0092] As an optional implementation, see Figure 2 As shown, in another embodiment of the present application, determining a boundary image frame of a video shot in the video data based on a difference between adjacent image frames in the video data includes the following steps:

[0093] S201 . Determine a first difference image between the first image frame and the second image frame based on a difference between grayscale values ​​of pixels of the first image frame and grayscale values ​​of pixels of the second image frame in video data.

[0094] In this embodiment, two adjacent image frames in the video data are used as the first image frame and the second image frame, wherein the first image frame is any image frame other than the first image frame in the video data, and the second image frame is the previous image frame adjacent to the first image frame. In this embodiment, the pixel gray value of each pixel in the first image frame and the pixel gray value of each pixel in the second image frame are first determined, and then the pixels in the first image frame and the second image frame are matched one by one, and the difference between the pixel gray value of the pixel in the first image frame and the pixel gray value of the pixel in the second image frame is calculated, so as to determine the pixel gray value difference corresponding to each pixel, and then according to the pixel gray value difference corresponding to each pixel, a difference image between the first image frame and the second image frame is constructed as the first difference image, that is, the gray value of each pixel in the first difference image is the pixel gray value difference corresponding to each pixel determined in advance.

[0095] S202 : performing binarization processing on each pixel in the first difference image based on a preset grayscale threshold to obtain a binarized image corresponding to the first difference image.

[0096] In this embodiment, a grayscale threshold is preset, and the grayscale threshold is preferably set to the middle value between the maximum grayscale value (e.g., grayscale value 255) and the minimum grayscale value (e.g., grayscale value 0). After constructing the first difference image through the above steps, it is necessary to perform binarization processing on each pixel in the first difference image, that is, perform binarization processing on the grayscale value of each pixel in the first difference image, replace the grayscale value of the pixel greater than the grayscale threshold with the maximum grayscale value, and replace the grayscale value of the pixel not greater than the grayscale threshold with the minimum grayscale value, and after binarization processing on the grayscale values ​​of all pixels in the first difference image, a binarized image corresponding to the first difference image is obtained, that is, the grayscale value of each pixel in the binarized image is the value of the grayscale value of each pixel in the first difference image after binarization processing.

[0097] S203: Determine a connected area in the binary image according to a preset connected gray value, and determine whether the first image frame is a boundary image frame of a video shot in the video data based on the connected area in the binary image.

[0098] In this embodiment, the pixel point with the maximum grayscale value (e.g., 255) in the binary image is taken as the lens conversion point, and the pixel point with the minimum grayscale value (e.g., 0) in the binary image is taken as the non-lens conversion point. In this embodiment, a preset connected grayscale value is preset, and the preset connected grayscale value is preferably set to the maximum grayscale value (e.g., 255). In this embodiment, the area composed of all pixel points with the preset connected grayscale value in the binary image is taken as the connected area in the binary image, that is, the area composed of all lens conversion points in the binary image is taken as the connected area. Since each lens conversion point in the binary image is not necessarily adjacent pixel points, the connected area composed of all lens conversion points may also include multiple.

[0099] This embodiment determines whether the first image frame is a boundary image frame of a video shot in the video data according to the connected areas in the binary image. Specifically, this embodiment mainly determines whether the first image frame is a boundary image frame according to the number of connected areas existing in the binary image and the size of each connected area. Among them, the size of the connected area is determined according to the number of pixels contained in the connected area. If it is determined that there is at least one connected area in the binary image, and the number of pixels contained in at least one connected area reaches a preset number, then the first image frame is determined to be a boundary image frame of the video shot in the video data. In other words, if there are one or more large connected areas in the binary image, it can be considered that these significant changes may be caused by shot switching, so that the first image frame is determined to be a boundary image frame. If no connected area is detected in the binary image, or the detected connected areas are very small and scattered (that is, the number of pixels contained in each connected area is less than the preset number), it can be considered that there is no significant visual change between the first image frame and the second image frame, the video content maintains continuity between the two image frames, no shot switching occurs, and the first image frame is not a boundary image frame.

[0100] Through the above steps, it is possible to detect whether any image frame except the first image frame in the video data is a boundary image frame, so that all boundary image frames in the video data can be extracted.

[0101] In addition, the present embodiment can also determine the area composed of non-lens transition points in the binary image, and use the area composed of non-lens transition points as the non-lens transition area. The non-lens transition area is regarded as a potential moving target. These areas show objects or scene parts that are relatively stable or continuously present from the second image frame to the first image frame. By determining the non-lens transition area in the image frame, the movement trajectory of the target in subsequent frames can be continuously tracked. Combined with other visual information, such as color, texture, etc., it can enhance the understanding of the scene content, thereby assisting in the reasoning and recognition of key image frames, and improving the generation accuracy of the description text of the key image frames.

[0102] As an optional implementation, see Figure 3 As shown, in another embodiment of the present application, a method for determining a video content tag is disclosed, which specifically includes the following steps:

[0103] S301: Cut the video data by detecting video shot boundaries in the video data to obtain a video segment corresponding to at least one video shot.

[0104] S302: extract key image frames from the video segment based on differences between adjacent image frames in the video segment.

[0105] S303: Identify the key image frame and generate description text of the key image frame.

[0106] This embodiment performs inference recognition on each key image frame extracted from each video clip, thereby obtaining the description text of each key image frame. In addition, this embodiment not only needs to determine the description text of the key image frame, but in order to further improve the coverage of the extracted video content label on the video data details, this embodiment can also extract the text content contained in the image frame in the video data, and / or, when the video data has corresponding audio, the text corresponding to the audio is also extracted. That is, this embodiment can also perform the following step S304 and / or step S305.

[0107] S304: Perform text region detection on at least one image frame in the video data, and perform character recognition on the detected text region to obtain image text corresponding to the video data.

[0108] In this embodiment, the image frame is extracted from the video data to obtain at least one image frame, wherein the video data can be parsed using opencv to decompose at least one image frame. In this embodiment, it is necessary to perform text area detection on the image frame extracted from the video data to determine the text area containing text content in the image frame. Among them, the image frame needs to be preprocessed first to perform text area detection on the image frame, for example, the image frame is subjected to denoising, graying, binarization and contrast enhancement, etc.; then the preprocessed image frame is subjected to feature extraction, and edge detection and contour detection, etc. are performed to determine the text area in the image frame. After the text area in the image frame is detected in this embodiment, the text area is subjected to character recognition using character recognition technology to obtain the image text in the image frame, and the image text recognized in each image frame is used as the image text corresponding to the video data. Among them, this embodiment can use the existing text area detection technology to perform text area detection on the image frame; this embodiment can also use the existing character recognition technology to perform character recognition on the text area of ​​the image frame, such as OCR technology.

[0109] S305: Perform speech recognition on the audio corresponding to the video data to obtain a speech text corresponding to the video data.

[0110] When the video data for which the video content tag is determined has corresponding audio, this embodiment can also perform speech recognition on the audio corresponding to the video data, thereby identifying the speech text corresponding to the video data. The technology for performing speech recognition on the audio in this embodiment can adopt existing speech recognition technology, for example, a method based on a neural network, that is, building a speech recognition model based on a neural network, or using an existing automatic speech recognition system (such as the OpenAI Whisper model), etc. This embodiment does not specifically describe the steps for performing speech recognition on the audio.

[0111] In addition, if the audio corresponding to the video data is long, it may increase the processing time and resources during speech recognition, thereby affecting the speech recognition efficiency. Therefore, in order to improve the speech recognition efficiency, the present embodiment can cut the audio corresponding to the video data according to the preset duration to obtain at least one audio segment, and then perform speech recognition on each audio segment to obtain the segment text corresponding to each audio segment, and finally merge the segment texts corresponding to each audio segment into a continuous text as the speech text corresponding to the video data according to the order of each audio segment.

[0112] S306: Extract video content tags corresponding to the video data based on the description text of the key image frame, and the image text corresponding to the video data and / or the voice text corresponding to the video data.

[0113] Through the above steps, this embodiment summarizes, corrects, merges and removes duplicate content of the description text of the key image frame, the image text corresponding to the video data and / or the voice text corresponding to the video data to obtain an integrated text, and then extracts the video content label corresponding to the video data from the integrated text. The specific execution method of extracting the video content label corresponding to the video data from the integrated text is the same as the specific execution method of extracting the video content label corresponding to the video data based on the description text of the key image frame in step S103 of the above embodiment, and will not be repeated in this embodiment.

[0114] Specifically, the present embodiment can also pre-construct a large language model, and input the pre-constructed label extraction instructions, the description text of the key image frame, and the image text corresponding to the video data and / or the voice text corresponding to the video data into the pre-trained large language model to obtain the video content label corresponding to the video data. The large language model can summarize, correct, merge and remove duplicate content from the description text of the key image frame, the image text corresponding to the video data and / or the voice text corresponding to the video data to obtain an integrated text, and then extract labels from the integrated text according to the pre-constructed label extraction instructions, and finally output the video content label corresponding to the video data.

[0115] The execution of steps S301 to S303 in this embodiment is the same as that of steps S101 to S103 in the above embodiment, and will not be repeated in this embodiment. This embodiment does not limit the execution order of steps S304 and S305, as long as steps S304 and S305 are executed before step S306.

[0116] Exemplary Devices

[0117] Accordingly, the present application also provides a video content tag determination device, see Figure 4 As shown, the device comprises:

[0118] The video cutting module 100 is used to cut the video data by detecting the video shot boundary in the video data to obtain a video segment corresponding to at least one video shot;

[0119] An image frame extraction module 110, configured to extract key image frames from a video segment based on differences between adjacent image frames in the video segment;

[0120] The tag determination module 120 is used to identify key image frames, generate description texts of the key image frames, and extract video content tags corresponding to the video data based on the description texts of the key image frames.

[0121] From the above introduction, it can be seen that the video content label determination device proposed in the embodiment of the present application can extract the image description text from the key image frame in the video data. The description text can cover the details of the video, so that the video content label extracted according to the description text is more accurate, and the video content label of the video data is automatically extracted, thereby improving the efficiency of determining the video content label.

[0122] As an optional implementation, another embodiment of the present application discloses that the video cutting module 100 includes: a boundary determination unit and a cutting unit.

[0123] A boundary determination unit, configured to determine a boundary image frame of a video shot in the video data based on a difference between adjacent image frames in the video data;

[0124] The cutting unit is used to cut the video data according to the boundary image frames to obtain a video segment corresponding to at least one video shot.

[0125] As an optional implementation manner, another embodiment of the present application discloses a boundary determination unit, which is specifically used to:

[0126] Determine a first difference image between the first image frame and the second image frame based on a difference between a pixel grayscale value of a first image frame and a pixel grayscale value of a second image frame in the video data; wherein the first image frame is any image frame other than the first image frame in the video data, and the second image frame is a previous image frame adjacent to the first image frame;

[0127] Based on a preset grayscale threshold, binarization is performed on each pixel in the first difference image to obtain a binarized image corresponding to the first difference image;

[0128] According to the preset connected grayscale value, a connected area in the binary image is determined, and based on the connected area in the binary image, it is judged whether the first image frame is a boundary image frame of a video shot in the video data.

[0129] As an optional implementation, another embodiment of the present application discloses that the image frame extraction module 110 includes: a difference determination unit and a query unit.

[0130] a difference determination unit, configured to determine a difference value corresponding to the third image frame based on a difference between a pixel value of a pixel point of the third image frame and a pixel value of a pixel point of the fourth image frame in the video segment; wherein the third image frame is any image frame in the video segment except the first image frame, and the fourth image frame is a previous image frame adjacent to the third image frame;

[0131] The query unit is used to query the local maximum value from the difference value sequence and use the image frame corresponding to the local maximum value as the key image frame in the video clip; the difference value sequence includes the difference values ​​corresponding to each image frame in the video clip except the first image frame.

[0132] As an optional implementation manner, another embodiment of the present application discloses a difference determination unit, which is specifically used to:

[0133] Determining a second difference image between the third image frame and the fourth image frame based on a difference between pixel values ​​of pixels of the third image frame and pixel values ​​of pixels of the fourth image frame in the video clip;

[0134] The average value of the pixel values ​​of all the pixels in the second difference image is used as the difference value corresponding to the third image frame.

[0135] As an optional implementation, another embodiment of the present application discloses that the tag determination module 120 includes: a description text determination unit.

[0136] The description text determination unit is used to input the key image frame and the pre-built description text generation instruction into the pre-trained multimodal large model to obtain the description text of the key image frame.

[0137] As an optional implementation, another embodiment of the present application discloses that the video content label determination device further includes: a character recognition module and a speech recognition module.

[0138] A character recognition module is used to perform text region detection on at least one image frame in the video data, and perform character recognition on the detected text region to obtain image text corresponding to the video data;

[0139] The speech recognition module is used to perform speech recognition on the audio corresponding to the video data to obtain the speech text corresponding to the video data.

[0140] As an optional implementation, another embodiment of the present application discloses that the tag determination module 120 further includes: a tag extraction unit.

[0141] The label extraction unit is used to extract the video content label corresponding to the video data based on the description text of the key image frame, the image text corresponding to the video data and / or the voice text corresponding to the video data.

[0142] As an optional implementation manner, another embodiment of the present application discloses a label extraction unit, which is specifically used to:

[0143] The pre-built label extraction instructions, the description text of the key image frames, and the image text corresponding to the video data and / or the speech text corresponding to the video data are input into the pre-trained large language model to obtain the video content label corresponding to the video data.

[0144] The video content label determination device provided in this embodiment belongs to the same application concept as the video content label determination method provided in the above embodiments of this application, and can execute the video content label determination method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the video content label determination method. For technical details not fully described in this embodiment, please refer to the specific processing content of the video content label determination method provided in the above embodiments of this application, and will not be repeated here.

[0145] Exemplary Electronic Devices

[0146] Another embodiment of the present application also provides an electronic device, see Figure 5 As shown, the device includes:

[0147] Memory 200 and processor 210;

[0148] The memory 200 is connected to the processor 210 and is used to store programs;

[0149] The processor 210 is configured to implement the video content tag determination method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0150] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0151] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0152] A bus may include a pathway that transfers information between components of a computer system.

[0153] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0154] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0155] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0156] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0157] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0158] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0159] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any video content tag determination method provided in the above embodiments of the present application.

[0160] Exemplary computer program products and storage media

[0161] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the video content tag determination method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0162] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0163] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the video content tag determination method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0164] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0165] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0166] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0167] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0169] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0171] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0172] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0173] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0174] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining a video content label, characterized in that: include: By detecting a video shot boundary in the video data, the video data is cut to obtain a video segment corresponding to at least one video shot; extracting key image frames from the video clip based on differences between adjacent image frames in the video clip; The key image frame is identified, a description text of the key image frame is generated, and based on the description text of the key image frame, a video content tag corresponding to the video data is extracted.

2. The method according to claim 1, characterized in that The method detects a video shot boundary in the video data and cuts the video data to obtain a video segment corresponding to at least one video shot, including: Determining a boundary image frame of a video shot in the video data based on a difference between adjacent image frames in the video data; The video data is cut according to the boundary image frame to obtain a video segment corresponding to at least one video shot.

3. The method according to claim 2, characterized in that Determining a boundary image frame of a video shot in the video data based on a difference between adjacent image frames in the video data includes: Determine a first difference image between the first image frame and the second image frame based on a difference between a pixel grayscale value of a first image frame and a pixel grayscale value of a second image frame in the video data; wherein the first image frame is any image frame other than the first image frame in the video data, and the second image frame is a previous image frame adjacent to the first image frame; Based on a preset grayscale threshold, binarization is performed on each pixel in the first difference image to obtain a binarized image corresponding to the first difference image; According to the preset connected grayscale value, a connected area in the binary image is determined, and based on the connected area in the binary image, it is judged whether the first image frame is a boundary image frame of a video shot in the video data.

4. The method according to claim 1, characterized in that Extracting key image frames from the video clip based on differences between adjacent image frames in the video clip includes: Determining a difference value corresponding to the third image frame based on a difference between a pixel value of a pixel point of a third image frame and a pixel value of a pixel point of a fourth image frame in the video clip; wherein the third image frame is any image frame in the video clip except the first image frame, and the fourth image frame is a previous image frame adjacent to the third image frame; A local maximum value is searched from a difference value sequence, and an image frame corresponding to the local maximum value is used as a key image frame in the video clip; the difference value sequence includes difference values ​​corresponding to each image frame in the video clip except the first image frame.

5. The method according to claim 4, characterized in that Determining a difference value corresponding to the third image frame based on a difference between a pixel value of a pixel point of a third image frame and a pixel value of a pixel point of a fourth image frame in the video clip includes: Determining a second difference image between the third image frame and the fourth image frame based on a difference between pixel values ​​of pixels of a third image frame and pixel values ​​of pixels of a fourth image frame in the video clip; An average value of pixel values ​​of all pixels in the second difference image is used as a difference value corresponding to the third image frame.

6. The method according to claim 1, characterized in that Identifying the key image frame and generating a description text of the key image frame includes: The key image frame and the pre-built description text generation instruction are input into a pre-trained multimodal large model to obtain the description text of the key image frame.

7. The method according to claim 1, characterized in that Before extracting the video content tag corresponding to the video data based on the description text of the key image frame, the method further includes: Performing text region detection on at least one image frame in the video data, and performing character recognition on the detected text region to obtain image text corresponding to the video data; and / or, performing speech recognition on the audio corresponding to the video data to obtain a speech text corresponding to the video data; Extracting a video content tag corresponding to the video data based on the description text of the key image frame includes: Based on the description text of the key image frame, and the image text corresponding to the video data and / or the voice text corresponding to the video data, the video content tag corresponding to the video data is extracted.

8. The method according to claim 7, characterized in that Extracting a video content tag corresponding to the video data based on the description text of the key image frame, and the image text corresponding to the video data and / or the voice text corresponding to the video data includes: The pre-built label extraction instructions, the description text of the key image frame, and the image text corresponding to the video data and / or the speech text corresponding to the video data are input into the pre-trained large language model to obtain the video content label corresponding to the video data.

9. A video content label determination device, characterized in that: include: A video cutting module, configured to cut the video data by detecting a video shot boundary in the video data to obtain a video segment corresponding to at least one video shot; An image frame extraction module, configured to extract key image frames from the video segment based on differences between adjacent image frames in the video segment; The tag determination module is used to identify the key image frame, generate a description text of the key image frame, and extract a video content tag corresponding to the video data based on the description text of the key image frame.

10. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the video content tag determination method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the video content tag determination method according to any one of claims 1 to 8.

Images