Video understanding method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video understanding method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence such as deep learning, computer vision, large models and natural language processing. The specific implementation scheme is as follows: performing shot segmentation on a to-be-processed video to obtain at least one shot clip of the to-be-processed video; performing target motion amplitude analysis on the at least one lens segment to obtain respective motion amplitude information of the at least one lens segment; based on the respective motion amplitude information of the at least one shot segment, extracting a key frame from the corresponding shot segment; and performing video content understanding on the to-be-processed video based on the multi-modal features of the respective key frames of the at least one shot clip to obtain description information of the to-be-processed video. By implementing the embodiment of the invention, the video understanding effect can be optimized.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, in particular to the fields of artificial intelligence technologies such as deep learning, computer vision, large models and natural language processing, and in particular to video understanding methods, devices, electronic devices, storage media and program products. Background Art

[0002] At present, large-scale image understanding models are emerging in an endless stream, which greatly improves the upper limit of image understanding effects. Most of the large-scale video-level understanding models in related technologies are based on image understanding models for frame extraction, taking multiple frames of images as input. However, the image-level tokens (words, or also called the basic data units processed by the model) are often relatively large, and the frame extraction images are too dense, which will exceed the maximum number of tokens that the large language model (LLM) can handle. On the contrary, if the frame extraction is too low, it will lead to insufficient understanding of fast-moving videos. Summary of the invention

[0003] The present disclosure provides a video understanding method, device, electronic device, storage medium and program product.

[0004] According to a first aspect of the present disclosure, a video understanding method is provided, comprising:

[0005] Segment the video to be processed to obtain at least one shot segment of the video to be processed;

[0006] Performing target motion amplitude analysis on the at least one shot segment respectively to obtain motion amplitude information of the at least one shot segment;

[0007] Extracting key frames from the corresponding shot segments based on the motion amplitude information of each of the at least one shot segment;

[0008] Based on the multimodal features of the key frames of each of the at least one shot segment, video content understanding is performed on the video to be processed to obtain description information of the video to be processed.

[0009] According to a second aspect of the present disclosure, a video understanding device is provided, comprising:

[0010] A segmentation module, used to perform shot segmentation on the video to be processed to obtain at least one shot segment of the video to be processed;

[0011] An analysis module, configured to perform target motion amplitude analysis on the at least one shot segment respectively to obtain motion amplitude information of each of the at least one shot segment;

[0012] An extraction module, configured to extract key frames from the corresponding shot segments based on the motion amplitude information of each of the at least one shot segments;

[0013] The content understanding module is used to understand the video content of the video to be processed based on the multimodal features of the key frames of each of the at least one shot segment, so as to obtain description information of the video to be processed.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video understanding method described in the first aspect.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the video understanding method described in the first aspect.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the video understanding method described in the first aspect when executed by a processor.

[0020] The technology disclosed in the present invention can improve the effect of overall video understanding.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a video understanding method provided according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of a video understanding method provided according to an embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of a video understanding method provided according to an embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of a video understanding method provided according to an embodiment of the present disclosure;

[0027] Figure 5 is a block diagram of a video understanding device provided according to an embodiment of the present disclosure;

[0028] Figure 6 is a block diagram of a video understanding device provided according to an embodiment of the present disclosure;

[0029] Figure 7 is a block diagram of a video understanding device provided according to an embodiment of the present disclosure;

[0030] Figure 8 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The embodiments of the present disclosure are not exhaustive, but are only illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, the solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined. For example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments. In each embodiment of the present disclosure, if there is no special explanation and logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0033] In some embodiments, the terms "at least one", "one or more", "a plurality of", "multiple" and the like can be used interchangeably. In the embodiments of the present disclosure, "plurality" refers to two or more.

[0034] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0036] It is worth noting that in the embodiments of the present disclosure, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0037] The embodiments of the present disclosure relate to artificial intelligence technology fields such as natural language processing, large models, deep learning, and computer vision.

[0038] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0039] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. It is a discipline that uses computer technology to analyze, understand and process natural language, that is, it uses computers as a powerful tool for language research, conducts quantitative research on language information with the support of computers, and provides language descriptions that can be used by humans and computers.

[0040] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0041] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0042] Computer vision refers to a technology that uses computers and related equipment to simulate biological vision. By processing the collected pictures or videos, a multi-dimensional understanding of the corresponding scene can be achieved. The main task of computer vision is to obtain three-dimensional information of the corresponding scene by processing the collected pictures or videos. The core of computer vision is to use cameras and computers to simulate the human visual system. Specifically, it is to install "eyes" (cameras) and "brains" (algorithms) on computers so that computers can perceive the environment. The research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0043] An agent is an agent that can perceive the environment and take actions to achieve specific goals. It can be software, hardware or a system with autonomy, adaptability and interaction. The agent perceives changes in the environment (such as through sensors or data input), makes judgments and decisions based on the knowledge and algorithms it has learned, and then performs actions to affect the environment or achieve predetermined goals.

[0044] Optical flow is a concept in object motion detection in the field of view, which is used to describe the motion of an observed object, surface or edge caused by the motion relative to the observer. The basic principle of optical flow is that when an object moves, the brightness pattern of its corresponding points on the image also moves accordingly. The apparent motion of the brightness pattern of the image is the optical flow. Optical flow is generated by camera motion, target motion in the scene, or both. The optical flow field is derived from optical flow, which refers to the two-dimensional instantaneous velocity field formed by the projection of the three-dimensional velocity vectors of visible pixels in the scene on the imaging surface. Specifically, the optical flow method estimates the motion of an object by analyzing the change in pixel intensity between two consecutive frames of images. This method does not require any prior knowledge of the scene and can be used for cameras in motion.

[0045] Background difference method is a method to detect moving objects by comparing the current frame with the background reference model. This method determines whether there is a moving target by calculating the difference between the current frame and the background image. The performance of background difference method depends on the background modeling technology used. The accuracy of the modeling and simulation of the background image directly affects the detection effect.

[0046] The inter-frame difference method is a method to obtain the contour of a moving target by performing a difference operation on two adjacent frames in a video image sequence. It marks the moving object by performing a difference operation on two adjacent frames in the video. When there is a moving object in the video, there will be a difference in grayscale between adjacent frames (or three adjacent frames). By calculating the absolute value of the grayscale difference between the two frames, when the absolute value exceeds a certain threshold, it can be judged as a moving target, thereby realizing the target detection function.

[0047] Principal Component Analysis (PCA) is a multivariate statistical analysis method that uses linear transformation to select a smaller number of important variables. It is also called principal component analysis. In statistics, principal component analysis PCA is a technique to simplify a data set. It is a linear transformation. This transformation transforms the data into a new coordinate system so that the first largest variance of any data projection is on the first coordinate (called the first principal component), the second largest variance is on the second coordinate (the second principal component), and so on. Principal component analysis is often used to reduce the dimensionality of a data set while retaining the features that contribute the most to the variance of the data set. This is done by retaining the lower-order principal components and ignoring the higher-order principal components. In this way, the lower-order components often retain the most important aspects of the data.

[0048] It is worth noting that large image understanding models are emerging in large numbers, which greatly improves the upper limit of image understanding. Most of the large video-level understanding models in related technologies are based on image understanding models for frame extraction, taking multiple frames of images as input. However, image-level tokens (words, or also called the basic data units processed by the model) are often relatively large, and the frame extraction images are too dense, which will exceed the maximum number of tokens that the large language model (LLM) can handle. On the contrary, if the frame extraction is too low, it will lead to insufficient understanding of fast-moving videos.

[0049] Therefore, the embodiments of the present disclosure provide a video understanding method, device, electronic device, storage medium and program product, which can distinguish and process fast-motion and slow-motion videos when extracting frames, realize adaptive frame extraction, and improve the video understanding effect when the maximum number of tokens of the LLM model is allowed.

[0050] The video understanding method, apparatus, electronic device, storage medium, and program product of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0051] It should be noted that the execution subject of the video understanding method of this embodiment is a video understanding device, which can be implemented by software and / or hardware, and can be configured in an electronic device, which can include but is not limited to a terminal, a server, etc. In some embodiments, the video understanding method can be implemented by an intelligent agent, but is not limited to this.

[0052] Figure 1 FIG. 1 is a flow chart of a video understanding method provided according to an embodiment of the present disclosure. Figure 1 As shown, the video understanding method may include but is not limited to the following steps.

[0053] In step 101, the video to be processed is segmented into shots to obtain at least one shot segment of the video to be processed.

[0054] In some embodiments, the video to be processed may be a film or television video, or may be at least a partial video clip in a film or television video, or may be other types of videos, such as promotional videos (such as advertising videos, local cultural promotional videos, etc.), but is not limited thereto.

[0055] In some embodiments, the format of the video to be processed can be MP4 (also called MPEG-4) format, but is not limited to this. For example, the format of the video to be processed can also be AVI (Audio Video Interleaved) format, or FLV (Flash Video) format, or MKV (Matroska) format, etc.

[0056] In some embodiments, a shot segmentation model can be used to perform shot segmentation on the video to be processed. In one possible implementation, the shot segmentation model can be a TransNetV2 model, which is a model based on 3D convolution. The TransNetV2 model can be used to perform shot segmentation on the video to be processed, so as to divide the video to be processed into different shot segments. Exemplarily, the video to be processed can be input into the preset TransNetV2 model for shot segmentation to obtain at least one shot segment of the video to be processed output by the TransNetV2 model. In some embodiments, the terms such as "shot segmentation", "shot cutting", and "shot cutting" can be interchangeable.

[0057] It should be noted that the above-mentioned implementation method of performing shot segmentation on the video to be processed by using the shot segmentation model is only an example given for the convenience of understanding by those skilled in the art, and cannot be used as a specific limitation of the present disclosure. That is to say, other methods can be used to implement shot segmentation of the video to be processed. Optionally, in some embodiments, a video editing tool (or video editing software) can be called to perform shot segmentation on the video to be processed to obtain at least one shot segment of the video to be processed. The present disclosure does not make any specific limitation on this and will not elaborate on it.

[0058] In step 102, target motion amplitude analysis is performed on at least one shot segment to obtain motion amplitude information of at least one shot segment.

[0059] In some embodiments, a target motion amplitude analysis method may be used to perform target motion amplitude analysis on at least one shot segment. In some embodiments, the target motion amplitude analysis method may include but is not limited to at least one of an optical flow method, a background difference method, and an inter-frame difference method. Exemplarily, for each shot segment in at least one shot segment, a target motion amplitude analysis method is used to perform target motion amplitude analysis on the shot segment to analyze the motion amplitude in the shot segment and obtain motion amplitude information of the shot segment. The motion amplitude information may indicate whether the content in the corresponding shot segment is fast-moving or slow-moving.

[0060] In some embodiments, a target motion amplitude analysis method may be used to perform target motion amplitude analysis on at least one shot segment respectively to obtain motion amplitude information of at least one shot segment.

[0061] In some embodiments, for each shot segment in at least one shot segment, multiple target motion amplitude analysis methods can be used to perform target motion amplitude analysis on the same shot segment to obtain multiple motion amplitude classification results of the shot segment, and the multiple motion amplitude classification results are fused to obtain motion amplitude information of the shot segment.

[0062] In some embodiments, the motion amplitude information may be a motion amplitude score, which can represent the motion amplitude of the shot segment. For example, the larger the motion amplitude, the larger the motion amplitude score, and vice versa. The smaller the motion amplitude, the smaller the motion amplitude score.

[0063] In step 103, based on the motion amplitude information of at least one shot segment, key frames are extracted from the corresponding shot segment.

[0064] In some embodiments, key frame extraction may be performed on each shot segment in at least one shot segment. Exemplarily, the motion amplitude information of the shot segment may be used to extract key frames from the shot segment. In some embodiments, the number of key frames extracted (or the number of key frame extractions) may be determined based on the motion amplitude information of the corresponding shot segment. For example, the greater the motion amplitude of the content in the shot segment, the greater the number of key frame extractions (e.g., when the content in the shot segment moves violently, the frame extractions are more intensive; when the motion is relatively gentle, the frame extractions are more sparse).

[0065] In some embodiments, the number of key frame reference frames (such as the number of key frames to be extracted) for each of the at least one shot fragment can be determined based on the motion amplitude information of each of the at least one shot fragment; and the corresponding number of key frames can be extracted from the corresponding shot fragment based on the number of key frame reference frames for each of the at least one shot fragment.

[0066] In step 104, based on the multimodal features of the key frames of at least one shot segment, the video content is understood for the video to be processed to obtain description information of the video to be processed.

[0067] In some embodiments, a multimodal feature extraction model may be used to extract features from the key frames of each shot segment in at least one shot segment to obtain multimodal features of the key frames of each shot segment. Based on the multimodal features of the key frames of the at least one shot segment, the video content of the video to be processed is understood to obtain description information of the video to be processed. The multimodal feature extraction model may be a CLIP (Contrastive Language Image Pre-training) model.

[0068] In a possible implementation, based on the multimodal features and preset questions of the key frames of at least one shot segment, the video content of the video to be processed can be understood by a large language model to obtain the description information of the video to be processed; wherein the large language model (LLM) can be used to understand the video content of the video to be processed based on the multimodal features and preset questions. It should be noted that the large language model can be a trained model, the input of the large language model can include the multimodal features of the image (such as the key frame) and the question information (such as the preset question), and the output of the large language model can be the description information of the video (also called the understanding information of the video). The preset question can be a prompt word with specific instructions or context information, which can guide the large language model to accurately perform the task. For example, the preset question can be "Please describe the event that occurred in the video", but is not limited to this. Exemplarily, the preset question can be text, or it can also be voice, but is not limited to this.

[0069] In some embodiments, when extracting key frames from a shot segment, the number of key frame extractions (or the number of key frame reference frames) can satisfy the maximum number of Tokens that can be processed (or accommodated) by the above-mentioned large language model. Exemplarily, when a large language model is used to understand the video content of a video to be processed, the number of key frame extractions (or the number of key frame reference frames) can be determined based on the motion amplitude information of the corresponding shot segment and the maximum number of Tokens that can be processed by the large language model (i.e., the maximum number of images that can be accommodated in the large language model), so as to extract key frames from the corresponding shot segment based on the number of key frame extractions (or the number of key frame reference frames).

[0070] By implementing the embodiments of the present disclosure, the target motion amplitude analysis can be performed on each shot segment in the processed video to obtain the motion amplitude information of each shot segment. The motion amplitude information is used to characterize the motion amplitude of the content in the shot segment, so as to facilitate the extraction of key frames from the corresponding shot segments according to the motion amplitude information of each shot segment, and the video content of the processed video is understood according to the multimodal features of the key frames of each shot segment to obtain the description information of the processed video. It can be seen that the embodiments of the present disclosure perform differentiated processing when extracting frames for fast-moving and slow-moving videos, which can achieve adaptive frame extraction and improve the effect of overall video understanding.

[0071] Figure 2 FIG. 1 is a flow chart of a video understanding method provided according to an embodiment of the present disclosure. Figure 2 As shown, the video understanding method may include but is not limited to the following steps.

[0072] In step 201, the video to be processed is segmented into shots to obtain at least one shot segment of the video to be processed.

[0073] In the embodiments of the present disclosure, step 201 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0074] In step 202, for each shot segment in at least one shot segment, at least one target motion amplitude analysis method is used to perform target motion amplitude analysis on the shot segment to obtain at least one dynamic feature of the shot segment, and the shot segment is motion-amplified based on the at least one dynamic feature to obtain at least one motion amplitude classification result of the shot segment.

[0075] In some embodiments, the motion amplitude classification may include multiple motion amplitude levels, for example, strenuous exercise level (such as in the case of fast video playback), moderate to heavy exercise level, moderate exercise level, light exercise level, static level, etc.

[0076] In some embodiments, the at least one target motion amplitude analysis method may include but is not limited to: at least one of an optical flow method, a background difference method, and an inter-frame difference method. Exemplarily, taking at least one target motion amplitude analysis method as an example, for each shot segment in at least one shot segment, the target motion amplitude analysis method may be used to perform target motion amplitude analysis on the shot segment to obtain dynamic features of the shot segment, and based on the dynamic features, the shot segment is subjected to motion amplitude classification to obtain a motion amplitude classification result of the shot segment.

[0077] Exemplarily, taking at least one target motion amplitude analysis method as an example, for example, two (such as optical flow method and background difference method), for each shot segment in at least one shot segment, the optical flow method can be used to analyze the target motion amplitude of the shot segment to obtain the dynamic characteristics of the shot segment, and the shot segment is classified by motion amplitude based on the dynamic characteristics to obtain the motion amplitude classification result of the shot segment. For the same shot segment, the background difference method can also be used to analyze the target motion amplitude of the shot segment to obtain the dynamic characteristics of the shot segment, and the shot segment is classified by motion amplitude based on the dynamic characteristics to obtain the motion amplitude classification result of the shot segment.

[0078] In some embodiments, the target motion amplitude analysis method may include an optical flow method, and the dynamic feature may include an optical flow feature; for each shot segment in at least one shot segment, the above-mentioned target motion amplitude analysis of the shot segment using the optical flow method to obtain the dynamic feature of the shot segment, and the implementation method of classifying the motion amplitude of the shot segment based on the dynamic feature may include the following steps: using the optical flow method to perform optical flow detection on the shot segment to obtain the optical flow feature of the shot segment; and classifying the motion amplitude of the shot segment based on the optical flow feature. That is, for each shot segment in at least one shot segment, the optical flow method can be used to perform motion amplitude analysis, and the motion amplitude analysis result of each shot segment can be obtained.

[0079] In some embodiments, the target motion amplitude analysis method may include a background difference method; for each shot segment in at least one shot segment, the above-mentioned target motion amplitude analysis is performed on the shot segment using the background difference method to obtain the dynamic characteristics of the shot segment, and the implementation method of classifying the motion amplitude of the shot segment based on the dynamic characteristics may include the following steps: selecting a plurality of consecutive image frames from the shot segment; performing pixel value averaging processing on the pixel points at corresponding positions in the plurality of consecutive image frames to obtain the pixel average value of the pixel points, and using the pixel average value as the grayscale value of the corresponding pixel point in the background image to obtain the background image; performing differential operations on the plurality of consecutive image frames and the background image respectively to obtain the respective foreground objects in the plurality of consecutive image frames; and classifying the motion amplitude of the shot segment based on the change sequence of the foreground object between the plurality of consecutive image frames. That is, for each shot segment in at least one shot segment, the background difference method may be used to perform motion amplitude analysis, and the motion amplitude analysis result of each shot segment may be obtained.

[0080] Exemplarily, the background difference method can be used to estimate the motion of the video. For example, since the video to be processed has completed the shot segmentation, the scene background in a single shot segment is basically the same, therefore, N consecutive image frames can be selected in the shot segment, and the average value of the N pixel values ​​of the multiple image frames at this point is calculated for each pixel point, and the average value is used as the grayscale value in the background image of the pixel point, and then the grayscale value in the background image is subtracted from each image frame to obtain the respective foreground object in each image frame. Based on the change sequence of the foreground object between the multiple image frames, a neural network is used for classification to obtain the motion amplitude classification result of the shot segment.

[0081] In step 203, based on at least one motion amplitude classification result of at least one shot segment, motion amplitude information of at least one shot segment is determined.

[0082] In some embodiments, a motion amplitude score corresponding to each motion amplitude classification result in at least one motion amplitude classification result is determined; and motion amplitude information of the shot segment is determined according to at least one motion amplitude score of the shot segment.

[0083] Exemplarily, the above-mentioned motion amplitude classification result may include the motion amplitude gear to which the lens clip belongs. The motion amplitude gear may include multiple motion amplitude gears, for example, a gear for vigorous exercise (such as in the case of video fast play), a gear for moderate to severe exercise, a gear for moderate exercise, a gear for light exercise, a gear for static exercise, etc., and each motion amplitude gear may correspond to a motion amplitude score. For example, taking the gear for vigorous exercise (such as in the case of video fast play), the gear for moderate to severe exercise, the gear for moderate exercise, the gear for light exercise, and the gear for static exercise as examples, the larger the motion amplitude, the larger the corresponding motion amplitude score may be, and vice versa, the smaller the motion amplitude, the smaller the corresponding motion amplitude score may be, for example, the motion amplitude score corresponding to the gear for vigorous exercise (such as in the case of video fast play) may be 5, the motion amplitude score corresponding to the gear for moderate to severe exercise may be 4, the motion amplitude score corresponding to the gear for moderate exercise may be 3, the motion amplitude score corresponding to the gear for light exercise may be 2, and the motion amplitude score corresponding to the static gear may be 1.

[0084] Exemplarily, when using at least one target motion amplitude analysis method to perform motion amplitude analysis on a shot clip to obtain at least one motion amplitude classification result of the shot clip, the motion amplitude score corresponding to each motion amplitude classification result can be determined based on the mapping relationship between the motion amplitude gear and the motion amplitude score, and then the motion amplitude information of the shot clip can be determined based on the at least one motion amplitude score of the shot clip.

[0085] In some embodiments, if the number of at least one target motion amplitude analysis method is one, the motion amplitude score of the shot segment can be used as the motion amplitude information of the shot segment; or, if the number of at least one target motion amplitude analysis method is multiple, the multiple motion amplitude scores of the shot segment can be fused, and the score obtained after the fusion process can be used as the motion amplitude information of the shot segment. Exemplarily, the fusion process can include but is not limited to: any one of averaging process, weighted averaging process, maximum value process, minimum value process, and intermediate value process. For example, the multiple motion amplitude scores can be averaged to obtain the motion amplitude information of the shot segment; or, the multiple motion amplitude scores can be weighted averaged to obtain the motion amplitude information of the shot segment, wherein the motion amplitude scores corresponding to different target motion amplitude analysis methods have different weights. For another example, the maximum value among the multiple motion amplitude scores can be used as the motion amplitude information of the shot segment. For example, the intermediate value among the multiple motion amplitude scores can be calculated, and the intermediate value can be used as the motion amplitude information of the shot segment.

[0086] For example, taking the optical flow method and the background difference method as examples, for any shot segment, the optical flow method can be used to analyze the motion amplitude of the shot segment to obtain the motion amplitude classification result a of the shot segment, and the background difference method can be used to analyze the motion amplitude of the shot segment to obtain the motion amplitude classification result b of the shot segment, and the motion amplitude score corresponding to the motion amplitude classification result a is determined to be score A, and the motion amplitude score corresponding to the motion amplitude classification result b is determined to be score B. The motion amplitude information of the shot segment can be determined based on the score A and the score B. Exemplarily, the average algorithm can be used to average the score A and the score B, and the average value obtained is used as the motion amplitude information of the shot segment.

[0087] In step 204, based on the motion amplitude information of at least one shot segment, key frames are extracted from the corresponding shot segment.

[0088] In some embodiments, based on the motion amplitude information of at least one shot segment, the number of key frame reference frames of at least one shot segment is determined; based on the number of key frame reference frames of at least one shot segment, key frames of a corresponding number of frames are extracted from the corresponding shot segment.

[0089] In a possible implementation, different motion amplitude information may correspond to different key frame reference frame numbers, that is, there is a mapping relationship between the motion amplitude information and the key frame reference frame number. For each shot segment in the video to be processed, when the motion amplitude information of the shot segment is obtained, the key frame reference frame number corresponding to the motion amplitude information may be determined from the mapping relationship as the key frame reference frame number of the shot segment.

[0090] In another possible implementation, if a large language model is used to understand the video content of the processed video, the number of key frame reference frames of each of the at least one shot segment can be determined based on the maximum number of images (maximum number of tokens) that can be accommodated in the large language model and the motion amplitude information of each of the at least one shot segment. In a possible implementation, the calculation formula of the key frame reference frame number can be as follows:

[0091]

[0092] Among them, m a is the number of key frame reference frames; i_max is the maximum number of images that can be accommodated in the large language model; s is the motion amplitude information of the shot segment (such as the motion amplitude score); n is a constant, which is related to the classification of the motion amplitude level. For example, if the motion amplitude is classified into 5 motion amplitude levels, the value of n can be 5 or 6, and the present disclosure does not make any specific limitation on this.

[0093] For example, the above formula (1) can be used to determine the number of key frame reference frames of each shot segment in the video to be processed, so as to extract key frames corresponding to the number of key frame reference frames from the shot segment. It can be seen that the number of key frame reference frames of the shot segment can be determined based on the motion amplitude information of the shot segment, so as to extract a corresponding number of key frames from the corresponding shot segment based on the number of key frame reference frames, and ensure that when the video moves violently, the frame extraction is more intensive, and when the motion is relatively gentle, the frame extraction is more sparse.

[0094] In some embodiments, after obtaining the number of key frame reference frames of a shot segment, a fixed FPS (frequency) may be used to extract key frames corresponding to the number of key frame reference frames from the shot segment. In some embodiments, after obtaining the number of key frame reference frames of a shot segment, the shot segment may be firstly subjected to uniformly spaced frame extraction to obtain candidate image frames of the shot segment, and then principal component analysis may be performed on the extracted candidate image frames to select key frames equal to the number of key frame reference frames, and the selected key frames may be more capable of representing the content of the shot segment.

[0095] In step 205, based on the multimodal features of the key frames of at least one shot segment, the video content is understood for the video to be processed to obtain description information of the video to be processed.

[0096] In the embodiments of the present disclosure, step 205 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0097] By implementing the embodiments of the present disclosure, the number of key frame reference frames of the shot clip can be determined based on the motion amplitude information of the shot clip, so that a corresponding number of key frames can be extracted from the corresponding shot clip based on the key frame reference frame number, which can ensure that when the video moves violently, the frame extraction is more intensive, and when the motion is relatively gentle, the frame extraction is more sparse, that is, differentiated processing is performed when extracting frames, and adaptive frame extraction can be achieved, thereby further improving the effect of video understanding when the maximum Token of the large language model allows.

[0098] Figure 3 FIG. 1 is a flow chart of a video understanding method provided according to an embodiment of the present disclosure. Figure 3 As shown, the video understanding method may include but is not limited to the following steps.

[0099] In step 301, the video to be processed is segmented into shots to obtain at least one shot segment of the video to be processed.

[0100] In the embodiments of the present disclosure, step 301 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0101] In step 302, target motion amplitude analysis is performed on at least one shot segment to obtain motion amplitude information of at least one shot segment.

[0102] In the embodiments of the present disclosure, step 302 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0103] In step 303, based on the motion amplitude information of at least one shot segment, the number of key frame reference frames of at least one shot segment is determined.

[0104] In the embodiments of the present disclosure, step 303 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0105] In step 304, for each shot segment in at least one shot segment, the shot segments are uniformly frame-decimated to obtain a plurality of candidate image frames of the shot segments.

[0106] In step 305, a feature vector of the shot segment is determined based on a plurality of candidate image frames of the shot segment.

[0107] In some embodiments, for each shot segment in at least one shot segment, a feature vector of the shot segment can be determined based on multiple candidate image frames of the shot segment. Exemplarily, for any candidate image frame of the shot segment, a multimodal feature extraction model (such as a CLIP model) can be used to extract features from the candidate image frame to obtain multimodal features of the candidate image frame, and then multiple multimodal features of multiple candidate image frames of the shot segment are fused to determine the feature vector of the shot segment. The fusion process can be a merging process, but is not limited thereto. For example, for each candidate image frame, a CLIP model is used to extract features and then reshape them to an n*1-dimensional vector, and then the n*1-dimensional vectors of the multiple candidate image frames of the shot segment are merged to obtain a [m*n] vector, which is the feature vector of the shot segment.

[0108] In step 306, principal component analysis is performed on the shot segment based on the feature vector to extract key frames having a number equal to the number of key frame reference frames from the shot segment.

[0109] In some embodiments, for each shot segment in at least one shot segment, principal component analysis can be performed on the shot segment using PCA technology based on the feature vector of the shot segment to extract key frames from the shot segment with a number equal to the number of key frame reference frames.

[0110] In step 307, based on the multimodal features of the key frames of at least one shot segment, the video content is understood for the video to be processed to obtain description information of the video to be processed.

[0111] In the embodiments of the present disclosure, step 307 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0112] For example, Figure 4 As shown, a video to be processed can be input, and the video to be processed can be segmented to obtain different shot segments of the video to be processed. For the shot segments, the LK (Lucas–Kanade) optical flow method can be used to detect the optical flow, and the motion amplitude level can be classified for the detected optical flow to obtain the motion amplitude classification result of the shot segment. For the same shot segment, the background difference method is used to estimate the motion of the video to obtain the motion amplitude classification result of the shot segment. The motion amplitude classification results obtained by the above two motion amplitude analysis methods are fused to obtain the motion amplitude information of the shot segment. Based on the motion amplitude information, the number of key frame reference frames corresponding to the shot segment is determined, and the number of key frames equal to the number of key frame reference frames is extracted from the shot segment. Based on the multimodal features of the key frames of all shot segments of the video to be processed, the video content of the video to be processed is understood to obtain the description information of the video to be processed.

[0113] By implementing the embodiments of the present disclosure, the shot fragment can be evenly frame-sampled to obtain multiple candidate image frames of the shot fragment, and principal component analysis can be performed on the shot fragment based on the multiple candidate image frames to extract key frames from the shot fragment with a number equal to the number of key frame reference frames, so that the selected key frames can better represent the content of the shot fragment, and the video understanding effect can be further improved.

[0114] Figure 5 is a block diagram of a video understanding device provided according to an embodiment of the present disclosure. Figure 5 As shown, the video understanding device may include but is not limited to: a segmentation module 510 , an analysis module 520 , an extraction module 530 and a content understanding module 540 .

[0115] The segmentation module 510 is used to perform shot segmentation on the video to be processed to obtain at least one shot segment of the video to be processed.

[0116] The analysis module 520 is used to perform target motion amplitude analysis on at least one shot segment respectively to obtain motion amplitude information of at least one shot segment respectively.

[0117] The extraction module 530 is used to extract key frames from the corresponding shot segments based on the motion amplitude information of each of the at least one shot segments.

[0118] The content understanding module 540 is used to understand the video content of the video to be processed based on the multimodal features of the key frames of each of the at least one shot segment, and obtain the description information of the video to be processed. In some embodiments, the content understanding module 540 is used to: understand the video content of the video to be processed through the large language model based on the multimodal features of the key frames of each of the at least one shot segment and the preset questions, and obtain the description information of the video to be processed; the large language model is used to understand the video content of the video to be processed based on the multimodal features and the preset questions.

[0119] In some embodiments, Figure 6 As shown, the analysis module 520 includes: an analysis unit 621 and a determination unit 622. The analysis unit 621 is used to analyze the target motion amplitude of each shot segment in at least one shot segment by using at least one target motion amplitude analysis method to obtain at least one dynamic feature of the shot segment, and classify the motion amplitude of the shot segment based on the at least one dynamic feature to obtain at least one motion amplitude classification result of the shot segment; the determination unit 622 is used to determine the motion amplitude information of the shot segment based on the at least one motion amplitude classification result of the shot segment. Figure 6 610, 630, 640 and Figure 5 510, 530, and 540 have the same function and structure.

[0120] In some embodiments, at least one target motion amplitude analysis method includes at least one of an optical flow method, a background difference method, and an inter-frame difference method.

[0121] In some embodiments, the target motion amplitude analysis method includes an optical flow method, and the dynamic features include optical flow features; the analysis unit 621 is used to: use the optical flow method to perform optical flow detection on the lens fragments to obtain the optical flow features of the lens fragments; and classify the motion amplitude of the lens fragments based on the optical flow features.

[0122] In some embodiments, the analysis unit 621 is used to: select a plurality of consecutive image frames from a shot segment; perform pixel value averaging on pixel points at corresponding positions in the plurality of consecutive image frames to obtain a pixel average value of the pixel points, and use the pixel average value as the grayscale value of the corresponding pixel points in the background image to obtain a background image; perform differential operations on the plurality of consecutive image frames and the background image respectively to obtain respective foreground objects in the plurality of consecutive image frames; and perform motion amplitude classification on the shot segment based on a change sequence of the foreground objects between the plurality of consecutive image frames.

[0123] In some embodiments, the determination unit 622 is used to: determine a motion amplitude score corresponding to each motion amplitude classification result in at least one motion amplitude classification result; and determine motion amplitude information of the shot segment according to at least one motion amplitude score of the shot segment.

[0124] In some embodiments, the determination unit 622 is used to: when the number of at least one target motion amplitude analysis method is one, a motion amplitude score of the shot clip is used as the motion amplitude information of the shot clip; or when the number of at least one target motion amplitude analysis method is multiple, multiple motion amplitude scores of the shot clip are fused and the score obtained after the fusion is used as the motion amplitude information of the shot clip.

[0125] In some embodiments, Figure 7 As shown, the extraction module 630 includes: a determination unit 631 and an extraction unit 632. The determination unit 631 is used to determine the number of key frame reference frames of at least one shot segment based on the motion amplitude information of at least one shot segment; the extraction unit 632 is used to extract the key frames of the corresponding frame number from the corresponding shot segment based on the number of key frame reference frames of at least one shot segment. Figure 7 710, 720, 740 and Figure 6 610, 620, and 640 have the same function and structure.

[0126] In some embodiments, the extraction unit 632 is further used to: for each shot segment in at least one shot segment, uniformly extract frames from the shot segment to obtain multiple candidate image frames of the shot segment; determine a feature vector of the shot segment based on the multiple candidate image frames of the shot segment; and perform principal component analysis on the shot segment based on the feature vector to extract key frames from the shot segment whose number is the number of key frame reference frames.

[0127] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] The embodiment of the present disclosure also proposes an intelligent agent. The intelligent agent includes an input module, a processing module and an output module. The input module is used to receive input information. Exemplarily, the input information can be a video to be processed. The processing module is used to determine a target task based on the input information received by the input module, and execute the video understanding method provided by the present disclosure based on the target task to obtain output information, that is, the output information can be description information of the video to be processed. The output module is used to output the output information obtained by the processing module.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0130] like Figure 8 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0131] like Figure 8 As shown, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 801 is taken as an example.

[0132] The memory 802 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the video understanding method provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the video understanding method provided by the present disclosure.

[0133] The memory 802 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the video understanding method in the embodiment of the present disclosure (for example, the attached Figure 5 The processor 801 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 802, that is, the video understanding method in the above method embodiment is implemented.

[0134] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] The electronic device may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0136] The input device 803 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 804 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0137] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0141] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video understanding method, comprising: Segment the video to be processed to obtain at least one shot segment of the video to be processed; Performing target motion amplitude analysis on the at least one shot segment respectively to obtain motion amplitude information of the at least one shot segment; Extracting key frames from the corresponding shot segments based on the motion amplitude information of each of the at least one shot segment; Based on the multimodal features of the key frames of each of the at least one shot segment, video content understanding is performed on the video to be processed to obtain description information of the video to be processed.

2. The method according to claim 1, wherein: The performing target motion amplitude analysis on the at least one shot segment respectively to obtain motion amplitude information of each of the at least one shot segment includes: For each shot segment in the at least one shot segment, respectively use at least one target motion amplitude analysis method to perform target motion amplitude analysis on the shot segment to obtain at least one dynamic feature of the shot segment, and perform motion amplitude classification on the shot segment based on the at least one dynamic feature to obtain at least one motion amplitude classification result of the shot segment; Based on at least one motion amplitude classification result of the shot segment, motion amplitude information of the shot segment is determined.

3. The method according to claim 2, wherein: The at least one target motion amplitude analysis method includes at least one of an optical flow method, a background difference method, and an inter-frame difference method.

4. The method according to claim 3, wherein: The target motion amplitude analysis method includes the optical flow method, and the dynamic feature includes the optical flow feature; The optical flow method is used to analyze the target motion amplitude of the shot segment to obtain the dynamic features of the shot segment, and the motion amplitude of the shot segment is classified based on the dynamic features, including: Using the optical flow method to perform optical flow detection on the shot segment to obtain optical flow features of the shot segment; The motion amplitude of the shot segment is classified based on the optical flow feature.

5. The method according to claim 3, wherein: The target motion amplitude of the shot segment is analyzed by using the background difference method to obtain the dynamic features of the shot segment, and the motion amplitude of the shot segment is classified based on the dynamic features, including: Selecting a plurality of continuous image frames from the shot segment; Perform pixel value averaging processing on pixel points at corresponding positions in the plurality of consecutive image frames to obtain a pixel average value of the pixel points, and use the pixel average value as the grayscale value of the corresponding pixel point in the background image to obtain the background image; Performing a difference operation between the plurality of continuous image frames and the background image respectively to obtain respective foreground objects in the plurality of continuous image frames; Based on the change sequence of the foreground object between the plurality of consecutive image frames, the motion amplitude of the shot segment is classified.

6. The method according to claim 2, wherein: The determining the motion amplitude information of the shot segment based on at least one motion amplitude classification result of the shot segment comprises: Determine a motion amplitude score corresponding to each of the motion amplitude classification results in the at least one motion amplitude classification result; The motion amplitude information of the shot segment is determined according to at least one motion amplitude score of the shot segment.

7. The method according to claim 6, wherein: The step of determining the motion amplitude information of the shot segment according to at least one motion amplitude score of the shot segment comprises: The number of the at least one target motion amplitude analysis method is one, and a motion amplitude score of the shot segment is used as the motion amplitude information of the shot segment; or The number of the at least one target motion amplitude analysis method is multiple, and the multiple motion amplitude scores of the shot segment are fused, and the scores obtained after the fusion process are used as the motion amplitude information of the shot segment.

8. The method according to claim 1, wherein: The step of extracting key frames from corresponding shot segments based on the motion amplitude information of each of the at least one shot segments comprises: Determining the number of key frame reference frames of each of the at least one shot segment based on the motion amplitude information of each of the at least one shot segment; Based on the key frame reference frame number of each of the at least one shot segment, key frames with a corresponding frame number are extracted from the corresponding shot segment.

9. The method according to claim 8, wherein: The step of extracting key frames of corresponding frame numbers from corresponding shot segments based on the key frame reference frame numbers of the at least one shot segment comprises: For each shot segment in the at least one shot segment, uniformly extracting frames from the shot segment to obtain a plurality of candidate image frames of the shot segment; Determining a feature vector of the shot segment based on a plurality of candidate image frames of the shot segment; A principal component analysis is performed on the shot segment based on the feature vector to extract key frames from the shot segment whose number is the number of key frame reference frames.

10. The method according to any one of claims 1 to 9, wherein: The step of understanding the video content of the video to be processed based on the multimodal features of the key frames of each of the at least one shot segment to obtain description information of the video to be processed includes: Based on the multimodal features of the key frames of each of the at least one shot segment and preset questions, the video content of the video to be processed is understood through a large language model to obtain description information of the video to be processed; the large language model is used to understand the video content of the video to be processed based on the multimodal features and preset questions.

11. A video understanding device, comprising: A segmentation module, used to perform shot segmentation on the video to be processed to obtain at least one shot segment of the video to be processed; An analysis module, configured to perform target motion amplitude analysis on the at least one shot segment respectively to obtain motion amplitude information of each of the at least one shot segment; An extraction module, configured to extract key frames from the corresponding shot segments based on the motion amplitude information of each of the at least one shot segments; The content understanding module is used to understand the video content of the video to be processed based on the multimodal features of the key frames of each of the at least one shot segment, so as to obtain description information of the video to be processed.

12. The device according to claim 11, wherein The analysis module comprises: an analyzing unit, configured to respectively perform target motion amplitude analysis on each shot segment of the at least one shot segment by using at least one target motion amplitude analysis method to obtain at least one dynamic feature of the shot segment, and perform motion amplitude classification on the shot segment based on the at least one dynamic feature to obtain at least one motion amplitude classification result of the shot segment; A determination unit is used to determine the motion amplitude information of the shot segment based on at least one motion amplitude classification result of the shot segment.

13. The device according to claim 12, wherein: The at least one target motion amplitude analysis method includes at least one of an optical flow method, a background difference method, and an inter-frame difference method.

14. The device according to claim 13, wherein: The target motion amplitude analysis method includes the optical flow method, and the dynamic feature includes the optical flow feature; The analysis unit is used for: Using the optical flow method to perform optical flow detection on the shot segment to obtain optical flow features of the shot segment; The motion amplitude of the shot segment is classified based on the optical flow feature.

15. The device according to claim 13, wherein: The analysis unit is used for: Selecting a plurality of continuous image frames from the shot segment; Perform pixel value averaging processing on pixel points at corresponding positions in the plurality of consecutive image frames to obtain a pixel average value of the pixel points, and use the pixel average value as the grayscale value of the corresponding pixel point in the background image to obtain the background image; Performing a difference operation between the plurality of continuous image frames and the background image respectively to obtain respective foreground objects in the plurality of continuous image frames; Based on the change sequence of the foreground object between the plurality of consecutive image frames, the motion amplitude of the shot segment is classified.

16. The device according to claim 12, wherein: The determining unit is used for: Determine a motion amplitude score corresponding to each of the motion amplitude classification results in the at least one motion amplitude classification result; The motion amplitude information of the shot segment is determined according to at least one motion amplitude score of the shot segment.

17. The device according to claim 16, wherein: The determining unit is used for: The number of the at least one target motion amplitude analysis method is one, and a motion amplitude score of the shot segment is used as the motion amplitude information of the shot segment; or The number of the at least one target motion amplitude analysis method is multiple, and the multiple motion amplitude scores of the shot segment are fused, and the scores obtained after the fusion process are used as the motion amplitude information of the shot segment.

18. The device according to claim 11, wherein The extraction module comprises: A determining unit, configured to determine the number of key frame reference frames of each of the at least one shot segment based on the motion amplitude information of each of the at least one shot segment; The extraction unit is used to extract key frames of a corresponding number of frames from the corresponding shot segment based on the key frame reference frame number of each of the at least one shot segment.

19. The device according to claim 18, wherein: The extraction unit is used for: For each shot segment in the at least one shot segment, uniformly extracting frames from the shot segment to obtain a plurality of candidate image frames of the shot segment; Determining a feature vector of the shot segment based on a plurality of candidate image frames of the shot segment; A principal component analysis is performed on the shot segment based on the feature vector to extract key frames from the shot segment whose number is the number of key frame reference frames.

20. The device according to any one of claims 11 to 19, wherein: The content understanding module is used to: Based on the multimodal features of the key frames of each of the at least one shot segment and the preset questions, the video content of the video to be processed is understood by a large language model to obtain description information of the video to be processed; The large language model is used to understand the video content of the video to be processed based on the multimodal features and preset questions.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video understanding method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the video understanding method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the computer program implements the steps of the video understanding method according to any one of claims 1 to 10.

Images