Video classification method and device, electronic device, and storage medium

Abstract

The application provides a video classification method and device, an electronic device and a storage medium, wherein the method comprises: encoding a first video to obtain a plurality of corresponding image frames, wherein the first video is any video in a plurality of videos to be classified; determining the complexity of the first video based on a target image frame in the plurality of image frames, wherein the complexity comprises spatial complexity and temporal complexity; and classifying the plurality of videos based on the complexity. Through the application, the technical problem that the prior art classifies by using the bottom features of a video and needs to rely on strong prior knowledge, resulting in poor video classification effect and weak robustness, is solved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video classification method and apparatus, electronic device and storage medium. Background Technology

[0002] Currently, in order to test video compression coding technology, a video classification method based on the underlying features of video content is adopted. This video classification method selects cartoons, commercials, movies, news and sports games as classification objects, selects color as the classification feature with the best classification effect from the underlying features of the video (such as color, texture and shape), and compares, analyzes and summarizes various classification algorithms based on color features, and proposes a classification method based on the fusion of underlying features.

[0003] The main shortcomings of video classification methods based on low-level video content features are: 1) These methods are limited to only a subset of low-level features, such as color, texture, and shape, and cannot encompass all features contained in the video content; 2) Different features in these methods exhibit biases towards different video classifications, requiring reliance on manually built expert systems for fusion; 3) These methods rely heavily on prior knowledge, resulting in weak robustness; 4) These methods require the extraction and fusion of multiple features, leading to a large computational burden. Summary of the Invention

[0004] The purpose of this application is to provide a video classification method, apparatus, electronic device, and storage medium, solving the problem that existing technologies classify videos based on low-level features, which relies heavily on prior knowledge, resulting in poor video classification performance and weak robustness. The specific technical solution is as follows:

[0005] In a first aspect of this application, a video classification method is provided, comprising: encoding a first video to obtain a plurality of corresponding image frames, wherein the first video is any one of a plurality of videos to be classified; determining the complexity of the first video based on a target image frame among the plurality of image frames, wherein the complexity includes spatial complexity and time complexity; and classifying the plurality of videos based on the complexity.

[0006] In a second aspect of this application, a video classification apparatus is also provided, comprising: an encoding module for encoding a first video to obtain a plurality of corresponding image frames, wherein the first video is any one of the plurality of videos to be classified; a determination module for determining the complexity of the first video based on a target image frame among the plurality of image frames, wherein the complexity includes spatial complexity and time complexity; and a classification module for classifying the plurality of videos based on the complexity.

[0007] In a third aspect of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the method described in the first aspect when executing the program stored in the memory.

[0008] In a fourth aspect of this application, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, implement the steps of the method described in the first aspect.

[0009] In this embodiment, the first video to be classified is first encoded to obtain multiple corresponding image frames. The complexity of the first video is determined based on the target frame among the multiple image frames. Since the first video is any one of the multiple videos, the complexity of each of the multiple videos to be classified can be obtained in the above manner. Then, the multiple videos can be classified according to the complexity. That is to say, the video classification method in this embodiment classifies the video based on the time complexity and space complexity of the video. It is not limited to classifying the video based solely on its low-level features, nor does it rely on prior human experience. Its classification is more efficient and more robust, thereby solving the problem in the prior art that classifies the video based on its low-level features and relies on strong prior knowledge, resulting in poor video classification performance and weak robustness. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0011] Figure 1 This is a flowchart of the video classification method in the embodiments of this application;

[0012] Figure 2 This is a schematic diagram of the time / space complexity after normalization in the coordinate system of the embodiments of this application;

[0013] Figure 3 This is a schematic diagram of the time / space complexity after logarithmic processing in a coordinate system in the embodiments of this application;

[0014] Figure 4 This is a schematic diagram of video classification based on time / space complexity characteristics in an embodiment of this application;

[0015] Figure 5 This is a schematic diagram of the structure of the video classification method apparatus in the embodiments of this application;

[0016] Figure 6This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the following description, suffixes such as "module" and "unit" used to denote elements are used only for the purposes of this application and have no specific meaning in themselves. Therefore, "module" and "component" can be used interchangeably.

[0019] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. The embodiments of this application provide a video classification method, such as... Figure 1 As shown, the method includes the following steps:

[0020] Step 102: Encode the first video to obtain multiple corresponding image frames, wherein the first video is any one of the multiple videos to be classified;

[0021] Step 104: Determine the complexity of the first video based on the target image frame among multiple image frames, wherein the complexity includes spatial complexity and time complexity;

[0022] Step 106: Classify multiple videos based on complexity.

[0023] Through steps 102 to 106 above, the first video to be classified is first encoded to obtain multiple corresponding image frames. The complexity of the first video is determined based on the target frame in the multiple image frames. Since the first video is any one of the multiple videos, the complexity of each of the multiple videos to be classified can be obtained in the above manner. Then, the multiple videos can be classified according to the complexity. That is to say, the video classification method in this application embodiment classifies the video based on the time complexity and space complexity of the video. It is not limited to classifying the video based solely on its low-level features, nor does it rely on prior human experience. Its classification is more efficient and more robust, thereby solving the problem in the prior art that classifies the video based on its low-level features and requires strong prior knowledge, resulting in poor video classification performance and weak robustness.

[0024] In an optional embodiment of this application, the method for determining the complexity of the first video based on a target image frame among multiple image frames involved in step 104 above may further include:

[0025] Step 11: When the target frame is an I-frame, determine the first average value of the I-frames among multiple image frames, and determine the spatial complexity of the first video based on the ratio of the first average value to the sum of the values ​​of the multiple image frames; or,

[0026] Step 12: When the target frame is P-frame, determine the second average value of P-frames among multiple image frames, and determine the time complexity of the first video based on the ratio of the second average value to the first average value.

[0027] It's important to note that after video compression, the resulting image frames include I-frames, P-frames, and B-frames. I-frames represent keyframes, meaning this frame's entire image needs to be preserved. Decoding only requires this frame's data (because it contains the complete image). P-frames represent the difference between this frame and a previous keyframe (or P-frame). Decoding requires overlaying the previously cached image with the differences defined in this frame to generate the final image. In other words, P-frames don't contain complete image data, only data showing the differences from the previous frame. B-frames are bidirectional difference frames, meaning they record the differences between this frame and the frames before and after them. In other words, decoding a B-frame requires not only retrieving the previous cached image but also decoding the subsequent image, obtaining the final image by overlaying the data from both frames.

[0028] For example, in a specific application scenario, the first video is encoded using a standard encoder (such as X264) to obtain multiple Groups of Pictures (GOPs). Each GOP includes one I-frame, multiple B-frames, and multiple P-frames. Therefore, to determine the average value of an I-frame, it is necessary to first calculate the sum of the sizes of all I-frames in all GOPs, and then determine the average value of the I-frame. The method for determining the average value of P-frames is similar to that for I-frames.

[0029] Furthermore, it should be noted that time complexity refers to the computational workload required to play the video, while space complexity refers to the memory space required to play the video. Time and space (i.e., registers) are both important aspects of computer resources, and complexity refers to the amount of resources required by the computer when playing a video. Therefore, in this application embodiment, using either space complexity or time complexity allows for a more comprehensive classification of videos, rather than being limited to the relatively simple classification methods in existing technologies, such as classification based on some low-level features, or by tags such as resolution, frame rate, bit rate, scene type, and business type. Additionally, in this application embodiment, either space complexity or time complexity can be used as the basis for classification.

[0030] In an optional embodiment of this application, the method of determining the first average value of I-frames among multiple image frames in step 11 can be further described as: normalizing the I-frames among multiple image frames and determining the first average value of the normalized I-frames.

[0031] In an optional embodiment of this application, the method for determining the second average value of P frames among multiple image frames in step 12 can be further described as: normalizing the P frames among multiple image frames and determining the second average value of the normalized P frames.

[0032] In other words, in this embodiment, I-frames and P-frames can be normalized first before determining their average value. Normalization further improves the accuracy of the complexity, enabling more refined video classification. Figure 2 As shown, a coordinate system is used with space complexity as the horizontal axis and time complexity as the vertical axis. This coordinate system can intuitively display the space complexity and time complexity of each video.

[0033] In this embodiment of the application, the method of classifying multiple videos based on complexity in step 106 may further include:

[0034] Step 21: Logarithmize the space complexity to obtain the first result;

[0035] Step 22: Logarithmize the time complexity to obtain the second result;

[0036] Step 23: Determine the position of the first video in a Cartesian coordinate system using the first and second results; where the vertical axis of the Cartesian coordinate system represents the time complexity and the horizontal axis represents the space complexity.

[0037] Step 24: Classify multiple videos based on their positions in a Cartesian coordinate system.

[0038] For steps 21 to 24 above, since the values ​​of I-frames and P-frames are relatively small after normalization, the space complexity and time complexity are logarithmically normalized, and the results of this logarithmic normalization are displayed in a coordinate system where the vertical axis represents time complexity and the horizontal axis represents space complexity. Figure 3 As shown, the complexity of each video can be more dispersed in this rectangular coordinate system, which means that the complexity of each video can be compared more intuitively.

[0039] In this embodiment of the application, the method of classifying multiple videos based on complexity involved in step 106 may further include:

[0040] Step 31: Classify multiple videos based on the preset range of values ​​into which the spatial complexity falls;

[0041] Step 32: Classify multiple videos based on the preset range of values ​​that the time complexity falls into;

[0042] Different preset range values ​​correspond to different video classification results.

[0043] As can be seen from steps 31 and 32 above, videos can be classified based on the preset range of values ​​that the spatial complexity or time complexity falls into, that is, videos that fall into the same preset range of values ​​are classified into one category.

[0044] The present application will now be explained in detail with reference to specific embodiments of the present application. These specific embodiments provide a video classification method based on time / space complexity characteristics. Figure 4 The method includes:

[0045] Step 401: Using the same quantization parameters, encode all video samples using the x264 encoder.

[0046] Step 402: Use the ratio of the average size of the normalized I-frame to the size of the original frame as the space complexity.

[0047] Step 403: Use the ratio of the normalized average size of P-frames to the average size of I-frames as the time complexity.

[0048] Step 404: Using space complexity as the x-axis and time complexity as the y-axis, obtain the position of the time / space complexity of all video samples in the coordinate system.

[0049] Step 405: Logarithmize the space complexity and time complexity respectively.

[0050] Step 406: Classify the space complexity and time complexity separately.

[0051] In this context, a lower gear indicates lower complexity, while a higher gear indicates higher complexity.

[0052] Through steps 401 to 406 above, a standard encoder is used to obtain the normalized I-frame size and P-frame size to measure spatial and temporal complexity. Logarithmic processing is then performed on each to classify the video. The video classification method of this specific implementation is not limited to some low-level features, has no preference for video classification, has better consistency, does not need to rely on manually established expert systems and prior knowledge, has strong robustness, and can directly reflect the complexity of information contained in the video content.

[0053] Corresponding to the above Figure 1 This application also provides a video classification device, such as... Figure 5 As shown, it includes:

[0054] The encoding module 52 is used to encode the first video to obtain multiple corresponding image frames, wherein the first video is any one of the multiple videos to be classified;

[0055] The determination module 54 is used to determine the complexity of the first video based on the target image frame among multiple image frames, wherein the complexity includes space complexity and time complexity.

[0056] Classification module 56 is used to classify multiple videos based on complexity.

[0057] The apparatus of this application embodiment first encodes a first video to be classified to obtain multiple corresponding image frames. The complexity of the first video is determined based on the target frame among the multiple image frames. Since the first video is any one of the multiple videos, the complexity of each of the multiple videos to be classified can be obtained in the above manner. Then, the multiple videos can be classified according to the complexity. That is to say, the video classification method in this application embodiment classifies videos based on the time complexity and space complexity of the videos. It is not limited to classifying only by the low-level features of the videos, nor does it rely on prior human experience for classification. Its classification is more efficient and more robust, thereby solving the problem in the prior art that classifies videos by low-level features and requires strong prior knowledge, resulting in poor video classification effect and weak robustness.

[0058] Optionally, the determining module 54 in this application embodiment may further include: a first determining unit, configured to determine a first average value of I-frames among multiple image frames when the target frame is an I-frame, and determine the spatial complexity of the first video based on the ratio of the first average value to the sum of the values ​​of the multiple image frames; or, a second determining unit, configured to determine a second average value of P-frames among multiple image frames when the target frame is a P-frame, and determine the time complexity of the first video based on the ratio of the second average value to the first average value.

[0059] In specific application scenarios, the first video is encoded using a standard encoder (such as X264) to obtain multiple Groups of Pictures (GOPs). Each GOP includes one I-frame, multiple B-frames, and multiple P-frames. Therefore, to determine the average value of an I-frame, it is necessary to first calculate the sum of the sizes of all I-frames in all GOPs, and then determine the average value of the I-frame. The method for determining the average value of P-frames is similar to that for I-frames.

[0060] Furthermore, it should be noted that time complexity refers to the computational workload required to play a video, while space complexity refers to the memory space required to play a video. Time and space (i.e., registers) are both important aspects of computer resources, and complexity refers to the amount of resources required by the computer when playing a video. Therefore, in this application's embodiments, using space complexity or time complexity allows for a more comprehensive classification of videos, rather than being limited to the relatively singular classification methods in existing technologies, such as classification based on some low-level features, or by tags such as resolution, frame rate, bit rate, scene type, and business type.

[0061] Optionally, the first determining unit in this application embodiment includes: a first processing subunit, configured to normalize I-frames in a plurality of image frames and determine a first average value of the normalized I-frames;

[0062] Optionally, the second determining unit in this application embodiment includes: a second processing subunit, used to normalize P frames in a plurality of image frames and determine a second average value of the normalized P frames.

[0063] In other words, in this embodiment, I-frames and P-frames can be normalized first before determining their average value. Normalization further improves the accuracy of the complexity, enabling more refined video classification. Figure 2 As shown, a coordinate system is used with space complexity as the horizontal axis and time complexity as the vertical axis. This coordinate system can intuitively display the space complexity and time complexity of each video.

[0064] Optionally, the classification module 56 in this embodiment may further include: a first processing unit, configured to perform logarithmic processing on the space complexity to obtain a first result; a second processing unit, configured to perform logarithmic processing on the time complexity to obtain a second result; a third determining unit, configured to determine the position of the first video in a Cartesian coordinate system based on the first result and the second result; wherein the vertical coordinate of the Cartesian coordinate system is the time complexity, and the horizontal coordinate of the Cartesian coordinate system is the space complexity; and a first classification unit, configured to classify multiple videos based on their positions in the Cartesian coordinate system.

[0065] Therefore, since the values ​​of I-frames and P-frames are relatively small after normalization, the space complexity and time complexity are logarithmically normalized. The results of this logarithmic normalization are then displayed on a coordinate system where the vertical axis represents time complexity and the horizontal axis represents space complexity. Figure 3 As shown, the complexity of each video can be more dispersed in this rectangular coordinate system, which means that the complexity of each video can be compared more intuitively.

[0066] Optionally, the classification module 56 in this embodiment may further include: a second classification unit, used to classify multiple videos based on a preset range of values ​​in which the spatial complexity falls; and a third classification unit, used to classify multiple videos based on a preset range of values ​​in which the time complexity falls; wherein different preset range values ​​correspond to different video classification results.

[0067] This application also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0068] Memory 603 is used to store computer programs;

[0069] Processor 601, when executing a program stored in memory 603, implements... Figure 1 The methods and steps described, and their functions and effects Figure 1 The methods and steps are the same as those in the previous section, so they will not be repeated here.

[0070] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0071] The communication interface is used for communication between the aforementioned terminal and other devices.

[0072] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0073] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0074] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the video classification methods described in the above embodiments.

[0075] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the video classification methods described in the above embodiments.

[0076] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0079] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method of classifying a video, characterized by, include: The first video is encoded to obtain multiple corresponding image frames, wherein the first video is any one of the multiple videos to be classified; The complexity of the first video is determined based on the target image frame among the plurality of image frames, wherein the complexity includes space complexity and time complexity; the time complexity refers to the amount of computational work required to play the video; the space complexity refers to the memory space required to play the video. The multiple videos are classified based on the complexity. The step of determining the complexity of the first video based on the target image frame among the plurality of image frames includes: encoding all video samples using an encoder with the same quantization parameters; using the ratio of the average size of the normalized I-frame to the original frame size as the spatial complexity; and using the ratio of the average size of the normalized P-frame to the average size of the I-frame as the time complexity.

2. The method of claim 1, wherein, The classification of the multiple videos based on the complexity includes: Logarithmic transformation of the space complexity yields the first result; Logarithmic transformation of the time complexity yields the second result; The first result and the second result are used to determine the position of the first video in a Cartesian coordinate system; wherein the vertical axis of the Cartesian coordinate system is the time complexity, and the horizontal axis of the Cartesian coordinate system is the space complexity. The multiple videos are classified based on their positions in the Cartesian coordinate system.

3. The method according to claim 1, characterized in that, The classification of the multiple videos based on the complexity includes: The multiple videos are classified based on the preset range of values ​​into which the spatial complexity falls; The multiple videos are classified based on the preset range of values ​​into which the time complexity falls; Different preset range values ​​correspond to different video classification results.

4. A video classification device, characterized in that, include: An encoding module is used to encode a first video to obtain multiple corresponding image frames, wherein the first video is any one of the multiple videos to be classified; The determination module is used to determine the complexity of the first video based on the target image frame among the plurality of image frames, wherein the complexity includes space complexity and time complexity; the time complexity refers to the amount of computational work required to play the video; the space complexity refers to the memory space required to play the video. A classification module is used to classify the multiple videos based on the complexity. The determining module is used to perform the following steps: using the same quantization parameters, encoding all video samples with an encoder; using the ratio of the normalized average size of I-frames to the original frame size as the space complexity; and using the ratio of the normalized average size of P-frames to the average size of I-frames as the time complexity.

5. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-3.

Images