Video file intelligent index generation system and method based on content analysis

Abstract

The system and method for generating intelligent indexes for video files based on content analysis relate to the field of video processing technology and are used to solve the problem that the existing technology cannot quickly jump to video images according to the video content. The technical points of the present invention include: processing the content of the video file to obtain multiple index frames and their corresponding valid information; establishing a mapping relationship between the index frames, valid information and the corresponding time points of the index frames, thereby generating an index file; jumping to play the video according to the index file; wherein the valid information includes index frame thumbnails and text information generated according to the index frames. The present invention can quickly and accurately locate the video content points that the user is interested in, improves the efficiency of finding key information in the video, and also enhances the user experience. The present invention can be applied to fast jumps during video playback.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a system and method for generating intelligent indexes of video files based on content analysis. Background Art

[0002] During video conferencing or remote video teaching, video files are usually recorded for viewing after the meeting or class. In addition to the video and audio of the participants, these video files also contain courseware or documents shared through computers during the meeting and course. When watching these video files, people usually hope to quickly locate the part they want to watch. The existing technology provides a time-based fast index positioning function, that is, a video file index is generated according to a fixed time interval, and the video thumbnails at the fixed time interval can be viewed and clicked to jump to the corresponding time position. This method can provide a certain degree of fast file jump playback, but because the index is generated according to a fixed time interval, it has little relevance to the actual content and cannot effectively help viewers quickly locate the content of the video file. For example, a video file describes three concepts, A, B, and C, with concept A lasting eight minutes, concept B lasting five minutes, and concept C lasting ten minutes. If a file index generated at a fixed time interval is used at one-minute intervals, a large number of file indexes will point to duplicated content in the video file locations, making it impossible to help viewers quickly and easily find the content they want to watch. Furthermore, when searching for the progress of a specified content in the video, the mouse needs to be repeatedly moved left and right in the playback progress bar for positioning, which is inefficient and not precise enough. Video file indexes with fixed time periods also contain excessive duplicate and invalid information. Summary of the Invention

[0003] In view of the above problems, the present invention proposes a system and method for generating an intelligent index of video files based on content analysis, so as to solve the problem that the prior art cannot quickly jump to the video screen according to the video content.

[0004] According to one aspect of the present invention, a video file intelligent index generation system based on content analysis is proposed, which includes: an index generation module for processing the content of the video file and generating an index file according to the processing result; an image change analysis submodule, an image valid information analysis submodule and a mapping establishment submodule; wherein the image change analysis submodule is used to process each frame of video image that changes continuously in the video file, and determine that the current frame image that has changed compared with the previous frame of the video picture is an index frame, thereby obtaining multiple index frames; the image valid information analysis submodule is used to extract valid information from multiple index frames, and obtain valid information corresponding to each index frame; the mapping establishment submodule is used to establish a mapping relationship between index frames, valid information and corresponding time points of index frames, thereby generating an index file; a video playback module is used to jump and play videos according to the index file.

[0005] Preferably, the valid information includes index frame thumbnails and text information generated according to the index frames.

[0006] Preferably, the process of determining in the image change analysis submodule that the current frame image of the video screen that has changed compared with the previous frame is an index frame includes: performing block processing on each frame of the video image, comparing each image sub-block at the corresponding position of the current frame and the previous frame, and determining the changed frame based on the comparison result, and the changed frame is the index frame.

[0007] Preferably, the process of determining the changed frame based on the comparison results in the image change analysis submodule includes: recording the image sub-blocks with exactly the same pixel data at the corresponding position of the current frame and the previous frame as the same sub-blocks, otherwise as different sub-blocks; when the percentage of the same sub-blocks in the video frame image exceeds a preset percentage threshold, the current frame is calibrated as a non-changed frame, otherwise it is calibrated as a changed frame.

[0008] Preferably, the video playback module includes an index input submodule and a jump playback submodule, the index input submodule is used to input or select valid information; the jump playback submodule is used to play the video according to the index frame time point corresponding to the valid information in the index file.

[0009] According to another aspect of the present invention, a method for generating an intelligent index for video files based on content analysis is proposed. The method comprises the following steps: Step 1: Processing the content of the video file and generating an index file based on the processing results; Step 2: Jumping to play the video according to the index file. Furthermore, the specific steps of Step 1 include: processing each continuously changing frame of the video image in the video file, determining that the current frame image that has changed compared to the previous frame is the index frame, thereby obtaining multiple index frames; extracting valid information from the multiple index frames to obtain valid information corresponding to each index frame; establishing a mapping relationship between the index frames, valid information, and the time points corresponding to the index frames, thereby generating an index file; the valid information includes index frame thumbnails and text information generated based on the index frames.

[0010] Preferably, the process of determining in step one that the current frame image of the video screen that has changed compared to the previous frame is the index frame includes: dividing each frame of the video image into blocks, comparing each image sub-block at the corresponding position of the current frame and the previous frame, and determining the changed frame based on the comparison result, and the changed frame is the index frame.

[0011] Preferably, the process of determining the changed frame based on the comparison result in step one includes: recording the image sub-blocks with exactly the same pixel data at the corresponding position of the current frame and the previous frame as the same sub-blocks, otherwise as different sub-blocks; when the percentage of the same sub-blocks in the video image exceeds a preset percentage threshold, the current frame is calibrated as a non-changed frame, otherwise it is calibrated as a changed frame.

[0012] Preferably, the specific steps of step 2 include: after the user inputs or selects valid information, playing the video according to the index frame time point corresponding to the valid information in the index file.

[0013] The beneficial technical effect of the present invention is: the present invention analyzes the content from the video screen and generates a video file index based on the content. When playing the video file, the file index generated based on the video content can quickly and accurately locate the video content point that the user is interested in, thereby improving the efficiency of finding key information in the video and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0015] Figure 1 This is a structural diagram of a video file intelligent index generation system based on content analysis according to one embodiment of the present application;

[0016] Figure 2is a flow chart of a method for generating an intelligent index of video files based on content analysis according to another embodiment of the present application;

[0017] Figure 3 is a schematic diagram of index generation in a method for intelligent index generation of video files based on content analysis according to another embodiment of the present application;

[0018] Figure 4 1 is a schematic diagram of video jump in a method for generating intelligent indexes of video files based on content analysis according to another embodiment of the present application;

[0019] Figure 5 is a structural diagram of a computing device according to another embodiment of the present application;

[0020] Figure 6 It is a structural diagram of a computer-readable storage medium according to another embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below with reference to the accompanying drawings. Obviously, the described embodiments or examples are only some of the embodiments or examples of the present invention, and not all of them. Based on the embodiments or examples of the present invention, all other embodiments or examples obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0022] For videos such as online conferences or course materials, whose content rarely changes, this paper proposes a system for intelligently indexing video files based on content analysis. During the recording of a video file, the system automatically generates a file index related to the content of the conference or course materials / documents. When watching such videos, users can quickly jump to a specific video location using the content-related index file.

[0023] Figure 1 This is a diagram of a video file intelligent index generation system based on content analysis according to an embodiment of the present application. Figure 1 The system includes an index generation module 10 and a video playback module 20; wherein,

[0024] The index generation module 10 is used to process the video file content and generate an index file based on the processing results; it includes an image change analysis submodule 110, an image valid information analysis submodule 120 and a mapping establishment submodule 130; wherein,

[0025] The image change analysis submodule 110 is used to process each frame of the video image that changes continuously in the video file, and determine that the current frame image that has changed compared to the previous frame is the index frame, thereby obtaining multiple index frames; the image valid information analysis submodule 120 is used to extract valid information from the multiple index frames, and obtain valid information corresponding to each index frame; the mapping establishment submodule 130 is used to establish a mapping relationship between the index frame, valid information and the time point corresponding to the index frame, thereby generating an index file;

[0026] The video playing module 20 is used to jump and play videos according to the index file.

[0027] In this embodiment, optionally, the valid information includes an index frame thumbnail and text information generated according to the index frame.

[0028] In this embodiment, optionally, the process of determining in the image change analysis submodule 110 that the current frame image that has changed compared to the previous frame of the video screen is an index frame includes: performing block processing on each frame of the video image, comparing each image sub-block at the corresponding position of the current frame and the previous frame, and determining the changed frame based on the comparison result, and the changed frame is the index frame.

[0029] In this embodiment, optionally, the process of determining the changed frame based on the comparison results in the image change analysis submodule 110 includes: recording the image subblocks with exactly the same pixel data at the corresponding position of the current frame and the previous frame as the same subblocks, otherwise as different subblocks; when the percentage of the same subblocks in the video frame image exceeds a preset percentage threshold, the current frame is calibrated as a non-changed frame, otherwise it is calibrated as a changed frame.

[0030] In this embodiment, optionally, the video playback module 20 includes an index input submodule 210 and a jump playback submodule 220. The index input submodule 210 is used to input or select valid information; the jump playback submodule 220 is used to play the video according to the index frame time point corresponding to the valid information in the index file.

[0031] Another embodiment of the present invention provides a method for generating an intelligent index of a video file based on content analysis. Figure 2 As shown, the method includes the following steps: Step 1, processing the content of the video file and generating an index file according to the processing result; Step 2, jumping to play the video according to the index file.

[0032] In this embodiment, optionally, the specific steps of step one include: processing each frame of video image that changes continuously in the video file, determining that the current frame image of the video screen that has changed compared to the previous frame is an index frame, thereby obtaining multiple index frames; extracting valid information from the multiple index frames, and obtaining valid information corresponding to each index frame; establishing a mapping relationship between the index frame, valid information and the corresponding time point of the index frame, thereby generating an index file; the valid information includes index frame thumbnails and text information generated based on the index frame.

[0033] In this embodiment, optionally, the process of determining in step one that the current frame image of the video screen that has changed compared to the previous frame is the index frame includes: dividing each frame of the video image into blocks, comparing each image sub-block at the corresponding position of the current frame and the previous frame, and determining the changed frame based on the comparison result, and the changed frame is the index frame.

[0034] In this embodiment, optionally, the process of determining the changed frame based on the comparison results in step one includes: recording the image sub-blocks with exactly the same pixel data at corresponding positions of the current frame and the previous frame as identical sub-blocks, otherwise as different sub-blocks; when the percentage of identical sub-blocks in the video image exceeds a preset percentage threshold, calibrating the current frame as a non-changed frame, otherwise calibrating it as a changed frame.

[0035] In this embodiment, the specific steps of step 2 optionally include: after the user inputs or selects valid information, playing the video according to the index frame time point corresponding to the valid information in the index file.

[0036] Another embodiment of the present invention provides a method for generating an intelligent index for video files based on content analysis. The method includes performing an intelligent analysis on the screen content captured by the sending end when recording the video. The analysis content includes: changes in the image, and effective information of the image (such as text); at the same time, the voice during the video recording process can be further analyzed, and a video index is generated by integrating the analysis results of the video and voice.

[0037] The method is divided into two parts. The first part is as follows. Figure 3 As shown, video recording generates intelligent indexes: When recording a video, the system automatically generates an index by analyzing the shared screen image or voice information. The index includes a thumbnail of the image, the time the image appeared, and the valid information of the image. This includes image change analysis, image valid information analysis, and intelligent index generation.

[0038] Image change analysis: Based on the content of the shared screen, select images with relatively long duration as indexes.

[0039] According to an embodiment of the present invention, first, each frame of the shared screen is read and each frame is divided into 16x16 sub-blocks; the current frame and the previous frame are taken, and the sub-block pixel data at the corresponding positions are compared. If all pixels are exactly the same, they are recorded as the same block, otherwise they are different blocks; the percentage of the same blocks in the total blocks of each frame is recorded. When the proportion of the same blocks is greater than a preset percentage threshold, such as 90%, the frame is marked as a non-changing frame, otherwise the frame is marked as a changing frame; wherein the first frame is always a changing frame. Then, the first changing frame is selected as the first index frame. Assuming that the nth index frame has been selected, if the images within the preset time threshold (for example, 3 seconds) after the nth index frame are not changing frames, then the first changing frame that appears after the preset time threshold of the nth index frame is the n+1th index frame; if there is a changing frame in the image within the preset time threshold after the nth index frame, then the frame at the preset time threshold position after the nth index frame is the n+1th index frame.

[0040] It should be noted that the above method for calibrating the change frame may also adopt other methods to calibrate the change frame, such as a histogram-based method.

[0041] Image effective information analysis: Use text recognition or voice recognition algorithms to extract text or voice information from the content of each index frame image. If there is no text or voice information in the index frame image content, it will not be extracted.

[0042] According to an embodiment of the present invention, the text in the video frame image is recognized using an existing text recognition algorithm to obtain text keyword information; the speech in the video frame sequence is recognized using an existing speech recognition algorithm to obtain text keyword information corresponding to the speech.

[0043] Generate smart index: Generate thumbnails for the selected index frame images, and establish a mapping relationship between the time when the index frame image appears, the index frame thumbnail image, and the text keyword information, and store the mapping relationship in the index file. An example of each record in the index file is as follows:

[0044] 00:00:00 Index 1 image file address "Index 1 contains text information"

[0045] 00:00:05 Index 2 image file address "Index 2 contains text information"

[0046] 00:01:02 Index 3 image file address

[0047] 00:03:00 Index 4 image file address "Index 4 contains text information"

[0048] …

[0049] The second part Figure 4As shown, video jump playback: when playing a video, the thumbnail and image valid information of the index frame are displayed on the video playback progress bar; the user selects the position he wants to watch based on the thumbnail or image valid information; the video player automatically jumps the video file to the corresponding time point for playback based on the correspondence between the thumbnail or valid information and time.

[0050] According to an embodiment of the present invention, the index file can exist in a flexible format, depending on the needs of video distribution. However, regardless of the format, it must be transmitted synchronously with the video file. The time point is time-calibrated information that strictly matches the timestamp in the video frame image. When watching a video, users can click to jump according to the prompted index information to jump to the time point of interest for viewing. The index information includes text keyword information, voice keyword information, and thumbnail information. For example, if a user enters the text keyword "Chapter 3" in the search box, the player will automatically jump to the time point corresponding to the first appearance of the index frame image containing the text "Chapter 3" based on the mapping relationship and continue playback. If a user enters the voice keyword "Chapter 3" in the search voice box, the player will first perform voice recognition to identify the text keyword information corresponding to the voice keyword, and then automatically jump to the time point corresponding to the first appearance of the index frame image containing the text "Chapter 3" based on the mapping relationship and continue playback. Users can also select any thumbnail from the provided thumbnails. The player will automatically jump to the time point corresponding to the thumbnail based on the mapping relationship between thumbnails and time.

[0051] The present application also provides a computing device, referring to Figure 5 The computing device includes a memory 1120, a processor 1110 and a computer program stored in the memory 1120 and capable of being executed by the processor 1110. The computer program is stored in a space 1130 for program code in the memory 1120. When the computer program is executed by the processor 1110, it is implemented to execute any one of the method steps 1131 according to the present invention.

[0052] The present application also provides a computer-readable storage medium. Figure 6 The computer-readable storage medium comprises a storage unit for program code, the storage unit being provided with a program 1131 ′ for executing the method steps according to the present invention, the program being executed by a processor.

[0053] The present application also provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is caused to execute the steps of the method according to the present invention.

[0054] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer loads and executes the computer program instructions, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0055] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] Those skilled in the art will appreciate that all or part of the steps in the above-described embodiment method can be performed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof.

[0057] Although the present invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the foregoing description, will appreciate that other embodiments are contemplated within the scope of the invention thus described. This disclosure is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. Video file intelligent index generation system based on content analysis, characterized by: include: The index generation module is used to process the video file content frame by frame in real time and automatically generate an index file based on the processing results; It includes image change analysis submodule, image effective information analysis submodule and mapping establishment submodule; wherein, The image change analysis submodule is used to process each frame of video image that changes continuously in the video file, determine whether the pixel data at the same position of the video screen has changed compared to the previous frame, and the current frame whose change ratio exceeds a preset percentage threshold is a changed frame, and if the time interval between the changed frame and the previous index frame is greater than a predetermined time threshold, then the changed frame is used as an index frame; if the time interval between the changed frame and the previous index frame is less than the predetermined time threshold, then the frame whose time interval between the changed frame and the previous index frame is the predetermined time threshold is used as an index frame, thereby obtaining multiple index frames; The image valid information analysis submodule is used to extract valid information from multiple index frames to obtain valid information corresponding to each index frame; The mapping establishment submodule is used to establish a mapping relationship between the index frame, the valid information and the time point corresponding to the index frame, thereby generating an index file; The video playback module is used to jump and play videos according to the index file.

2. The video file intelligent index generation system based on content analysis according to claim 1 is characterized in that: The valid information includes an index frame thumbnail and text information generated according to the index frame.

3. The video file intelligent index generation system based on content analysis according to claim 2 is characterized in that: The process of determining in the image change analysis submodule that pixel data at the same position of the video picture has changed compared to the previous frame, and the current frame whose change ratio exceeds a preset percentage threshold is an index frame includes: Each frame of video image is divided into blocks, and each image sub-block at the corresponding position of the current frame and the previous frame is compared. The changed frame is determined according to the comparison result. The changed frame is the index frame.

4. The video file intelligent index generation system based on content analysis according to claim 3 is characterized in that: The process of determining the changed frame according to the comparison result in the image change analysis submodule includes: The image sub-blocks with the same pixel data at the corresponding position in the current frame and the previous frame are recorded as the same sub-blocks, otherwise they are different sub-blocks; When the percentage of the same sub-blocks in the video frame image exceeds a preset percentage threshold, the current frame is marked as a non-changing frame, otherwise it is marked as a changing frame.

5. The video file intelligent index generation system based on content analysis according to claim 4 is characterized in that: The video playback module includes an index input submodule and a jump playback submodule, wherein the index input submodule is used to input or select valid information; The jump play submodule is used to play the video according to the index frame time point corresponding to the valid information in the index file.

6. A method for generating an intelligent index of video files based on content analysis, characterized in that: The following steps are involved: Step 1: Process the video file content frame by frame in real time and automatically generate an index file based on the processing results; Step 2: Jump to play video according to the index file; It is characterized in that the specific steps of step one include: Processing each frame of a continuously changing video image in a video file, determining that pixel data at the same position of the video image has changed compared to that of the previous frame, and that a current frame whose change ratio exceeds a preset percentage threshold is a changed frame, and if the time interval between the changed frame and the previous index frame is greater than a predetermined time threshold, then taking the changed frame as an index frame; if the time interval between the changed frame and the previous index frame is less than the predetermined time threshold, then taking a frame whose time interval between the changed frame and the previous index frame is the predetermined time threshold as an index frame, thereby obtaining multiple index frames; Extracting valid information from multiple index frames to obtain valid information corresponding to each index frame; establishing a mapping relationship between the index frame, valid information, and the time point corresponding to the index frame, thereby generating an index file; The valid information includes an index frame thumbnail and text information generated according to the index frame.

7. The method for generating an intelligent index of a video file based on content analysis according to claim 6, characterized in that: The process of determining in step 1 that the pixel data of the video image at the same position has changed compared to the previous frame, and the current frame whose change ratio exceeds a preset percentage threshold is the index frame includes: Each frame of video image is divided into blocks, and each image sub-block at the corresponding position of the current frame and the previous frame is compared. The changed frame is determined according to the comparison result. The changed frame is the index frame.

8. The method for generating intelligent index of video files based on content analysis according to claim 7, characterized in that: The process of determining the changed frame based on the comparison results in step 1 includes: The image sub-blocks with the same pixel data at the corresponding position in the current frame and the previous frame are recorded as the same sub-blocks, otherwise they are different sub-blocks; When the percentage of the same sub-blocks in the video image exceeds a preset percentage threshold, the current frame is marked as a non-changing frame, otherwise it is marked as a changing frame.

9. The method for generating intelligent index of video files based on content analysis according to claim 8, characterized in that: The specific steps of step 2 include: After the user enters or selects valid information, the video is played according to the index frame time point corresponding to the valid information in the index file.

Images