Video note generation method and device based on AI

Abstract

The invention provides a video note generation method and device based on AI. Relates to the technical field of computers. The method comprises the steps of generating a note text, a text tree, a voice tree and an image tree based on a to-be-processed video; matching the third abstracts with the second abstract and the first abstract, and determining one or more matched abstracts corresponding to each third abstract; creating an index matched with the key time node and a key text string corresponding to the third abstract corresponding to the key time node, and displaying the key text string in a target state in a note display interface; and responding to a video viewing trigger operation for the target key text string, starting to play the to-be-processed video from the target matching key time node corresponding to the key text string, and ending at the next matching key time node. Linkage of notes and videos can be realized in sequence, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an AI-based video note generation method and device. Background Art

[0002] Currently, when users need to take notes while watching a video using a video player, they need to use a note-taking tool to edit the notes. However, the software isolation between the video player and the note-taking tool requires users to frequently switch between the video player and the note-taking tool, making note-taking inconvenient. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide an AI-based video note generation method and device to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides an AI-based video note generation method, comprising: Generate note text, a text tree, a voice tree, and an image tree based on the video to be processed, wherein the voice tree includes a plurality of first key time nodes of the video to be processed and a first summary corresponding to each first key time node, the image tree includes a plurality of second key time nodes of the video to be processed and a second summary corresponding to each second key time node, and the text tree includes a plurality of key text strings of the note text and a third summary corresponding to each key text string; Matching the third digest with the second digest and the first digest to determine one or more matching digests corresponding to each of the third digests, wherein the first key time node or the second key time node corresponding to the matching digest is a matching key time node; Creating an index of a key text string corresponding to the matching key time node and the corresponding third summary, and displaying the key text string as a target state in the note display interface; In response to a video viewing trigger operation for a target key text string, the video to be processed starts playing from a target matching key time node corresponding to the key text string and ends at a next matching key time node.

[0005] In an optional embodiment, in response to a video viewing trigger operation for a target key text string, playing the to-be-processed video from a target matching key time node corresponding to the key text string and pausing at a next matching key time node includes: When the target key text string corresponds to multiple target matching time nodes, in response to the video viewing trigger operation for the target key text string, the video corresponding to each target matching key time node is played in sequence, and the video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

[0006] In an optional implementation manner, generating a speech tree based on the video to be processed includes: Calculate the speech node score; each speech node is the starting node of a paragraph, and the score of each speech node is composed of three parts: the first part is obtained by weighting the one-dimensional convolution importance predictor and the audio Mel-frequency cepstral coefficient features; the second part is the alignment score of the speech-to-text result and the video content; the third part is the time weight calculated based on the Gaussian distribution of the timestamp; The speech node whose score exceeds the first threshold is selected as the first key time node.

[0007] In an optional implementation, generating an image tree based on the video to be processed includes: Calculate the visual saliency score; each visual node is the starting node of a video, and the visual saliency score of each visual node consists of three parts: the first part is the frame saliency score calculated based on the saliency model; the second part is the alignment score of the image and subtitles; the third part is the score calculated based on contextual similarity; The visual node whose score exceeds the second threshold is selected as the second key time node.

[0008] In an optional embodiment, the text tree is generated based on the video to be processed: Calculate the text criticality score; each text node is the starting node of a text segment, and the text criticality score of each text node consists of three parts: the first part is the score calculated based on TF-IDF and position weight; the second part is the score calculated using the cross-modal attention mechanism; and the third part is the score calculated based on graph centrality. The text nodes having scores exceeding a third threshold are selected as key text strings.

[0009] In an optional implementation, a hierarchical caching strategy is also included: L0 cache: stores the original data of the current node and adjacent frames; L1 cache: stores the visual feature vectors of key frames; L2 cache: stores the indexes of the speech tree and image tree; Preload timing calculation: The preload timing is dynamically calculated based on the current time, the next key node time, and network delay.

[0010] In an optional implementation, matching the third digest with the second digest and the first digest to determine one or more matching digests corresponding to each of the third digests includes: Based on the cross-modal similarity and spatiotemporal consistency between the third abstract and the second abstract and the first abstract, one or more matching abstracts corresponding to each of the third abstracts are determined.

[0011] In a second aspect, the present invention provides an AI-based video note generation device, comprising: a generation module, configured to generate a note text, a text tree, a voice tree, and an image tree based on the video to be processed, wherein the voice tree includes a plurality of first key time nodes of the video to be processed and a first summary corresponding to each first key time node; the image tree includes a plurality of second key time nodes of the video to be processed and a second summary corresponding to each second key time node; and the text tree includes a plurality of key text strings of the note text and a third summary corresponding to each key text string; a matching module, configured to match the third digest with the second digest and the first digest, and determine one or more matching digests corresponding to each of the third digests, wherein the first key time node or the second key time node corresponding to the matching digest is a matching key time node; a creation module, configured to create an index of a key text string corresponding to the matching key time node and the corresponding third summary, and display the key text string as a target state in a note display interface; The viewing module is used to respond to the video viewing trigger operation for the target key text string, start playing the to-be-processed video from the target matching key time node corresponding to the key text string, and end at the next matching key time node.

[0012] In an optional embodiment, the viewing module is specifically configured to: When the target key text string corresponds to multiple target matching time nodes, in response to the video viewing trigger operation for the target key text string, the video corresponding to each target matching key time node is played in sequence, and the video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

[0013] In an optional embodiment, the generating module is specifically configured to: Calculate the speech node score; each speech node is the starting node of a paragraph, and the score of each speech node is composed of three parts: the first part is obtained by weighting the one-dimensional convolution importance predictor and the audio Mel-frequency cepstral coefficient features; the second part is the alignment score of the speech-to-text result and the video content; the third part is the time weight calculated based on the Gaussian distribution of the timestamp; The speech node whose score exceeds the first threshold is selected as the first key time node.

[0014] In an optional embodiment, the generating module is specifically configured to: Calculate the visual saliency score; each visual node is the starting node of a video, and the visual saliency score of each visual node consists of three parts: the first part is the frame saliency score calculated based on the saliency model; the second part is the alignment score of the image and subtitles; the third part is the score calculated based on contextual similarity; The visual node whose score exceeds the second threshold is selected as the second key time node.

[0015] In an optional embodiment, the generating module is specifically configured to: Calculate the text criticality score; each text node is the starting node of a text segment, and the text criticality score of each text node consists of three parts: the first part is the score calculated based on TF-IDF and position weight; the second part is the score calculated using the cross-modal attention mechanism; and the third part is the score calculated based on graph centrality. The text nodes having scores exceeding a third threshold are selected as key text strings.

[0016] In an optional embodiment, a tiered caching strategy: L0 cache: stores the original data of the current node and adjacent frames; L1 cache: stores the visual feature vectors of key frames; L2 cache: stores the indexes of the speech tree and image tree; Preload timing calculation: The preload timing is dynamically calculated based on the current time, the next key node time, and network delay.

[0017] In an optional embodiment, the matching module is used to: Based on the cross-modal similarity and spatiotemporal consistency between the third abstract and the second abstract and the first abstract, one or more matching abstracts corresponding to each of the third abstracts are determined.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the method described in any one of the aforementioned embodiments is implemented.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method described in any one of the aforementioned embodiments.

[0020] The present invention provides an AI-based video note generation method and device. By generating note text, text tree, voice tree and image tree based on the video to be processed; matching the third summary with the second summary and the first summary, determining one or more matching summaries corresponding to each third summary; creating an index of the key text string corresponding to the matching key time node and its corresponding third summary, and displaying the key text string as the target state in the note display interface; responding to the video viewing trigger operation for the target key text string, playing the video to be processed from the target matching key time node corresponding to the key text string, and ending at the next matching key time node. The linkage between notes and videos can be realized in sequence, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an AI-based video note generation method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an AI-based video note generation device provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0024] Figure 1 A flow chart of a method for generating video notes based on AI provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps: S110, generating note text, text tree, voice tree and image tree based on the video to be processed.

[0025] Among them, the voice tree includes multiple first key time nodes of the video to be processed and a first summary corresponding to each first key time node; the image tree includes multiple second key time nodes of the video to be processed and a second summary corresponding to each second key time node; the text tree includes multiple key text strings of the note text and a third summary corresponding to each key text string.

[0026] A speech tree can be generated based on the following steps: calculating the speech node score; wherein each speech node is the starting node of a paragraph, and the score of each speech node consists of three parts; the first part is obtained by weighting the one-dimensional convolution importance predictor and the Mel-frequency cepstral coefficient feature of the audio; the second part is the alignment score of the speech-to-text result and the video content; the third part is the time weight calculated based on the Gaussian distribution of the timestamp; select the speech node with a score exceeding the first threshold as the first key time node.

[0027] The speech node score can be calculated based on the following formula:

[0028] in, Indicates the The score of the speech node.

[0029] Indicates the The Mel-Frequency Cepstral Coefficients (MFCC) feature vector of the audio corresponding to the speech node elements, n is the dimension of the MFCC feature vector, MFCC features are a type of feature extracted from audio signals that can reflect the spectral envelope information of the audio and are commonly used in speech recognition and audio analysis. These features are usually extracted from audio signals through signal processing technology; Indicates the The speech-to-text result corresponding to the speech node With video content The cross-modal alignment score between . CLIP (Contrastive Language–Image Pre-training) is a cross-modal alignment model that can learn the correspondence between images and text. Here, It is a variant of the CLIP model, specifically for speech-image alignment. Indicates that the The result of converting the audio corresponding to the speech node into text.

[0030] For the The audio corresponding to the speech node The acoustic feature weight of each element; and μ represent the cross-modal and time distribution weight coefficients, respectively, which are used to balance the weights between the speech-to-text result and the video content alignment score and the MFCC feature weighted score. The value of is usually between 0 and 1, and and The sum of is 1.

[0031] It represents the audio or related information of the audio corresponding to the i-th speech node. For example, The first Mel-frequency cepstral coefficient (MFCC) feature vector of the audio corresponding to the i-th speech node elements; Indicates the result of converting the audio corresponding to the i-th speech node into text.

[0032] An image tree can be generated based on the following steps: calculating a visual saliency score; wherein each visual node is the starting node of a video, and the visual saliency score of each visual node consists of three parts; the first part is a frame saliency score calculated based on a saliency model; the second part is an alignment score between the image and the subtitle; the third part is a score calculated based on contextual similarity; and selecting a visual node with a score exceeding a second threshold as the second key time node.

[0033] The visual saliency score can be calculated based on the following formula:

[0034] in, is the visual saliency score, which is used to measure the visual saliency of a frame in a video. significance.

[0035] α, β, and γ are adaptive weights used to balance the contributions of different parts in the formula. + + = 1. These weights can be trained to adapt to different video content and subtitle requirements.

[0036] For the The saliency score of a frame reflects the degree to which the frame visually attracts the viewer's attention. It can be calculated using a saliency detection algorithm.

[0037] is the image-text alignment model score, used to measure the frame Its corresponding subtitles The CLIP model is a multimodal visual and text learning model that can establish correspondence between images and text through training. The score reflects the degree of match between the frame and the subtitle.

[0038] is the context similarity score used to measure the frame Its adjacent frames This helps identify consecutive and related scenes in a video. Contextual similarity can be calculated by comparing feature vectors between frames.

[0039] A text tree can be generated based on the following steps: calculating a text criticality score; wherein each text node is the starting node of a text segment, and the text criticality score of each text node consists of three parts; the first part is a score calculated based on TF-IDF and position weight; the second part is a score calculated by a cross-modal attention mechanism; the third part is a score calculated based on graph centrality; and selecting text nodes whose scores exceed the third threshold as key text strings.

[0040] The text criticality score can be calculated based on the following formula:

[0041] in, For words The text criticality score.

[0042] Represents words in the text The term frequency is the inverse text frequency. It measures the importance of a word in the text.

[0043] is the position weight, according to the word The position in the text can be calculated using the following formula:

[0044] Should is the adjustment coefficient, is the mean and is the standard deviation of word positions, r is the The position in the document.

[0045] is the cross-modal weight, which is an adjustable parameter used to adjust the impact of cross-modal attention on the final score.

[0046] For cross-modal attention, it can be calculated by the softmax function, which can be calculated by the following formula:

[0047] in is a document word The query vector, is the key vector, which can be obtained from the multimodal features Projection obtained, is a value vector, also from Projection, but with Use different weight matrices. These vectors are usually obtained through pre-trained models or task-specific training; Is the dimension of the vector, where the query vector, key vector and vector dimensions can all be equal to ,For example =512.

[0048] A knowledge graph built based on notes is a graph structure that contains concepts, entities and their relationships in notes.

[0049] is the graph centrality, which is used to measure the words in the document In the knowledge graph The degree of centrality in a graph. Common graph centrality measures include degree centrality, closeness centrality, betweenness centrality, etc. These centrality values ​​are usually calculated using graph algorithms.

[0050] S120 : Match the third digest with the second digest and the first digest to determine one or more matching digests corresponding to each third digest, and use the first key time node or the second key time node corresponding to the matching digest as the matching key time node.

[0051] In some embodiments, one or more matching abstracts corresponding to each third abstract may be determined based on the cross-modal similarity and spatiotemporal consistency between the third abstract and the second abstract and the first abstract.

[0052] The cross-modal similarity can be calculated based on the following formula :

[0053] is the third summary of the text tree; is the first abstract (speech) or the second abstract (image); Embed is a dedicated embedding model used to convert different types of input (such as text, speech, and images) into embedding vectors; BERT is used for text, Wav2Vec is used for speech, and CLIP is used for images.

[0054] It is the application of Embed model on text data. For a given text (such as ), Use the BERT (Bidirectional Encoder Representations from Transformers) model or other text embedding techniques to convert text into embedding vectors. BERT is a pre-trained deep bidirectional model that captures the context of text.

[0055] It is the application of Embed model on modal data (such as speech, image). For a given speech or image (such as ), Convert the modal data into embedding vectors using the corresponding embedding model (such as Wav2Vec for speech and CLIP for images). Wav2Vec is a pre-trained model for speech recognition that captures the acoustic characteristics of speech. CLIP is a multimodal vision and text learning model for image recognition that maps images and text into the same vector space.

[0056] cos represents the cosine similarity function, which is used to calculate the similarity between two embedding vectors. The cosine similarity range is [-1, 1], and larger values ​​indicate more similarity between the two vectors.

[0057] Among them, spatiotemporal consistency verification can include temporal consistency score and spatial consistency score: The temporal consistency score is used to evaluate whether the timing of the text key string appears as expected. It can be determined based on the following formula: Where e is the base of the natural logarithm. δ is the attenuation coefficient, which controls the penalty for time lag. This parameter is typically determined through experimentation or experience, and the specific value needs to be adjusted based on the application scenario. The time when the text keyword appears. This parameter is obtained through video analysis or text recognition technology and indicates the time when the target text appears in the video. The reference time point or expected occurrence time. This parameter is set according to the specific application scenario and may be derived from user input, preset rules, or other information sources.

[0058] is the spatial consistency score. This score is used to evaluate whether the spatial position of the text key string in the video is consistent with the expectation. It can be determined based on the following formula: Among them, IoU: Intersection over Union, is used to measure the degree of overlap between two regions. The bounding box of the spatial position of the text key string in the video. This parameter is obtained through video analysis or text recognition technology and represents the position of the target text in the video frame. The bounding box of the reference spatial location or expected location. This parameter is also set according to the specific application scenario and may come from user input, preset rules, or other information sources.

[0059] In summary, the temporal consistency score is calculated by calculating the time difference between the target text's appearance time and the reference time point, combined with the decay coefficient δ. The spatial consistency score is obtained by calculating the intersection-over-union ratio of the target text's bounding box and the reference bounding box. These parameters together form the basis for spatiotemporal consistency verification.

[0060] S130: Create an index of a key text string that matches the key time node and its corresponding third summary, and display the key text string as a target state in the note display interface.

[0061] An index is a data structure used to quickly find and locate information. It associates key time points with corresponding summary content. These key text strings (i.e., content related to the time points and summaries) are displayed in a specific target state in the note display interface, enabling efficient retrieval and display.

[0062] The target state refers to the form or format in which information is ultimately presented to the user. For example, in a note-taking interface, key text strings might be highlighted so that users can immediately notice them. This might be the core information further refined from an existing summary. Key text strings refer to specific text content associated with time nodes and summaries. These contents are usually the key parts of a document and can help users quickly understand the context. The note display interface is the visual area where users view the contents of notes or documents. It usually uses design (such as color, font, etc.) to enhance the readability and usability of information.

[0063] S140 , in response to a video viewing trigger operation for a target key text string, playing the video to be processed starts from a target matching key time node corresponding to the key text string and ends at a next matching key time node.

[0064] In some embodiments, when the target key text string corresponds to multiple target matching time nodes, in response to the video viewing trigger operation for the target key text string, the video corresponding to each target matching key time node is played in sequence, and the video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

[0065] When the user enters a view operation for the target key text string, the time nodes related to the text string will be found and the video will be played from these time nodes until the next matching time node. If the key text string corresponds to multiple time nodes, the clips corresponding to each time node will be played in sequence.

[0066] A video view trigger action can be an action performed by the user, such as clicking a button or entering text, that initiates the seek and play functionality of a video.

[0067] After generating these tree structures, they can be further integrated to form a comprehensive video note. Specifically, key nodes and their summaries in the audio, image, and text trees can be cross-referenced and linked to ensure the coherence and integrity of the note content. For example, if the audio summary of a first key time node and the image summary of a second key time node are highly relevant in content, they can be linked together to form a multimedia note entry.

[0068] You can also optimize note generation efficiency by utilizing a tiered caching strategy. This strategy stores note content in a tiered cache based on the importance and frequency of use of the node. Frequently accessed or highly important nodes can be stored in a high-speed cache for fast access. Infrequently accessed or less important nodes can be stored in a low-speed cache to conserve storage space.

[0069] In some embodiments, a tiered caching strategy is also included: L0 cache: stores the original data of the current node and adjacent frames; L1 cache: stores the visual feature vectors of key frames; L2 cache: stores the indexes of the speech tree and image tree; Preload timing calculation: The preload timing is dynamically calculated based on the current time, the next key node time, and network delay.

[0070] During the note matching and display phase, the matching module can be used to filter the most relevant information from the generated notes based on the user's needs and preferences. For example, when a user needs to find notes related to a specific topic, the matching module searches the key text strings, voice excerpts, and image excerpts in the notes to find the note entries that best match the topic and display them to the user.

[0071] In summary, the present invention generates comprehensive video notes by constructing a voice tree, an image tree, and a text tree, and integrating these tree structures, thereby providing users with an efficient, accurate, and multimedia note-taking experience.

[0072] Figure 2This is a schematic diagram of the structure of an AI-based video note generation device provided in an embodiment of the present application. The device may include: A generation module 201 is configured to generate a note text, a text tree, a voice tree, and an image tree based on the video to be processed, wherein the voice tree includes a plurality of first key time nodes of the video to be processed and a first summary corresponding to each first key time node, the image tree includes a plurality of second key time nodes of the video to be processed and a second summary corresponding to each second key time node, and the text tree includes a plurality of key text strings of the note text and a third summary corresponding to each key text string; A matching module 202 is configured to match the third digest with the second digest and the first digest, determine one or more matching digests corresponding to each third digest, and use the first key time node or the second key time node corresponding to the matching digest as the matching key time node; A creation module 203 is configured to create an index of a key text string corresponding to the key time node and the third summary thereof, and display the key text string as a target state in the note display interface; The viewing module 204 is configured to respond to a video viewing trigger operation for a target key text string, start playing the video to be processed from a target matching key time node corresponding to the key text string, and end at the next matching key time node.

[0073] In some embodiments, the viewing module 204 is specifically configured to: When the target key text string corresponds to multiple target matching time nodes, in response to the video viewing trigger operation for the target key text string, the video corresponding to each target matching key time node is played in sequence, and the video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

[0074] In some embodiments, the generating module 201 is specifically configured to: Calculate the speech node score; each speech node is the starting node of a paragraph, and the score of each speech node is composed of three parts: the first part is obtained by weighting the one-dimensional convolution importance predictor and the audio Mel-frequency cepstral coefficient features; the second part is the alignment score of the speech-to-text result and the video content; the third part is the time weight calculated based on the Gaussian distribution of the timestamp; The speech node whose score exceeds the first threshold is selected as the first key time node.

[0075] In some embodiments, the generating module 201 is specifically configured to: Calculate the visual saliency score; each visual node is the starting node of a video, and the visual saliency score of each visual node consists of three parts: the first part is the frame saliency score calculated based on the saliency model; the second part is the alignment score of the image and subtitles; the third part is the score calculated based on contextual similarity; The visual node whose score exceeds the second threshold is selected as the second key time node.

[0076] In some embodiments, the generating module 201 is specifically configured to: Calculate the text criticality score; each text node is the starting node of a text segment, and the text criticality score of each text node consists of three parts: the first part is the score calculated based on TF-IDF and position weight; the second part is the score calculated using the cross-modal attention mechanism; and the third part is the score calculated based on graph centrality. The text nodes having scores exceeding a third threshold are selected as key text strings.

[0077] In some embodiments, the matching module 202 is configured to: Based on the cross-modal similarity and spatiotemporal consistency between the third abstract and the second abstract and the first abstract, one or more matching abstracts corresponding to each third abstract are determined.

[0078] See Figure 3 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the AI-based video note generation method provided in the embodiment of the present application is implemented.

[0079] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0080] Memory 302 may include a readable storage medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023. Memory 302 may also include program tools 3025 having a set (at least one) of program modules 3024. Program modules 3024 include, but are not limited to, an operating subsystem, one or more application programs, other program modules, and program data. Each of these examples, or some combination thereof, may include an implementation of a network environment.

[0081] The processor 301 may be a single processing element or a collective term for multiple processing elements. For example, the processor 301 may be a central processing unit (CPU) or one or more integrated circuits configured to implement the AI-based video note generation method provided in the embodiments of the present application. Specifically, the processor 301 may be a general-purpose processor, including but not limited to a CPU, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0082] The electronic device 300 can communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phone, computer, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). Such communication can be performed through an input / output (I / O) interface 305. In addition, the electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 306. Figure 3 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk array (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0083] It should be noted that Figure 3 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0084] The following is an introduction to the computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium provided in an embodiment of the present application stores computer instructions, and when the computer instructions are executed by a processor, the AI-based video note generation method provided in an embodiment of the present application is implemented. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the AI-based video note generation method provided in an embodiment of the present application by executing the built-in or installed computer instructions.

[0085] In addition, the AI-based video note generation method provided in the embodiment of the present application can also be implemented as a computer program product, which includes program code. When the program code runs on a processor, it implements the AI-based video note generation method provided in the embodiment of the present application.

[0086] The computer program product provided in the embodiments of the present application may adopt one or more computer-readable storage media, and the computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. Specifically, more specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0087] The computer program product provided in the embodiments of the present application may be a CD-ROM and include program code, and may also be run on electronic devices such as road management equipment. However, the computer program product provided in the embodiments of the present application is not limited thereto. In the embodiments of the present application, the computer-readable storage medium may be any tangible medium containing or storing program code, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0088] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0089] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0090] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0091] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include such modifications and variations.

Claims

1. A video note generation method based on AI, characterized in that: include: Generate note text, a text tree, a voice tree, and an image tree based on the video to be processed, wherein the voice tree includes a plurality of first key time nodes of the video to be processed and a first summary corresponding to each first key time node, the image tree includes a plurality of second key time nodes of the video to be processed and a second summary corresponding to each second key time node, and the text tree includes a plurality of key text strings of the note text and a third summary corresponding to each key text string; Matching the third digest with the second digest and the first digest to determine one or more matching digests corresponding to each of the third digests, wherein the first key time node or the second key time node corresponding to the matching digest is a matching key time node; Creating an index of a key text string corresponding to the matching key time node and the corresponding third summary, and displaying the key text string as a target state in the note display interface; In response to a video viewing trigger operation for a target key text string, the video to be processed starts playing from a target matching key time node corresponding to the key text string and ends at a next matching key time node.

2. The method according to claim 1, characterized in that In response to a video viewing trigger operation for a target key text string, playing the to-be-processed video from a target matching key time node corresponding to the key text string and pausing at a next matching key time node includes: When the target key text string corresponds to multiple target matching time nodes, in response to the video viewing trigger operation for the target key text string, the video corresponding to each target matching key time node is played in sequence, and the video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

3. The method according to claim 1, characterized in that Generating a speech tree based on the video to be processed includes: Calculate the speech node score; each speech node is the starting node of a paragraph, and the score of each speech node is composed of three parts: the first part is obtained by weighting the one-dimensional convolution importance predictor and the audio Mel-frequency cepstral coefficient features; the second part is the alignment score of the speech-to-text result and the video content; the third part is the time weight calculated based on the Gaussian distribution of the timestamp; The speech node whose score exceeds the first threshold is selected as the first key time node.

4. The method according to claim 1, wherein Generating an image tree based on the video to be processed includes: Calculate the visual saliency score; each visual node is the starting node of a video, and the visual saliency score of each visual node consists of three parts: the first part is the frame saliency score calculated based on the saliency model; the second part is the alignment score of the image and subtitles; the third part is the score calculated based on contextual similarity; The visual node whose score exceeds the second threshold is selected as the second key time node.

5. The method according to claim 1, wherein The text tree is generated based on the video to be processed: Calculate the text criticality score; each text node is the starting node of a text segment, and the text criticality score of each text node consists of three parts: the first part is the score calculated based on TF-IDF and position weight; the second part is the score calculated using the cross-modal attention mechanism; and the third part is the score calculated based on graph centrality. The text nodes having scores exceeding a third threshold are selected as key text strings.

6. The method according to claim 1, wherein Also includes a tiered caching strategy: L0 cache: stores the original data of the current node and adjacent frames; L1 cache: stores the visual feature vectors of key frames; L2 cache: stores the indexes of the speech tree and image tree; Preload timing calculation: The preload timing is dynamically calculated based on the current time, the next key node time, and network delay.

7. The method according to claim 1, characterized in that Matching the third digest with the second digest and the first digest to determine one or more matching digests corresponding to each of the third digests includes: Based on the cross-modal similarity and spatiotemporal consistency between the third abstract and the second abstract and the first abstract, one or more matching abstracts corresponding to each of the third abstracts are determined.

8. An AI-based video note generation device, characterized in that: include: a generation module, configured to generate a note text, a text tree, a voice tree, and an image tree based on the video to be processed, wherein the voice tree includes a plurality of first key time nodes of the video to be processed and a first summary corresponding to each first key time node; the image tree includes a plurality of second key time nodes of the video to be processed and a second summary corresponding to each second key time node; and the text tree includes a plurality of key text strings of the note text and a third summary corresponding to each key text string; a matching module, configured to match the third digest with the second digest and the first digest, and determine one or more matching digests corresponding to each of the third digests, wherein the first key time node or the second key time node corresponding to the matching digest is a matching key time node; a creation module, configured to create an index of a key text string corresponding to the matching key time node and the corresponding third summary, and display the key text string as a target state in a note display interface; The viewing module is used to respond to the video viewing trigger operation for the target key text string, start playing the to-be-processed video from the target matching key time node corresponding to the key text string, and end at the next matching key time node.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.

Images