AI-based video note generation method and apparatus

Abstract

The application provides an AI-based video note generation method and device. It relates to the technical field of computers. The method comprises: generating a note text, a text tree, a voice tree and an image tree based on a to-be-processed video; matching the third abstract with the second abstract and the first abstract to determine one or more matching abstracts corresponding to each third abstract; creating an index of a matching key time node and a key text string corresponding to the third abstract corresponding to the matching key time node, and displaying the key text string as a target state in a note display interface; in response to a video viewing trigger operation for the target key text string, playing the to-be-processed video from the target matching key time node corresponding to the key text string, and ending at the next matching key time node. In turn, the linkage of notes and videos can be realized, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a video note generation method and device based on AI. BACKGROUND

[0002] At present, when a user watches a video by using a video player and needs to make a note, the user needs to use a note tool to edit the note in the note tool. However, there is software isolation between the video player and the note tool, which causes the user to frequently switch between the video player and the note tool, and the note is inconvenient to use. SUMMARY

[0003] Embodiments of the present application aim to provide a video note generation method and device based on AI to solve the above technical problems.

[0004] In a first aspect, the present application provides a video note generation method based on AI, comprising:

[0005] generating a note text, a text tree, a voice tree and an image tree based on a to-be-processed video, the voice tree comprising a plurality of first key time nodes of the to-be-processed video and a first abstract corresponding to each first key time node, the image tree comprising a plurality of second key time nodes of the to-be-processed video and a second abstract corresponding to each second key time node, and the text tree comprising a plurality of key text strings of the note text and a third abstract corresponding to each key text string;

[0006] matching the third abstract with the second abstract and the first abstract to determine one or more matching abstracts corresponding to each third abstract, the first key time node or the second key time node corresponding to the matching abstract being a matching key time node;

[0007] creating an index of the matching key time node and the key text string corresponding to the third abstract corresponding to the matching key time node, and displaying the key text string as a target state in a note display interface;

[0008] responding to a video viewing trigger operation for a target key text string, playing the to-be-processed video starting from the target matching key time node corresponding to the key text string, and ending at the next matching key time node.

[0009] In an optional implementation, responding to a video viewing trigger operation for a target key text string, playing the to-be-processed video starting from the target matching key time node corresponding to the key text string, and pausing at the next matching key time node, comprises:

[0010] When the target key text string corresponds to multiple target matching time nodes, in response to a video viewing trigger operation for the target key text string, videos corresponding to each target matching key time node are played in sequence, the video corresponding to the target matching key time node starting from the target matching key time node and ending at the next matching key time node.

[0011] In an optional implementation, the generating a speech tree based on the to-be-processed video comprises:

[0012] calculating a speech node score; wherein each speech node is a starting node of a paragraph, and the score of each speech node is composed of three parts; the first part is obtained by weighting the mel-frequency cepstral coefficient feature of the audio through a one-dimensional convolution importance predictor; the second part is an alignment score of the speech-to-text result and the video content; and the third part is a time weight calculated based on a timestamp Gaussian distribution;

[0013] selecting a first key time node with a speech node score exceeding a first threshold value.

[0014] In an optional implementation, the generating an image tree based on the to-be-processed video comprises:

[0015] calculating a visual saliency score; wherein each visual node is a starting node of a video, and the visual saliency score of each visual node is composed of three parts; the first part is a frame saliency score calculated based on a saliency model; the second part is an alignment score of the image and the subtitle; and the third part is a score calculated based on a context similarity;

[0016] selecting a second key time node with a visual node score exceeding a second threshold value.

[0017] In an optional implementation, the generating a text tree based on the to-be-processed video comprises:

[0018] calculating a text keyness score; wherein each text node is a starting node of a text, and the text keyness score of each text node is composed of three parts; the first part is a score calculated based on TF-IDF and position weight; the second part is a score calculated through a cross-modal attention mechanism; and the third part is a score calculated based on graph centrality;

[0019] selecting a key text string with a text node score exceeding a third threshold value.

[0020] In an optional implementation, the method further comprises a hierarchical caching strategy:

[0021] L0 cache: storing original data of a current node and adjacent frames;

[0022] L1 cache: storing visual feature vectors of key frames;

[0023] L2 cache: store the index of voice tree and image tree;

[0024] Preloading timing calculation: the preloading timing is dynamically calculated according to the current time, the next key node time and the network delay.

[0025] In an optional implementation, matching the third abstract with the second abstract and the first abstract, determining one or more matching abstracts corresponding to each of the third abstracts, comprises:

[0026] Based on the cross-modal similarity and the spatio-temporal consistency of the third abstract with the second abstract and the first abstract, determining one or more matching abstracts corresponding to each of the third abstracts.

[0027] In a second aspect, the present application provides an AI-based video note generation device, comprising:

[0028] A generation module is configured to generate a note text, a text tree, a voice tree and an image tree based on a to-be-processed video, the voice tree comprising a plurality of first key time nodes of the to-be-processed video and a first abstract corresponding to each first key time node, the image tree comprising a plurality of second key time nodes of the to-be-processed video and a second abstract corresponding to each second key time node, and the text tree comprising a plurality of key text strings of the note text and a third abstract corresponding to each key text string.

[0029] A matching module is configured to match the third abstract with the second abstract and the first abstract, and determine one or more matching abstracts corresponding to each of the third abstracts, wherein the first key time node or the second key time node corresponding to the matching abstract is a matching key time node.

[0030] A creation module is configured to create an index of the matching key time node and the key text string corresponding to the third abstract corresponding to the matching key time node, and display the key text string as a target state in a note display interface.

[0031] A viewing module is configured to, in response to a video viewing trigger operation for a target key text string, play the to-be-processed video starting from a target matching key time node corresponding to the key text string and ending at a next matching key time node.

[0032] In an optional implementation, the viewing module is specifically configured to:

[0033] When the target key text string corresponds to multiple target matching time nodes, in response to a video viewing trigger operation for the target key text string, videos corresponding to each target matching key time node are played in sequence, the video corresponding to the target matching key time node starting from the target matching key time node and ending at the next matching key time node.

[0034] In an optional implementation, the generating module is specifically configured to:

[0035] Calculate a speech node score; wherein each speech node is a starting node of a paragraph, and the score of each speech node is composed of three parts; the first part is obtained by weighting the mel-frequency cepstral coefficient feature of the audio by a one-dimensional convolution importance predictor; the second part is an alignment score of the speech-to-text result and the video content; and the third part is a time weight calculated based on a timestamp Gaussian distribution;

[0036] Select a first key time node with a speech node score exceeding a first threshold value.

[0037] In an optional implementation, the generating module is specifically configured to:

[0038] Calculate a visual saliency score; wherein each visual node is a starting node of a video, and the visual saliency score of each visual node is composed of three parts; the first part is a frame saliency score calculated based on a saliency model; the second part is an alignment score of the image and the subtitle; and the third part is a score calculated based on context similarity;

[0039] Select a second key time node with a visual node score exceeding a second threshold value.

[0040] In an optional implementation, the generating module is specifically configured to:

[0041] Calculate a text keyness score; wherein each text node is a starting node of a text, and the text keyness score of each text node is composed of three parts; the first part is a score calculated based on TF-IDF and position weight; the second part is a score calculated by a cross-modal attention mechanism; and the third part is a score calculated based on graph centrality;

[0042] Select a key text string with a text node score exceeding a third threshold value.

[0043] In an optional implementation, the hierarchical caching strategy is:

[0044] L0 cache: store the original data of the current node and adjacent frames;

[0045] L1 cache: store the visual feature vector of the key frame;

[0046] L2 cache: store the index of voice tree and image tree;

[0047] Preloading timing calculation: the preloading timing is dynamically calculated according to the current time, the next key node time and the network delay.

[0048] In an optional embodiment, the matching module is configured to:

[0049] Based on the cross-modal similarity and the spatio-temporal consistency of the third abstract, the second abstract and the first abstract, determine one or more matching abstracts corresponding to each third abstract.

[0050] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program executable on the processor, and the processor executes the computer program to realize the method of any one of the preceding embodiments.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to execute the method of any one of the preceding embodiments.

[0052] The present application provides an AI-based video note generation method and device. By generating note text, text tree, voice tree and image tree based on the to-be-processed video, matching the third abstract with the second abstract and the first abstract to determine one or more matching abstracts corresponding to each third abstract, creating an index of the matching key time node and the key text string corresponding to the third abstract, and displaying the key text string as a target state in the note display interface, and in response to the video viewing trigger operation for the target key text string, playing the to-be-processed video from the target matching key time node corresponding to the key text string and ending at the next matching key time node. In turn, the linkage of notes and videos can be realized, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0054] Figure 1 A flowchart of an AI-based video note generation method provided by the embodiments of the present application is shown in the figure.

[0055] Figure 2 A structure schematic diagram of an AI-based video note generation device provided for an embodiment of the present application;

[0056] Figure 3 A structure schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0058] Figure 1 A flowchart of an AI-based video note generation method provided for an embodiment of the present application. As shown in the figure, the method comprises the following steps: Figure 1

[0059] S110, generating a note text, a text tree, a speech tree and an image tree based on a to-be-processed video.

[0060] The speech tree comprises a plurality of first key time nodes of the to-be-processed video and a first abstract corresponding to each first key time node; the image tree comprises a plurality of second key time nodes of the to-be-processed video and a second abstract corresponding to each second key time node; and the text tree comprises a plurality of key text strings of the note text and a third abstract corresponding to each key text string.

[0061] The speech tree can be generated based on the following steps: calculating a speech node score; wherein each speech node is a starting node of a paragraph, and the score of each speech node is composed of three parts; the first part is obtained by weighting the Mel-frequency cepstral coefficient (MFCC) feature vector of the audio through a one-dimensional convolution importance predictor; the second part is the alignment score of the speech-to-text result and the video content; and the third part is the time weight calculated based on the Gaussian distribution of the timestamp; and selecting the first key time node whose speech node score exceeds a first threshold.

[0062] The speech node score can be calculated based on the following formula:

[0063]

[0064] wherein, represents the score of the i th speech node.

[0065] represents the i th element of the Mel-frequency cepstral coefficient (MFCC) feature vector of the audio corresponding to the i th speech node, n is the dimension of the MFCC feature vector, ​​​​MFCC features are features extracted from audio signals that reflect the spectral envelope information of the audio, and are commonly used in speech recognition and audio analysis. These features are typically extracted from audio signals using signal processing techniques.

[0066] Indicates the first The speech-to-text result corresponding to each speech node With video content The cross-modal alignment score between images and text. CLIP (Contrastive Language–Image Pre-training) is a cross-modal alignment model that learns the correspondence between images and text. Here, It is a variant of the CLIP model, specifically designed for speech-image alignment. Indicates the first The result of converting the audio corresponding to each voice node into text.

[0067] For the first The audio corresponding to the voice node is the _th Acoustic feature weights of each element;

[0068] μ and μ represent the cross-modal and temporal distribution weighting coefficients, respectively, used to balance the weights between the speech-to-text result and the video content alignment score and the MFCC feature weighted score. The value of is usually between 0 and 1, and and The sum of is 1.

[0069] This represents the audio or related information corresponding to the i-th speech node. For example, The eigenvector representing the Mel-frequency cepstral coefficients (MFCCs) of the audio corresponding to the i-th speech node. One element; This represents the result of converting the audio corresponding to the i-th speech node into text.

[0070] An image tree can be generated based on the following steps: calculating visual saliency scores; where each visual node is the starting node of a video segment, and the visual saliency score of each visual node consists of three parts: the first part is the frame saliency score calculated based on the saliency model; the second part is the alignment score between the image and the caption; and the third part is the score calculated based on contextual similarity. Visual nodes whose scores exceed a second threshold are selected as the second key time nodes.

[0071] The visual salience score can be calculated based on the following formula:

[0072]

[0073] where, is the visual saliency score, which measures the saliency of a frame in the video.

[0074] α, β, γ are adaptive weights that balance the contributions of different parts in the formula, + + = 1. These weights can be obtained through training to adapt to different video content and caption needs.

[0075] is the saliency score of the i-th frame, reflecting the degree to which the frame visually attracts the attention of the audience. It can be calculated by a saliency detection algorithm.

[0076] is the text-alignment model score, which measures the semantic consistency between the frame and its corresponding caption . The CLIP model is a multi-modal visual and textual learning model that can establish a correspondence between images and texts through training. The score here reflects the matching degree between the frame and the caption.

[0077] is the context similarity score, which measures the content similarity between the frame and its neighboring frames . This helps to identify continuous and relevant scenes in the video. The context similarity can be calculated by comparing the feature vectors between frames.

[0078] The text tree can be generated based on the following steps: calculating the text importance score; where each text node is the starting node of a piece of text, and the text importance score of each text node is composed of three parts; the first part is the score calculated based on TF-IDF and position weight; the second part is the score calculated by the cross-modal attention mechanism; the third part is the score calculated based on graph centrality; selecting the text node whose score exceeds the third threshold as the key text string.

[0079] The text importance score can be calculated based on the following formula:

[0080]

[0081] where, is the text importance score of the word .

[0082] ​​​tf-idf of a word in a document It measures the importance of a word in a document.

[0083] is the position weight, which is calculated according to the position of the word in the text. Specifically, it can be calculated by the following formula:

[0084]

[0085] is the adjustment coefficient, is the mean of the word position and is the standard deviation of the word position, and r is the position of the word in the document.

[0086] is the cross-modal weight, which is an adjustable parameter used to adjust the influence of cross-modal attention on the final score.

[0087] is the cross-modal attention, which can be calculated by the softmax function. Specifically, it can be calculated by the following formula:

[0088]

[0089] where is the query vector of the document word , is the key vector, which can be projected from the multi-modal feature , is the value vector, which is also projected from but uses a different weight matrix than . These vectors are usually obtained through pre-trained models or specific task training; is the dimension of the vector, where the dimension of the query vector, the key vector, and the value vector can all be equal to , for example = 512.

[0090] is the knowledge graph constructed based on the notes, which is a graph structure containing concepts, entities, and their relationships in the notes.

[0091] is the graph centrality, which is used to measure the centrality of the word in the knowledge graph in the document. Common graph centrality measures include degree centrality, closeness centrality, betweenness centrality, etc. These centrality values are usually calculated by graph algorithms.

[0092] S120, matching the third abstracts with the second abstracts and the first abstracts, determining one or more matching abstracts corresponding to each third abstract, and the first key time node or the second key time node corresponding to the matching abstract is a matching key time node.

[0093] In some embodiments, one or more matching abstracts corresponding to each third abstract can be determined based on the cross-modal similarity and the spatio-temporal consistency of the third abstracts with the second abstracts and the first abstracts.

[0094] The cross-modal similarity can be calculated based on the following formula :

[0095]

[0096] The third abstract is a text tree;

[0097] The first abstract is a text (speech) or a second abstract (image);

[0098] Embed is a dedicated embedding model used to convert different types of input (such as text, speech, and image) into embedding vectors; where text uses BERT, speech uses Wav2Vec, and image uses CLIP.

[0099] is the application of the Embed model on text data. For a given text (such as ), BERT (Bidirectional Encoder Representations from Transformers) model or other text embedding techniques are used to convert text into embedding vectors. BERT is a pre-trained deep bidirectional model that can capture the context information of text.

[0100] is the application of the Embed model on modal data (such as speech, image). For a given speech or image (such as ), The corresponding embedding model (such as Wav2Vec for speech and CLIP for image) is used to convert modal data into embedding vectors. Wav2Vec is a pre-trained model for speech recognition that can capture acoustic features of speech; CLIP is a multi-modal visual and textual learning model for image recognition that can map images and text into the same vector space.

[0101] cos represents the cosine similarity function, which is used to calculate the similarity between two embedding vectors. The value range of cosine similarity is [-1, 1], and the larger the value, the more similar the two vectors are.

[0102] wherein the spatiotemporal consistency verification can include a temporal consistency score and a spatial consistency score:

[0103] is the temporal consistency score. This score is used to evaluate whether the time of the appearance of the text key string meets the expectations, which can be determined based on the following formula: wherein e is the base of the natural logarithm. δ is the decay coefficient, which is used to control the punishment degree of the time difference. This parameter is usually determined through experiments or experience, and the specific value needs to be adjusted according to the application scenario. is the time of the appearance of the text key string. This parameter is obtained through video analysis or text recognition technology, indicating the time point of the appearance of the target text in the video. is the reference time point or the expected appearance time. This parameter is set according to the specific application scenario, which may come from user input, preset rules or other information sources.

[0104] is the spatial consistency score. This score is used to evaluate whether the spatial position of the text key string in the video meets the expectations, which can be determined based on the following formula: wherein IoU: Intersection over Union, which is used to measure the overlap degree of two regions. is the bounding box of the spatial position of the text key string in the video. This parameter is obtained through video analysis or text recognition technology, indicating the position of the target text in the video frame. is the reference spatial position or the bounding box of the expected appearance position. This parameter is also set according to the specific application scenario, which may come from user input, preset rules or other information sources.

[0105] In summary, the temporal consistency score is calculated by calculating the time difference between the appearance time of the target text and the reference time point, combined with the decay coefficient δ; the spatial consistency score is obtained by calculating the intersection over union of the bounding box of the target text and the reference bounding box. These parameters together constitute the basis of spatiotemporal consistency verification.

[0106] S130, create an index of the key text string corresponding to the matching key time node and its corresponding third summary, and display the key text string as the target state in the note display interface.

[0107] An index is a data structure that allows for fast lookup and location of information. The purpose of the index is to associate key time nodes with corresponding summary content, and these key text strings (i.e., content related to time nodes and summaries) are presented in a certain target state in the note display interface, enabling efficient retrieval and display.

[0108] A target state refers to the form or style in which information is ultimately presented to the user. For example, in a note interface, key text strings may be highlighted so that the user can notice them at a glance. These may be core information further refined from existing summaries.

[0109] A key text string refers to specific textual content associated with a time node and a summary, which is often the highlight of the document and can help users quickly understand the context.

[0110] A note display interface is the visual area where users view note or document content, often with design elements (such as color, font, etc.) to enhance readability and usability of information.

[0111] In response to a video viewing trigger operation for the target key text string, the video to be processed is played from the target matching key time node corresponding to the key text string and ends at the next matching key time node.

[0112] In some embodiments, when the target key text string corresponds to multiple target matching time nodes, in response to a video viewing trigger operation for the target key text string, the video corresponding to each target matching key time node is played in turn, starting from the target matching key time node and ending at the next matching key time node.

[0113] When the user inputs a viewing operation for the target key text string, the time nodes related to the text string are found, and the video is played from these time nodes until the next matching time node. If the key text string corresponds to multiple time nodes, the video corresponding to each time node is played in turn.

[0114] A video viewing trigger operation can refer to an operation performed by the user, such as clicking a button or entering text, to initiate the positioning and playing of the video.

[0115] After generating these tree structures, the trees can be further integrated to form a comprehensive video note. Specifically, key nodes and their summaries in the speech tree, image tree, and text tree can be cross-referenced and linked to ensure the coherence and completeness of the note content. For example, when the speech summary of a first key time node is highly related in content to the image summary of a second key time node, they can be linked together to form a multimedia note entry.

[0116] The generation efficiency of notes can also be optimized using a hierarchical caching strategy. This strategy stores note content in the cache according to the importance and frequency of use of nodes. For frequently accessed or important nodes, they can be stored in the cache for fast access. For less frequently accessed or less important nodes, they can be stored in the low-speed cache to save storage space.

[0117] In some embodiments, a hierarchical caching strategy is also included:

[0118] L0 cache: stores the original data of the current node and adjacent frames;

[0119] L1 cache: stores the visual feature vector of the key frame;

[0120] L2 cache: stores the index of the speech tree and the image tree;

[0121] Preloading timing calculation: the preloading timing is dynamically calculated according to the current time, the next key node time and the network delay.

[0122] In the matching and display stage of the notes, the matching module can be used to filter out the most relevant information from the generated notes according to the user's needs and preferences. For example, when the user needs to find notes related to a certain topic, the matching module searches the key text string, speech summary and image summary in the notes to find the most matching note entry for the topic and displays it to the user.

[0123] In summary, the present application constructs a speech tree, an image tree and a text tree, and integrates these tree structures to generate comprehensive video notes, providing users with efficient, accurate and multimedia note experience.

[0124] Figure 2 A structure schematic diagram of an AI-based video note generation device provided by an embodiment of the present application. The device can include:

[0125] The generation module 201 is configured to generate note text, a text tree, a speech tree and an image tree based on the to-be-processed video, the speech tree including a plurality of first key time nodes of the to-be-processed video and a first summary corresponding to each first key time node, the image tree including a plurality of second key time nodes of the to-be-processed video and a second summary corresponding to each second key time node, and the text tree including a plurality of key text strings of the note text and a third summary corresponding to each key text string;

[0126] The matching module 202 is configured to match the third abstracts with the second abstracts and the first abstracts, determine one or more matching abstracts corresponding to each third abstract, and determine a matching key time node corresponding to the first key time node or the second key time node of the matching abstracts;

[0127] The creating module 203 is configured to create an index of the key text string corresponding to the matching key time node and the third abstract corresponding to the matching key time node, and display the key text string as a target state in a note display interface.

[0128] The viewing module 204 is configured to, in response to a video viewing trigger operation for the target key text string, play the to-be-processed video starting from the target matching key time node corresponding to the key text string and ending at the next matching key time node.

[0129] In some embodiments, the viewing module 204 is specifically configured to:

[0130] When the target key text string corresponds to a plurality of target matching time nodes, in response to a video viewing trigger operation for the target key text string, the videos corresponding to each target matching key time node are played in sequence, and the video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

[0131] In some embodiments, the generating module 201 is specifically configured to:

[0132] Calculate a speech node score, wherein each speech node is a starting node of a paragraph, and the score of each speech node is composed of three parts; the first part is obtained by weighting the mel-frequency cepstral coefficient feature of the audio by a one-dimensional convolution importance predictor; the second part is an alignment score of the speech-to-text result and the video content; and the third part is a time weight calculated based on a timestamp Gaussian distribution;

[0133] Select a speech node score that exceeds a first threshold value as a first key time node.

[0134] In some embodiments, the generating module 201 is specifically configured to:

[0135] Calculate a visual saliency score, wherein each visual node is a starting node of a video, and the visual saliency score of each visual node is composed of three parts; the first part is a frame saliency score calculated based on a saliency model; the second part is an alignment score of an image and a subtitle; and the third part is a score calculated based on a context similarity;

[0136] Select a visual node score that exceeds a second threshold value as a second key time node.

[0137] In some embodiments, the generating module 201 is specifically configured to:

[0138] calculating a text importance score; wherein each text node is a start node of a piece of text, and the text importance score of each text node is composed of three parts; the first part is a score calculated based on TF-IDF and position weight; the second part is a score calculated through a cross-modal attention mechanism; and the third part is a score calculated based on graph centrality;

[0139] selecting a key text string as a text node score exceeding a third threshold value.

[0140] In some embodiments, the matching module 202 is configured to:

[0141] based on the cross-modal similarity and the spatio-temporal consistency of the third abstract with the second abstract and the first abstract, determining one or more matching abstracts corresponding to each third abstract.

[0142] Referring to Figure 3 The electronic device 300 provided by the embodiments of the present application at least includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the AI-based video note generation method provided by the embodiments of the present application when executing the computer program.

[0143] The electronic device 300 provided by the embodiments of the present application can further include a bus 303 connecting different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0144] The memory 302 can include a readable storage medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023. The memory 302 can also include a program tool 3025 having a set of (at least one) program modules 3024, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of these examples or some combination thereof can include the implementation of a network environment.

[0145] The processor 301 can be one processing element or a collective term for multiple processing elements. For example, the processor 301 can be a central processing unit (CPU), or one or more integrated circuits configured to implement the AI-based video note generation method provided by the embodiments of the present application. Specifically, the processor 301 can be a general-purpose processor, including but not limited to a CPU, an application specific integrated circuit (ASIC), a ready-made programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0146] The electronic device 300 can communicate with one or more external devices 304 (such as a keyboard, a remote controller, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 300 (such as a mobile phone, a computer, etc.), and / or any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 305. In addition, the electronic device 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) through a network adapter 306. As shown, the network adapter 306 communicates with other modules of the electronic device 300 through the bus 303. It should be understood that although Figure 3 the network adapter 306 is shown as a separate component, the network adapter 306 can be an integral part of the bus 303, the processor 301, or the electronic device 300. Figure 3 It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, redundant arrays of independent disks (RAID) subsystems, tape drives, and data backup storage subsystems, etc.

[0147] It should be noted that Figure 3 The electronic device 300 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0148] The computer readable storage medium provided by the embodiments of the present application is introduced as follows. The computer readable storage medium provided by the embodiments of the present application stores computer instructions, and the computer instructions are executed by a processor to implement the AI-based video note generation method provided by the embodiments of the present application. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the AI-based video note generation method provided by the embodiments of the present application by executing the built-in or installed computer instructions.

[0149] In addition, the AI-based video note generation method provided by the embodiments of the present application can also be implemented as a computer program product, which includes program codes that implement the AI-based video note generation method provided by the embodiments of the present application when running on a processor.

[0150] The computer program product provided by the embodiments of the present application can adopt one or more computer readable storage media, and the computer readable storage media can be, but are not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any suitable combination of the above. Specifically, more specific examples (non-exhaustive list) of the computer readable storage media include an electrical connection with one or more conductive wires, a portable disc, a hard disk, a RAM, a ROM, an Erasable Programmable Read Only Memory (EPROM), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0151] The computer program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on an electronic device such as a road management device. However, the computer program product provided by the embodiments of the present application is not limited to this. In the embodiments of the present application, the computer readable storage medium can be any tangible medium containing or storing program codes, which can be used or combined with an instruction execution system, device or component.

[0152] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is only exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by multiple units.

[0153] Furthermore, although the operations of the method(s) herein can be described in a particular, sequential order, this order is not meant to be a limitation and one or more of the operations described can be performed in parallel, or in a different order, including before or after other operations described. The various steps described can be implemented in hardware, software, or a combination thereof. The subject specification can be implemented by computer software implemented by one or more processors of a computing device.

[0154] While the preferred embodiments of the application have been described above, it should be understood that they have been presented by way of example only, and not limitation. Numerous changes to the methods and materials can be made by those skilled in the art without departing from the spirit and scope of the application. Thus, the breadth and scope of the present application should not be limited by any of the above described embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0155] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An AI-based video note generation method, characterized in that, include: The video to be processed generates notes, a text tree, a speech tree, and an image tree. The speech tree includes multiple first key time nodes of the video to be processed and a first summary corresponding to each first key time node. The image tree includes multiple second key time nodes of the video to be processed and a second summary corresponding to each second key time node. The text tree includes multiple key text strings of the notes and a third summary corresponding to each key text string. The third summary is matched with the second summary and the first summary to determine one or more matching summaries corresponding to each third summary, and the first key time node or the second key time node corresponding to the matching summary is the matching key time node; Create an index for the key time node that matches the key text string corresponding to the third summary, and display the key text string as the target state in the note display interface; In response to a video viewing trigger operation targeting a key text string, the video to be processed is played starting from the target matching key time node corresponding to the key text string and ending at the next matching key time node. The process of generating a speech tree based on the video to be processed includes: Calculate the speech node score; where each speech node is the starting node of a segment, and the score of each speech node consists of three parts; the first part is obtained by weighting the audio’s Mel frequency cepstral coefficient features through a one-dimensional convolutional importance predictor; the second part is obtained by aligning the speech-to-text result with the video content; and the third part is the time weight calculated based on the Gaussian distribution of timestamps. Select the voice node whose score exceeds the first threshold as the first key time node.

2. The method according to claim 1, characterized in that, In response to a video viewing trigger operation targeting a key text string, the video to be processed is played starting from the target matching key time node corresponding to the key text string, and paused at the next matching key time node, including: When the target key text string corresponds to multiple target matching time nodes, the video viewing trigger operation for the target key text string is responded to, and the video corresponding to each target matching key time node is played in sequence. The video corresponding to the target matching key time node starts from the target matching key time node and ends at the next matching key time node.

3. The method according to claim 1, characterized in that, The method of generating an image tree based on the video to be processed includes: Calculate the visual saliency score; where each visual node is the starting node of a video segment, and the visual saliency score of each visual node consists of three parts: the first part is the frame saliency score calculated based on the saliency model; the second part is the alignment score between the image and the caption; and the third part is the score calculated based on contextual similarity. Select visual nodes whose scores exceed the second threshold as the second key time nodes.

4. The method according to claim 1, characterized in that, The text tree is generated based on the video to be processed: Calculate the text criticality score; where each text node is the starting node of a text segment, and the text criticality score of each text node consists of three parts: the first part is the score calculated based on TF-IDF and positional weights; the second part is the score calculated through a cross-modal attention mechanism; and the third part is the score calculated based on graph centrality. Text nodes with scores exceeding the third threshold are selected as key text strings.

5. The method according to claim 1, characterized in that, It also includes a tiered caching strategy: L0 cache: Stores the original data of the current node and adjacent frames; L1 buffer: Stores the visual feature vectors of keyframes; L2 cache: Stores indexes for the speech tree and image tree; Preloading timing calculation: The preloading timing is dynamically calculated based on the current time, the time of the next critical node, and network latency.

6. The method according to claim 1, characterized in that, Matching the third digest with the second digest and the first digest to determine one or more matching digests corresponding to each third digest includes: Based on the cross-modal similarity and spatiotemporal consistency between the third summary and the second and first summaries, one or more matching summaries corresponding to each third summary are determined.

7. An AI-based video note-taking device, characterized in that, include: The generation module is used to generate note text, text tree, speech tree and image tree based on the video to be processed. The speech tree includes multiple first key time nodes of the video to be processed and a first summary corresponding to each first key time node. The image tree includes multiple second key time nodes of the video to be processed and a second summary corresponding to each second key time node. The text tree includes multiple key text strings of the note text and a third summary corresponding to each key text string. The matching module is used to match the third summary with the second summary and the first summary to determine one or more matching summaries corresponding to each third summary, wherein the first key time node or the second key time node corresponding to the matching summary is the matching key time node; A creation module is used to create an index of the key text string corresponding to the matching key time node and its corresponding third summary, and to display the key text string as the target state in the note display interface; The viewing module is used to respond to video viewing trigger operations for target key text strings, start playing the video to be processed from the target matching key time node corresponding to the key text string, and end at the next matching key time node. Specifically, the generation module is used to perform the following steps during the process of generating a speech tree based on the video to be processed: Calculate the speech node score; where each speech node is the starting node of a segment, and the score of each speech node consists of three parts; the first part is obtained by weighting the audio’s Mel frequency cepstral coefficient features through a one-dimensional convolutional importance predictor; the second part is obtained by aligning the speech-to-text result with the video content; the third part is the time weight calculated based on the Gaussian distribution of timestamps; select the speech node whose score exceeds the first threshold as the first key time node.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images