Live video key point marking method and device, equipment and storage medium
By acquiring and integrating audio, video and interactive data of live videos, the event detection model is used to automatically identify and mark key events in live broadcasts, solving the problem that viewers find it difficult to capture important information, and improving the accuracy and user experience of event detection.
Patent Information
- Application Number
- CN202411070781.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
During the live broadcast, it is difficult for viewers to concentrate, making it difficult to capture and review important knowledge points or key events. The existing recording methods require manual search and labor-consuming, and users are eager to quickly obtain key information.
By obtaining the audio data, video data and interactive data of live videos, feature extraction and fusion are performed, key events are identified using event detection models, and their importance is screened to achieve automatic marking.
It improves the accuracy and reliability of identifying key events in live videos, reduces the time and labor for users to manually mark, and meets the need to quickly obtain key information.
Smart Images

Figure CN119071520A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video live broadcast technology, and in particular to a method, device, equipment and storage medium for marking key points of a live video. Background Art
[0002] With the rapid development of Internet technology, online education and live broadcast platforms are becoming more and more popular, especially in the field of education, where online live courses have become one of the mainstream teaching methods. However, during the live broadcast, due to the richness and real-time nature of the content, it is often difficult for viewers to focus on all the content, especially during long live broadcasts, and it is easy to miss important knowledge points or key events. Therefore, how to help viewers better understand and review the live broadcast content has become an urgent problem to be solved.
[0003] Traditional recording methods often record the entire live broadcast, recording both useful information and unnecessary content. Afterwards, if users need to review specific content, they must manually search and mark it, which is time-consuming and labor-intensive. Moreover, with the continuous increase in live broadcast content, users' need to quickly obtain key information is becoming increasingly urgent.
[0004] In summary, the problems existing in the prior art need to be solved urgently. Summary of the invention
[0005] The present invention provides a method, device, equipment and storage medium for marking key points of live video, so as to solve the defects in the prior art and realize automatic recognition and marking of key events in live video.
[0006] The present invention provides a method for marking key points of a live video, comprising:
[0007] Obtain audio data, video data and interactive data of live video;
[0008] Extracting features from the audio data, the video data, and the interactive data to obtain audio data features, video data features, and interactive data features;
[0009] Performing feature fusion on the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature;
[0010] Inputting the multimodal fusion features into an event detection model to obtain key event information;
[0011] The key event information is screened according to importance to obtain target key events.
[0012] According to a method for marking key points of a live video provided by the present invention, after the step of obtaining audio data, video data and interactive data of the live video, the method further includes:
[0013] The audio data, the video data and the interactive data are preprocessed, wherein the preprocessing includes denoising, performing speech recognition on the audio data, adjusting the frame rate of the video data, and performing data statistics on the interactive data.
[0014] According to a method for marking key points of a live video provided by the present invention, the audio data includes the host audio data and the audience audio data, and the video data includes the host video data and the audience video data;
[0015] The host audio data is obtained through a microphone arranged on the host terminal;
[0016] The audience audio data is obtained through a microphone arranged at the audience terminal;
[0017] The anchor video data is obtained through a camera set on the anchor terminal;
[0018] The audience video data is obtained through a camera arranged at the audience terminal.
[0019] According to a method for marking key points of a live video provided by the present invention, the audio data features include voice features, emotion features and keyword features, the video data features include facial expression features, body language features and scene features, and the interactive data features include text features and behavior features.
[0020] According to a method for marking key points of a live video provided by the present invention, the step of fusing the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature specifically includes:
[0021] Aligning the timestamps of the audio data features, the video data features, and the interactive data features;
[0022] The aligned audio data features, video data features and interactive data features are fused to obtain multimodal fusion features.
[0023] According to a method for marking key points of a live video provided by the present invention, the step of screening the key event information by importance to obtain the target key event specifically includes:
[0024] Performing cluster analysis on the key event information to obtain candidate key events;
[0025] The candidate key events are screened by dimensionality reduction to obtain target key events.
[0026] According to a method for marking key points of a live video provided by the present invention, the step of screening the key event information by importance to obtain the target key event specifically includes:
[0027] Scoring the key event information based on preset feature weights to obtain an importance score corresponding to the key event information;
[0028] When the importance score is greater than a preset threshold, the corresponding key event information is used as a target key event.
[0029] The present invention also provides a key point marking device for live video, comprising:
[0030] A data acquisition module, used to acquire audio data, video data and interactive data of live video;
[0031] A feature extraction module, used to extract features from the audio data, the video data and the interactive data to obtain audio data features, video data features and interactive data features;
[0032] A feature fusion module, used for fusing the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature;
[0033] An event detection module, used for inputting the multimodal fusion features into an event detection model to obtain key event information;
[0034] The event screening module is used to screen the key event information according to importance to obtain target key events.
[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for marking key points of a live video as described in any one of the above is implemented.
[0036] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for marking key points of a live video as described in any one of the above is implemented.
[0037] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for marking key points of a live video.
[0038] The method, device, equipment and storage medium for marking key points of live video provided by the present invention obtain audio data, video data and interactive data of the live video; extract features of the audio data, video data and interactive data to obtain audio data features, video data features and interactive data features; fuse the audio data features, video data features and interactive data features to obtain multimodal fusion features; input the multimodal fusion features into an event detection model to obtain key event information; and perform importance screening on the key event information to obtain target key events. The present invention uses multimodal fusion features to more comprehensively capture key information of different dimensions in the live broadcast process by comprehensively analyzing audio data, video data and interactive data. In this way, key events in live broadcasts can be identified more accurately, improving the accuracy and reliability of event detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 It is a flow chart of the key point marking method of live video provided by the present invention;
[0041] Figure 2 It is a structural schematic diagram of a key point marking device for live video provided by the present invention;
[0042] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] Traditional recording methods often record the entire live broadcast, recording both useful information and unnecessary content. Afterwards, if users need to review specific content, they must manually search and mark it, which is time-consuming and labor-intensive. Moreover, with the continuous increase in live broadcast content, users' need to quickly obtain key information is becoming increasingly urgent.
[0045] In order to solve these problems, some auxiliary tools have emerged in the prior art, such as recording software that provides a manual marking function, which allows users to manually mark the content they think is important during the live broadcast. However, this manual operation still cannot fully meet user needs, especially when the live broadcast stream is long or the user participation is not high, manual marking may not effectively capture all important events.
[0046] In order to solve the problems in the prior art, the present invention proposes a method for marking key points of a live video to realize automatic recognition of video key points. The key point marking of the live video is described below. Figure 1 As shown, including but not limited to the following steps:
[0047] Step 110: Obtain audio data, video data, and interactive data of the live video.
[0048] In step 110, it is necessary to obtain audio data, video data and interactive data of the live video. Specifically, for audio data, the audio data can be obtained in real time through the audio acquisition module of the live platform during the live broadcast process. The audio data usually includes the lecturer's voice, background music and other ambient sounds. For video data, the video data can be collected in real time through the camera equipment or user terminal equipment of the live broadcast platform. The video data includes visual information such as the picture in the live broadcast scene, the lecturer's expression, gestures, scene transitions, etc. For interactive data, the text information posted by the audience in the comment area and the question and answer area during the live broadcast process can be collected in real time. These data are usually obtained through the API interface of the live broadcast platform.
[0049] In the process of obtaining the above-mentioned audio data, video data and interactive data, each data type is attached with timestamp information to ensure the synchronization of multimodal data. These data can be stored in a database or cloud storage system in real time for subsequent processing and analysis. For data with inconsistent formats, necessary format conversions can be performed before storage, such as transcribing audio data into text form, or extracting key frames in video data as static images. Through the above steps, the present invention can obtain comprehensive and synchronized multimodal live broadcast data, providing data support for subsequent feature extraction and key event detection.
[0050] Step 120: extract features from the audio data, the video data and the interactive data to obtain audio data features, video data features and interactive data features.
[0051] For feature extraction of audio data, first, the acquired audio data is preprocessed, including noise reduction, normalization and other operations to improve the quality of the audio data. Then, feature extraction is performed on the preprocessed audio data. The specific steps are as follows:
[0052] Speech feature extraction: Mel frequency cepstral coefficients (MFCC) are used to extract the spectral features of the audio. MFCC can effectively capture the frequency information of speech and is a commonly used feature in speech analysis.
[0053] Emotional feature extraction: By analyzing changes in pitch, speaking rate, and volume, the emotional state of the lecturer in the audio can be identified, such as emotional expressions such as emphasis, excitement, or calmness.
[0054] Keyword detection: Using natural language processing (NLP) technology, perform speech recognition (ASR) on audio data, transcribe the speech content into text, and use keyword extraction algorithms to identify keywords or phrases that the lecturer repeatedly mentions or emphasizes in the speech.
[0055] For feature extraction of video data, the acquired video data is preprocessed, including grayscale, denoising and enhancement of the image, to improve the clarity and contrast of the video image. Feature extraction is performed on the preprocessed video data. The specific steps are as follows:
[0056] Facial expression feature extraction: Use facial expression recognition algorithms to analyze the lecturer's facial expressions and extract emotional features such as anger, joy, and surprise. These expression features can reflect the lecturer's emotional state when explaining certain content.
[0057] Body language feature extraction: Through the pose estimation algorithm, the lecturer's gestures and body movements are identified and body language features are extracted. The lecturer's gestures and body movements can reflect the key points or emphasized parts of the lecture content.
[0058] Scene change detection: Analyze scene changes in the video, such as the switching of slides, changes in scene background, etc. These changes can serve as an important reference for marking key events.
[0059] For feature extraction of interactive data, the audience's interactive data is processed and analyzed to extract the following features:
[0060] Text feature extraction: Use natural language processing technology to perform sentiment analysis on audience comments and questions, extract positive, negative or neutral sentiment features, and extract keywords and topics to identify hot topics that audiences are concerned about.
[0061] Behavioral feature extraction: Count the audience's interaction frequency, including the number of comments, questions, likes, etc., to evaluate the audience's participation. Analyze the audience's viewing time and exit time to extract the audience's attention characteristics.
[0062] Voting and feedback feature extraction: Analyze the audience's voting behavior and feedback, count the concentration of voting results and the satisfaction characteristics of feedback content, and identify the audience's preferences for live content.
[0063] Step 130: Fusing the audio data features, the video data features, and the interactive data features to obtain multimodal fusion features.
[0064] In step 130, before the audio data features, video data features, and interactive data features are subjected to feature fusion, each type of feature is first standardized. Since the acquisition frequency and time dimension of audio, video, and interactive data may be different, these features need to be time-aligned. Before feature fusion, feature selection can be performed on the extracted features to remove redundant or irrelevant features to reduce the impact of noise on the fusion results. Subsequently, the audio data features, video data features, and interactive data features can be directly spliced together to form a high-dimensional feature vector. The features of audio, video, and interactive data can also be processed separately (for example, using independent neural networks or classifiers), and then the respective output results are weighted averaged or cascaded to finally form multimodal fusion features.
[0065] Through the above steps, the audio data features, video data features and interactive data features are effectively fused to obtain comprehensive multimodal fusion features. These fusion features have stronger expression capabilities and can comprehensively reflect the interactive information of multimodal data during the live broadcast process, providing a more accurate and comprehensive basis for subsequent event detection and key content marking.
[0066] Step 140: Input the multimodal fusion features into an event detection model to obtain key event information.
[0067] In step 140, in order to realize event detection of multimodal fusion features, the present invention preferably uses a deep learning model that can process multimodal data, such as a model based on a long short-term memory network (LSTM), a model combining a convolutional neural network (CNN) and a recurrent neural network (RNN), or a model based on a Transformer architecture. These models can effectively capture the temporal and spatial relationships between multimodal features, thereby accurately identifying key events in live broadcasts.
[0068] The multimodal fusion feature vector obtained in step 130 is input into the event detection model. The input feature vector can be a vector of fixed length or a serialized data input, depending on the model structure.
[0069] The event detection model identifies various events in live video based on the input multimodal fusion features. For example, the model can detect the change in the lecturer's tone when emphasizing an important point, the audience's enthusiastic reaction at a certain moment, etc. After identifying the event, the model classifies it into different types of key events. These events may include "key explanation events", "audience strong reaction events", "important content switching events", etc. Different types of event classification can provide a basis for subsequent processing and presentation.
[0070] For the identified key events, the model will annotate them and generate corresponding event information, including the time of the event, event type, event description, etc. This annotated information will be used for subsequent screening and display.
[0071] Through the above steps, the system inputs the multimodal fusion features into the event detection model, which can effectively identify and classify various key event information during the live broadcast. The identified key event information includes event type, occurrence time, related audio, video and interactive data features, etc., which provides a basis for subsequent key event screening and target event display.
[0072] Step 150: Screen the key event information by importance to obtain target key events.
[0073] In step 150, for each key event information output from the event detection model, the system scores the importance according to preset rules or algorithms. These rules may include: output confidence scoring based on the model, the higher the confidence, the higher the importance score of the event. Alternatively, events may be weighted and scored based on reaction indicators such as the number of likes, comments, and votes from the audience in the interactive data. The stronger the audience reaction to the event, the higher its importance score. Alternatively, by analyzing the changes in the lecturer's tone, body language, and expressions in the audio and video data, higher scores may be given to content that the lecturer particularly emphasizes.
[0074] The system sets a series of screening rules based on specific needs to screen out the most valuable target key events. During the screening process, in order to avoid repeated recording of the same event or redundancy caused by similar events at different time points, the system will deduplicate and merge the screened events. Events with high similarity will be merged into one target event, and their frequency or time period information will be retained. For the screened target key events, the system sorts them according to their importance scores. After sorting, the system can generate a list of key events for users to view and use. Finally, the system can provide a user feedback mechanism to allow users to evaluate and adjust the screened target events. User feedback will be used for model iteration and optimization of screening rules to improve the accuracy and applicability of future event screening.
[0075] The method, device, equipment and storage medium for marking key points of live video provided by the present invention obtain audio data, video data and interactive data of the live video; extract features of the audio data, video data and interactive data to obtain audio data features, video data features and interactive data features; fuse the audio data features, video data features and interactive data features to obtain multimodal fusion features; input the multimodal fusion features into an event detection model to obtain key event information; and perform importance screening on the key event information to obtain target key events. The present invention uses multimodal fusion features to more comprehensively capture key information of different dimensions in the live broadcast process by comprehensively analyzing audio data, video data and interactive data. In this way, key events in live broadcasts can be identified more accurately, improving the accuracy and reliability of event detection.
[0076] As a further optional embodiment, after the step of obtaining the audio data, video data and interactive data of the live video, the method further includes:
[0077] The audio data, the video data and the interactive data are preprocessed, wherein the preprocessing includes denoising, performing speech recognition on the audio data, adjusting the frame rate of the video data, and performing data statistics on the interactive data.
[0078] In this embodiment, the collected audio data needs to be denoised to reduce the impact of background noise and ensure the accuracy of subsequent speech analysis. Common denoising methods include frequency domain filtering, time domain filtering, or noise suppression algorithms based on deep learning. Using speech recognition technology, the audio data is converted into text data. This step can extract the text information in the live broadcast, which is convenient for subsequent keyword detection and sentiment analysis. Common speech recognition models include Google Speech-to-Text, Microsoft Azure Speech, etc. The frame rate of the video data is adjusted as needed, and the frame rate of the video stream is unified to a standard frame rate (such as 30fps or 60fps). This helps the subsequent feature extraction and event detection process to ensure the consistency of the video data. Perform preliminary statistics and aggregation on the interactive data, such as counting the number of comments, likes, or votes per minute, and calculating the peak time period of interaction. These statistical results will provide support for subsequent feature extraction and importance analysis.
[0079] As a further optional embodiment, the audio data includes host audio data and audience audio data, and the video data includes host video data and audience video data;
[0080] The host audio data is obtained through a microphone arranged on the host terminal;
[0081] The audience audio data is obtained through a microphone arranged at the audience terminal;
[0082] The anchor video data is obtained through a camera set on the anchor terminal;
[0083] The audience video data is obtained through a camera arranged at the audience terminal.
[0084] In this embodiment, the host audio data is obtained through a microphone set at the host terminal. The microphone should be able to capture clear sound and be configured near the host's explanation position to ensure that the host's voice can be accurately recorded. The audience audio data is obtained through a microphone set at the audience terminal. The audience's microphone should be able to record the audience's voice within a certain distance, including comments, discussions or other interactive sounds. The host video data is obtained through a camera set at the host terminal. The camera should be able to provide high-resolution video and clearly capture the host's facial expressions and body language. The audience video data is obtained through a camera set at the audience terminal. The camera captures the audience's real-time video feedback, including the audience's expressions and reactions.
[0085] As a further optional embodiment, the audio data features include voice features, emotion features and keyword features, the video data features include facial expression features, body language features and scene features, and the interactive data features include text features and behavior features.
[0086] In this embodiment, audio signal processing technology is used to extract speech features, including pitch, volume, speech rate, intonation, etc. These features help analyze the host's voice changes, such as whether a certain point is emphasized. Emotional analysis technology is used to identify emotional states in audio, such as excitement, anger, sadness, etc. This can be achieved by analyzing the changes in tone, speech rate and volume in the speech. Natural language processing (NLP) technology is used to identify and extract keywords or phrases from the audio. These keywords are usually the content that the host repeatedly mentions or emphasizes.
[0087] For video data features, the facial expressions of the host and the audience are analyzed to identify their emotional states and expression changes. For example, smile, frown, surprise, etc. By analyzing the body movements in the video, such as gestures, posture changes, etc., the host's level of involvement or the key points of the content can be judged. The scene information in the video is extracted, including the background environment, perspective changes, etc. These features help to understand the context of the video.
[0088] For interactive data, we analyze the audience's comments, questions, feedback and other text data to extract key information, sentiment and theme features from the text. We extract the audience's behavioral features from the interactive data, such as the number of likes, comments, voting results, etc. These behavioral features reflect the audience's attention and reaction intensity to specific content.
[0089] As a further optional embodiment, the step of fusing the audio data features, the video data features, and the interactive data features to obtain a multimodal fusion feature specifically includes:
[0090] Aligning the timestamps of the audio data features, the video data features, and the interactive data features;
[0091] The aligned audio data features, video data features and interactive data features are fused to obtain multimodal fusion features.
[0092] In this embodiment, when acquiring audio data, video data, and interactive data, a timestamp is assigned to each piece of data. The timestamp records the acquisition time of each piece of data to ensure the time consistency of different data sources. The timestamp of the audio data feature is matched with the timestamp of the video data feature and the interactive data feature to ensure the time consistency of all features. The timestamp of the interactive data feature is aligned with the timestamp of the audio data feature and the video data feature.
[0093] Through the above implementation, the time alignment of audio data features, video data features and interactive data features can be ensured, and comprehensive multimodal fusion features can be obtained through an effective feature fusion method. This method can make full use of information from different data sources and improve the ability to identify key events in live content.
[0094] As a further optional embodiment, the step of screening the key event information by importance to obtain the target key event specifically includes:
[0095] Performing cluster analysis on the key event information to obtain candidate key events;
[0096] The candidate key events are screened by dimensionality reduction to obtain target key events.
[0097] In this embodiment, the key event information obtained from the event detection model includes feature vectors of each event. These feature vectors may include comprehensive information of audio features, video features, and interactive data features. The feature vectors of the key event information are input into a clustering algorithm for clustering analysis. The clustering algorithm groups events into different clusters, each cluster representing a group of similar events. Based on the clustering results, the most representative events are selected from each cluster as candidate key events. These events can be selected using the average feature vector within the cluster or other representative metrics.
[0098] Next, prepare the feature vector set of candidate key events obtained by cluster analysis. These feature vectors need to be processed by dimensionality reduction to reduce the dimension of the data and retain the main information. According to the feature representation after dimensionality reduction, the events are sorted or further analyzed to determine which events have high significance and importance in the principal component space. Those events that stand out in the feature space after dimensionality reduction are selected as target key events. These events are the most likely to have a significant impact on the live broadcast content.
[0099] Through the above implementation, cluster analysis and PCA dimensionality reduction screening of key event information can effectively identify target key events with high importance from numerous events. This method not only reduces the complexity of data, but also improves the accuracy and efficiency of event screening, providing strong support for subsequent key event processing and analysis.
[0100] As a further optional embodiment, the step of screening the key event information by importance to obtain the target key event specifically includes:
[0101] Scoring the key event information based on preset feature weights to obtain an importance score corresponding to the key event information;
[0102] When the importance score is greater than a preset threshold, the corresponding key event information is used as a target key event.
[0103] In this embodiment, it is necessary to define the importance weights of features based on actual application requirements and experience in specific fields. These weights can be determined based on expert knowledge, historical data analysis, or model training results. For example, audio features, video features, and interactive data features can be assigned different weight values. Next, the defined feature weights are applied to the features of each key event. The specific operation is to multiply each feature value by its corresponding weight, and then sum all weighted feature values to obtain the total score of each key event. According to the application scenario and data characteristics, a preset threshold for importance scoring is set. The threshold can be determined by data analysis, experimental testing, or expert advice. When the score of an event is greater than the preset threshold, the event is marked as a target key event. In this way, those events with higher importance and significance can be screened out from a large number of key events.
[0104] Through the above implementation, by scoring the key event information based on the feature weight and comparing it with the preset threshold, it is possible to effectively screen out the target key events with higher importance. This method can make the key events finally marked have higher relevance and practical value, providing strong support for subsequent analysis and application.
[0105] The key point marking device for live video provided by the present invention is described below. Figure 2 As shown, the focus marking device for live video described below and the focus marking method for live video described above can correspond to each other.
[0106] A device for marking key points of a live video, comprising:
[0107] The data acquisition module 210 is used to acquire the audio data, video data and interactive data of the live video;
[0108] A feature extraction module 220, configured to extract features from the audio data, the video data, and the interactive data to obtain audio data features, video data features, and interactive data features;
[0109] A feature fusion module 230, configured to fuse the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature;
[0110] An event detection module 240 is used to input the multimodal fusion features into an event detection model to obtain key event information;
[0111] The event screening module 250 is used to screen the key event information according to importance to obtain target key events.
[0112] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the key marking method of the live video, and the method includes:
[0113] Obtain audio data, video data and interactive data of live video;
[0114] Extracting features from the audio data, the video data, and the interactive data to obtain audio data features, video data features, and interactive data features;
[0115] Performing feature fusion on the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature;
[0116] Inputting the multimodal fusion features into an event detection model to obtain key event information;
[0117] The key event information is screened according to importance to obtain target key events.
[0118] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0119] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the key marking method of the live video provided by the above methods, the method includes:
[0120] Obtain audio data, video data and interactive data of live video;
[0121] Extracting features from the audio data, the video data, and the interactive data to obtain audio data features, video data features, and interactive data features;
[0122] Performing feature fusion on the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature;
[0123] Inputting the multimodal fusion features into an event detection model to obtain key event information;
[0124] The key event information is screened according to importance to obtain target key events.
[0125] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for marking key points of a live video provided by the above methods is implemented, and the method comprises:
[0126] Obtain audio data, video data and interactive data of live video;
[0127] Extracting features from the audio data, the video data, and the interactive data to obtain audio data features, video data features, and interactive data features;
[0128] Performing feature fusion on the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature;
[0129] Inputting the multimodal fusion features into an event detection model to obtain key event information;
[0130] The key event information is screened according to importance to obtain target key events.
[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for marking key points of a live video, characterized in that: include: Obtain audio data, video data and interactive data of live video; Extracting features from the audio data, the video data, and the interactive data to obtain audio data features, video data features, and interactive data features; Performing feature fusion on the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature; Inputting the multimodal fusion features into an event detection model to obtain key event information; The key event information is screened according to importance to obtain target key events.
2. The method for marking key points of a live video according to claim 1, characterized in that: After the step of obtaining the audio data, video data and interactive data of the live video, the method further includes: The audio data, the video data and the interactive data are preprocessed, wherein the preprocessing includes denoising, performing speech recognition on the audio data, adjusting the frame rate of the video data, and performing data statistics on the interactive data.
3. The method for marking key points of a live video according to claim 1, characterized in that: The audio data includes the host audio data and the audience audio data, and the video data includes the host video data and the audience video data; The host audio data is obtained through a microphone arranged on the host terminal; The audience audio data is obtained through a microphone arranged at the audience terminal; The anchor video data is obtained through a camera set on the anchor terminal; The audience video data is obtained through a camera arranged at the audience terminal.
4. The method for marking key points of a live video according to claim 1, characterized in that: The audio data features include voice features, emotion features and keyword features, the video data features include facial expression features, body language features and scene features, and the interactive data features include text features and behavior features.
5. The method for marking key points of a live video according to claim 1, characterized in that: The step of fusing the audio data features, the video data features and the interactive data features to obtain multimodal fusion features specifically includes: Aligning the timestamps of the audio data features, the video data features, and the interactive data features; The aligned audio data features, video data features and interactive data features are fused to obtain multimodal fusion features.
6. The method for marking key points of a live video according to claim 1, characterized in that: The step of screening the key event information by importance to obtain the target key event specifically includes: Performing cluster analysis on the key event information to obtain candidate key events; The candidate key events are screened by dimensionality reduction to obtain target key events.
7. The method for marking key points of a live video according to claim 1, characterized in that: The step of screening the key event information by importance to obtain the target key event specifically includes: Scoring the key event information based on preset feature weights to obtain an importance score corresponding to the key event information; When the importance score is greater than a preset threshold, the corresponding key event information is used as a target key event.
8. A device for marking key points of a live video, characterized in that: include: A data acquisition module, used to acquire audio data, video data and interactive data of live video; A feature extraction module, used to extract features from the audio data, the video data and the interactive data to obtain audio data features, video data features and interactive data features; A feature fusion module, used for fusing the audio data features, the video data features and the interactive data features to obtain a multimodal fusion feature; An event detection module, used for inputting the multimodal fusion features into an event detection model to obtain key event information; The event screening module is used to screen the key event information according to importance to obtain target key events.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for marking key points of a live video as described in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for marking key points of a live video as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Video processing method and device, computer equipment and storage medium
CN114758271A
Live broadcast interaction method and device, computer equipment and storage medium
CN116939230A
Live broadcast monitoring method and device and electronic equipment
CN117097921A
Video summarization method and device, model training method and device and electronic equipment
CN118172712A
Non-operational interactive live broadcast data intelligent analysis system
CN118413708A
Cited By
Live broadcast data processing method and device, and storage medium
CN119996721A
Highlight combination set generation method and system
CN120321474A
A high photosynthetic yield generation method and system
CN120321474B
Stateless broadcasting method and system based on multi-modal time sequence prediction
CN120529260A
Key audio and video extraction method
CN120730094A
