Video processing method, device and equipment and readable storage medium

Abstract

The invention discloses a video processing method, device and equipment and a readable storage medium, and belongs to the technical field of computers. The method comprises the steps of performing abstract extraction on text content of a to-be-processed video to obtain multiple pieces of target abstract content; for any target abstract content, identifying a target text fragment of which the semantic similarity with the target abstract content is higher than a similarity threshold value from the text content, and carrying out semantic integration on the target text fragment and the associated fragment to obtain an integrated text fragment, the associated fragment is at least one of a previous text or a next text of a target text fragment in the text content; screening out a key text fragment from the plurality of integrated text fragments; and based on the timestamp of the key text fragment in the to-be-processed video, editing a video fragment where the key text fragment is located from the to-be-processed video. According to the invention, the video clip where the wonderful content is located can be rapidly obtained, the complexity of video processing is reduced, and the video processing efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video processing method, apparatus, device, and readable storage medium. Background Art

[0002] With the development of computer technology, video has become one of the main ways to transmit information. In some video-based application scenarios, multiple segments can be extracted from the original video to generate videos that can be shared on social media software based on these segments.

[0003] In related technologies, video clip extraction and processing rely on manual operations. Typically, the subject first watches the original video, records the start and end times of each clip, and then extracts each clip from the original video based on these start and end times. This manual process of extracting clips from a video is cumbersome and results in low video processing efficiency. Summary of the Invention

[0004] The present application provides a video processing method, apparatus, device and readable storage medium, which can quickly extract segments containing exciting content from a video, reduce the complexity of video processing, and improve video processing efficiency. The technical solution includes the following contents.

[0005] In one aspect, a video processing method is provided, the method comprising:

[0006] Obtaining a video to be processed and obtaining text content of the video to be processed;

[0007] Summarizing and refining the text content to obtain multiple target summary contents;

[0008] For any target summary content, identifying a target text segment from the text content whose semantic similarity with the target summary content is greater than a similarity threshold, and semantically integrating the target text segment with an associated segment to obtain an integrated text segment, wherein the associated segment is at least one of the preceding and following contexts of the target text segment in the text content;

[0009] Filter out key text fragments from multiple integrated text fragments;

[0010] Based on the timestamp of the key text segment in the video to be processed, a video segment containing the key text segment is clipped from the video to be processed.

[0011] In another aspect, a video processing device is provided, comprising:

[0012] An acquisition module, used to acquire a video to be processed and acquire text content of the video to be processed;

[0013] A refining module, configured to perform summary refining on the text content to obtain a plurality of target summary contents;

[0014] an identification and integration module configured to, for any target summary content, identify, from the text content, a target text segment having a semantic similarity with the target summary content greater than a similarity threshold, and semantically integrate the target text segment with an associated segment to obtain an integrated text segment, wherein the associated segment is at least one of the preceding and following contexts of the target text segment in the text content;

[0015] A screening module, used to screen out key text fragments from multiple integrated text fragments;

[0016] The editing module is used to edit the video segment containing the key text segment from the video to be processed based on the timestamp of the key text segment in the video to be processed.

[0017] In a possible implementation, the text content includes multiple sentence texts;

[0018] The acquisition module is used to acquire audio information from the video to be processed; perform sentence segmentation on the audio information to obtain multiple sentence audios; and perform speech recognition on each sentence audio to obtain each sentence text.

[0019] In one possible implementation, the acquisition module is used to frame the audio information to obtain multiple audio frames; for any audio frame, feature extraction is performed on the any audio frame to obtain audio frame features, and the audio frame features are classified to obtain classification results, and the classification results are used to characterize whether the any audio frame is a speech frame; based on the classification results of each audio frame, the multiple sentence audios are determined.

[0020] In one possible implementation, the acquisition module is used to determine each speech frame from the multiple audio frames based on the classification results of each audio frame; for any speech frame other than the first speech frame, if the time interval between any speech frame and the previous speech frame of any speech frame is not greater than a set time, then the any speech frame and the previous speech frame are merged into the same sentence audio; if the time interval between any speech frame and the previous speech frame is greater than the set time, then the any speech frame and the previous speech frame are divided into different sentence audios.

[0021] In a possible implementation, any sentence audio includes multiple speech frames;

[0022] The acquisition module is used to extract features of each speech frame included in any sentence audio to obtain features of each speech frame; map each speech frame feature to a corresponding character; and integrate the characters corresponding to each speech frame feature to obtain a sentence text.

[0023] In one possible implementation, the refining module is used to segment the text content to obtain multiple first text segments; for any first text segment, perform summary refinement on the any first text segment to obtain reference summary content; and integrate each reference summary content to obtain multiple target summary contents.

[0024] In one possible implementation, the refining module is used to construct a prompt text related to the content of the video to be processed; perform feature extraction on the prompt text and any one of the first text fragments to obtain a first text feature, wherein the first text feature is used to characterize the semantics of the prompt text and the semantics of any one of the first text fragments; and generate reference summary content semantically related to the first text feature.

[0025] In one possible implementation, the recognition and integration module is configured to perform feature extraction on any target summary content to obtain a target summary feature, where the target summary feature is used to characterize the semantics of any target summary content; segment the text content to obtain a plurality of second text segments; perform feature extraction on any second text segment to obtain a second text feature, and determine the semantic similarity between any target summary content and any second text segment based on the target summary feature and the second text feature, where the second text feature is used to characterize the semantics of any second text segment; and select a target text segment from the plurality of second text segments whose semantic similarity is higher than a similarity threshold.

[0026] In a possible implementation, the screening module is configured to determine a quality index of each integrated text segment; and screen out key text segments whose quality indexes meet index conditions from the integrated text segments.

[0027] In a possible implementation, the screening module is configured to determine, for any integrated text segment, a quality indicator of the any integrated text segment based on at least one of the first indicator, the second indicator, or the third indicator;

[0028] Among them, the first indicator is used to characterize the content importance of any integrated text segment in the scene to which the video to be processed belongs, the second indicator is used to characterize the amount of object interaction data in the video segment where any integrated text segment is located, and the third indicator is used to characterize the content similarity between any integrated text segment and the corresponding target summary content.

[0029] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned video processing methods.

[0030] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned video processing methods.

[0031] On the other hand, a computer program is further provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement any of the above-mentioned video processing methods.

[0032] On the other hand, a computer program product is further provided. The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned video processing methods.

[0033] The technical solution provided by this application brings at least the following beneficial effects:

[0034] In the technical solution provided by this application, the target summary content is obtained by summarizing and refining the text content of the video to be processed. Since the target summary content is the theme content of the text content, and the video to be processed and the text content correspond to the same theme content, the theme content of the video can be quickly determined by determining the target summary content. Then, a target text segment whose semantic similarity with the target summary content is higher than a similarity threshold is identified from the text content, thereby quickly locating the segment containing the theme content. Then, the target text segment and the context are semantically integrated to obtain an integrated text segment, thereby enriching the semantics of the integrated text segment in the context and ensuring the semantic coherence and integrity. This allows the key text segments to be accurately screened from the integrated text segment, improving the quality of the key text segments and ensuring that the key text segments are the segments containing the exciting content. Afterwards, based on the timestamps of the key text segments in the video to be processed, the video segments containing the key text segments are clipped from the video to be processed, thereby quickly clipping out the video segments containing the exciting content. This process does not require manual editing, which can reduce the complexity of video processing and improve video processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 This is a schematic diagram of an implementation environment of a video processing method provided in an embodiment of the present application;

[0037] Figure 2 This is a flowchart of a video processing method provided by an embodiment of the present application;

[0038] Figure 3 This is a schematic diagram of determining multiple sentence texts based on audio information provided by an embodiment of the present application;

[0039] Figure 4 This is a schematic diagram of determining the content of a target summary provided by an embodiment of the present application;

[0040] Figure 5 This is a schematic diagram of determining a target text segment provided in an embodiment of the present application;

[0041] Figure 6 This is a schematic diagram of a framework of a video processing method provided in an embodiment of the present application;

[0042] Figure 7 is a schematic diagram of a video processing provided by an embodiment of the present application;

[0043] Figure 8 is a structural diagram of a video processing device provided in an embodiment of the present application;

[0044] Figure 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0045] Figure 10 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first," "second," and the like in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0048] First, the abbreviations and key terms involved in the embodiments of the present application are defined.

[0049] Large Language Model (LLM): A model trained using large-scale text datasets based on deep learning technology. It can learn information such as the structure, grammar, semantics, or context of natural language and has the ability to process and generate natural language.

[0050] Natural Language Processing (NLP): A machine learning technique that enables computers to interpret, process, and understand language.

[0051] Automatic Speech Recognition (ASR): A technology used to convert speech signals into text data.

[0052] Voice Activity Detection (VAD): A technique for detecting active and inactive parts of a speech signal. It can be used for front-end preprocessing and noise suppression in speech processing applications.

[0053] Convolutional Neural Network (CNN): It includes at least one layer of network. Some networks use convolution operations instead of general matrix multiplication. It can be used to process pixel data to realize image recognition and processing.

[0054] Generative Pre-trained Transformer (GPT): A large language model based on the Transformer architecture that can be used to process and generate natural language.

[0055] In the field of computer technology, video is one of the primary means of information transmission. In some video-based applications, subjects are required to first watch the original video and record the start and end times of each segment. These segments are then extracted from the original video based on their start and end times, and then used to generate videos that can be shared on social media platforms. However, this manual process of extracting segments from videos is cumbersome and inefficient.

[0056] Based on the above reasons, the embodiment of the present application provides a video processing method that can quickly locate the wonderful content segments in the video and extract the video segments where the wonderful content segments are located. This not only reduces the complexity of the operation but also improves the efficiency of video processing.

[0057] Figure 1 This is a schematic diagram of an implementation environment of a video processing method provided in an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal device 101 and a server 102. The video processing method in the embodiment of the present application can be executed by the terminal device 101, or by the server 102, or by the terminal device 101 and the server 102 together.

[0058] The terminal device 101 can be a smart phone, a game console, a desktop computer, a tablet computer, a laptop computer, a smart TV, a smart car device, an intelligent voice interaction device, a smart home appliance, etc. The server 102 can be a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 can be connected to the terminal device 101 via a communication network, which is a wired network or a wireless network. The server 102 can have functions such as data processing, data storage, and data transmission and reception, which are not limited in the embodiments of the present application. The number of terminal devices 101 and servers 102 is not limited and can be one or more.

[0059] Optionally, the terminal device 101 or the server 102 may obtain the video to be processed and process the video to be processed to obtain multiple video segments. Alternatively, when the terminal device 101 obtains the video to be processed, it sends the video to be processed to the server 102 via the communication network. After receiving the video to be processed, the server 102 processes the video to be processed to obtain multiple video segments and sends the multiple video segments to the terminal device 101 via the communication network.

[0060] The following steps can be used to process a video to obtain multiple video segments. First, the video to be processed is processed to obtain text content. Then, multiple target summary contents are determined based on the text content. Next, multiple target text segments are determined from the text content based on the multiple target summary contents. Finally, multiple video segments are determined from the video to be processed based on the multiple target text segments.

[0061] The various optional embodiments of the present application can be implemented based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0062] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] like Figure 2 As shown, Figure 2 This is a flowchart of a video processing method provided by an embodiment of the present application. For ease of description, the terminal device 101 or the server 102 that executes the video processing method in the embodiment of the present application is referred to as an electronic device. In other words, the method in the embodiment of the present application is executed by an electronic device. Figure 2 As shown, the method includes the following steps.

[0064] Step 201: Obtain a video to be processed and obtain the text content of the video to be processed.

[0065] The video to be processed can be any video. For example, the video to be processed can be at least one of a game video, a live broadcast video, a film and television video, a dance video, a music video, etc. The embodiment of the present application does not limit the method for obtaining the video to be processed. For example, the electronic device can use any video input by the object as the video to be processed, or the electronic device can collect the video to be processed, or the electronic device can use any video obtained through the Internet as the video to be processed.

[0066] The text content can be obtained based on the video to be processed, and the embodiments of the present application do not limit the method for obtaining the text content. For example, for action videos, such as dance videos and sports videos, the action detection module can be used to detect the action in the action video and describe the action in the form of text to obtain the text content. For another example, for music videos, such as singing videos and instrument performance videos, the melody can be detected by the melody detection module and the melody or the emotion expressed can be described in the form of text to obtain the text content.

[0067] In a possible implementation, the text content includes multiple sentence texts. The "obtaining text content of the video to be processed" in step 201 includes steps 2011 to 2013 (not shown in the figure).

[0068] Step 2011: Acquire audio information in the video to be processed.

[0069] Generally, the video being processed contains content in both visual and auditory dimensions. Image sequences represent the visual content, while audio information represents the auditory content. The diversity of video content is primarily reflected in these two dimensions. For speech-based videos, audio often contains a denser amount of information. For example, in live streams with frequent dialogue or explanations, audio contains more information than image sequences. Therefore, audio information can be extracted from the video being processed.

[0070] The method for extracting audio information is not limited here. For example, some video editing software has an audio separation function. The video to be processed is imported into this video editing software, and the audio information is separated from the video to be processed using the audio separation function. Alternatively, the video to be processed is played and the sound during playback is recorded using recording software to obtain the audio information. Alternatively, if the video and audio correspond to different file formats, the audio information can be obtained by modifying the file format of the video to be processed to the file format of the audio.

[0071] Step 2012: Sentence segmentation is performed on the audio information to obtain multiple sentence audios.

[0072] In an embodiment of the present application, the electronic device divides the audio information into multiple sentence audios in units of sentences, and each sentence audio is an audio containing several sentences, which can express complete semantics. In addition, after the electronic device divides any sentence audio from the audio information, it can also obtain the start and end time of the sentence audio in the audio information, so as to associate the various sentence audios according to the start and end time. Generally, there is a long pause between sentences. Therefore, the segmentation method based on sentences is not only more in line with human auditory habits and achieves a better division of language expression, but also can more accurately record the start and end time of each sentence audio. It can be understood that the start and end time include the start time and the end time.

[0073] The embodiments of the present application do not limit the sentence segmentation method. For example, some audio processing software has the function of audio segmentation. The audio information is imported into these audio processing software, and the audio segmentation function is used to segment individual sentence audios from the audio information.

[0074] In an exemplary embodiment, step 2012 includes: dividing the audio information into frames to obtain multiple audio frames; for any audio frame, extracting features of any audio frame to obtain audio frame features, classifying the audio frame features to obtain classification results, and the classification results are used to characterize whether any audio frame is a speech frame; based on the classification results of each audio frame, determining multiple sentence audios.

[0075] In an embodiment of the present application, an electronic device can obtain a speech behavior detection model and use the speech behavior detection model to identify multiple pause locations in the audio information. The audio information is segmented based on the respective pause locations to obtain multiple sentence audios. The structure of the speech behavior detection model is not limited herein. Exemplarily, the speech behavior detection model includes at least one of a convolutional layer, an activation layer, a linear layer, an attention layer, and the like.

[0076] Optionally, the voice activity detection model is used to segment the audio information into frames according to a sliding window and a sliding step size, generating multiple audio frames. The duration of each audio frame is the duration of the sliding window. For example, if the sliding window is 25 milliseconds, the duration of the audio frame is 25 milliseconds. The sliding step size is less than or equal to the sliding window. If the sliding step size is equal to the sliding window, there is no overlap between adjacent audio frames. If the sliding step size is less than the sliding window, there is overlap between adjacent audio frames.

[0077] Next, feature extraction is performed on any audio frame through the voice behavior detection model to obtain audio frame features. The feature extraction method is not limited here. For example, the audio frame can be first converted into any spectrogram such as a mel-spectrogram or a spectrogram, and the spectrogram is subjected to at least one of Fourier transform, convolution processing, or attention processing to obtain audio frame features. The audio frame features are used to represent the information of the audio frame in the time domain and / or frequency domain. Optionally, the audio frame features can represent at least one of the bandwidth, energy, short-time zero-crossing rate, fundamental frequency period, etc. of the audio frame.

[0078] The audio frame features are then activated using a speech activity detection model to map them into probabilities, which represent the probability that the audio frame is a speech frame. If the probability is greater than a threshold, the audio frame classification result indicates that the audio frame is a speech frame. If the probability is not greater than the threshold, the audio frame classification result indicates that the audio frame is not a speech frame. Based on the foregoing, the audio frame features are classified to obtain a classification result for the audio frame. A speech frame is an audio frame containing speech.

[0079] According to the above method, the classification results of each audio frame can be determined by the voice behavior detection model. Then, multiple sentence audios are determined based on the classification results of each audio frame. The determination method is not limited here.

[0080] Optionally, based on the classification results of each audio frame, multiple sentence audios are determined, including: based on the classification results of each audio frame, each speech frame is determined from multiple audio frames; for any speech frame other than the first speech frame, if the time interval between any speech frame and the previous speech frame of any speech frame is not greater than a set time, any speech frame and the previous speech frame are merged into the same sentence audio; if the time interval between any speech frame and the previous speech frame is greater than the set time, any speech frame and the previous speech frame are divided into different sentence audios.

[0081] In an embodiment of the present application, if the classification result of the audio frame indicates that the audio frame is not a speech frame, the audio frame is determined to be a non-speech frame; if the classification result of the audio frame indicates that the audio frame is a speech frame, the audio frame is determined to be a speech frame.

[0082] The first speech frame can be grouped into one set. For any speech frame other than the first speech frame, if the time interval between the first speech frame and the previous speech frame is no greater than a set time, the first speech frame and the previous speech frame are grouped into the same set. If the time interval between the first speech frame and the previous speech frame is greater than a set time, the first speech frame and the previous speech frame are grouped into different sets. In this way, each speech frame is grouped into at least one set, and the speech frames in a set are continuous.

[0083] For any set, the individual speech frames in the set can be concatenated and merged to obtain a single sentence audio. Optionally, if the number of speech frames in the set is less than a threshold, the set is discarded. If the number of speech frames in the set is not less than the threshold, the individual speech frames in the set are concatenated and merged to obtain a single sentence audio.

[0084] The set time or quantity threshold may be a value set based on manual experience or a value obtained through experimental verification, which will not be described in detail here. For example, the set time may be 0.5 seconds and the quantity threshold may be 15.

[0085] It should be noted that the above method for determining sentence audio is only one possible implementation method. In actual applications, other implementation methods are possible. For example, multiple consecutive speech frames located between two non-speech frames can be spliced ​​into a single sentence audio. For example, if audio frames 1 to 5 are a non-speech frame, a speech frame, a speech frame, a speech frame, and a non-speech frame, respectively, then audio frames 2 to 4 can be spliced ​​into a single sentence audio.

[0086] It can be understood that non-speech frames are frames where pauses in the audio information are located. By determining whether each audio frame is a speech frame, pause locations can be detected from the audio information. Pause locations often indicate the boundaries of sentences or speech paragraphs. By setting a time to determine whether to merge adjacent speech frames into a continuous sentence audio, the probability of misjudgment is reduced and the accuracy of the sentence audio is improved.

[0087] Step 2013: Perform speech recognition on the audio of each sentence to obtain the text of each sentence.

[0088] In an embodiment of the present application, an electronic device can perform speech recognition on any sentence audio to obtain sentence text, and the method of speech recognition is not limited here. For example, some audio processing software has a speech recognition function, and the sentence audio is imported into the audio processing software, and the speech recognition function is used to recognize the sentence text based on the sentence audio. The sentence audio and the sentence text correspond to the same sentence, and both express the same content, semantics, etc.

[0089] In an exemplary embodiment, any sentence audio includes multiple speech frames. Step 2013 includes: for any sentence audio, extracting features from each speech frame included in the sentence audio to obtain features of each speech frame; mapping each speech frame feature to a corresponding character; and integrating the characters corresponding to each speech frame feature to obtain a sentence text.

[0090] In an embodiment of the present application, the electronic device can obtain a speech recognition model, which is obtained by training a large amount of speech data, and can recognize the speech content corresponding to at least one language, and convert the speech content into corresponding text. That is to say, the embodiment of the present application can convert the sentence audio into the corresponding sentence text through the speech recognition model. The embodiment of the present application does not limit the structure of the speech recognition model. Exemplarily, the speech recognition model includes at least one of a convolutional layer, an activation layer, a linear layer, an attention layer, etc. Optionally, the speech recognition model is a Whisper model, and the Whisper model includes multiple Transformer layers, and any Transformer layer includes at least one of an Embedding layer, a Position Encoding layer, an Encoder, a Decoder, an Attention layer, etc.

[0091] Optionally, through the speech recognition model, feature extraction is performed on each speech frame included in any sentence audio to obtain speech frame features of each speech frame. The method for determining the speech frame features is similar to the method for determining the audio frame features mentioned above. Please refer to the description of step 2012 and will not be repeated here. The speech frame features are used to represent information of the speech frame in the time domain and / or frequency domain. Optionally, the speech frame features can represent at least one of the bandwidth, energy, short-time zero-crossing rate, fundamental frequency period, etc. of the speech frame.

[0092] Then, for any speech frame feature, the speech frame feature is mapped to a prediction feature through the speech recognition model. The prediction feature is a feature vector predicted by the model to characterize the character. In addition, the electronic device can obtain a dictionary, which includes character features of multiple characters. The prediction feature is matched with each character feature in the dictionary, and the character corresponding to the character feature that matches the prediction feature is determined as the character corresponding to the speech frame feature. The way in which the prediction feature and the character feature are matched is not limited here. For example, each element in the prediction feature and each element in the character feature are compared, and whether the prediction feature and the character feature match is determined based on the element comparison result. Alternatively, the similarity between the prediction feature and the character feature is calculated, and whether the prediction feature and the character feature match is determined based on the similarity.

[0093] Afterwards, the characters corresponding to the features of each speech frame are integrated to perform deduplication, correction, and other processing on the characters to obtain the sentence text, thereby improving the accuracy of the sentence text. The integration method is not limited here. For example, the sentence text can be obtained by integrating the characters according to the template of the sentence text. Alternatively, the character features of each character are fused through a network model to obtain a fused feature, which is used to represent the semantics of the sentence text and is mapped to the sentence text.

[0094] Optionally, the start and end time of any sentence audio in the audio information is determined as the start and end time of the sentence text corresponding to the sentence audio.

[0095] Through the above method, it is possible to determine each sentence text and the corresponding start and end time in the audio information based on the audio information in the video to be processed, quickly extract sentence information from the video, and ensure that these sentence information can be accurately aligned with the time in the video, so that the clips in the video can be quickly located according to the start and end time of the sentence text, which is conducive to subsequent video editing, video synthesis and other processing.

[0096] like Figure 3 As shown, Figure 3 It is a schematic diagram of determining multiple sentence texts based on audio information provided by an embodiment of the present application. In the embodiment of the present application, the video to be processed is a long video. A long video is a video whose duration exceeds a set duration threshold. For example, a video with a duration of more than two hours is a long video. The voice extraction step can be performed first, that is, extracting audio information from the long video. Then the voice cutting step is performed, that is, obtaining sentence audio based on the pause segmentation in the audio information, and recording the start and end time of each sentence audio, and the interval between two adjacent sentence audios is not less than 0.5 seconds. Afterwards, the voice recognition step is performed, that is, converting each sentence audio into a sentence text based on the ASR model, and the form of the i-th (i is a positive integer) sentence text is: [start and end time i: sentence text i]. Through the above three steps, the audio information in the video is converted into multiple sentence texts with high precision and efficiency, and the position of each sentence text in the video is accurately located.

[0097] Step 202: extract the summary of the text content to obtain multiple target summary contents.

[0098] In the embodiments of the present application, the target summary content is used to represent a summary of one aspect of the text content, or a summary of a portion of the text content. For example, when the text content is a game commentary, one target summary content may represent a summary of the game resources, while another target summary content may represent a summary of the team battle commentary within the game commentary.

[0099] The embodiment of the present application does not limit the method of abstract extraction. In an exemplary embodiment, step 202 includes steps 2021 to 2023 (not shown in the figure).

[0100] Step 2021: segment the text content to obtain multiple first text segments.

[0101] The embodiments of the present application do not limit the method for segmenting the text content. For example, the text content can be divided into a set number of first text segments, or the text content can be divided into multiple first text segments in a manner where each first text segment has a set number of characters. Two adjacent first text segments may or may not have overlapping characters.

[0102] For example, a large language model can be used for summarization and refinement. For a large language model, the length of text processed at a single time is often limited. For example, the length of text processed at a single time by a large language model is 4096 (about 4000) characters. Based on this, the text content can be divided into multiple first text segments, each of which contains about 3000 characters. There is about 10% character overlap between two adjacent first text segments to prevent information loss and maintain the semantic coherence of the first text segments. By having overlapping characters between two adjacent first text segments, it can be ensured that the information at the boundary of overlapping characters and non-overlapping characters can be fully understood and processed in the context.

[0103] Step 2022: For any first text segment, perform summary extraction on the first text segment to obtain reference summary content.

[0104] In this embodiment of the present application, a large language model can be used to summarize and refine any first text segment to obtain a reference summary. This embodiment of the present application does not limit the structure of the large language model. Exemplarily, the large language model includes at least one of a convolutional layer, an activation layer, a linear layer, and an attention layer. Optionally, the large language model is a ChatGPT (ChatGenerative Pre-trained Transformer) model.

[0105] Optionally, any first text segment includes multiple characters. By mapping each character in the first text segment to a corresponding character vector, an embedding vector for the first text segment is obtained, where the embedding vector includes the character vector of each character. The embedding vector of the first text segment is input into a large language model, which includes an encoding network and a decoding network. The encoding network is used to extract features from the embedding vector of the first text segment, obtaining text features output by the encoding network. The text features are used to represent the semantics of the first text segment. The decoding network generates reference summary content based on the text features output by the encoding network.

[0106] Optionally, step 2022 includes: constructing a prompt text related to the content of the video to be processed; performing feature extraction on the prompt text and any first text segment to obtain a first text feature, the first text feature being used to characterize the semantics of the prompt text and any first text segment; and generating reference summary content semantically related to the first text feature.

[0107] In an embodiment of the present application, the electronic device can obtain a personalized policy library, which includes multiple candidate words and at least one prompt word corresponding to each candidate word. For example, if the candidate word is an item, the item can correspond to prompt words: production process, color, taste, origin, etc. The electronic device can obtain the label of the video to be processed, and display the theme, field, etc. of the video to be processed through the label. The electronic device matches the label with each candidate word in the personalized policy library, and constructs a prompt text based on the prompt word corresponding to the candidate word matching the label, so that the prompt text is related to the content of the video to be processed. For example, if the content of the video to be processed is related to an item, the prompt text "Please summarize the color of the item" can be constructed based on the prompt word "color" corresponding to the item.

[0108] The prompt text includes multiple characters. Each character in the prompt text is mapped to a corresponding character vector to obtain an embedding vector for the prompt text, which includes the character vector of each character. The embedding vector of the first text segment and the embedding vector of the prompt text are concatenated and input into the large language model. The concatenated vector is then subjected to feature extraction by the encoding network to obtain a first text feature. The first text feature is used to represent the semantics of the prompt text and the semantics of the first text segment.

[0109] Optionally, the encoding network includes multiple encoding blocks. The first encoding block performs feature extraction on the concatenated vector to obtain the text features output by the first encoding block. For non-first encoding blocks, the non-first encoding blocks perform feature extraction on the text features output by the previous encoding block to obtain the text features output by the non-first encoding block. The text features output by the last encoding block are the first text features.

[0110] Each encoding block performs feature extraction in a similar manner. The following uses the first encoding block's feature extraction of the concatenated vector as an example to illustrate the principle of feature extraction in encoding blocks.

[0111] The first encoding block includes a self-attention network layer, a layer normalization network layer, a feedforward layer, and a layer normalization network layer connected in series. First, the self-attention network layer performs attention processing on the spliced ​​vector based on the attention mechanism to obtain the attention feature. Then, the spliced ​​vector and the attention feature are spliced, and the spliced ​​features are subjected to layer normalization processing through the first layer normalization network layer to obtain a first processing result. Then, the first processing result is linearly mapped or nonlinearly mapped through the feedforward layer to obtain a mapping result. After that, the first processing result and the mapping result are spliced, and the spliced ​​features are subjected to layer normalization processing through the second layer normalization network layer to obtain a second processing result. The second processing result is the text feature output by the first encoding block, which is used to characterize the semantics of the content text and the semantics of the prompt text.

[0112] The feature extraction process is also a semantic fusion process. Specifically, the text features output by the first encoding block represent the resulting fused semantics, which incorporates the semantics of the content text and the hint text. This fused semantics encompasses both the semantics of the content text and the hint text. Continuous feature extraction enhances the representational power of the resulting first text features, improving the accuracy of subsequent processing.

[0113] After determining the first text feature, a decoding network may be used to generate reference summary content based on the first text feature. Optionally, the reference summary content includes multiple words, each of which includes at least one character. For the first word in the reference summary content, the decoding network decodes the first text feature to obtain the first word. For non-first words in the reference summary content, the decoding network decodes the first text feature and each word preceding the non-first word to obtain the non-first words.

[0114] The decoding network includes a masked multi-head attention network layer, a layer normalization network layer, a multi-head attention network layer, a layer normalization network layer, a feedforward layer, and a layer normalization network layer connected in series. First, through the masked multi-head attention network layer, based on the attention mechanism, the features of the starting character (a set character, located before the first word generated by the decoding network) or each word before the non-first word are subjected to attention processing to obtain a first attention feature. Then, the features of the starting character or each word before the non-first word are spliced ​​with the first attention feature, and the spliced ​​features are subjected to layer normalization processing through the first layer normalization network layer to obtain a first processing result. Then, through the multi-head attention network layer, based on the attention mechanism, the first processing result and the first text feature are subjected to attention processing to obtain a second attention feature. Then, the first processing result is spliced ​​with the second attention feature, and the spliced ​​features are subjected to layer normalization processing through the second layer normalization network layer to obtain a second processing result. Afterwards, the second processing result is linearly mapped or nonlinearly mapped through the feedforward layer to obtain a mapping result. Afterwards, the second processing result and the mapping result are spliced ​​together, and the spliced ​​features are subjected to layer normalization processing through the third layer normalization network layer to obtain the third processing result.

[0115] For the first word, the third processing result obtained by the decoding network is used to represent the semantics obtained by fusing the semantics of the content text and the semantics of the prompt text. For non-first words, the third processing result obtained by the decoding network is used to represent the semantics obtained by fusing the semantics of each word before the non-first word, the semantics of the content text, and the semantics of the prompt text.

[0116] The decoding network includes a word library, which includes semantic features of multiple candidate words. The semantic correlation between the third processing result and the semantic features of each candidate word is calculated. The candidate word with the greatest semantic correlation is used as the first word or a non-first word.

[0117] Through the above-described method, the individual words in the reference summary content are gradually determined, thereby determining the reference summary content. The large language model performs semantic understanding and information compression on the first text segment, enabling the model to identify the key points of the first text segment and condense it into the reference summary content. Because the prompt text is related to the content of the video to be processed, the reference summary content obtained by summarizing and refining the first text segment using the prompt text can more accurately capture and summarize summary information related to the video content, making the reference summary content both comprehensive and targeted.

[0118] Step 2023: Integrate the reference summary contents to obtain multiple target summary contents.

[0119] In an embodiment of the present application, each reference summary content can be integrated through a large language model to obtain multiple target summary contents. As mentioned above, the large language model includes an encoding network and a decoding network. The encoding network is first used to extract features of each reference summary content to obtain a first summary feature, and the first summary feature is used to represent the semantics of each reference summary content. Optionally, the encoding network is used to extract features of each reference summary content and prompt text to obtain a first summary feature, and the first summary feature is used to represent the semantics of each reference summary content and the semantics of the prompt text. Then, the first summary feature is decoded through the decoding network to obtain the target summary content. Among them, the principle of feature extraction by the encoding network and the principle of decoding by the decoding network have been described above and will not be repeated here.

[0120] By integrating and sorting the reference summaries using a large language model, duplicate content is merged and redundant content is discarded, resulting in a coherent and comprehensive target summary. This integration process, enabled by a large language model, becomes more than just simple information splicing; it becomes a highly intelligent semantic synthesis, cross-validating and integrating information from different reference summaries to ensure that the target summary neither misses key information nor includes redundant information. This improves the target summary's information fluency, logical coherence, and natural semantic expression.

[0121] In an exemplary embodiment, the reference summary content includes a first content part and a second content part, wherein the first content part is the main point of the second content part, and the second content part is the summary of the first text segment. Optionally, the electronic device extracts the summary of any first text segment using a large language model to obtain the second content part. Then, the second content part is extracted from the main points using the large language model to obtain the first content part, and the reference summary content is determined based on the first content part and the second content part. The methods of extracting the main points and extracting the summary are similar, as can be seen in the description of step 2022, and will not be repeated here.

[0122] Optionally, when integrating the various reference summary contents, for any two reference summary contents, if the semantic similarity between the first content parts of the two reference summary contents is higher than a first threshold, one of the two reference summary contents is determined as the first summary content; if the semantic similarity between the first content parts of the two reference summary contents is lower than a second threshold (the second threshold is not greater than the first threshold), both of the reference summary contents are regarded as the first summary content. In this way, preliminary deduplication of the various reference summary contents is achieved, and multiple semantically dissimilar first summary contents are obtained, which reduces the number of first summary contents and is conducive to improving the efficiency of subsequent integration. In addition, since the data volume of the first content part is small, deduplication through the first content part can improve the deduplication efficiency.

[0123] Next, the electronic device integrates the second content portions from the multiple first summary contents to obtain a third content portion. By integrating the second content portions, key information is extracted and redundant information is removed. The integrated third content portion is a summary of the text content, including only the key information from the text content. Optionally, a large language model is used to integrate the second content portions to obtain the third content portion. The integration method is described in step 2023 and is not further described here.

[0124] Next, the electronic device extracts key points from the third content section to obtain a fourth content section. The fourth content section is the key points of the third content section and includes only the key information from the third content section. Based on the third and fourth content sections, the target summary content is determined, ensuring the conciseness and comprehensiveness of the target summary content. The methods for extracting key points and extracting the summary are similar, as described in step 2022 and will not be repeated here.

[0125] As you can understand, the large language model is fully pre-trained and ready for immediate use, requiring no additional training. This simplifies model deployment and application. Based on deep learning algorithms for language processing, the large language model analyzes dialogue, explanations, and other content in videos to identify and extract key information and key takeaways. The large language model not only identifies important facts, opinions, and arguments, but also understands the relationships between them and their significance within the overall narrative. It converts this information into a clear and concise text summary, providing users with a quick and accurate overview of the video content.

[0126] like Figure 4 As shown, Figure 4: This is a schematic diagram of determining a target summary content provided by an embodiment of the present application. In the embodiment of the present application, the video to be processed is a long video. The electronic device can obtain the text content of the long video and perform text preprocessing on the text content. Among them, text preprocessing includes segmentation, that is, the text content can be segmented to obtain N first text segments, where N is a positive integer. Then, the electronic device obtains a personalized strategy library and constructs a prompt text based on the content of the long video. For example, if the video involves a certain item, prompt text about the production process, taste and color, and origin characteristics can be constructed. Then, based on the prompt text, the electronic device uses a large language model to summarize and refine each first text segment to obtain reference summary content. The reference summary content is content extracted from the first text segment that is semantically related to the prompt text. Then, all reference summary contents (i.e., N reference summary contents) are integrated using the large language model to merge similar content and remove duplicate content to obtain M target summary contents. Among them, M is a positive integer less than N.

[0127] Optionally, both the reference summary and the target summary content are formatted as "Key Points: Summary." "Key Points" corresponds to the first or fourth content section mentioned above, and "Summary" corresponds to the second or third content section mentioned above. After processing by the large language model, a refined list of key points and corresponding summary text is generated, providing an accurate and clear content navigation for subsequent processing. This efficient key point summarization capability significantly improves the speed and quality of information retrieval, enhancing its application value in processing large amounts of video information and knowledge management efficiency.

[0128] In step 203, for any target summary content, a target text segment having a semantic similarity with any target summary content greater than a similarity threshold is identified from the text content, and the target text segment and the associated segment are semantically integrated to obtain an integrated text segment, where the associated segment is at least one of the preceding and following contexts of the target text segment in the text content.

[0129] As mentioned above, the target summary content is a summary of the text content and is high-quality information. However, text content is generally long and contains a lot of redundant information. Based on this, by identifying target text segments from the text content whose semantic similarity with the target summary content exceeds a similarity threshold, we can identify segments containing high-quality information within the content text, which is beneficial for identifying highlights in the video being processed. Furthermore, identifying target text segments at the semantic level improves the accuracy of the target text segments.

[0130] In an exemplary embodiment, the step 203 of “identifying a target text segment having a semantic similarity with any target summary content higher than a similarity threshold from the text content” includes steps 2031 to 2034 (not shown in the figure).

[0131] Step 2031 : extract features from any target summary content to obtain target summary features. The target summary features are used to represent the semantics of any target summary content.

[0132] In the embodiments of the present application, a target text segment can be identified by a semantic model, and the structure of the semantic model is not limited herein. Exemplarily, the semantic model includes at least one of a convolutional layer, an activation layer, a linear layer, an attention layer, and the like. Optionally, the semantic model includes any one of BERT (Bidirectional Encoder Representations from Transformers), GPT, or a deep learning model.

[0133] The target summary content includes multiple characters. Each character can be mapped to a corresponding character vector through a semantic model to obtain a second summary feature of the target summary content, and the second summary feature includes the character vector of each character in the target summary content. The second summary feature can be feature extracted through the semantic model, and the feature extraction method is not limited here. Exemplarily, the semantic model includes an encoding network, and the target summary feature is obtained by extracting the second summary feature through the encoding network. The method of feature extraction by the encoding network has been described above and will not be repeated here.

[0134] Alternatively, the character vector of the set starting character can be spliced ​​before the character vector of the first character in the second summary feature, for example, the starting character is the character [CLS]. And / or, the character vector of the set ending character can be spliced ​​after the character vector of the last character in the second summary feature, for example, the ending character is the character [SEP]. For the starting character (or ending character), the character vector of the starting character (or ending character) and the character vectors of each character in the second summary feature can be fused through the semantic model to obtain the encoding feature of the starting character (or ending character). Since the encoding feature of the starting character (or ending character) fuses the character vectors of each character in the second summary feature, the encoding feature of the starting character (or ending character) can be used as the target summary feature.

[0135] The target summary feature is derived from feature extraction of the second summary feature. This feature extraction maps the shallow features corresponding to the second summary feature to a high-dimensional space, encoding the semantics of the target summary content into a vector form. This results in the target summary feature. In other words, the target summary feature represents the semantics of the target summary content.

[0136] Step 2032: segment the text content to obtain multiple second text segments.

[0137] The embodiments of the present application do not limit the manner in which the text content is segmented. For example, the text content can be divided into a set number of second text segments, or the text content can be divided into multiple second text segments in a manner where each set number of characters constitutes one second text segment. Two adjacent second text segments may have overlapping characters or no overlapping characters. It is understandable that any second text segment may be one of the first text segments mentioned above, or may have some character overlap with a first text segment mentioned above.

[0138] For example, the text content can be segmented according to a specific strategy, which is: dividing the text content into multiple second text segments with a length of approximately 150 characters, with approximately 50% character overlap between two adjacent second text segments, so as to maintain semantic coherence and integrity and optimize semantic understanding and retrieval efficiency.

[0139] Optionally, multiple candidate text segments are obtained after segmenting the text content. If any candidate text segment includes a keyword in the target summary content, the candidate text segment is used as the second text segment. By filtering and screening the candidate text segments, segments containing keywords are screened out, and segments that do not contain keywords are filtered out, thereby improving the quality of the second text segments and reducing the number of second text segments. This facilitates the subsequent rapid determination of the target text segment from the second text segments, thereby improving video processing efficiency.

[0140] Step 2033: For any second text segment, feature extraction is performed on the second text segment to obtain a second text feature. Based on the target summary feature and the second text feature, the semantic similarity between any target summary content and any second text segment is determined. The second text feature is used to characterize the semantics of any second text segment.

[0141] In this embodiment of the present application, the second text segment includes multiple characters. Each character can be mapped to a corresponding character vector using a semantic model to obtain an embedding vector for the second text segment. The embedding vector for the second text segment includes the character vectors for each character in the second text segment. Feature extraction is performed on the embedding vector of the second text segment using the semantic model. The principle of feature extraction using the semantic model has been described in step 2031 and will not be repeated here.

[0142] By extracting features from the embedding vector of any second text segment and mapping the shallow features corresponding to the embedding vector into a high-dimensional space, the semantics of the second text segment is encoded into a vector form, thereby obtaining the second text features. In other words, the second text features can represent the semantics of the second text segment, and both the second text features and the target summary features are in the semantic space.

[0143] Next, semantic similarity is calculated based on the target summary features and the second text features according to a similarity algorithm. This embodiment of the application does not limit the similarity algorithm. For example, the similarity algorithm includes but is not limited to a cosine similarity algorithm, a Euclidean distance algorithm, or other similarity measurement algorithms.

[0144] Step 2034 : Select a target text segment having a semantic similarity higher than a similarity threshold from the plurality of second text segments.

[0145] In an embodiment of the present application, the similarity threshold is a numerical value set based on experience, or the similarity threshold is data input by a user, or the similarity threshold is the maximum semantic similarity among the semantic similarities between any target summary content and each second text segment.

[0146] For any second text segment, if the semantic similarity corresponding to the second text segment is higher than the similarity threshold, the second text segment is used as the target text segment. i , where i is 0 or a positive integer, for example, i=0, 1, 2... The target text segment D can be determined according to the following formula (1): j .

[0147]

[0148] Among them, f(·) represents the feature extraction operation, which can convert the text into a feature vector in the semantic space. f(Q) represents the target summary features obtained by extracting the features of the target summary content. f(D j ) represents the jth second text segment D j The second text feature is obtained by performing feature extraction. Characterize the target summary feature f(Q) and the second text feature f(D j ), if the semantic similarity is the largest, then the second text segment D j The target text segment D j .

[0149] like Figure 5 As shown, Figure 5 1 is a schematic diagram of determining a target text segment provided by an embodiment of the present application. On the one hand, an electronic device sequentially extracts M target summary contents to obtain the kth target summary content. Here, M is a positive integer, and k is any positive integer between 1 and M. On the other hand, text preprocessing is performed on the text content of a long video to segment the text content to obtain M second text segments. The number of second text segments and the number of target summary contents may be the same or different.

[0150] Next, feature extraction is performed on the kth target summary content to obtain a target summary feature. Feature extraction is also performed on the M second text segments to obtain M second text features. Semantic similarity is calculated based on the target summary feature and each second text feature. Based on the semantic similarity, the target text segment with the highest semantic similarity is retrieved from the M second text segments.

[0151] In some cases, text content contains informal or colloquial expressions. For example, the text content of live video often includes a large number of colloquial expressions, which affects the accuracy of text retrieval. In the embodiment of the present application, the target text segment is retrieved from the second text segment by measuring the semantic similarity between the target summary feature and each second text feature. This achieves the semantic relevance between the target summary content and the second text segment by characterizing the semantic similarity, capturing the relevance between the target summary content and the second text segment at the semantic level, rather than just superficial text matching, thereby improving retrieval efficiency and the accuracy of retrieval results.

[0152] After a target text segment is identified from the text content, the preceding and / or following context of the target text segment in the text content can be identified as associated segments of the target text segment, wherein the associated segments include at least one sentence text. After locating a target text segment that is highly relevant to the content summary, a context expansion strategy is employed to enhance the completeness and coherence of the information. By extracting text segments directly related to the content summary and expanding to include the context of the text segment, a more comprehensive information perspective is achieved, facilitating understanding of the position and significance of the target text segment within the overall context.

[0153] Optionally, any target text segment and associated segments are integrated using a large language model. The large language model includes an encoding network and a decoding network. The encoding network extracts features from the target text segment and associated segments, obtaining text features output by the encoding network. These text features represent the semantics of the target text segment and the semantics of the associated segments. The decoding network generates an integrated text segment based on the text features output by the encoding network. The implementation principles of the encoding network and the decoding network have been described above and will not be repeated here.

[0154] The target text segment is a text segment that is highly semantically relevant to the content summary of the video. The integrated text segment is obtained by integrating the target text segment and the related segments, which realizes the integration of contextual information and improves the text quality. On the basis of retaining the content summary, redundant information irrelevant to the content summary is identified and eliminated, and the logical coherence and integrity of the sentence are enhanced, so that the integrated text segment remains closely related to the original content summary.

[0155] Step 204 : Filter out key text segments from the multiple integrated text segments.

[0156] By screening the integrated text segments, poor-quality integrated text segments are removed, while high-quality integrated text segments are retained. This not only improves the quality of the key text segments, but also improves the quality of the video segments containing the key text segments when they are subsequently cut from the video to be processed.

[0157] The present embodiment does not limit the screening method. For example, if the integrated text segment includes a keyword in the target summary content, the integrated text segment is used as the key text segment.

[0158] In an exemplary embodiment, step 204 includes steps 2041 and 2042 (not shown in the figure).

[0159] Step 2041 , determining the quality index of each integrated text segment.

[0160] Since the quality of each integrated text segment varies, a quality index of each integrated text segment can be determined to filter the integrated text segments based on the quality index. The embodiment of the present application does not limit the method for determining the quality index of the integrated text segment. For example, the quality index can be determined based on the number of keywords included in the integrated text segment.

[0161] Alternatively, step 2041 includes: for any integrated text segment, determining a quality indicator of any integrated text segment based on at least one of the first indicator, the second indicator, or the third indicator.

[0162] First, the method for determining the first indicator is introduced. In an embodiment of the present application, the electronic device can obtain the label of the video to be processed, and determine the scene to which the video to be processed belongs based on the label. The importance of the same content in different scenes varies. For example, in the scene of item introduction, more attention is paid to the appearance description of the item, the resources required to obtain the item, the usage experience of the item, etc., but the game live broadcast scene does not need to pay attention to the aforementioned content. The electronic device can obtain a personalized policy library, which includes various scenes and corresponding content of concern. The specified content of concern to the scene to which the video to be processed belongs can be determined based on the personalized policy library, and the first indicator corresponding to the integrated text segment can be determined based on the correlation between the integrated text segment and the specified content. The first indicator is used to characterize the content importance of any integrated text segment in the scene to which the video to be processed belongs.

[0163] The embodiment of the present application does not limit the method for determining the first indicator. For example, the characteristics of the specified content and the characteristics of the integrated text segment can be determined separately by a deep learning model. The characteristic distance between the characteristics of the specified content and the characteristics of the integrated text segment is calculated, and the characteristic distance is mapped to the first indicator corresponding to the integrated text segment. Optionally, the smaller the characteristic distance, the higher the correlation between the specified content and the integrated text segment, and the larger the first indicator.

[0164] Alternatively, the TF-IDF (term frequency-inverse document frequency) of each word in the specified content or the integrated text segment can be counted, and a weighted calculation can be performed based on the TF-IDF (term frequency-inverse document frequency) of each word to obtain a calculation result. Based on the difference between the calculation result of the specified content and the calculation result of the integrated text segment, the first index corresponding to the integrated text segment is determined. Optionally, the smaller the difference, the higher the correlation between the specified content and the integrated text segment, and the larger the first index.

[0165] Alternatively, a semantic embedding distance between the specified content and the integrated text segment is calculated, and the semantic embedding distance is mapped to a first index corresponding to the integrated text segment. Optionally, a smaller semantic embedding distance indicates a higher correlation between the specified content and the integrated text segment, and a larger first index.

[0166] Next, the method for determining the second indicator is introduced. In general, the object watching the video can perform operations on the video, and the data of these operations are the object interaction data in the embodiment of the present application. For example, the object can perform operations on the video, including but not limited to likes, collections, posting barrages, posting comments, etc. Based on these operations, the number of likes, collections, barrage data, comment data, etc. can be obtained. Since the integrated text segment corresponds to the target text segment and its context in the text content, and the text content corresponds to the video to be processed, it is possible to determine the video segment where the target text segment is located and the video segment where the context is located, and determine the object interaction data of these video segments. Then, the object interaction data is mapped to the second indicator of the integrated text segment, and the mapping method is not limited here. It can be understood that the second indicator is positively correlated with the number of object interaction data. That is to say, the second indicator is used to characterize the number of object interaction data of the video segment where any integrated text segment is located.

[0167] Next, the method for determining the third indicator is introduced. In an embodiment of the present application, the third indicator can be determined based on the integrated text segment and the target summary content. Optionally, the method for determining the third indicator is similar to the method for determining the first indicator based on the integrated text segment and the specified content, and will not be repeated here. The third indicator is used to characterize the content similarity between any integrated text segment and the corresponding target summary content. The third indicator can reflect the degree of semantic consistency between the integrated text segment and the target summary content. The higher the third indicator is, the higher the degree of semantic consistency between the integrated text segment and the target summary content is.

[0168] Afterwards, at least one of the first index, the second index and the third index can be averaged or weighted summed to obtain a quality index of the integrated text segment. It is understandable that the determination method of the quality index is only schematic, and in practical applications, other determination methods can also be provided. For example, the fourth index can also be determined based on at least one of the emotions expressed by the video segment where any integrated text segment is located (such as the emotions expressed by the dance video), the emotions expressed by the audio segment of the video segment (such as the emotions expressed by the background audio), and the emotions expressed by the integrated text segment. The quality index of the integrated text segment is determined based on at least one of the first index, the second index, the third index or the fourth index.

[0169] By determining the quality indicators of integrated text segments, the quality of integrated text segments can be accurately evaluated, which is conducive to screening out more exciting text segments, thereby obtaining higher quality video segments and improving the user experience.

[0170] Step 2042 , screening out key text segments whose quality indicators meet the indicator conditions from each integrated text segment.

[0171] The present embodiment does not limit the content of the quality indicator satisfying the indicator condition. For example, the quality indicator satisfying the indicator condition includes: the quality indicator is not less than the indicator threshold value set based on experience, or the quality indicator is greater than one of the quality indicators. For any integrated text segment, if the quality indicator corresponding to the integrated text segment meets the indicator condition, the integrated text segment is regarded as a key text segment.

[0172] By screening the integrated text snippets, we achieve quantitative evaluation and prioritization of these snippets, ensuring that the selected key text snippets focus on the most exciting and relevant content in the video. Subsequent editing of the video clip containing the key text snippets from the processed video not only improves editing efficiency but also ensures that the video clip is the most informative segment in the long video, enhancing the user viewing experience and bringing higher user engagement and conversion rates for video providers.

[0173] Step 205 : based on the timestamp of the key text segment in the video to be processed, clip the video segment where the key text segment is located from the video to be processed.

[0174] It is understandable that there is at least one key text segment. As mentioned above, the text content includes multiple sentence texts and the start and end time of each sentence text in the video to be processed. For any key text segment, the key text segment corresponds to a target text segment and an associated segment. Since the associated segment is the above and / or below the target text segment, the target text segment and the associated segment are continuous segments, including multiple continuous sentence texts. Based on this, the timestamp of the key text segment includes: the start time of the first sentence text and the end time of the last sentence text in the multiple sentence texts corresponding to the key text segment.

[0175] After determining the timestamp of any key text segment, the video segment containing the key text segment in the video being processed is determined, and the start and end times of the video segment containing the key text segment in the video being processed are determined. Therefore, using the timestamp, a video editing tool can be used to accurately extract the video segment containing the key text segment from the video being processed.

[0176] Electronic devices can display individual video clips so that a video editing object can use them to synthesize videos and edit the video being processed. Optionally, the electronic device can also display tags for each video clip, reflecting auxiliary information such as the video clip's theme, summary, and scene, to enhance the editing experience of the video editing object. Displaying tags enhances the comprehensibility of the video clips and provides rich context, which helps the video editing object perform efficient editing.

[0177] In addition, the display order of each video clip can be adjusted based on the needs or preferences of the video clip object, or based on the quality index of each video clip, to ensure that the best video clips are displayed first, which is conducive to improving the efficiency of video editing. Among them, the quality index of the video clip is the quality index of the key text segment corresponding to the video clip.

[0178] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the video, audio information, text content, etc. to be processed involved in this application are all obtained with full authorization.

[0179] In the above method, the target summary content is obtained by summarizing and refining the text content of the video to be processed. Since the target summary content is the theme of the text content, and the video to be processed and the text content correspond to the same theme, determining the target summary content allows for rapid determination of the video's theme. Next, target text segments whose semantic similarity with the target summary content exceeds a similarity threshold are identified from the text content, allowing for rapid location of the segments containing the theme content. Next, the target text segments are semantically integrated with the context to obtain integrated text segments. This enriches the semantics of the integrated text segments within the context, ensuring semantic coherence and integrity. This allows for accurate subsequent screening of key text segments from the integrated text segments, improving the quality of key text segments and ensuring that they contain interesting content. Subsequently, based on the timestamps of the key text segments in the video to be processed, video segments containing the key text segments are clipped from the video to be processed, allowing for rapid identification of the video segments containing interesting content. This process eliminates the need for manual editing, reduces the complexity of video processing, and improves video processing efficiency.

[0180] The above describes the video processing method of the embodiment of the present application from the perspective of method steps. The following is a systematic and comprehensive explanation in combination with the scenarios. The video processing solution of the present application can be applied to any scenario related to video, for example, it can be applied to live broadcast scenarios, game scenarios, knowledge dissemination scenarios, etc. In the live broadcast scenario, the video to be processed mentioned above can be a live broadcast video. In the game scenario, the video to be processed can include game commentary videos, game running videos, etc. In the knowledge dissemination scenario, the video to be processed can include teaching videos, etc. The video to be processed can be a language video, an action video, or a singing video, etc.

[0181] According to the video processing method of the embodiment of the present application, the video to be processed is processed to obtain a video clip, which can be used in the fields of advertising production, video clipping and video editing. Figure 6 As shown, in the field of video editing, the video processing task can be the task of editing long video highlights. By inputting a long video file into the highlight content recognition system, the highlight content recognition system can determine and output the highlight content segments according to the video processing method of the embodiment of the application.

[0182] The following uses a live broadcast scenario as an example to systematically illustrate the video processing method of an embodiment of the present application. In a live broadcast scenario, the duration of a live video is often several hours or even more than ten hours. If a live video is manually edited, the editor needs to quickly browse the entire video file and mark the start and end times of each video file so that the editing can be based on the start and end times. This process is both time-consuming and laborious.

[0183] The method of the embodiments of the present application can automatically analyze live video and identify highlights in long videos. Whether the video is a few hours long or more than ten hours long, it can accurately analyze and find the highlights. Not only can the core themes and summaries of these clips be intelligently extracted, but the highlights can also be organized into complete video clips based on these themes, improving the efficiency of editing and production, reducing the technical difficulty of video production, and enabling users to extract and generate high-quality content from large amounts of video data in a short period of time.

[0184] Specifically, if Figure 7 As shown, the electronic device can obtain the original long-running live video file, which can be as long as several hours. It can perform speech recognition on the audio information in the video to obtain text content. Because text can more comprehensively express and locate the main content of the live video, obtaining text content helps improve processing efficiency and facilitates subsequent analysis and processing.

[0185] Next, using the summarization capabilities of GPT or other NLP models, the text content is summarized and refined to produce a target summary. Based on the target summary, a target text segment is retrieved from the text content, identifying the most exciting content segments in the video, such as climaxes, key moments, or other important events. Using the summarization capabilities of GPT or other NLP models, the target text segment is integrated with contextual information to complete the semantics of the target text segment within the text content, resulting in an integrated text segment.

[0186] Next, the integrated text segment is scored to obtain a matching score and a key point score. The matching score can be obtained by matching the keywords in the integrated text segment with the keywords in the target summary content, calculating the feature similarity between the integrated text segment and the target summary content, and mapping the feature similarity to a key point score. Alternatively, data analysis can be performed on interactive data such as the number of comments and likes in the video to obtain an interaction score. Data analysis can be performed on sales data such as the number of orders placed in the video to obtain a sales score. Still further, other scores can be determined based on the emotions or other information expressed in the video.

[0187] For a long video, an electronic device may recognize dozens or even hundreds of integrated text segments. The video segments containing each integrated text segment can be scored using the aforementioned method. The integrated text segments can then be filtered based on matching scores, key point scores, interaction scores, sales scores, and other scores. During the filtering process, the integrated text segments can be sorted, deduplicated, and assembled.

[0188] After filtering, the key text snippets are obtained. The video clips containing the key text snippets are then clipped from the live video file. These video clips can be displayed on electronic devices, allowing users to use them for post-production based on their needs. For example, users can edit these video clips into a short video and share it on social media, or integrate the video clips into a larger video project.

[0189] In an exemplary embodiment, the electronic device can access a library of personalized live broadcast policies to replace personalized policies for different types of live broadcasts, thereby implementing summary refinement and rating filtering based on the policies. After determining the video clip, the electronic device can also determine auxiliary information of the video clip, such as the video clip's theme, tags, and scene, so that the user can conveniently use the auxiliary information for post-production creation.

[0190] Typically, manually editing a long video takes about 3 to 4 days, which significantly limits production efficiency. However, after adopting the method of the embodiment of the present application, the editing cycle of a long video can be shortened to about half a day, reducing the workload of manual editing. Table 1 below shows some data after processing live video files based on the method of the embodiment of the present application in e-commerce and education scenarios.

[0191] Table 1

[0192] Live video files E-commerce scenario (about 6.5 hours) Educational scene (approximately 3 hours) Video clips 37 8 The fragment adopted by the object 29 8

[0193] As can be seen from Table 1, in the e-commerce scenario, by processing a live video file of approximately 6.5 hours using the method of an embodiment of the present application, 37 video clips can be obtained, of which 29 video clips are adopted by the subjects. In the education scenario, by processing a live video file of approximately 3 hours using the method of an embodiment of the present application, 8 video clips can be obtained, and all of these video clips are adopted by the subjects. It can be seen that the method of an embodiment of the present application is not only efficient, but also can obtain high-quality video clips, thereby improving the effectiveness and practical value of the method in practical applications.

[0194] Figure 8 FIG. 1 is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 8 As shown, the device includes:

[0195] An acquisition module 801 is used to acquire a video to be processed and obtain text content of the video to be processed;

[0196] The refining module 802 is used to extract the summary of the text content and obtain multiple target summary contents;

[0197] The identification and integration module 803 is configured to identify, from the text content, a target text segment whose semantic similarity with any target summary content exceeds a similarity threshold, and semantically integrate the target text segment with associated segments to obtain an integrated text segment, where the associated segment is at least one of the preceding and following contexts of the target text segment in the text content.

[0198] A screening module 804 is used to screen out key text segments from the plurality of integrated text segments;

[0199] The clipping module 805 is configured to clip the video segment containing the key text segment from the video to be processed based on the timestamp of the key text segment in the video to be processed.

[0200] In a possible implementation, the text content includes multiple sentence texts;

[0201] The acquisition module 801 is used to obtain audio information in the video to be processed; perform sentence segmentation on the audio information to obtain multiple sentence audios; and perform speech recognition on each sentence audio to obtain each sentence text.

[0202] In one possible implementation, the acquisition module 801 is used to frame the audio information to obtain multiple audio frames; for any audio frame, feature extraction is performed on any audio frame to obtain audio frame features, and the audio frame features are classified to obtain classification results, and the classification results are used to characterize whether any audio frame is a speech frame; based on the classification results of each audio frame, multiple sentence audios are determined.

[0203] In one possible implementation, the acquisition module 801 is used to determine each speech frame from multiple audio frames based on the classification results of each audio frame; for any speech frame other than the first speech frame, if the time interval between any speech frame and the previous speech frame of any speech frame is not greater than the set time, then any speech frame and the previous speech frame are merged into the same sentence audio; if the time interval between any speech frame and the previous speech frame is greater than the set time, then any speech frame and the previous speech frame are divided into different sentence audios.

[0204] In a possible implementation, any sentence audio includes multiple speech frames;

[0205] The acquisition module 801 is used to extract features of each speech frame included in any sentence audio to obtain features of each speech frame; map each speech frame feature to a corresponding character; and integrate the characters corresponding to each speech frame feature to obtain a sentence text.

[0206] In one possible implementation, the refining module 802 is used to segment the text content to obtain multiple first text segments; for any first text segment, perform summary refinement on any first text segment to obtain reference summary content; and integrate the reference summary contents to obtain multiple target summary contents.

[0207] In one possible implementation, the refining module 802 is used to construct a prompt text related to the content of the video to be processed; perform feature extraction on the prompt text and any first text segment to obtain a first text feature, where the first text feature is used to characterize the semantics of the prompt text and any first text segment; and generate reference summary content related to the semantics of the first text feature.

[0208] In one possible implementation, the identification and integration module 803 is configured to perform feature extraction on any target summary content to obtain a target summary feature, which is used to characterize the semantics of any target summary content; segment the text content to obtain multiple second text segments; perform feature extraction on any second text segment to obtain a second text feature, and determine the semantic similarity between any target summary content and any second text segment based on the target summary feature and the second text feature, where the second text feature is used to characterize the semantics of any second text segment; and select a target text segment from the multiple second text segments whose semantic similarity is higher than a similarity threshold.

[0209] In a possible implementation, the screening module 804 is configured to determine the quality index of each integrated text segment; and screen out key text segments whose quality indexes meet the index conditions from each integrated text segment.

[0210] In one possible implementation, the screening module 804 is configured to determine, for any integrated text segment, a quality indicator of any integrated text segment based on at least one of the first indicator, the second indicator, or the third indicator;

[0211] Among them, the first indicator is used to characterize the content importance of any integrated text segment in the scene to be processed, the second indicator is used to characterize the number of object interaction data in the video segment where any integrated text segment is located, and the third indicator is used to characterize the content similarity between any integrated text segment and the corresponding target summary content.

[0212] In the above-described device, target summary content is obtained by summarizing and refining the text content of the video to be processed. Since the target summary content is the subject matter of the text content, and the video to be processed and the text content correspond to the same subject matter, determining the target summary content allows for rapid determination of the video's subject matter. Next, target text segments whose semantic similarity with the target summary content exceeds a similarity threshold are identified from the text content, allowing for rapid location of the segment containing the subject matter. Next, the target text segment is semantically integrated with the context to obtain an integrated text segment, enriching the semantics of the integrated text segment within the context and ensuring semantic coherence and integrity. This allows for accurate subsequent screening of key text segments from the integrated text segment, improving the quality of the key text segments and ensuring that they contain the most compelling content. Subsequently, based on the timestamps of the key text segments in the video to be processed, video segments containing the key text segments are clipped from the video to be processed, allowing for rapid identification of the video segments containing the most compelling content. This process eliminates the need for manual editing, reduces the complexity of video processing, and improves video processing efficiency.

[0213] It should be understood that the above Figure 8 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0214] Figure 9 FIG2 shows a block diagram of a terminal device 900 provided by an exemplary embodiment of the present application. The terminal device 900 includes a processor 901 and a memory 902 .

[0215] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0216] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is executed by the processor 901 to implement the video processing method provided in the method embodiment of the present application.

[0217] In some embodiments, the terminal device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0218] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0219] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuits related to Near Field Communication (NFC), which is not limited in this application.

[0220] The display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 905 is a touch screen, it is also capable of collecting touch signals on or above the surface of the display screen 905. These touch signals can be input as control signals to the processor 901 for processing. In this case, the display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 905, located on the front panel of the terminal device 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of the terminal device 900 or in a foldable design. In other embodiments, the display screen 905 can be a flexible display, located on a curved or foldable surface of the terminal device 900. Furthermore, the display screen 905 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0221] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0222] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the terminal device 900. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0223] The power supply 908 is used to power the various components in the terminal device 900. The power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0224] In some embodiments, the terminal device 900 further includes one or more sensors 909 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , an optical sensor 914 , and a proximity sensor 915 .

[0225] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal device 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0226] The gyroscope sensor 912 can detect the body orientation and rotation angle of the terminal device 900. The gyroscope sensor 912 can work with the acceleration sensor 911 to collect the user's 3D movements of the terminal device 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (such as changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0227] The pressure sensor 913 can be set on the side frame of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal device 900, it can detect the user's grip signal of the terminal device 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0228] Optical sensor 914 is used to detect ambient light intensity. In one embodiment, processor 901 can control the display brightness of display screen 905 based on the ambient light intensity detected by optical sensor 914. Specifically, when the ambient light intensity is high, the display brightness of display screen 905 is increased; when the ambient light intensity is low, the display brightness of display screen 905 is decreased. In another embodiment, processor 901 can also dynamically adjust the shooting parameters of camera assembly 906 based on the ambient light intensity detected by optical sensor 914.

[0229] The proximity sensor 915, also known as a distance sensor, is typically located on the front panel of the terminal device 900. The proximity sensor 915 is used to detect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state. When the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0230] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the terminal device 900, and the terminal device 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0231] Figure 10This is a schematic diagram of the structure of the server provided in an embodiment of the present application. The server 1000 may vary significantly due to different configurations or performance, and may include one or more processors 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, which is loaded and executed by the one or more processors 1001 to implement the video processing methods provided in the above-mentioned various method embodiments. Exemplarily, the processor 1001 is a CPU. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.

[0232] In an exemplary embodiment, a computer-readable storage medium is further provided. The storage medium stores at least one computer program. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned video processing methods.

[0233] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0234] In an exemplary embodiment, a computer program is further provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned video processing methods.

[0235] In an exemplary embodiment, a computer program product is further provided. The computer program product stores at least one computer program. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned video processing methods.

[0236] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0237] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0238] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Obtaining a video to be processed and obtaining text content of the video to be processed; Summarizing and refining the text content to obtain multiple target summary contents; For any target summary content, identifying a target text segment from the text content whose semantic similarity with the target summary content is greater than a similarity threshold, and semantically integrating the target text segment with an associated segment to obtain an integrated text segment, wherein the associated segment is at least one of the preceding and following contexts of the target text segment in the text content; Filter out key text fragments from multiple integrated text fragments; Based on the timestamp of the key text segment in the video to be processed, a video segment containing the key text segment is clipped from the video to be processed.

2. The method according to claim 1, characterized in that The text content includes a plurality of sentence texts; and obtaining the text content of the video to be processed includes: Acquiring audio information from the video to be processed; Segmenting the audio information to obtain multiple sentence audios; Perform speech recognition on the audio of each sentence to obtain the text of each sentence.

3. The method according to claim 2, characterized in that The step of performing sentence segmentation on the audio information to obtain a plurality of sentence audios includes: framing the audio information to obtain a plurality of audio frames; For any audio frame, extract features of the audio frame to obtain audio frame features, classify the audio frame features to obtain a classification result, and use the classification result to indicate whether the audio frame is a speech frame; The plurality of sentence audios are determined based on the classification results of the respective audio frames.

4. The method according to claim 3, characterized in that The step of determining the plurality of sentence audios based on the classification results of the respective audio frames includes: Determining each speech frame from the plurality of audio frames based on the classification results of each audio frame; For any speech frame other than the first speech frame, if the time interval between the any speech frame and the previous speech frame of the any speech frame is not greater than the set time, then merge the any speech frame and the previous speech frame into the same sentence audio; If the time interval between any one of the speech frames and the previous speech frame is greater than the set time, the any one of the speech frames and the previous speech frame are divided into different sentence audios.

5. The method according to claim 2, characterized in that Any sentence audio includes multiple speech frames; the speech recognition is performed on each sentence audio to obtain each sentence text, including: For any of the sentence audios, performing feature extraction on each speech frame included in the any of the sentence audios to obtain features of each speech frame; Map each speech frame feature to the corresponding character; The characters corresponding to the features of each speech frame are integrated to obtain a sentence text.

6. The method according to claim 1, characterized in that The text content is summarized and refined to obtain multiple target summary contents, including: Segmenting the text content to obtain a plurality of first text segments; For any first text segment, performing summary extraction on the first text segment to obtain reference summary content; The reference summary contents are integrated to obtain multiple target summary contents.

7. The method according to claim 6, characterized in that The step of extracting a summary of any one of the first text segments to obtain reference summary content includes: Constructing prompt text related to the content of the video to be processed; Performing feature extraction on the prompt text and any one of the first text segments to obtain a first text feature, where the first text feature is used to characterize the semantics of the prompt text and any one of the first text segments; Generate reference summary content semantically related to the first text feature.

8. The method according to claim 1, characterized in that The step of identifying a target text segment from the text content, the target text segment having a semantic similarity with any target summary content being higher than a similarity threshold, includes: Performing feature extraction on any target summary content to obtain target summary features, wherein the target summary features are used to represent the semantics of any target summary content; Segmenting the text content to obtain a plurality of second text segments; For any second text segment, feature extraction is performed on the second text segment to obtain a second text feature, and based on the target summary feature and the second text feature, a semantic similarity between the target summary content and the second text segment is determined, where the second text feature is used to represent the semantics of the second text segment; A target text segment having a semantic similarity higher than a similarity threshold is selected from the plurality of second text segments.

9. The method according to any one of claims 1 to 8, characterized in that The step of screening out key text segments from the plurality of integrated text segments includes: Determine quality indicators for each integrated text segment; From the integrated text segments, key text segments whose quality indicators meet the indicator conditions are screened out.

10. The method according to claim 9, characterized in that The quality indicators of each integrated text segment are determined, including: For any integrated text segment, determining a quality indicator of the any integrated text segment based on at least one of the first indicator, the second indicator, or the third indicator; Among them, the first indicator is used to characterize the content importance of any integrated text segment in the scene to which the video to be processed belongs, the second indicator is used to characterize the amount of object interaction data in the video segment where any integrated text segment is located, and the third indicator is used to characterize the content similarity between any integrated text segment and the corresponding target summary content.

11. A video processing device, characterized in that: The device comprises: An acquisition module, used to acquire a video to be processed and acquire text content of the video to be processed; A refining module, configured to perform summary refining on the text content to obtain a plurality of target summary contents; an identification and integration module configured to, for any target summary content, identify, from the text content, a target text segment having a semantic similarity with the target summary content greater than a similarity threshold, and semantically integrate the target text segment with an associated segment to obtain an integrated text segment, wherein the associated segment is at least one of the preceding and following contexts of the target text segment in the text content; A screening module, used to screen out key text fragments from multiple integrated text fragments; The editing module is used to edit the video segment containing the key text segment from the video to be processed based on the timestamp of the key text segment in the video to be processed.

12. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, so that the electronic device implements the video processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement the video processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement the video processing method according to any one of claims 1 to 10.

Images