Audio and video subtitle generation method and device, electronic equipment and medium

Abstract

Provided are a method and device for generating audio-video subtitles, an electronic device, and a medium, which can be applied in the fields of big data and artificial intelligence. The method comprises: obtaining an audio-video stream in real time based on a current dialogue; performing a first preprocessing operation based on the audio-video stream to obtain audio-video numerical features; inputting the audio-video numerical features into a pre-trained audio-video information analysis model to obtain acoustic features and visual features, wherein the audio-video information analysis model is trained based on a multi-encoder transformer deep learning model; and obtaining a real-time corresponding subtitle for the audio-video stream based on the acoustic features and visual features and a pre-trained subtitle generation model. According to the method provided in the application, the delay of subtitle generation can be reduced, and the user experience is effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and big data technologies, and more specifically to a method, apparatus, electronic device, and medium for generating audio and video subtitles. Background Technology

[0002] With the development of digital and intelligent technologies, the demand for audio and video captioning systems in bank counter services is becoming increasingly urgent. Effective communication and information exchange between customers and bank staff are crucial in bank counter services. However, traditional verbal communication methods may present challenges in certain situations, such as with hearing-impaired customers, customers with foreign language difficulties, or in noisy environments.

[0003] To address these issues, audio and video captioning systems have been introduced into bank counter services to provide real-time captioning, helping customers better understand and participate in the process. However, existing automatic audio captioning systems are primarily designed and implemented for relatively complete audio segments, without optimizing for real-time audio and video interaction. This results in noticeable latency during real-time audio and video calls, creating a time lag between the captioning system and the video feed, thus inconveniencing the user experience. Furthermore, traditional captioning systems still face challenges such as low accuracy and a lack of predictive ability for specific business keywords, leading to inefficient use of audio and video information and relatively low caption quality. Summary of the Invention

[0004] In view of the above problems, according to a first aspect of the present invention, a method for generating audio and video subtitles is provided, characterized in that the method includes: acquiring an audio and video stream in real time based on a current dialogue; performing a first preprocessing operation on the audio and video stream to obtain audio and video numerical features; inputting the audio and video numerical features into a pre-trained audio and video information analysis model to obtain acoustic features and visual features, wherein the audio and video information analysis model is trained based on a multi-encoder transformer deep learning model; and acquiring real-time corresponding subtitles for the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model.

[0005] According to some exemplary embodiments, the first preprocessing operation based on the audio and video stream to obtain audio and video numerical features specifically includes: performing audio preprocessing on the audio stream in the audio and video stream to obtain audio data, wherein the audio preprocessing includes noise reduction, echo removal, and equalization; performing video preprocessing on the video stream in the audio and video stream to obtain video data, wherein the video preprocessing includes cropping, noise reduction, and frame rate control; and performing feature extraction based on the audio data and the video data to obtain audio and video numerical features.

[0006] According to some exemplary embodiments, the audio and video information analysis model includes an embedding layer and N encoder layers. The step of inputting the numerical features of the audio and video into the pre-trained audio and video information analysis model to obtain acoustic features and visual features specifically includes: using the embedding layer to convert the numerical features of the audio and video into a feature sequence; and inputting the feature sequence into the N encoder layers to obtain acoustic features and visual features using the self-attention mechanism and convolution operation of the N encoder layers, where N is a positive integer.

[0007] According to some exemplary embodiments, obtaining the real-time corresponding subtitles of the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model specifically includes: generating initial subtitle combination features based on the acoustic features, visual features, and an initial subtitle generation model.

[0008] According to some exemplary embodiments, the training process of the initial subtitle generation model specifically includes: collecting unlabeled data; obtaining a training dataset and a test dataset based on the unlabeled data; establishing an initial pre-trained model based on a transformer model; performing unsupervised pre-training on the initial pre-trained model using the training dataset to obtain an intermediate pre-trained model; and evaluating the intermediate pre-trained model using the test dataset, wherein when the evaluation value of the intermediate pre-trained model meets a preset threshold, the intermediate pre-trained model is used as the initial subtitle generation model.

[0009] According to some exemplary embodiments, obtaining real-time corresponding subtitles for the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model specifically includes: filtering based on initial subtitle combination features using a pre-trained keyword prediction model to obtain optimized subtitle combination features, wherein the keyword prediction model is trained based on a transformer model.

[0010] According to some exemplary embodiments, the step of filtering based on initial subtitle combination features using a pre-trained keyword prediction model specifically includes: inputting the initial subtitle combination features into the keyword prediction model to obtain predicted keywords; generating filtering rules based on the predicted keywords; and deleting features in the initial subtitle combination features that are unrelated to the predicted keywords based on the predicted keywords and the filtering rules.

[0011] According to some exemplary embodiments, obtaining real-time corresponding subtitles for the audio-video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model specifically includes: obtaining historical subtitle information of previous time nodes, wherein the previous time nodes are all time nodes where the dialogue occurred before the current dialogue or the time node before the dialogue occurred before the current dialogue; and obtaining real-time corresponding subtitles for the audio-video stream based on the historical subtitle information, optimized subtitle combination features, and the subtitle generation model.

[0012] According to some exemplary embodiments, after obtaining the real-time corresponding subtitles of the audio and video stream, the method includes: obtaining current dialogue records and user feedback information, wherein the current dialogue records include audio and video data and corresponding subtitle information; obtaining the user's satisfaction level based on the user feedback information; if the satisfaction level is higher than a first threshold, using the current dialogue records as tag data; and if the satisfaction level is lower than a second threshold, obtaining the correct subtitle information of the audio and video data, and retraining the audio and video information analysis model and the subtitle generation model based on the audio and video data and the correct subtitle information.

[0013] According to a second aspect of the present invention, an audio-visual subtitle generation apparatus is provided, the apparatus comprising: an audio-visual stream acquisition module, configured to: acquire an audio-visual stream in real time based on a current dialogue; an audio-visual numerical feature acquisition module, configured to: perform a first preprocessing operation based on the audio-visual stream to obtain audio-visual numerical features; an acoustic feature and visual feature acquisition module, configured to: input the audio-visual numerical features into a pre-trained audio-visual information analysis model to obtain acoustic features and visual features, wherein the audio-visual information analysis model is trained based on a multi-encoder transformer deep learning model; and a real-time corresponding subtitle acquisition module, configured to: acquire real-time corresponding subtitles for the audio-visual stream based on the acoustic features and visual features and a pre-trained subtitle generation model.

[0014] According to some exemplary embodiments, the audio and video numerical feature acquisition module may include an audio preprocessing unit, a video preprocessing unit, and an audio and video numerical feature extraction unit.

[0015] According to some exemplary embodiments, the audio preprocessing unit can be used to perform audio preprocessing on the audio stream in the audio and video stream to obtain audio data, wherein the audio preprocessing includes noise reduction, echo removal and equalization.

[0016] According to some exemplary embodiments, the video preprocessing unit can be used to perform video preprocessing on the video stream in the audio and video stream to obtain video data, wherein the video preprocessing includes cropping, noise reduction and frame rate control.

[0017] According to some exemplary embodiments, the audio and video numerical feature extraction unit can be used to extract features based on the audio data and the video data to obtain audio and video numerical features.

[0018] According to some exemplary embodiments, the acoustic and visual feature acquisition module may include a feature sequence acquisition unit and an acoustic and visual feature extraction unit.

[0019] According to some exemplary embodiments, the feature sequence acquisition unit can be used to convert the audio and video numerical features into feature sequences using the embedding layer.

[0020] According to some exemplary embodiments, the acoustic and visual feature extraction unit can be used to input the feature sequence into the N encoder layers and obtain acoustic and visual features using the self-attention mechanism and convolution operation of the N encoder layers, where N is a positive integer.

[0021] According to some exemplary embodiments, the real-time corresponding subtitle acquisition module may include an initial subtitle generation model training module, a keyword filtering module, and a subtitle data expansion module.

[0022] According to some exemplary embodiments, the initial subtitle generation model training module may include a data acquisition unit, a dataset acquisition unit, an initial pre-trained model building unit, a training unit, and an evaluation unit.

[0023] According to some exemplary embodiments, the data acquisition unit can be used to collect unlabeled data.

[0024] According to some exemplary embodiments, the dataset acquisition unit can be used to acquire training datasets and test datasets based on the unlabeled data.

[0025] According to some exemplary embodiments, the initial pre-trained model building unit can be used to build an initial pre-trained model based on a transformer model.

[0026] According to some exemplary embodiments, the training unit can be used to perform unsupervised pre-training on the initial pre-trained model using the training dataset to obtain an intermediate pre-trained model.

[0027] According to some exemplary embodiments, the evaluation unit can be used to evaluate the intermediate subtitle generation model using the test dataset, wherein when the evaluation value of the intermediate subtitle generation model meets a preset threshold, the intermediate subtitle generation model to be used as the initial subtitle generation model.

[0028] According to some exemplary embodiments, the keyword filtering module may include a predicted keyword acquisition unit, a filtering rule acquisition unit, and a subtitle text deletion unit.

[0029] According to some exemplary embodiments, the predicted keyword acquisition unit can be used to input the initial subtitle combination features into the keyword prediction model to obtain predicted keywords.

[0030] According to some exemplary embodiments, the filtering rule acquisition unit can be used to generate filtering rules based on the predicted keywords.

[0031] According to some exemplary embodiments, the subtitle text deletion unit can be used to delete features in the initial subtitle combination features that are unrelated to the predicted keywords based on the predicted keywords and filtering rules.

[0032] According to some exemplary embodiments, the subtitle data expansion module may include a historical subtitle information acquisition unit and an optimized real-time corresponding subtitle generation unit.

[0033] According to some exemplary embodiments, the historical caption information acquisition unit can be used to acquire historical caption information of previous time nodes, wherein the previous time nodes are all time nodes in which the dialogue occurred before the current dialogue or the previous time node in which the dialogue occurred before the current dialogue.

[0034] According to some exemplary embodiments, the optimized real-time corresponding subtitle generation unit can be used to obtain real-time corresponding subtitles for the audio and video stream based on the historical subtitle information, optimized subtitle combination features and the subtitle generation model.

[0035] According to some exemplary embodiments, the audio and video subtitle generation apparatus may further include a feedback information acquisition unit, a satisfaction level acquisition unit, a first judgment unit, and a second judgment unit.

[0036] According to some exemplary embodiments, the feedback information acquisition unit can be used to acquire the current dialogue record and user feedback information, wherein the current dialogue record includes audio and video data and corresponding subtitle information.

[0037] The satisfaction level acquisition unit can be used to acquire the user's satisfaction level based on the user feedback information.

[0038] According to some exemplary embodiments, the first determination unit may be used to treat the current dialogue record as tag data if the satisfaction level is higher than a first threshold.

[0039] According to some exemplary embodiments, the second judgment unit can be used to obtain the correct subtitle information of the audio and video data if the satisfaction level is lower than a second threshold, and retrain the audio and video information analysis model and the subtitle generation model based on the audio and video data and the correct subtitle information.

[0040] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0041] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0042] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0043] The above-described one or more embodiments have the following advantages or beneficial effects: According to the audio and video subtitle generation method provided by the present invention, by using an audio and video information analysis model and a subtitle generation model, it is possible to better understand the semantics and contextual relationships of the input data, which helps to generate more coherent and consistent text, thereby reducing the delay in subtitle generation, ensuring real-time performance, and thus effectively improving the user experience; in addition, it is possible to make full use of the complementarity between different modalities, thereby improving the comprehensive understanding and analysis capabilities of audio and video data. Attached Figure Description

[0044] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0045] Figure 1 The illustration schematically depicts an application scenario of the method, apparatus, device, and medium for generating audio and video subtitles according to embodiments of the present invention.

[0046] Figure 2 A flowchart illustrating a method for generating audio and video subtitles according to an embodiment of the present invention is shown.

[0047] Figure 3 A flowchart illustrating a method for a first preprocessing operation according to an embodiment of the present invention is shown schematically.

[0048] Figure 4 A flowchart illustrating a method for acquiring acoustic and visual features according to an embodiment of the present invention is shown.

[0049] Figure 5 A flowchart illustrating a training method for an initial subtitle generation model according to an embodiment of the present invention is shown.

[0050] Figure 6 The flowchart illustrates a method for screening based on a keyword prediction model according to an embodiment of the present invention.

[0051] Figure 7 A flowchart illustrating a method for optimizing and generating subtitles based on historical subtitle information according to an embodiment of the present invention is shown.

[0052] Figure 8 The flowchart illustrates a method for optimizing audio and video subtitle generation based on user satisfaction according to an embodiment of the present invention.

[0053] Figure 9 A schematic block diagram of an audio / video subtitle generation apparatus according to an embodiment of the present invention is shown.

[0054] Figure 10 A block diagram of an electronic device suitable for an audio / video subtitle generation method according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0055] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0057] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0058] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0059] In the technical solution of this invention, the acquisition, storage and application of user personal information all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0060] First, the technical terms used in this article are explained and clarified as follows.

[0061] Transformer: A deep learning model based on self-attention mechanism, widely used in natural language processing tasks such as machine translation, text generation, and language modeling. The core idea of ​​Transformer is to capture global dependencies in the input sequence through self-attention mechanism, without relying on traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs). It introduces attention mechanisms and multi-head attention mechanisms, as well as positional encoding to process sequential data.

[0062] Convolutional Neural Network (CNN): A deep learning model particularly well-suited for tasks involving grid-structured data, such as images and audio. The core idea of ​​CNNs is to extract features from input data through convolution and pooling operations. It introduces convolutional and pooling layers, as well as fully connected layers for classification or regression tasks.

[0063] Multi-encoder Transformer: An extension of the Transformer model that introduces multiple encoder layers to increase the model's representational and learning capabilities. In a traditional Transformer model, only one encoder layer processes the input sequence, while the multi-encoder Transformer builds a deeper model by stacking multiple encoder layers.

[0064] With the development and popularization of audio and video communication technology, real-time audio and video interaction has become an important part of people's daily life and work. In certain scenarios, such as bank counter services, real-time caption generation systems are introduced to provide better service and auxiliary functions, thereby helping customers with special circumstances, such as hearing impairments, difficulties in foreign language communication, or communication in noisy environments.

[0065] However, existing automatic audio captioning systems have the following problems: They are primarily designed and implemented for relatively complete audio segments, without optimizing for real-time audio-visual interaction. This results in noticeable latency during real-time audio-visual calls, creating a time difference between the captioning system and the video feed, inconveniencing the user experience. Furthermore, these systems only utilize the audio information from the audio and video streams, neglecting to use video information such as gestures, expressions, and actions that could improve caption quality, leading to inefficient use of audio and video information. Additionally, they directly generate captions from audio information without considering specialized vocabulary related to keywords in the audio. Therefore, due to the existence of homophones and other semantic features in Chinese, the quality of the generated captions is relatively low.

[0066] Based on this, embodiments of the present invention provide a method for generating audio and video subtitles. The method includes: acquiring an audio and video stream in real time based on the current dialogue; performing a first preprocessing operation on the audio and video stream to obtain audio and video numerical features; inputting the audio and video numerical features into a pre-trained audio and video information analysis model to obtain acoustic features and visual features, wherein the audio and video information analysis model is trained based on a multi-encoder transformer deep learning model; and obtaining real-time corresponding subtitles for the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model. According to the audio and video subtitle generation method provided by the present invention, by using the audio and video information analysis model and the subtitle generation model, the semantics and contextual relationships of the input data can be better understood, which helps to generate more coherent and consistent text, thereby reducing the delay in subtitle generation, ensuring real-time performance, and effectively improving the user experience. Furthermore, it can fully utilize the complementarity between different modalities to improve the comprehensive understanding and analysis capabilities of audio and video data.

[0067] It should be noted that the audio and video subtitle generation method, apparatus, device, and medium defined in this invention can be used in the fields of artificial intelligence technology and big data technology, as well as in the financial field, and can also be used in a variety of fields other than artificial intelligence technology, big data technology, and the financial field. The application fields of the audio and video subtitle generation method, apparatus, device, and medium provided in the embodiments of this invention are not limited.

[0068] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0069] In the technical solution of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0070] Figure 1 The illustration schematically depicts an application scenario of the method, apparatus, device, and medium for generating audio and video subtitles according to embodiments of the present invention.

[0071] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0072] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0073] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0074] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0075] It should be noted that the audio and video subtitle generation method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the gesture recognition-based particle image device provided in this embodiment of the invention can generally be located in server 105. The audio and video subtitle generation method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the gesture recognition-based particle image device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0076] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0077] Figure 2 A flowchart illustrating a method for generating audio and video subtitles according to an embodiment of the present invention is shown.

[0078] like Figure 2 As shown, the audio and video subtitle generation method 200 of this embodiment may include operations S210 to S240.

[0079] When operating S210, audio and video streams are acquired in real time based on the current conversation.

[0080] In embodiments of the present invention, voice signals from the current conversation can be acquired in real time using a microphone or other audio input device, wherein an audio interface or API can be used to obtain the real-time audio stream; simultaneously, video signals from the current conversation can be acquired in real time using a camera or other video input device, which involves obtaining the real-time video stream using a video interface or API.

[0081] In embodiments of the present invention, by combining information from different modalities, the correlation and interaction between audio and video can be better captured, so as to avoid the video information such as gestures, expressions, and actions in the video stream that may have a certain effect on the subtitle generation system and can improve the quality of the generated subtitles being unused, thereby generating subtitles that are more in line with the actual situation.

[0082] In operation S220, a first preprocessing operation is performed based on the audio and video stream to obtain numerical features of the audio and video.

[0083] In an embodiment of the present invention, in order to achieve data standardization and facilitate computer processing and analysis, a first preprocessing operation is required on the audio and video streams.

[0084] Figure 3 A flowchart illustrating a method for a first preprocessing operation according to an embodiment of the present invention is shown schematically.

[0085] like Figure 3 As shown, the method of the first preprocessing operation in this embodiment may include operations S310 to S330.

[0086] In operation S310, audio preprocessing is performed on the audio stream in the audio and video stream to obtain audio data. The audio preprocessing includes noise reduction, echo removal, and equalization.

[0087] Specifically, noise reduction removes background noise and interference, which can be achieved by applying noise reduction algorithms, such as spectral subtraction and wavelet denoising. Noise reduction can improve audio quality, making subsequent feature extraction more accurate and reliable. In some audio scenarios, there is an echo problem, which is a repetitive signal generated by the reflection of the audio signal. Removing echo is to eliminate the repetitive signal in the audio and improve the clarity and intelligibility of the audio. Audio equalization is to adjust the frequency response of the audio signal to have a balanced volume and sound quality across different frequency bands. This can be achieved by applying an equalizer, such as a graphic equalizer or an adaptive equalizer.

[0088] It should be noted that the methods listed here are merely exemplary and are not intended to limit the reference methods for audio preprocessing in the embodiments of the present invention. That is, other methods may also be used for performing the audio preprocessing in the embodiments of the present invention.

[0089] In operation S320, video preprocessing is performed on the video stream in the audio and video stream to obtain video data. The video preprocessing includes cropping, noise reduction, and frame rate control.

[0090] Specifically, a common operation in video preprocessing is cropping, which involves selecting the video region of interest and removing irrelevant parts. Cropping can be achieved by specifying the video's temporal range and spatial coordinates to retain task-relevant content. Noise reduction in video preprocessing aims to reduce noise, artifacts, and other visual noise in the video. This can be achieved by applying noise reduction algorithms, such as video smoothing filters or denoising neural network models. Noise reduction helps improve the visual quality and clarity of the video. Frame rate control adjusts the video's frame rate, which can be achieved by increasing or decreasing the number of frames. Frame rate control can be used to adjust the video's playback speed, smoothness, and storage requirements to suit different application scenarios and device requirements.

[0091] It should be noted that the methods listed here are merely exemplary and are not intended to limit the reference methods for video preprocessing in the embodiments of the present invention. That is, other methods may also be used for video preprocessing in the embodiments of the present invention.

[0092] In operation S330, feature extraction is performed based on the audio data and the video data to obtain numerical features of the audio and video.

[0093] In embodiments of the present invention, feature extraction based on audio and video data involves transforming the original audio and video information into numerical feature representations. These feature representations can be used for model training and application to achieve subsequent audio and video subtitle generation tasks. Specifically, basic time-domain features of the audio signal, such as amplitude, energy, and zero-crossing rate, can be extracted through time-domain analysis, and frequency-domain features and / or spectral features can be extracted through Fourier transform. Frame-level features can be obtained by extracting each frame in the video sequence, and motion features in the video can be obtained by comparing the differences between consecutive frames.

[0094] Return to reference Figure 2 In operation S230, the numerical features of the audio and video are input into a pre-trained audio and video information analysis model to obtain acoustic features and visual features. The audio and video information analysis model is trained based on a multi-encoder transformer deep learning model.

[0095] Figure 4 A flowchart illustrating a method for acquiring acoustic and visual features according to an embodiment of the present invention is shown.

[0096] like Figure 4 As shown, the method for acquiring acoustic and visual features in this embodiment may include operations S410 to S420.

[0097] In operation S410, the embedding layer is used to convert the audio and video numerical features into a feature sequence.

[0098] In embodiments of the present invention, an embedding layer is a technique for mapping discrete symbols to a continuous vector space. In audio and video processing, an embedding layer can be used to transform the numerical features of audio and video into continuous embedded vector representations. These embedded vectors have semantic relevance in the vector space and can capture the semantic information of audio and video.

[0099] In embodiments of the present invention, after converting the numerical features of audio and video into embedding vectors through an embedding layer, a feature sequence can be obtained. Specifically, the feature sequence can arrange the features of audio and video in chronological order to form sequential data, with each time step corresponding to an embedding vector.

[0100] It should be noted that the methods listed here are merely exemplary and are not intended to limit the method of forming the feature sequence in the embodiments of the present invention. That is, the formation of the feature sequence in the embodiments of the present invention can also refer to other methods.

[0101] According to embodiments of the present invention, the embedding layer can map high-dimensional discrete input data into low-dimensional continuous vector representations, thereby reducing the complexity and number of parameters of the model; at the same time, by learning the distributed representation of the embedding vectors, the semantic relationships and similarities between input data can be captured, enabling the model to better understand the semantics of the input data.

[0102] In operation S420, the feature sequence is input into the N encoder layers, and acoustic and visual features are obtained by using the self-attention mechanism and convolution operation of the N encoder layers, where N is a positive integer.

[0103] In embodiments of the present invention, each of the N encoder layers consists of a self-attention mechanism and a feedforward neural network. By stacking N encoder layers, the audio-visual information analysis model can progressively extract and integrate the feature information of audio and video.

[0104] Specifically, self-attention mechanisms can be used to establish relationships within audio and video sequences. They work by calculating attention weights between different positions in the feature sequence, allowing information from that position to interact and aggregate with information from other positions. In this way, the embedding vector at each position can incorporate semantic information from surrounding positions, thus capturing dependencies between different positions in the feature sequence and extracting global features from the audio and video.

[0105] Specifically, convolution operations can be used to extract local features from feature sequences, capturing local patterns and details in audio and video. Specifically, local feature representations can be obtained by performing a sliding window convolution operation on the neighboring locations of the feature sequence.

[0106] In embodiments of the present invention, by combining a self-attention mechanism and a convolution operation, N encoder layers can further acquire acoustic and visual features.

[0107] According to embodiments of the present invention, information extraction from audio and video streams is achieved using a multi-encoder transformer deep learning model, which enables the extraction of audio and video stream information with lower latency and higher quality.

[0108] Return to reference Figure 2 In operation S240, based on the acoustic and visual features and the pre-trained subtitle generation model, the real-time corresponding subtitles of the audio and video stream are obtained.

[0109] In embodiments of the present invention, a subtitle generation model can be pre-trained based on a convolutional neural network and generated by connecting it to the output layer.

[0110] Furthermore, in order to improve the quality and accuracy of subtitle combinations, as well as to ensure contextual consistency and semantic relevance, operation S240 can be improved by using a pre-trained model, a keyword prediction model, and an improved scheme that incorporates historical subtitle information from preceding time nodes.

[0111] It should be noted that the above-mentioned improvement schemes, which use pre-trained models, keyword prediction models, and historical subtitle information from preceding time nodes, can be used in parallel or in a certain order; they can be used individually or two of the improvement schemes can be applied.

[0112] The following explanation uses the complete improvement scheme, which incorporates a pre-trained model, a keyword prediction model, and historical subtitle information from preceding time nodes, as an example.

[0113] In an embodiment of the present invention, obtaining the real-time corresponding subtitles of the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model specifically includes: generating initial subtitle combination features based on the acoustic features, visual features, and an initial subtitle generation model.

[0114] In embodiments of this invention, the pre-trained model is a technique for training on large-scale unlabeled data. It acquires a representation with strong generalization ability by learning the statistical structure and semantic information of the data. During unsupervised pre-training, the model is trained using unlabeled data and does not rely on any task-specific labels or guidance signals. Therefore, during data collection, large-scale unlabeled data can be collected, which can be text, images, audio, or other forms of data. This data is typically obtained from sources such as the Internet, text corpora, and image databases.

[0115] Figure 5 A flowchart illustrating a training method for an initial subtitle generation model according to an embodiment of the present invention is shown.

[0116] like Figure 5 As shown, the training method for the initial subtitle generation model in this embodiment may include operations S510 to S550.

[0117] When operating the S510, unlabeled data is collected.

[0118] In operation S520, training datasets and test datasets are obtained based on the unlabeled data.

[0119] When operating the S530, an initial pre-trained model is built based on the transformer model.

[0120] In operation S540, the initial pre-trained model is unsupervised pre-trained using the training dataset to obtain an intermediate pre-trained model.

[0121] In embodiments of the present invention, a model is trained to learn the intrinsic representation or features of unlabeled data. A common method in unsupervised pre-training is to learn statistical features of language and vision, as well as the latent structure in unlabeled data, by maximizing the likelihood of the data.

[0122] In operation S550, the intermediate subtitle generation model is evaluated using the test dataset. When the evaluation value of the intermediate subtitle generation model meets a preset threshold, the intermediate subtitle generation model to be used as the initial subtitle generation model.

[0123] In embodiments of the present invention, model performance can be evaluated by assessing the quality of generated samples. Specifically, the following metrics can be used to evaluate the model: accuracy, precision, recall, F1 score, etc. Accuracy refers to the proportion of correctly predicted samples out of the total number of samples, calculated as: Accuracy = (Number of correctly predicted samples) / (Total number of samples). Accuracy is one of the most commonly used evaluation metrics, particularly suitable for situations where the sample categories are evenly distributed; alternatively, automated evaluation metrics such as BLEU and ROUGE can be used to measure the quality of text generation.

[0124] In embodiments of the present invention, the advantage of unsupervised pre-training is that it can utilize a large amount of unlabeled data for model training, learn the latent representation of the data, and help extract more useful features, which can bring performance improvements in subsequent tasks, especially when labeled data is scarce.

[0125] In an embodiment of the present invention, obtaining the real-time corresponding subtitles of the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model further includes: using a pre-trained keyword prediction model to filter based on the initial subtitle combination features to obtain optimized subtitle combination features, wherein the keyword prediction model is trained based on a transformer model.

[0126] In embodiments of the present invention, the keyword prediction model can predict keywords that may be related to the audio and video content based on the characteristics and contextual information of the audio and video content. These keywords may cover proprietary terms in the relevant field, the theme of the audio, the theme and sentiment of the video, and other information.

[0127] Figure 6 The flowchart illustrates a method for screening based on a keyword prediction model according to an embodiment of the present invention.

[0128] like Figure 6 As shown, the method for filtering based on a keyword prediction model in this embodiment may include operations S610 to S630.

[0129] In operation S610, the initial subtitle combination features are input into the keyword prediction model to obtain predicted keywords.

[0130] In operation S620, filtering rules are generated based on the predicted keywords.

[0131] In operation S630, based on the predicted keywords and filtering rules, features in the initial subtitle combination features that are unrelated to the predicted keywords are deleted.

[0132] In embodiments of the present invention, the filtering rules are conditions set based on predicted keywords and task requirements, used to filter and delete subtitle text unrelated to the predicted keywords in the initial subtitle combination features. The filtering rules may include the following: considering specialized vocabulary in the keyword's domain, retaining subtitle text highly relevant to the predicted keywords and deleting text unrelated to the keywords; performing sentiment filtering on the subtitle text based on the sentiment tendency of the predicted keywords, deleting text that does not meet the requirements; performing topic filtering on the subtitle text based on the theme or topic of the predicted keywords, deleting text unrelated to the theme; setting a range for the length of the subtitle text, deleting text that is too long or too short; and checking the grammatical correctness of the subtitle text, deleting text that does not conform to the grammatical rules.

[0133] In embodiments of the present invention, by applying filtering rules, the parts of the initial subtitle combination features that are irrelevant to the predicted keywords can be filtered out, retaining the optimal subtitle text that is relevant to the audio and video content and meets the task requirements. This can improve the accuracy and quality of subsequent subtitle generation, making the generated subtitles more precise and in line with user expectations.

[0134] Figure 7 A flowchart illustrating a method for optimizing and generating subtitles based on historical subtitle information according to an embodiment of the present invention is shown.

[0135] like Figure 7 As shown, the method for optimizing and generating subtitles based on historical subtitle information in this embodiment may include operations S710 to S720.

[0136] In operation S710, historical caption information of previous time nodes is obtained, wherein the previous time nodes are all time nodes in which the dialogue occurred before the current dialogue or the previous time node in which the dialogue occurred before the current dialogue.

[0137] In operation S720, based on the historical subtitle information, optimized subtitle combination features, and the subtitle generation model, the real-time corresponding subtitles of the audio and video stream are obtained.

[0138] In embodiments of the present invention, historical subtitle information can be used for contextual understanding and coherence maintenance, providing more comprehensive and coherent content when generating real-time corresponding subtitles.

[0139] In an embodiment of the present invention, a feature fusion layer can be introduced into the subtitle generation model to fuse features extracted from historical subtitle information and optimized subtitle combination features, and finally generate subtitles through the output layer of the subtitle generation model.

[0140] In addition, to further improve the accuracy of subtitle generation, relevant business data generated during business processing can be obtained based on the evaluations and feedback from tellers and customers regarding the system, and the audio and video information analysis model and subtitle generation model can be continuously trained and optimized.

[0141] Figure 8 The flowchart illustrates a method for optimizing audio and video subtitle generation based on user satisfaction according to an embodiment of the present invention.

[0142] like Figure 8 As shown, the method for optimizing audio and video subtitle generation based on user satisfaction in this embodiment may include operations S810 to S840.

[0143] In operation S810, the current dialogue record and user feedback information are obtained, wherein the current dialogue record includes audio and video data and corresponding subtitle information.

[0144] In embodiments of the present invention, user feedback can be obtained in various ways, such as user ratings, user comments, or other forms of feedback mechanisms.

[0145] In operation S820, the user's satisfaction level is obtained based on the user feedback information.

[0146] In embodiments of the present invention, user feedback information can be used to evaluate the accuracy and quality of the model, and to understand users' needs and satisfaction with the subtitles. Satisfaction levels can be quantitative rating indicators or qualitative user opinions and suggestions. If the satisfaction level is a qualitative user opinion or suggestion, it can be converted into a quantitative representation.

[0147] In operation S830, if the satisfaction level is higher than the first threshold, the current dialogue record is used as tag data.

[0148] In an embodiment of the present invention, if the satisfaction level is higher than a set first threshold, it means that the user is satisfied with the current dialogue record and the generated subtitles. In this case, the current dialogue record can be used as tag data for training and improving the model, which can increase the amount of training data for the model and improve model performance and generalization ability.

[0149] In operation S840, if the satisfaction level is lower than the second threshold, the correct subtitle information of the audio and video data is obtained, and the audio and video information analysis model and the subtitle generation model are retrained based on the audio and video data and the correct subtitle information.

[0150] In an embodiment of the present invention, if the satisfaction level is lower than a set second threshold, it means that the user is dissatisfied with the current dialogue record and the generated subtitles. In this case, correct subtitle information can be obtained based on audio and video data. Based on the correct subtitle information and audio and video data, the audio and video information analysis model and the subtitle generation model can be retrained. Through retraining, the models can learn and correct previous errors to improve the accuracy and quality of generated subtitles.

[0151] The audio-visual subtitle generation method provided by this invention overcomes the shortcomings of current automatic audio subtitle generation systems trained on large-scale audio datasets, such as high latency, low utilization of audio-visual information, and lack of domain-specific information in the generated subtitles. By pre-training audio-visual records to obtain a low-latency multi-encoder transformer audio-visual feature extraction model, and obtaining a subtitle generation deep learning model from the corresponding text records of the audio-visual content, this method can flexibly meet the need for real-time subtitle generation based on audio-visual information streams during various counter service transactions, and specifically brings the following beneficial effects:

[0152] 1. A multi-encoder transformer deep learning model was used to analyze audio and video stream information, which can generate visual features and better understand the semantics and contextual relationships of the input data. This helps to generate more coherent and consistent text, thereby reducing the delay in subtitle generation and effectively improving the user experience.

[0153] 2. By combining audio and video information and using multiple models for subtitle generation, the error of a single model can be reduced, resulting in more accurate subtitle output;

[0154] 3. By combining information from different modalities, the correlation and interaction between audio and video can be better captured, generating subtitles that are more relevant to the actual situation;

[0155] 4. Based on a large amount of data such as audio and video information flow and dialogue text records in the business process, supplemented by the evaluation and feedback of business personnel and customers, a subtitle generation model for counter business processing can be obtained through extensive training. There is no need to manually mark training data and parameters, which makes it highly easy to use.

[0156] Figure 9 A schematic block diagram of an audio / video subtitle generation apparatus according to an embodiment of the present invention is shown.

[0157] like Figure 9As shown, the audio and video subtitle generation apparatus 900 according to this embodiment includes an audio and video stream acquisition module 910, an audio and video numerical feature acquisition module 920, an acoustic feature and visual feature acquisition module 930, and a real-time corresponding subtitle acquisition module 940.

[0158] The audio and video stream acquisition module 910 can be used to acquire audio and video streams in real time based on the current dialogue. In one embodiment, the audio and video stream acquisition module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0159] The audio / video numerical feature acquisition module 920 can be used to perform a first preprocessing operation based on the audio / video stream to obtain audio / video numerical features. In one embodiment, the audio / video numerical feature acquisition module 920 can be used to execute the operation S220 described above, which will not be repeated here.

[0160] The acoustic and visual feature acquisition module 930 can be used to input the numerical features of the audio and video into a pre-trained audio and video information analysis model to obtain acoustic and visual features. The audio and video information analysis model is trained based on a multi-encoder transformer deep learning model. In one embodiment, the acoustic and visual feature acquisition module 930 can be used to perform the operation S230 described above, which will not be repeated here.

[0161] The real-time corresponding subtitle acquisition module 940 can be used to acquire real-time corresponding subtitles for the audio and video stream based on the acoustic features, visual features, and a pre-trained subtitle generation model. In one embodiment, the real-time corresponding subtitle acquisition module 940 can be used to perform the operation S240 described above, which will not be repeated here.

[0162] According to an embodiment of the present invention, the audio and video numerical feature acquisition module 920 may include an audio preprocessing unit, a video preprocessing unit, and an audio and video numerical feature extraction unit.

[0163] The audio preprocessing unit can be used to perform audio preprocessing on the audio stream in the audio / video stream to obtain audio data. The audio preprocessing includes noise reduction, echo removal, and equalization. In one embodiment, the audio preprocessing unit can be used to perform the operation S310 described above, which will not be repeated here.

[0164] The video preprocessing unit can be used to perform video preprocessing on the video stream in the audio and video stream to obtain video data. The video preprocessing includes cropping, noise reduction, and frame rate control. In one embodiment, the video preprocessing unit can be used to perform the operation S320 described above, which will not be repeated here.

[0165] The audio and video numerical feature extraction unit can be used to extract features based on the audio data and the video data to obtain audio and video numerical features. In one embodiment, the audio and video numerical feature extraction unit can be used to perform the operation S330 described above, which will not be repeated here.

[0166] According to an embodiment of the present invention, the acoustic and visual feature acquisition module 930 may include a feature sequence acquisition unit and an acoustic and visual feature extraction unit.

[0167] The feature sequence acquisition unit can be used to convert the audio and video numerical features into a feature sequence using the embedding layer. In one embodiment, the feature sequence acquisition unit can be used to perform the operation S410 described above, which will not be repeated here.

[0168] The acoustic and visual feature extraction unit can be used to input the feature sequence into the N encoder layers, and obtain acoustic and visual features using the self-attention mechanism and convolution operation of the N encoder layers, where N is a positive integer. In one embodiment, the acoustic and visual feature extraction unit can be used to perform the operation S420 described above, which will not be repeated here.

[0169] According to an embodiment of the present invention, the real-time corresponding subtitle acquisition module 940 may include an initial subtitle generation model training module, a keyword filtering module, and a subtitle data expansion module.

[0170] The initial subtitle generation model training module may include a data acquisition unit, a dataset acquisition unit, an initial pre-trained model building unit, a training unit, and an evaluation unit.

[0171] The data acquisition unit can be used to collect unlabeled data. In one embodiment, the data acquisition unit can be used to perform the operation S510 described above, which will not be repeated here.

[0172] The dataset acquisition unit can be used to acquire training and testing datasets based on the unlabeled data. In one embodiment, the dataset acquisition unit can be used to perform the operation S520 described above, which will not be repeated here.

[0173] The initial pre-trained model building unit can be used to build an initial pre-trained model based on the transformer model. In one embodiment, the initial pre-trained model building unit can be used to perform the operation S530 described above, which will not be repeated here.

[0174] The training unit can be used to perform unsupervised pre-training on the initial pre-trained model using the training dataset to obtain an intermediate pre-trained model. In one embodiment, the training unit can be used to perform the operation S540 described above, which will not be repeated here.

[0175] The evaluation unit can be used to evaluate the intermediate subtitle generation model using the test dataset. When the evaluation value of the intermediate subtitle generation model meets a preset threshold, the model to be used as the initial subtitle generation model. In one embodiment, the evaluation unit can be used to perform the operation S550 described above, which will not be repeated here.

[0176] The keyword filtering module may include a predicted keyword acquisition unit, a filtering rule acquisition unit, and a subtitle text deletion unit.

[0177] The predicted keyword acquisition unit can be used to input the initial subtitle combination features into the keyword prediction model to obtain predicted keywords. In one embodiment, the predicted keyword acquisition unit can be used to perform the operation S610 described above, which will not be repeated here.

[0178] The filtering rule acquisition unit can be used to generate filtering rules based on the predicted keywords. In one embodiment, the filtering rule acquisition unit can be used to perform the operation S620 described above, which will not be repeated here.

[0179] The subtitle text deletion unit can be used to delete features in the initial subtitle combination features that are unrelated to the predicted keywords, based on the predicted keywords and filtering rules. In one embodiment, the subtitle text deletion unit can be used to perform the operation S630 described above, which will not be repeated here.

[0180] The subtitle data extension module may include a historical subtitle information acquisition unit and an optimized real-time corresponding subtitle generation unit.

[0181] The historical caption information acquisition unit can be used to acquire historical caption information of previous time nodes, wherein the previous time nodes are all time nodes where the dialogue occurred before the current dialogue or the time node before the dialogue occurred before the current dialogue. In one embodiment, the historical caption information acquisition unit can be used to perform the operation S710 described above, which will not be repeated here.

[0182] The optimized real-time corresponding subtitle generation unit can be used to obtain real-time corresponding subtitles for the audio and video stream based on the historical subtitle information, optimized subtitle combination features, and the subtitle generation model. In one embodiment, the optimized real-time corresponding subtitle generation unit can be used to perform the operation S720 described above, which will not be repeated here.

[0183] According to an embodiment of the present invention, the audio and video subtitle generation device 900 may further include a feedback information acquisition unit, a satisfaction level acquisition unit, a first judgment unit, and a second judgment unit.

[0184] The feedback information acquisition unit can be used to acquire the current dialogue record and user feedback information, wherein the current dialogue record includes audio and video data and corresponding subtitle information. In one embodiment, the feedback information acquisition unit can be used to perform the operation S810 described above, which will not be repeated here.

[0185] The satisfaction level acquisition unit can be used to acquire the user's satisfaction level based on the user feedback information. In one embodiment, the satisfaction level acquisition unit can be used to perform the operation S820 described above, which will not be repeated here.

[0186] The first determination unit can be used to treat the current dialogue record as tag data if the satisfaction level is higher than a first threshold. In one embodiment, the first determination unit can be used to perform the operation S830 described above, which will not be repeated here.

[0187] The second judgment unit can be used to obtain the correct subtitle information of the audio and video data if the satisfaction level is lower than a second threshold, and retrain the audio and video information analysis model and the subtitle generation model based on the audio and video data and the correct subtitle information. In one embodiment, the second judgment unit can be used to perform the operation S840 described above, which will not be repeated here.

[0188] Figure 10 A block diagram of an electronic device suitable for an audio / video subtitle generation method according to an embodiment of the present invention is shown schematically.

[0189] like Figure 10 As shown, an electronic device 1000 according to an embodiment of the present invention includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0190] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0191] According to an embodiment of the present invention, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0192] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0193] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0194] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0195] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0196] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0197] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0198] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0200] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for generating audio and video subtitles, characterized in that, The method includes: Based on the current conversation, acquire audio and video streams in real time; Based on the audio and video streams, a first preprocessing operation is performed to obtain numerical features of the audio and video. The numerical features of the audio and video are input into a pre-trained audio and video information analysis model to obtain acoustic and visual features. The audio and video information analysis model is trained based on a multi-encoder transformer deep learning model. Based on the acoustic and visual features and a pre-trained subtitle generation model, real-time corresponding subtitles for the audio and video stream are obtained, including: generating initial subtitle combination features; filtering based on the initial subtitle combination features using a pre-trained keyword prediction model to obtain optimized subtitle combination features, wherein the keyword prediction model is trained based on a transformer model; obtaining historical subtitle information of previous time nodes, wherein the previous time nodes are all time nodes where dialogues occurred before the current dialogue or the time node preceding the current dialogue; and obtaining real-time corresponding subtitles for the audio and video stream based on the historical subtitle information, optimized subtitle combination features, and the subtitle generation model. The step of filtering based on initial subtitle combination features using a pre-trained keyword prediction model specifically includes: inputting the initial subtitle combination features into the keyword prediction model to obtain predicted keywords; generating filtering rules based on the predicted keywords; and deleting features from the initial subtitle combination features that are irrelevant to the predicted keywords based on the predicted keywords and the filtering rules. After obtaining the real-time corresponding subtitles of the audio and video stream, the method includes: obtaining the current dialogue record and user feedback information, wherein the current dialogue record includes audio and video data and corresponding subtitle information; obtaining the user's satisfaction level based on the user feedback information; if the satisfaction level is higher than a first threshold, using the current dialogue record as tag data; and if the satisfaction level is lower than a second threshold, obtaining the correct subtitle information of the audio and video data, and retraining the audio and video information analysis model and the subtitle generation model based on the audio and video data and the correct subtitle information.

2. The method according to claim 1, characterized in that, The first preprocessing operation based on the audio and video stream to obtain numerical features of the audio and video specifically includes: The audio stream in the audio and video stream is preprocessed to obtain audio data, wherein the audio preprocessing includes noise reduction, echo removal and equalization; The video stream in the audio and video stream is preprocessed to obtain video data, wherein the video preprocessing includes cropping, noise reduction, and frame rate control; and Based on the audio data and the video data, feature extraction is performed to obtain numerical features of the audio and video.

3. The method according to claim 1, characterized in that, The audio-visual information analysis model includes an embedding layer and N encoder layers. The step of inputting the numerical features of the audio and video into the pre-trained audio-visual information analysis model to obtain acoustic and visual features specifically includes: Using the embedding layer, the numerical features of the audio and video are transformed into a feature sequence; and The feature sequence is input into the N encoder layers, and acoustic and visual features are obtained by utilizing the self-attention mechanism and convolution operation of the N encoder layers, where N is a positive integer.

4. The method according to claim 1, characterized in that, The features for generating the initial subtitle combination specifically include: Based on the acoustic and visual features and the initial subtitle generation model, initial subtitle combination features are generated.

5. The method according to claim 4, characterized in that, The training process of the initial subtitle generation model specifically includes: Collect unlabeled data; Based on the unlabeled data, obtain the training dataset and the test dataset; Based on the transformer model, an initial pre-trained model is established; Using the training dataset, the initial pre-trained model is unsupervised pre-trained to obtain an intermediate pre-trained model; and The intermediate pre-trained model is evaluated using the test dataset. When the evaluation value of the intermediate pre-trained model meets a preset threshold, the intermediate pre-trained model is used as the initial subtitle generation model.

6. An audio / video subtitle generation device, characterized in that, The device includes: The audio and video stream acquisition module is used to: acquire audio and video streams in real time based on the current conversation; The audio and video numerical feature acquisition module is used to: perform a first preprocessing operation based on the audio and video stream to obtain audio and video numerical features; The acoustic and visual feature acquisition module is used to: input the numerical features of the audio and video into a pre-trained audio and video information analysis model to obtain acoustic and visual features, wherein the audio and video information analysis model is trained based on a multi-encoder transformer deep learning model; The real-time corresponding subtitle acquisition module is used to: acquire real-time corresponding subtitles for the audio and video stream based on the acoustic and visual features and a pre-trained subtitle generation model, including: generating initial subtitle combination features; and using a pre-trained keyword prediction model to filter and obtain optimized subtitle combination features based on the initial subtitle combination features, wherein the keyword prediction model is trained based on a transformer model; the real-time corresponding subtitle acquisition module includes a keyword filtering module and a subtitle data expansion module. The keyword filtering module includes a predicted keyword acquisition unit, a filtering rule acquisition unit, and a subtitle text deletion unit. The predicted keyword acquisition unit is used to: input the initial subtitle combination features into the keyword prediction model to obtain predicted keywords; the filtering rule acquisition unit is used to: generate filtering rules based on the predicted keywords; and the subtitle text deletion unit is used to: delete features in the initial subtitle combination features that are unrelated to the predicted keywords based on the predicted keywords and the filtering rules. The subtitle data extension module includes a historical subtitle information acquisition unit and an optimized real-time corresponding subtitle generation unit. The historical subtitle information acquisition unit is used to: acquire historical subtitle information of previous time nodes, wherein the previous time nodes are all time nodes in which the dialogue occurred before the current dialogue or the time node before the dialogue occurred before the current dialogue; the optimized real-time corresponding subtitle generation unit is used to: acquire real-time corresponding subtitles of the audio and video stream based on the historical subtitle information, optimized subtitle combination features and the subtitle generation model. The feedback information acquisition unit is used to: acquire the current dialogue record and user feedback information, wherein the current dialogue record includes audio and video data and corresponding subtitle information; The satisfaction level acquisition unit is used to: acquire the user's satisfaction level based on the user feedback information; The first judgment unit is used to: if the satisfaction level is higher than the first threshold, use the current dialogue record as tag data; The second judgment unit is used to: if the satisfaction level is lower than the second threshold, obtain the correct subtitle information of the audio and video data, and retrain the audio and video information analysis model and the subtitle generation model based on the audio and video data and the correct subtitle information.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images