Real-time subtitle generation method and system based on AI and voice mouse

Abstract

The invention discloses a real-time subtitle generation method and system based on AI and a voice mouse, and the method comprises the steps: collecting an audio signal of a video stream based on the voice mouse, and obtaining a video stream data set; performing data processing on the acquisition result of the video stream data set according to the acquisition result of the first audio analysis model of the voice mouse on the video stream data set and the audio change information, and then determining a target data set; and after target detection is carried out on the audio of the video stream collected in real time by using the second audio analysis model, a real-time scene subtitle analysis result is generated, and the scene subtitle analysis result is judged. The video stream data can be obtained based on the voice mouse, and the analysis result is generated through analysis after the data is called through the AI; and corresponding judgment and generation are performed according to the analysis result, and a subtitle optimization strategy is added in the AI analysis process, so that the generation effect of the video streams in different scene environments is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a real-time subtitle generation method based on AI and voice mouse and a system thereof. Background Art

[0002] With the development of mobile Internet, short video live broadcasts and recorded programs have gradually been accepted by the public and widely used in all aspects of life. Corresponding live broadcast tools, systems and functions have also emerged. In order to meet people's more usage needs, these applications and systems have set up AI subtitle functions, such as: AI subtitle recognition for recorded live videos after the live broadcast. At present, there are tools to assist video creators in generating subtitles based on the captured videos, but this method requires users to import the captured videos into the tool, and use the tool to analyze the sound in the video to generate subtitles. It is impossible to generate real-time subtitles when the user is recording a video or broadcasting live.

[0003] At the same time, the live broadcast tools and systems currently on the market can meet the subtitle requirements for live and recorded broadcasts, and the technology is based on AI voice recognition and OCR text recognition. However, AI voice recognition and OCR text recognition are both highly accurate in specific scenarios, but when a new application scenario is changed, the deviation rate of the recognition effect will increase to 80% or even 100%. The live broadcast field requires real-time subtitles, and the live broadcast scene is constantly changing. The scene switching and the change of the voice person make the recognition of live subtitles very difficult and often not as expected.

[0004] The existing patent (publication number: CN108818681A) discloses a method for automatically generating subtitles based on a wireless mouse. The method uses real-time data processing on the text sent by the wireless mouse to the computer and combines it with a corpus to identify subtitles, thereby solving the defects of slow subtitle input speed and high error rate.

[0005] Publication number: CN108818681A discloses an AI translation method and system for automatically converting dialects into subtitles, including the following steps: obtaining a video stream and an audio stream, and preprocessing the audio stream; generating speech features of the preprocessed audio stream, and comparing them with the dialect speech feature library, translating the audio stream into Mandarin; converting the translated Mandarin into TTS speech and text, and adding the text as subtitles to the video stream. The present invention designs a translation system based on various dialects and Mandarin, which can translate various dialects into Mandarin, convert them into TTS speech and subtitles, and facilitate the reviewer to proofread.

[0006] Therefore, for the above problems, firstly, live subtitles are generated after the live broadcast is recorded, which are offline subtitles and cannot be generated. Secondly, the scene switching recognition rate is very poor or even unrecognized, as well as comprehensive problems such as cost. For this reason, we propose a real-time subtitle generation method and system based on AI and voice mouse. Summary of the invention

[0007] In view of the above problems existing in the existing video subtitle generation, the present invention is proposed.

[0008] One of the purposes of the present invention is to provide a real-time subtitle generation method and system based on AI and voice mouse, which obtains video stream data based on voice mouse, and generates parsing results after calling the data through AI, makes corresponding judgments and generates subtitles according to the parsing results, and adds subtitle optimization strategies in the process of AI parsing, thereby improving the generation effect of video streams in different scene environments.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0010] On the one hand, the present invention provides a real-time subtitle generation method based on AI and voice mouse, comprising:

[0011] The audio signal of the video stream is collected based on the voice mouse to obtain the video stream data set;

[0012] According to the first audio analysis model of the voice mouse, the acquisition result of the video stream data set and the audio change information are processed, and a target data set is determined; the target data set is used to represent a set of scene change audio data in the acquisition result;

[0013] Sending the target data set and the collection results in the video stream data set to the cloud server of the voice mouse;

[0014] Receive a second audio analysis model sent by the cloud server, and update the first audio analysis model to the second audio analysis model; the second audio analysis model is obtained by the voice mouse cloud server by calling AI analysis and adjusting and training the collection results in the target data set and the video stream data set;

[0015] After using the second audio analysis model to perform target detection on the audio of the real-time collected video stream, a real-time scene subtitle analysis result is generated, and it is determined whether the scene subtitle analysis result matches the preset scene parameters and key sentences. If they match, a subtitle file is generated; the subtitle file is overlaid on the area position of the video screen in the video stream in the form of a subtitle layer for display.

[0016] As a preferred solution of the present invention, the audio signal of the video stream is collected based on the voice mouse, specifically, the original data is collected through the acoustic vocoder of the voice mouse, a data comparison table is generated based on the original data, the audio signal in the window is Fourier transformed so that the audio data is converted from the time domain to the frequency domain, the first audio analysis model is analyzed according to the collection result of the video stream data set and the frequency domain data of the audio change information to generate the first recognition result data; a video stream audio signal in a continuous time domain is given by the first audio analysis model, wherein the Fourier transform is expressed as follows:

[0017]

[0018] Where x(f) is a complex-valued function with frequency f, x(t) is the audio signal at the tth time point in the continuous time domain, e is a natural constant, i is an imaginary unit representing the frequency domain signal, f is the frequency in the continuous time domain, and dt is the differential symbol.

[0019] As a preferred solution of the present invention, the first audio analysis model calculates the audio change feature value of the segmented frequency corresponding to the spectrum data of the audio change time period according to the similarity of the frequency domain signal change, as follows:

[0020] p i =G i ∫|x i (t)| 2 dt

[0021] Among them, p i is the audio change feature data of the i-th frequency domain signal, and is converted into vector data as the feature parameter value of the audio change information, G i is the similarity value of the ith frequency domain signal, x i (t) is the original data value of the audio change on the i-th frequency domain signal.

[0022] As a preferred solution of the present invention, when determining the target data set, it also includes removing abnormal video stream data, wherein the abnormal video stream data is abnormal audio recognition information of the audio signal in the video stream, specifically, audio change information whose similarity of frequency domain signal changes exceeds a preset threshold through a neural network, and abnormal audio recognition features that do not conform to normal speech states, wherein the frequency domain features in the abnormal audio recognition features are discriminated by Mel-cephalogram coefficients.

[0023] As a preferred solution of the present invention, the second audio analysis model is based on the first recognition result data of the first audio analysis model, and is fused after being parsed by the AI ​​voice large model to generate a second recognition result. Specifically, the cloud server receives and responds to the target data set and the collection result uploaded by the voice mouse, and then parses and processes the target data set and the collection result according to the second audio analysis model to generate an AI parsing processing data set and corresponding audio change parsing data, combines the audio change parsing data in the voice mouse target data set and the AI ​​parsing processing data set, performs support and correlation analysis, and then fused through the second audio analysis model to generate the second recognition result data. The second audio analysis model is expressed as follows:

[0024]

[0025] Among them, U P,q It represents the fusion data after the analysis and processing of the voice mouse and the AI ​​voice model, that is, the second recognition result data, p i is the audio change feature data of the i-th frequency domain signal, P represents the set of audio change information processed by the first audio analysis model; q represents the set of AI speech large model analysis data; q j represents the jth audio change feature data after being parsed and processed by the AI ​​speech model; supp(p i ,q) represents the support data of the audio change characteristics of the i-th frequency domain signal for the AI ​​speech model parsing event; corr(q j ,p i ) represents the correlation data between the audio change analysis data after AI voice big model analysis and the voice mouse audio change feature data.

[0026] As a preferred solution of the present invention, the AI ​​speech large model analysis is performed through speech synthesis TTS and speech recognition ASR methods.

[0027] As a preferred solution of the present invention, when collecting audio of a video stream based on a voice mouse, it also includes processing the audio signal of the video stream collected based on a voice mouse through multiple channels.

[0028] As a preferred solution of the present invention, wherein: when judging whether the scene subtitle parsing result matches the preset scene parameters and key sentences, if they match, the second recognition result data is used to generate the current subtitle file;

[0029] If there is no match, the match fails, and the current subtitle file is generated by acquiring and based on the first recognition result data.

[0030] As a preferred solution of the present invention, the audio change characteristics of the audio change information include: content, volume, time sequence, and scale;

[0031] The audio signal of the video stream includes online video recording, offline video recording, online video playback recording, offline video playback recording, and online audio and video call recording.

[0032] On the one hand, the present invention provides a real-time subtitle generation system based on AI and voice mouse, comprising:

[0033] A voice mouse acquisition module is used to collect the audio signal of the video stream based on the voice mouse to obtain a video stream data set;

[0034] A voice mouse processing module, used for determining a target data set after performing data processing on the collection result of the video stream data set and the audio change information according to the first audio analysis model of the voice mouse;

[0035] A voice mouse sending module, used for sending the target data set and the acquisition results in the video stream data set to the cloud server of the voice mouse;

[0036] The cloud server processing module is used to receive the second audio analysis model sent by the cloud server, and update the first audio analysis model to the second audio analysis model; after performing target detection on the audio of the real-time collected video stream using the second audio analysis model, a real-time scene subtitle analysis result is generated;

[0037] The voice mouse judgment module is used to judge whether the scene subtitle analysis result matches the preset scene parameters and key sentences. If they match, a subtitle file is generated;

[0038] The execution module is used to overlay the subtitle file in the form of a subtitle layer in the area of ​​the video screen in the video stream for display.

[0039] Beneficial effects of the present invention: The present invention obtains video stream data based on a voice mouse, and generates a parsing result by parsing the data after calling it through AI, makes corresponding judgments and generates subtitles according to the parsing result, and adds a subtitle optimization strategy in the process of AI parsing, thereby improving the generation effect of video streams in different scene environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0041] Figure 1 This is an application scenario diagram of the real-time subtitle generation method based on AI and voice mouse of the present invention;

[0042] Figure 2 The present invention is a flowchart of a method for generating real-time subtitles based on AI and voice mouse;

[0043] Figure 3 The figure is a schematic diagram of the modular structure of the real-time subtitle generation system based on AI and voice mouse of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0045] Reference Figure 1 and Figure 2 as well as Figure 3 , is an embodiment of the present invention, which provides a real-time subtitle generation method and system based on AI and voice mouse, such as Figure 1 As shown, a voice mouse 10 and a corresponding cloud server 20 and a cloud database cynosDB are provided, wherein the voice mouse 10 is connected to a user's computer, mobile device, or other electronic device for recording live video via Bluetooth. When the user performs Internet video recording, video-on-demand recording, and live video recording, the real-time subtitle generation service of the voice mouse 10 can be awakened by pressing a button or voice. In addition, the cloud server 20 can form a corresponding feature training database and an abnormal feature library by calling AI analysis. The abnormal feature library includes scene anomalies, fusion anomalies, voice anomalies, etc., and performs subsequent training analysis and processing operations. It should be emphasized that the core invention of the present invention is that by combining a voice mouse, an AI analysis tool, and a subtitle display technology, real-time subtitles of live or recorded video content are realized, which greatly improves the accessibility of live content and the audience experience. To further explain, the technical solution of this embodiment is mainly to use a voice mouse as a medium. When the user is recording a video, the voice mouse records the user's voice and transmits it to the AI ​​tool on the computer or mobile phone. The AI ​​tool parses and recognizes the voice, and performs semantic analysis on the voice based on the user's language expression habits to generate high-quality text, and then generates a subtitle file through the tool and displays it on the computer or mobile phone screen.

[0046] like Figure 2As shown, the specific method steps of the real-time subtitle generation method based on AI and voice mouse in this embodiment include:

[0047] Step S201, collecting the audio signal of the video stream based on the voice mouse 10 to obtain a video stream data set;

[0048] Step S202, after performing data processing on the collection result of the video stream data set and the audio change information according to the first audio analysis model of the voice mouse 10, a target data set is determined; the target data set is used to indicate a collection of audio data with scene changes in the collection result;

[0049] Step S203, sending the target data set and the acquisition result of the video stream data set to the cloud server 20 of the voice mouse 10;

[0050] Step S204, receiving the second audio analysis model sent by the cloud server 20, and updating the first audio analysis model to the second audio analysis model; the second audio analysis model is obtained by the cloud server 20 of the voice mouse 10 by calling AI analysis and adjusting and training the collection results in the target data set and the video stream data set;

[0051] Step S205, after performing target detection on the audio of the real-time collected video stream using the second audio analysis model, a real-time scene subtitle analysis result is generated, and it is determined whether the scene subtitle analysis result matches the preset scene parameters and key sentences. If they match, a subtitle file is generated;

[0052] Step S206: Overlay the subtitle file in the form of a subtitle layer at a region of the video screen in the video stream for display.

[0053] Specifically, in the above method of this embodiment, the audio signal of the video stream is collected based on the voice mouse 10, and the original data is collected by the acoustic vocoder of the voice mouse 10. A data comparison table is generated based on the original data, and the audio data in the window is converted from the time domain to the frequency domain by Fourier transform. The first audio analysis model analyzes the frequency domain data of the video stream data set and the audio change information according to the collection result of the video stream data set and generates the first recognition result data; a video stream audio signal in a continuous time domain is given by the first audio analysis model, wherein the Fourier transform is expressed as follows:

[0054]

[0055] Where x(f) is a complex-valued function with frequency f, x(t) is the audio signal at the tth time point in the continuous time domain, e is a natural constant, i is an imaginary unit representing the frequency domain signal, f is the frequency in the continuous time domain, and dt is the differential symbol.

[0056] This embodiment further illustrates that the first audio analysis model specifically calculates the audio change feature value of the segmented frequency corresponding to the spectrum data of the audio change time period according to the similarity of the frequency domain signal change, as follows:

[0057] p i =G i ∫|x i (t)| 2 dt

[0058] Among them, p i is the audio change feature data of the i-th frequency domain signal, and is converted into vector data as the feature parameter value of the audio change information, G i is the similarity value of the ith frequency domain signal, x i (t) is the original data value of the audio change on the i-th frequency domain signal.

[0059] In addition, when determining the target data set, this embodiment also includes removing abnormal video stream data. The abnormal video stream data is abnormal audio recognition information of the audio signal in the video stream, specifically, audio change information whose similarity of frequency domain signal changes exceeds a preset threshold through a neural network, and abnormal audio recognition features that do not conform to the normal speech state. The frequency domain features in the abnormal audio recognition features are distinguished by Mel-cephalogram coefficients.

[0060] In the above method of this embodiment, the second audio analysis model is based on the first recognition result data of the first audio analysis model, and is combined with the AI ​​voice model for analysis and fusion to generate the second recognition result. Specifically, the cloud server 20 receives and responds to the target data set and the collection result uploaded by the voice mouse 10, and then analyzes and processes the target data set and the collection result according to the second audio analysis model to generate an AI analysis processing data set and corresponding audio change analysis data, and combines the audio change analysis data in the target data set of the voice mouse 10 and the AI ​​analysis processing data set to perform support and correlation analysis, and then fuses through the second audio analysis model to generate the second recognition result data. The second audio analysis model is expressed as follows:

[0061]

[0062] Among them, U P,q It represents the fusion data after the analysis and processing of the voice mouse and the AI ​​voice model, that is, the second recognition result data, p i is the audio change feature data of the i-th frequency domain signal, P represents the set of audio change information processed by the first audio analysis model; q represents the set of AI speech large model analysis data; q j represents the jth audio change feature data after being parsed and processed by the AI ​​speech model; supp(p i,q) represents the support data of the audio change characteristics of the i-th frequency domain signal for the AI ​​speech model parsing event; corr(q j ,p i ) represents the correlation data between the audio change analysis data after AI voice big model analysis and the voice mouse audio change feature data.

[0063] Based on the above, this embodiment realizes the recognition and fusion of voice through two analysis models, voice mouse and cloud, and can generate subtitles in real time during short video live broadcast, screen recording or video communication dialogue. The subtitles are matched with the voice in the video in real time, and real-time subtitles are accurately generated. It has a good effect in the recognition process of scene switching and changes in the voice person. In the process of AI analysis, a subtitle optimization strategy is added to improve the generation effect of video streams in different scene environments; there is no need to add subtitles after the recording is completed, which saves time and cost of post-production; in addition, by setting a voice mouse, the operation process is simplified and the efficiency of subtitle generation is improved.

[0064] Specifically in one embodiment, the AI ​​speech large model is parsed through speech synthesis TTS and speech recognition ASR methods.

[0065] Specifically, in one embodiment, when the audio of the video stream is collected based on the voice mouse 10, it also includes multi-channel processing of the audio signal of the video stream collected based on the voice mouse 10. This improves the ability to identify different sound sources and further improves the accuracy of subtitle generation.

[0066] In one embodiment, preferably, when determining whether the scene subtitle parsing result matches the preset scene parameters and key sentences, if they match, the second recognition result data is used to generate the current subtitle file;

[0067] If there is no match, the match fails, and the current subtitle file is generated by obtaining and generating the first recognition result data. More accurate and scene-adaptive voice subtitles can be generated, and if the match fails, the recognition result of the voice mouse can be called and displayed accordingly, thereby ensuring the scene switching recognition rate.

[0068] Specifically in one embodiment, the audio change features of the audio change information include: content, volume, time sequence, and scale;

[0069] The audio signals of the video stream include online video recording, offline video recording, online video playback recording, offline video playback recording, and online audio and video call recording.

[0070] In addition, this embodiment specifically provides specific application operation steps of the voice mouse during live broadcasting, as follows:

[0071] Step A: Place the voice mouse in the live broadcast scene to record the entire live broadcast scene. It should be noted that when using the voice mouse to record, the screen content being recorded is recorded in the set area, and the set area refers to the part of the screen where subtitles need to be added.

[0072] Step B, operate the voice mouse to start the live broadcast, and then start real-time recording of the voice, and send the recorded voice data to the AI ​​analysis tool through the network in real time; it should be noted that the scenario in step A refers to: the voice scenario where subtitles need to be added during the live broadcast.

[0073] Step C, the AI ​​analysis tool analyzes the voice data in the scene at the same time, and obtains the analysis results based on the analysis content, and transmits the analysis results back to the voice mouse through the network in real time; it should be noted that the analysis in this manual refers to: analyzing the voice data content.

[0074] Step D: Based on the received analysis result, the voice mouse determines whether the analysis result matches the set key sentence. If so, the current subtitle file is generated according to the scene subtitle analysis result fused by AI analysis; if not, the match fails, and the current subtitle file is generated according to the recognition result data of the voice mouse.

[0075] Step E: Execute and generate a subtitle file, which is overlaid on the screen of the recorded video in the form of a subtitle layer.

[0076] Step F: The subtitle file is combined with the voice data in the recorded video data, and a short video can be automatically generated after the recording is completed.

[0077] Also refer to Figure 3 As shown, this embodiment further provides a real-time subtitle generation system based on AI and voice mouse according to the above-mentioned real-time subtitle generation method based on AI and voice mouse, which is applied to the corresponding voice mouse 10 electronic device and the cloud server 20, and the system includes:

[0078] The voice mouse acquisition module 101 is used to collect the audio signal of the video stream based on the voice mouse 10 to obtain a video stream data set;

[0079] The voice mouse processing module 102 is used to determine the target data set after performing data processing on the collection result of the video stream data set and the audio change information according to the first audio analysis model of the voice mouse 10;

[0080] The voice mouse sending module 103 is used to send the target data set and the acquisition results of the video stream data set to the cloud server 20 of the voice mouse 10;

[0081] The cloud server processing module 201 is used to receive the second audio analysis model sent by the cloud server 20, and update the first audio analysis model to the second audio analysis model; after performing target detection on the audio of the real-time collected video stream using the second audio analysis model, a real-time scene subtitle analysis result is generated;

[0082] The voice mouse judgment module 104 is used to judge whether the scene subtitle analysis result matches the preset scene parameters and key sentences, and if so, generate a subtitle file;

[0083] The execution module 105 is used to overlay the subtitle file in the form of a subtitle layer in the area of ​​the video screen in the video stream for display.

[0084] In summary, the present invention obtains video stream data based on voice mouse, and generates parsing results by calling data through AI, makes corresponding judgments and generates subtitles according to the parsing results, and adds subtitle optimization strategies in the process of AI parsing, thereby improving the generation effect of video streams in different scene environments. The beneficial effects of the present invention are:

[0085] 1. It realizes the real-time generation of subtitles during short video live broadcast, screen recording or video communication dialogue. The subtitles are matched with the voice in the video in real time, improving the viewing experience of the live broadcast.

[0086] 2. There is no need to add subtitles after recording is completed, saving time and cost for post-production.

[0087] 3. By setting up the voice mouse, the operation process is simplified and the efficiency of subtitle generation is improved.

[0088] 4. Use AI parsing tools to parse voice data, which improves the accuracy and speed of subtitle generation.

[0089] 5. The use of multi-channel processing improves the ability to recognize different sound sources and further improves the accuracy of subtitle generation.

[0090] 6. The voice mouse analyzes the voice data through an acoustic vocoder, converting the audio data from the time domain to the frequency domain, thereby improving the accuracy of data analysis.

[0091] 7. The generated subtitle file is overlaid on the recorded video screen in the form of a subtitle layer, which does not affect the viewing of the video screen.

[0092] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0093] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0094] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0095] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0096] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0097] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0098] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0099] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A real-time subtitle generation method based on AI and voice mouse, characterized in that: include: The audio signal of the video stream is collected based on the voice mouse to obtain the video stream data set; Determine the target data set after performing data processing on the collection result of the video stream data set and the audio change information according to the first audio analysis model of the voice mouse; The target data set is used to represent a set of audio data with scene changes in the acquisition result; Sending the target data set and the collection results in the video stream data set to the cloud server of the voice mouse; Receive a second audio analysis model sent by the cloud server, and update the first audio analysis model to the second audio analysis model; the second audio analysis model is obtained by the voice mouse cloud server by calling AI analysis and adjusting and training the collection results in the target data set and the video stream data set; After using the second audio analysis model to perform target detection on the audio of the real-time collected video stream, a real-time scene subtitle analysis result is generated, and it is determined whether the scene subtitle analysis result matches the preset scene parameters and key sentences. If they match, a subtitle file is generated; the subtitle file is overlaid on the area position of the video screen in the video stream in the form of a subtitle layer for display.

2. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 1, characterized in that: The audio signal of the video stream is collected based on the voice mouse, specifically, the original data is collected through the acoustic vocoder of the voice mouse, a data comparison table is generated based on the original data, and the audio data is converted from the time domain to the frequency domain by Fourier transforming the audio signal in the window. The first audio analysis model analyzes the frequency domain data of the acquisition result of the video stream data set and the audio change information to generate the first recognition result data; a video stream audio signal in a continuous time domain is given by the first audio analysis model, wherein the Fourier transform is expressed as follows: Where x(f) is a complex-valued function with frequency f, x(t) is the audio signal at the tth time point in the continuous time domain, e is a natural constant, i is an imaginary unit representing the frequency domain signal, f is the frequency in the continuous time domain, and dt is the differential symbol.

3. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 2, characterized in that: The first audio analysis model specifically calculates the audio change feature value of the segmented frequency corresponding to the spectrum data of the audio change time period according to the similarity of the frequency domain signal change, as follows: p i =G i ∫|x i (t)| 2 dt Among them, p i is the audio change feature data of the i-th frequency domain signal, and is converted into vector data as the feature parameter value of the audio change information, G i is the similarity value of the i-th frequency domain signal, x i (t) is the original data value of the audio change on the i-th frequency domain signal.

4. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 3, characterized in that: When determining the target data set, it also includes removing abnormal video stream data, wherein the abnormal video stream data is abnormal audio recognition information of the audio signal in the video stream, specifically, audio change information whose similarity of frequency domain signal changes exceeds a preset threshold through a neural network, and abnormal audio recognition features that do not conform to the normal speech state, wherein the frequency domain features in the abnormal audio recognition features are discriminated by Mel-cephalogram coefficients.

5. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 3, characterized in that: The second audio analysis model is based on the first recognition result data of the first audio analysis model, and is combined with the AI ​​voice large model for analysis and fusion to generate a second recognition result. Specifically, the cloud server receives and responds to the target data set and the collection result uploaded by the voice mouse, and then parses and processes the target data set and the collection result according to the second audio analysis model to generate an AI analysis processing data set and corresponding audio change analysis data, combines the audio change analysis data in the voice mouse target data set and the AI ​​analysis processing data set, performs support and correlation analysis, and then fuses through the second audio analysis model to generate the second recognition result data. The second audio analysis model is expressed as follows: Among them, U P,q It represents the fusion data after the analysis and processing of the voice mouse and the AI ​​voice model, that is, the second recognition result data, p i is the audio change feature data of the i-th frequency domain signal, P represents the set of audio change information processed by the first audio analysis model; q represents the set of AI speech large model analysis data; q j represents the jth audio change feature data after being parsed and processed by the AI ​​speech model; supp(p i ,q) represents the support data of the audio change characteristics of the i-th frequency domain signal for the AI ​​speech model parsing event; corr(q j ,p i ) represents the correlation data between the audio change analysis data after AI voice big model analysis and the voice mouse audio change feature data.

6. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 5, characterized in that: The AI ​​speech large model analysis is performed through speech synthesis TTS and speech recognition ASR methods.

7. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 1, characterized in that: When the audio of the video stream is collected based on the voice mouse, it also includes processing the audio signal of the video stream collected based on the voice mouse through multiple channels.

8. The method for generating real-time subtitles based on AI and voice mouse as claimed in claim 5, characterized in that: Determining whether the scene subtitle parsing result matches the preset scene parameters and key sentences, and if so, generating a current subtitle file with the second recognition result data; If there is no match, the match fails, and the current subtitle file is generated by acquiring and based on the first recognition result data.

9. The real-time subtitle generation method based on AI and voice mouse as claimed in claim 3, characterized in that: The audio change features of the audio change information include: content, volume, time sequence, and scale; The audio signal of the video stream includes online video recording, offline video recording, online video playback recording, offline video playback recording, and online audio and video call recording.

10. A system using the method for generating real-time subtitles based on AI and voice mouse as claimed in claim 1, characterized in that: include: A voice mouse acquisition module is used to collect the audio signal of the video stream based on the voice mouse to obtain a video stream data set; A voice mouse processing module, used for determining a target data set after performing data processing on the collection result of the video stream data set and the audio change information according to the first audio analysis model of the voice mouse; A voice mouse sending module, used for sending the target data set and the acquisition results in the video stream data set to the cloud server of the voice mouse; The cloud server processing module is used to receive the second audio analysis model sent by the cloud server, and update the first audio analysis model to the second audio analysis model; after performing target detection on the audio of the real-time collected video stream using the second audio analysis model, a real-time scene subtitle analysis result is generated; The voice mouse judgment module is used to judge whether the scene subtitle analysis result matches the preset scene parameters and key sentences. If they match, a subtitle file is generated; The execution module is used to overlay the subtitle file in the form of a subtitle layer in the area of ​​the video screen in the video stream for display.

Images