Sign language animation conversion method and device based on set top box audio, equipment and medium

Abstract

The invention relates to the technical field of audio processing, and discloses a sign language animation conversion method and device based on set top box audio, equipment and a medium. The method comprises the following steps: acquiring original audio and video data, separating the original audio and video data into original video data and original audio data, and copying the original audio data to obtain copied audio data; processing the original video data and the original audio data by using a decoder to obtain decoded video data and audio data; extracting voice text fragment data from the copied audio data, performing noise reduction on the voice text fragment data, and extracting human voice data; converting the human voice data into a language text, mapping the language text into a sign language sequence, and rendering the sign language sequence through a local database and a cloud database to generate a sign language animation; and displaying the decoded video data, the decoded audio data and the sign language animation on a display interface. The method is wide in coverage range, accurate in semantic analysis and high in sign language accuracy, and the barrier-free access experience of digital media is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, and in particular to a sign language animation conversion method and device based on set-top box audio, equipment and medium. BACKGROUND

[0002] Currently, under the background of the accelerated development of digital inclusive technology, audio-visual content barrier-free access has become a global issue. Research shows that 68% of people with congenital hearing loss have difficulty reading text, and this group has great difficulty in watching TV shows and video content. Their mother tongue communication needs are significantly mismatched with existing subtitle solutions. Therefore, sign language, as the natural language of this group, is the key path to achieving information equity.

[0003] In the prior art, pre-recorded artificial sign language translation videos are generally directly embedded and packaged in the original video stream for display. The disadvantage of this method is that the content coverage is very low, the production cost is high, and it cannot cover most of the film and television resources. Or only provide basic subtitle services. The disadvantage of this method is that it cannot meet the needs of hearing-impaired people who use sign language as their main means of communication. SUMMARY

[0004] Therefore, the purpose of the present application is to overcome the deficiencies in the prior art and provide a sign language animation conversion method and device based on set-top box audio, equipment and medium.

[0005] The present application provides the following technical solutions: In a first aspect, the present application provides a sign language animation conversion method based on set-top box audio, which comprises: Obtaining original audio and video data of a set-top box, separating the original audio and video data into original video data and original audio data using a demultiplexer, and copying the original audio data to obtain copied audio data; processing the original video data using a video decoder to obtain decoded video data, and processing the original audio data using an audio decoder to obtain decoded audio data; extracting speech text segment data from the copied audio data, denoising the speech text segment data, and extracting human voice data from the denoised speech text segment data; Converting the human voice data into language text, mapping the language text into sign language sequences, and rendering the sign language sequences through a local database and a cloud database to generate sign language animation; Displaying the decoded video data and the decoded audio data on a display interface, and constructing a target window on the display interface to synchronously display the sign language animation through the target window.

[0006] In an optional implementation, the extracting speech text segment data from the copied audio data comprises: identifying speech data and non-speech data in the copied audio data using a voice activity detection technology; when the speech data is detected, recording a start timestamp of the speech data, and when the non-speech data is detected, determining whether there is a record of the start timestamp; when there is a record of the start timestamp, recording a latest end timestamp before the start of the non-speech data; according to the start timestamp of the speech data and the latest end timestamp before the start of the non-speech data, cutting out the speech text segment data, and taking the start timestamp of the speech data and the latest end timestamp before the start of the non-speech data as a time anchor pair of the speech text segment data.

[0007] In an optional implementation, the de-noising the speech text segment data and extracting human voice data in the de-noised speech text segment data comprises: using a DeepFilterNet de-noising model to remove environmental sound and non-human voice noise in the speech text segment data, and retaining human voice and background sound, to obtain the de-noised speech text segment data; using a Demucs model to remove background sound in the de-noised speech text segment data, to obtain the human voice data, and dynamically binding the human voice data with the time anchor pair of the speech text segment data.

[0008] In an optional implementation, the converting the human voice data into language text comprises: using a Whisper architecture to build an automatic speech recognition model, and using a Common Voice data set to train the automatic speech recognition model, to obtain a trained automatic speech recognition model; using the trained automatic speech recognition model to identify the language of the human voice data, and converting the language of the human voice data into a target language, to obtain the language text, and dynamically binding the language text with the time anchor pair of the human voice data.

[0009] In an optional implementation, the mapping the language text into a sign language sequence comprises: using a natural language processing algorithm to extract a main part of the language text, and realizing real-time conversion of text grammar to sign language structure on the main part, to obtain a converted main part; The converted stem part is disassembled to a minimum morpheme unit by using a dependency syntax analyzer to generate a plurality of words, and each of the words is combined into the sign language sequence.

[0010] In an optional embodiment, the rendering of the sign language sequence by the local database and the cloud database to generate a sign language animation comprises: A preset perplexity rule is used to determine the perplexity of each word in the sign language sequence. According to the perplexity of each word, each word is sent to the local database or the cloud database, and each word is rendered by the local database and the cloud database to generate a corresponding cloud skeleton animation or a local skeleton animation. A lightweight generation model in the local database is used to generate a transition skeleton animation between each word. The total duration of the sign language sequence is calculated according to the time anchor points, and the number of rendering frames is determined according to the total duration and a preset frame rate. According to the cloud skeleton animation, the local skeleton animation and the transition skeleton animation, an opneGL rendering technology is used to generate the sign language animation with the number of rendering frames.

[0011] In an optional embodiment, according to the perplexity of each word, each word is sent to the local database or the cloud database, and each word is rendered by the local database and the cloud database to generate a corresponding cloud skeleton animation or a local skeleton animation, which comprises: Words with a perplexity greater than a preset perplexity threshold are determined as complex words, the complex words and the sign language sequence are sent to the cloud database, the cloud database is used to analyze the complex words and the sign language sequence to generate the cloud skeleton animation, and the cloud skeleton animation is sent to the local database. Words with a perplexity less than or equal to the preset perplexity threshold are determined as basic words, and the basic words are sent to the local database, and the local database is used to analyze the basic words to generate the local skeleton animation.

[0012] In a second aspect, the present application provides a sign language animation conversion device based on a set-top box audio, which comprises: A separation module is configured to obtain original audio and video data of a set-top box, separate the original audio and video data into original video data and original audio data by using a demultiplexer, and copy the original audio data to obtain copied audio data. decode the original video data by using a video decoder to obtain decoded video data, and decode the original audio data by using an audio decoder to obtain decoded audio data; extracting a speech text segment data from the copied audio data, denoising the speech text segment data, and extracting a human voice data from the denoised speech text segment data; rendering the human voice data into a language text, mapping the language text into a sign language sequence, and rendering the sign language sequence by using a local database and a cloud database to generate a sign language animation; displaying the decoded video data and the decoded audio data on a display interface, and constructing a target window on the display interface to synchronously display the sign language animation through the target window.

[0013] In a third aspect, a computer device is provided in the embodiments of the present disclosure. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the sign language animation conversion method based on audio of a set-top box in the first aspect are implemented.

[0014] In a fourth aspect, a computer readable storage medium is provided in the embodiments of the present disclosure. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the sign language animation conversion method based on audio of a set-top box in the first aspect are implemented.

[0015] Advantages of the present application: The sign language animation conversion method based on audio of a set-top box provided by the embodiments of the present application realizes conversion of audio data in audio and video data (for example, live programs, television / film programs, local video content, etc.) of a set-top box into a sign language animation by using real-time audio translation technology, and the sign language animation is intelligently adapted to a right lower area of an original picture in a picture-in-picture small window form. The method has wide coverage, accurate semantic analysis, high sign language accuracy, wide language recognition range, and significantly improves the barrier-free access experience of digital media.

[0016] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor. In the various drawings, similar components are denoted by similar reference numerals.

[0018] Figure 1 A flow chart of a sign language animation conversion method based on set top box audio provided by an embodiment of the present application is shown; Figure 2 A structural schematic diagram of a sign language animation conversion device based on set top box audio provided by an embodiment of the present application is shown; Figure 3 A structural schematic diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The embodiments of the present application will be described in detail below, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation on the present application.

[0020] It should be noted that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the template herein are only for the purpose of describing the specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0022] Embodiment 1 As Figure 1 shown, a flow chart of a sign language animation conversion method based on set top box audio in an embodiment of the present application, the sign language animation conversion method based on set top box audio provided by the present embodiment includes the following steps: Step S110, obtaining original audio and video data of the set-top box, separating the original audio and video data into original video data and original audio data by using a demux, and copying the original audio data to obtain copied audio data.

[0023] In the embodiment, when the original audio and video data is obtained and parsed, the original audio and video data is separated into original video data and original audio data by using a demux, and an audio auxiliary channel is created synchronously, the original audio data is copied to the channel to obtain copied audio data, which is used for subsequent audio processing and sign language animation generation.

[0024] Step S120, processing the original video data by using a video decoder to obtain decoded video data, and processing the original audio data by using an audio decoder to obtain decoded audio data.

[0025] Meanwhile, the original video data and the original audio data flow to an audio main channel, the original video data is processed by using a video decoder of the audio main channel to obtain decoded video data, and the original audio data is processed by using an audio decoder of the audio main channel to obtain decoded audio data.

[0026] The above method copies audio data based on the demux, and realizes an audio double-channel processing mechanism: the main channel is directly connected to the audio and video decoder, and the auxiliary channel enters a sign language animation generation processing channel. The two channels do not interfere with each other, which can maximize the real-time performance of video playing and achieve subsequent synchronous display.

[0027] Step S130, extracting speech text segment data from the copied audio data, denoising the speech text segment data, and extracting human voice data in the denoised speech text segment data.

[0028] Understandably, since the effective human voice in the audio data of the film and television resource is not continuous, in order to reduce the amount of calculation, save system resources, and facilitate subsequent model processing, segmentation is performed here to extract the speech text segment data as much as possible.

[0029] Specifically, voice activity detection technology (VAD) is used to accurately identify the speech data and non-speech data in the copied audio data. When speech data is detected, the start time stamp apts_start of the speech data is recorded. When non-speech data is detected, it is determined whether there is a record of the start time stamp apts_start. When there is a record of the start time stamp apts_start, the nearest end time stamp apts_end before the non-speech data is recorded.

[0030] Further, according to the start timestamp apts_start of the speech data and the nearest end timestamp apts_end before the start of the non-speech data, the corresponding speech text segment data is intercepted. At the same time, the start timestamp apts_start of the speech data and the nearest end timestamp apts_end before the start of the non-speech data are taken as the time anchor pair of the speech text segment data {speech text segment data1: [apts_start, apts_end]} to establish a mapping relationship, so as to ensure the speech text segment data and the original audio and video frame level alignment, and realize the subsequent synchronization function.

[0031] Further, by using the DeepFilterNet denoising model, the environmental sound (such as wind, car horn, footsteps, etc.) and non-human voice noise in the speech text segment data are removed through time-frequency mask prediction, and the human voice and background sound (such as music) are retained, so as to obtain the denoised speech text segment data.

[0032] The principle of the DeepFilterNet denoising model is as follows: ①Spectrum analysis: using short-time Fourier transform (STFT) to convert the input speech text segment data into time-frequency domain; ②ERB processing: using an equivalent rectangular bandwidth (ERB) filter, based on the human ear perception limit characteristics and different perception resolution in high and low frequencies, the 48K (481 dimensions) spectrum is compressed to 32 dimensions ERB frequency band; ③Real gain prediction: a set of hearing perception oriented frequency band compression coefficients are dynamically generated through a neural network, and energy suppression is performed on different frequency bands, which is equivalent to installing an intelligent volume knob for each ERB frequency band; ④Deep filtering: constructing an N-order neural network filter coefficient in the complex domain based on the DNS4 dataset superimposed with various environmental noise (traffic, wind, etc.), mainly focusing on enhancing mechanism below 5kHz (including harmonic coherence utilization, phase alignment, noise suppression based on the randomness of noise in frequency domain), and mainly random noise above 5kHz, with low actual semantic contribution; ⑤Signal reconstruction and enhancement: for the harmonic characteristics of speech (such as fundamental frequency formant), the fine spectral structure destroyed by noise is restored, and the naturalness of speech is improved.

[0033] Then, by using the Demucs model, the background sound in the denoised speech text segment data is removed through time-frequency feature decoupling, the pure human voice data is extracted, and the extracted human voice data is dynamically bound with the time anchor pair of the speech text segment data, so as to output the structured data {human voice data1: [apts_start, apts_end]} and ensure the synchronization consistency of the downstream modules.

[0034] Among them, the principle of the Demucs model is as follows: The Demucs model is based on an improved U-Net architecture. Using the original waveform, it extracts features layer by layer through a neural network: ① Use the encoder for downsampling, adopting one-dimensional convolution (Conv1D) operations to gradually extract more abstract, high-level, and discriminative feature representations; ② Use a bidirectional LSTM to receive a series of high-level feature vector sequences output by the encoder and output a new feature sequence that integrates global temporal context, which endows the model with the ability to understand the global structure of music and long-distance temporal dependencies and can understand the position and meaning of music; ③ Use the decoder for upsampling to gradually reconstruct the waveforms of the separated pure sound sources (such as vocals, drums, etc.).

[0035] When extracting speech text segment data, the above method uses an efficient VAD scheme, which can effectively extract the parts with speech, thereby reducing unnecessary audio processing and saving resources. Using an efficient noise reduction model and a voice separation model for speech text segment data can improve the accuracy of speech recognition.

[0036] Step S140: Convert the voice data into a language text, map the language text to a sign language sequence, and render the sign language sequence through a local database and a cloud database to generate a sign language animation.

[0037] Specifically, build an Automatic Speech Recognition (ASR) model using the Whisper architecture and train the ASR model using the Common Voice dataset to obtain a trained ASR model. Common Voice is a multilingual and open-source speech dataset created by Mozilla. It not only includes mainstream foreign languages such as English, Japanese, and Chinese but also can include mainstream dialect languages such as Cantonese to improve the recognition accuracy of various languages.

[0038] Then use the trained ASR model to identify the language of the voice data (supporting multiple languages such as Mandarin / Cantonese / English / Japanese / Chinese, etc.) and convert the language of the voice data into the target language to obtain a language text. For example: "The temperature today is 25 degrees" → "今天天气25度"; "今日の気温は25度です" → "今天天气25度", and finally output the language text in Chinese. At the same time, also dynamically bind the generated language text and the time anchor pair of the voice data, and output structured data {language text 1: [apts_start, apts_end]}, inheriting the temporal anchor of the voice data to ensure the lip synchronization accuracy of the sign language animation.

[0039] Further, a natural language processing algorithm (NLP) is used to extract the main part of the language text and implement real-time conversion of the main part from text syntax (e.g., subject-verb-object) to sign language structure (e.g., topic-explanation), to obtain the converted main part, for example, Chinese: “I eat” → sign language: “food I eat”, which is a word order conversion and translation. Then, the converted main part is disassembled into the smallest morpheme unit by using a dependency syntax analyzer to generate multiple words, and the words are combined into a sign language sequence, for example, “weather 25 degrees today” → “weather” “25 degrees” “today”, which is used for subsequent word-level sign language animation generation.

[0040] Further, a predetermined perplexity rule is used to extract features and evaluate the complexity of each word in the sign language sequence to determine the perplexity (PPL) of each word. Taking “weather 25 degrees today” as an example, the PPL of “today” is 35, the PPL of “weather” is 60, and the PPL of “25 degrees” is 89.

[0041] Words with a perplexity greater than a predetermined perplexity threshold (e.g., 85) are determined to be complex words (e.g., “25 degrees” is a compound structure containing numbers and units), and the complex words and the entire sign language sequence are sent to the cloud database for analysis, decomposition into “2” “5” “degrees” three elements, and then the cloud database is queried for the corresponding actions of the three elements, to generate cloud skeleton animation, and the cloud skeleton animation is returned to the local database. Words with a perplexity less than or equal to the predetermined perplexity threshold (e.g., 85) are determined to be basic words (e.g., “today”, “weather”), and the basic words are sent to the local database. The local database is preloaded with a skeleton database of basic sign language animation materials, which contains a large number of basic words, and the data type is generally BVH (BioVision Hierarchy) standard unified data format, which stores skeleton frame information and action information. Therefore, the local database is used to analyze the basic words and generate local skeleton animation.

[0042] In the local database, a lightweight generation model is also deployed, which can calculate and generate transition skeleton animation between words, including transition fill actions (e.g., conjunction gestures), basic spatial reference actions (e.g., here, there), etc.

[0043] It should be noted that in the present embodiment, a cache can be maintained in the local database, and the latest cloud skeleton animation returned by the cloud database is saved in the cache, for example, for 72 hours, and subsequent processing of the words by the cloud database can be queried in the local database cache to determine whether the corresponding animation exists, and if so, the animation can be directly called, and if not, the words are sent to the cloud database for processing, thereby improving the efficiency of data processing.

[0044] Preferably, a local database updating mechanism can also be provided, and the local database is updated once every preset time (for example, 72 hours), and the cloud database can be used to analyze user request hot words, and the words frequently used by the user are added to the local database, and the words rarely used by the user are excluded from the local database.

[0045] Further, the total duration T of the sign language sequence is calculated according to the time anchor points, the frame number is allocated according to the preset frame rate f (for example, 30 fps), and the rendering frame number N is determined according to the total duration T and the preset frame rate.

[0046] Preferably, for the continuity between different skeleton animations, a transition animation of a preset frame number (for example, 2-5 frames) is generated by difference based on the total duration and the rendering frame number. For example, animation A and animation B are two consecutive animations:

[0047] The above language recognition uses an improved Whisper architecture, supports multiple language interpretation, and has greatly improved coverage and practicality compared with the prior art. In the generation of sign language animation, intelligent decision-making is used to determine whether to generate locally or in the cloud, and basic words are directly called to generate animation by the local database and local model to ensure real-time performance, and complex words or compound words are calculated by the cloud database to generate more accurate and delicate animations. At the same time, the local data resources can be updated regularly and in real time to meet the user's usage scenarios.

[0048] Step S150, displaying the decoded video data and the decoded audio data on the display interface, and constructing a target window on the display interface to synchronously display the sign language animation through the target window.

[0049] Finally, the decoded video data and the decoded audio data are displayed on the display interface, synchronized based on the audio timestamp, and the rendering and display of the sign language animation are triggered when the video is played to apts_start. The first frame of the video can be delayed for a preset time period (e.g., 500 ms) to compensate for the time delay of all the sign language animation rendering processes in the foregoing steps. The sign language animation is displayed in a target window on the display interface by default. The target window is generally placed in the lower right corner of the screen, and is implemented by layer superposition through a SurfaceView. The size, range, and coordinates of the target window can be set, and the sign language animation can be scaled in real time according to the size of the window.

[0050] Finally, the continuous animation generated by fusing the two-end animations is output, and the continuous sign language animation with local response speed and cloud modeling accuracy is obtained, which greatly improves the viewing experience and life quality of the hearing-impaired person.

[0051] The sign language animation conversion method based on the audio of a set-top box provided in the embodiments of the present application realizes conversion of audio data in audio and video data (for example, live programs, television / film programs, local video content, etc.) of a set-top box into sign language animation through real-time audio translation technology and intelligent adaptation of the sign language animation in the form of a picture-in-picture small window to the right lower area of the original picture. The sign language animation conversion method has wide coverage, accurate semantic analysis, high sign language accuracy, wide language recognition range, and significantly improves the barrier-free access experience of digital media.

[0052] Embodiment 2 As shown in Figure 2 Fig. 1 is a structural schematic diagram of a sign language animation conversion device 200 based on the audio of a set-top box in the embodiments of the present application. The device includes: a separation module 210, configured to obtain original audio and video data of a set-top box, separate the original audio and video data into original video data and original audio data by using a demultiplexer, and copy the original audio data to obtain copied audio data; a decoding module 220, configured to process the original video data by using a video decoder to obtain decoded video data, and process the original audio data by using an audio decoder to obtain decoded audio data; an extraction module 230, configured to extract speech text segment data from the copied audio data, perform noise reduction on the speech text segment data, and extract human voice data in the noise-reduced speech text segment data; a rendering module 240, configured to convert the human voice data into language text, map the language text into a sign language sequence, and render the sign language sequence by using a local database and a cloud database to generate a sign language animation; The display module 250 is configured to display the decoded video data and the decoded audio data on a display interface, and construct a target window on the display interface, and display the sign language animation through the target window.

[0053] The sign language animation conversion device based on set-top box audio provided by the embodiments of the present application can implement each process of the sign language animation conversion method based on set-top box audio corresponding to the embodiment 1, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0054] The sign language animation conversion device based on set-top box audio provided by the embodiments of the present application can implement each process of the sign language animation conversion method based on set-top box audio corresponding to the embodiment 1, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0055] Embodiment 3 The embodiments of the present application also provide a computer device. For details, please refer to Figure 3 , Figure 3 The basic structure block diagram of the computer device of the present embodiment is shown in FIG. 3.

[0056] The computer device 3 includes a memory 31, a processor 32, and a network interface 33 which are connected to each other through a system bus. It should be noted that only the computer device 3 with the memory 31, the processor 32, and the network interface 33 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device that can automatically perform numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0057] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and other computing devices. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and the like.

[0058] The memory 31 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or D slot compatible memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 31 can be an internal storage unit of the computer device 3, such as a hard disk or a memory of the computer device 3. In other embodiments, the memory 31 can also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 3. Of course, the memory 31 can also include both the internal storage unit and the external storage device of the computer device 3. In this embodiment, the memory 31 is generally used to store an operating system and various application software installed on the computer device 3, such as computer readable instructions of the slot compatibility test method, etc. In addition, the memory 31 can also be used to temporarily store various data that have been output or will be output.

[0059] The processor 32 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other hand sign animation conversion chip based on set top box audio in some embodiments. The processor 32 is generally used to control the overall operation of the computer device 3. In this embodiment, the processor 32 is used to run computer readable instructions or process data stored in the memory 31, such as computer readable instructions of the slot compatibility test method.

[0060] The network interface 33 can include a wireless network interface or a wired network interface, and the network interface 33 is generally used to establish a communication connection between the computer device 3 and other electronic devices.

[0061] The computer device provided in this embodiment can execute the hand sign animation conversion method based on set top box audio described above. Here, the hand sign animation conversion method based on set top box audio can be the hand sign animation conversion method based on set top box audio of each of the above embodiments.

[0062] Embodiment 4 This embodiment also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the hand sign animation conversion method based on set top box audio in the embodiments.

[0063] In this embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer readable storage medium is generally used to store an operating system and various application software installed on the computer device. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0064] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only illustrative, for example, the flowcharts and structural diagrams in the drawings show the possible implementation architecture, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in alternative implementation, the functions noted in the block can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the structural diagram and / or flowchart, and the combination of blocks in the structural diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0065] In addition, each functional module or unit in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0066] If the functions are implemented in the form of software function modules and sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium can be a non-volatile storage medium or a volatile storage medium, for example, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0067] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A sign language animation conversion method based on set-top box audio, characterized by, The method comprises: Obtaining original audio and video data of a set-top box, separating the original audio and video data into original video data and original audio data by using a demultiplexer, and copying the original audio data to obtain copied audio data; Processing the original video data by using a video decoder to obtain decoded video data, and processing the original audio data by using an audio decoder to obtain decoded audio data; Extracting speech text segment data from the copied audio data, denoising the speech text segment data, and extracting human voice data from the denoised speech text segment data; Converting the human voice data into language text, mapping the language text into sign language sequences, and rendering the sign language sequences by using a local database and a cloud database to generate sign language animation; Displaying the decoded video data and the decoded audio data on a display interface, and constructing a target window on the display interface to synchronously display the sign language animation through the target window.

2. The set-top box audio-based sign language animation conversion method of claim 1, wherein, The extracting of the speech text segment data from the copied audio data comprises: Identifying speech data and non-speech data in the copied audio data by using a voice activity detection technology; When the speech data is detected, recording a start time stamp of the speech data, and when the non-speech data is detected, judging whether there is a record of the start time stamp; When there is a record of the start time stamp, recording a nearest end time stamp before the start of the non-speech data; According to the start time stamp of the speech data and the nearest end time stamp before the start of the non-speech data, the speech text segment data is intercepted, and the start time stamp of the speech data and the nearest end time stamp before the start of the non-speech data are taken as a time anchor point pair of the speech text segment data.

3. The set-top box audio-based sign language animation conversion method of claim 2, wherein, The denoising of the speech text segment data and the extracting of human voice data from the denoised speech text segment data comprise: Removing environmental sound and non-human voice noise in the speech text segment data by using a DeepFilterNet denoising model, retaining human voice and background sound, and obtaining the denoised speech text segment data; Removing background sound in the denoised speech text segment data by using a Demucs model to obtain the human voice data, and dynamically binding the human voice data with the time anchor point pair of the speech text segment data.

4. The set-top box audio-based sign language animation conversion method of claim 2, wherein, The converting of the human voice data into language text comprises: Building an automatic speech recognition model by using a Whisper architecture, training the automatic speech recognition model by using a Common Voice data set, and obtaining a trained automatic speech recognition model; Identifying the language of the human voice data by using the trained automatic speech recognition model, converting the language of the human voice data into a target language to obtain the language text, and dynamically binding the language text with the time anchor point pair of the human voice data.

5. The set-top box audio-based sign language animation conversion method of claim 2, wherein, The mapping of the language text into sign language sequences comprises: extracting a main part of the language text by using a natural language processing algorithm, and realizing real-time conversion of text syntax to sign language structure on the main part to obtain a converted main part; dissolving the converted main part into minimum morpheme units by using a dependency syntax analyzer to generate a plurality of words, and combining each of the words into the sign language sequence.

6. The set-top box audio-based sign language animation conversion method of claim 2, wherein, The rendering of the sign language sequence by the local database and the cloud database to generate a sign language animation includes: determining a perplexity of each of the words in the sign language sequence by using a preset perplexity rule; sending each of the words to the local database or the cloud database according to the perplexity of each of the words, and rendering each of the words by the local database and the cloud database to generate a corresponding cloud skeleton animation or a local skeleton animation; generating transition skeleton animations between each of the words by using a lightweight generation model in the local database; calculating a total duration of the sign language sequence according to the time anchor points, and determining a rendering frame number according to the total duration and a preset frame rate; generating the sign language animation of the rendering frame number by using an opneGL rendering technology according to the cloud skeleton animation, the local skeleton animation and the transition skeleton animations.

7. The set-top box audio-based sign language animation conversion method of claim 6, wherein, The sending of each of the words to the local database or the cloud database according to the perplexity of each of the words, and the rendering of each of the words by the local database and the cloud database to generate a corresponding cloud skeleton animation or a local skeleton animation includes: determining a complex word whose perplexity is greater than a preset perplexity threshold as the complex word, sending the complex word and the sign language sequence to the cloud database, and analyzing the complex word and the sign language sequence by the cloud database to generate the cloud skeleton animation and send the cloud skeleton animation to the local database; determining a basic word whose perplexity is less than or equal to the preset perplexity threshold as the basic word, and sending the basic word to the local database to analyze the basic word by the local database to generate the local skeleton animation.

8. A sign language animation conversion apparatus based on set-top box audio, characterized by, The device includes: a separation module configured to obtain original audio and video data of a set top box, separate the original audio and video data into original video data and original audio data by using a demultiplexer, and copy the original audio data to obtain copied audio data; a decoding module configured to process the original video data by using a video decoder to obtain decoded video data, and process the original audio data by using an audio decoder to obtain decoded audio data; an extraction module configured to extract speech text segment data from the copied audio data, de-noise the speech text segment data, and extract human voice data in the de-noised speech text segment data; a rendering module configured to convert the human voice data into language text, map the language text into a sign language sequence, and render the sign language sequence by a local database and a cloud database to generate a sign language animation. A display module is configured to display the decoded video data and the decoded audio data on a display interface and to construct a target window on the display interface, and the sign language animation is displayed synchronously through the target window.

9. A computer device, comprising: A computer readable storage medium stores a computer program, and a processor executes the computer program to implement the steps of the sign language animation conversion method based on set top box audio according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and a processor executes the computer program to implement the steps of the sign language animation conversion method based on set top box audio according to any one of claims 1-7.

Images