Voice and sign language conversion method and apparatus

Abstract

Embodiments of the present disclosure provide a voice and sign language conversion method and apparatus. The method comprises at least one of the following: acquiring a first sign language action video sent by a first terminal in a state of the first terminal making a call with a second terminal, converting a first sign language action in the first sign language action video into a first voice, and sending first interaction information to the second terminal; and acquiring a second voice sent by the second terminal in the state of the first terminal making a call with the second terminal, converting the second voice into a second sign language action, generating a second sign language action video, and sending second interaction information to the first terminal.

Description

Method and device for converting speech and sign language

[0001] Cross-references to related publications

[0002] This disclosure is based on Chinese patent disclosure CN202310790326.8, filed on June 28, 2023, entitled “Method and device for conversion between speech and sign language”, and claims the priority of the patent disclosure, and all the contents disclosed therein are incorporated into this disclosure by reference. Technical Field

[0003] The present disclosure relates to the field of communications, and more particularly, to a method and apparatus for converting speech into sign language. Background Art

[0004] Currently, when deaf-mute people communicate with others remotely, they usually use specific software, applets, professional equipment, service intermediaries (human intermediaries, artificial intelligence (AI) intermediaries), etc. to convert text to speech or speech to text to achieve remote communication. These communication methods are either limited by expensive equipment, or limited by friend relationships, or are essentially text-to-speech communication, which is not the most familiar communication method for deaf-mute people. All of these have led to many limitations in the communication methods of the deaf-mute group and the hearing-impaired group, and they cannot truly chat on their phones like normal people.

[0005] Currently, the main ways for deaf-mute people to communicate with healthy people are as follows:

[0006] Sign language is converted into speech through a human intermediary. The smartphone software arranges professional sign language interpreters in the background. Users can use the "dial call" or "call customer service" function, and the sign language interpreter will complete the translation of spoken language into sign language or sign language into spoken language, so that hearing-impaired people can have online conversations with hearing people.

[0007] Translate sign language into text using an AI sign language translator. This AI sign language translator, armed with the world's largest Chinese sign language recognition dataset, can recognize most everyday sign language expressions used by hearing-impaired individuals. Simply facing a camera, the translator's screen quickly converts the sign language into text via a high-speed computational process.

[0008] Open the hearing-impaired protection applet through WeChat, open a video call and invite WeChat friends to join the video call. The deaf person sends text in the subtitles, and the system converts the normal person's speech into text and displays it in the subtitles, so that you can see the text chat history while watching the video.

[0009] Use offline translation media to translate sign language into speech, speech into sign language, and text into speech, but do not support direct remote video real-time call translation.

[0010] Summarizing the above methods, we can see that the current technological means for deaf-mute people to communicate with the world primarily rely on speech-to-text, text-to-speech, intermediary sign language interpretation, and device-based sign language interpretation. While the introduction of these technologies is improving the communication experience for deaf-mute people, they still cannot achieve barrier-free mobile conversations with more hearing people without assistive devices.

[0011] Summary of the Invention

[0012] The embodiments of the present disclosure provide a method and apparatus for converting speech to sign language, so as to at least solve the problem in the related art that sign language to speech conversion cannot be performed during a call.

[0013] According to one embodiment of the present disclosure, a method for converting speech and sign language is provided, comprising at least one of the following: obtaining a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, converting a first sign language action in the first sign language action video into a first speech, and sending first interaction information to the second terminal, wherein the first interaction information includes the first speech; obtaining a second speech sent by the second terminal when the first terminal is in a call with the second terminal, converting the second speech into a second sign language action and generating a second sign language action video, and sending second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0014] According to another embodiment of the present disclosure, a device for converting speech and sign language is provided, including: a first conversion module, configured to obtain a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert a first sign language action in the first sign language action video into a first speech, and send first interaction information to the second terminal, wherein the first interaction information includes the first speech; a second conversion module, configured to obtain a second speech sent by the second terminal when the first terminal is in a call with the second terminal, convert the second speech into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0015] According to another embodiment of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when running.

[0016] According to another embodiment of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a hardware structure block diagram of a computer terminal according to a method for converting speech to sign language according to an embodiment of the present disclosure;

[0018] FIG2 is a schematic diagram of a system for converting speech to sign language according to an embodiment of the present disclosure;

[0019] FIG3 is a flow chart of a method for converting speech to sign language according to an embodiment of the present disclosure;

[0020] FIG4 is a structural block diagram of a device for converting speech and sign language according to an embodiment of the present disclosure;

[0021] FIG5 is a structural block diagram of a device for converting speech and sign language according to yet another embodiment of the present disclosure;

[0022] FIG6 is a schematic diagram of user data transmission according to an embodiment of the present disclosure;

[0023] FIG7 is a schematic diagram of a terminal display interface according to an embodiment of the present disclosure;

[0024] FIG8 is a flow chart of a method for converting speech and sign language according to yet another embodiment of the present disclosure;

[0025] FIG9 is a flow chart of a method for converting speech to sign language according to yet another embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and in conjunction with embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] The method embodiments provided in the embodiments disclosed in the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking operation on a computer terminal as an example, FIG1 is a hardware structure block diagram of a computer terminal of a method for converting speech to sign language in an embodiment of the present disclosure. As shown in FIG1 , the computer terminal may include one or more (only one is shown in FIG1 ) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 configured to store data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 configured to have a communication function. It will be understood by those skilled in the art that the structure shown in FIG1 is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include more or fewer components than those shown in FIG1 , or have a configuration different from that shown in FIG1 .

[0029] The memory 104 can be configured to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for converting speech to sign language in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0031] The disclosed embodiments of the present invention can run on the voice and sign language conversion system shown in Figure 2. As shown in Figure 2, the system includes a new call operation management module, a new call sign language translation service application server (Application Server, AS), a VoNR capability network element, a VoNR+ media plane, a media capability platform, and a terminal.

[0032] New call operation management: responsible for the activation and setting of sign language interpretation service functions;

[0033] New call sign language interpretation service AS: responsible for the logical scheduling of sign language interpretation services;

[0034] VoNR capability network element: responsible for providing signaling forwarding and media capability interface;

[0035] VoNR+ media interface: Provides media stream replication, user data interaction, a graphics processor-based cloud rendering center for sign language digital human generation, and real-time subtitle synthesis.

[0036] Media capability platform: responsible for AI speech-to-text, AI text-to-sign language encoding, AI sign language motion recognition, AI expression recognition, and AI lip shape recognition;

[0037] Terminal: Responsible for capturing audio and video in real time and streaming them to the media interface, interacting with users, submitting interactive data to the media interface, controlling terminal display, multi-grid expansion of video windows, and subtitle broadcasting.

[0038] In this embodiment, a method for converting speech to sign language is provided. FIG3 is a flow chart of the method for converting speech to sign language according to an embodiment of the present disclosure. As shown in FIG3 , the flow chart includes the following steps:

[0039] Step S302: Obtain a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert the first sign language action in the first sign language action video into a first voice, and send first interaction information to the second terminal, wherein the first interaction information includes the first voice; and / or, obtain a second voice sent by the second terminal when the first terminal is in a call with the second terminal, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0040] In step S302, converting the first sign language action in the first sign language action video into a first voice includes: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text; and / or converting the second voice into a second sign language action and generating a second sign language action video includes: converting the second voice into a second text, and converting the second text into a second sign language action video, wherein the second interaction information includes the second text.

[0041] By converting sign language movements into text and sending it to the second terminal, the user of the second terminal can refer to the text information while receiving the voice generated by the sign language conversion, and transmit communication information in multiple dimensions, making communication more accurate and smooth.

[0042] In step S302, the first interaction information is sent to the second terminal, including: based on the 5G ultra-high-definition audio and video call service VoNR, the first interaction information is sent to the second terminal through the data channel of the Internet Protocol Multimedia Subsystem IMS; and / or, the second interaction information is sent to the first terminal, including: based on the VoNR technology, the second interaction information is sent to the first terminal through the data channel of the IMS.

[0043] Using the traditional 5G ultra-high-definition audio and video call service VoNR technology can achieve high-quality calls with low connection delay. Based on the VoNR technology, the interactive information generated in the embodiment of the present disclosure is sent to the first terminal and the second terminal via the IMS data channel, which can achieve barrier-free communication of voice and sign language while ensuring the original call.

[0044] In step S302, the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal is obtained, and the first sign language action in the first sign language action video is converted into a first voice, including: media stream copying the video stream of the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, identifying and extracting the first sign language action in the first sign language action video, converting the first sign language action into a sign language word sequence, converting the sign language word sequence into a first text, and converting the first text into the first voice.

[0045] In step S302, a second voice sent by the second terminal when the first terminal is in a call with the second terminal is obtained, the second voice is converted into a second sign language action and a second sign language action video is generated, including: media stream copying the audio stream of the second voice sent by the second terminal when the first terminal is in a call with the second terminal, converting the spoken language logic of the second voice into corresponding sign language logic, generating a corresponding digital human sign language action video according to the sign language logic, adding a digital human sign language action display window for the first terminal, and sending the digital human sign language action video to the digital human sign language action display window.

[0046] In the process of converting speech to sign language, the spoken language logic of the speech is first converted into the corresponding sign language logic. Then, the conversion is performed. This can ensure the accuracy of the generated sign language action video, avoid conversion errors caused by the difference between spoken language logic and sign language logic, and improve the conversion quality between sign language and speech.

[0047] In step S302, the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal is obtained, including: judging whether the sign language and voice conversion service is activated, and when at least one of the first terminal and the second terminal has activated the sign language and voice conversion service, obtaining the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal; and / or, obtaining the second voice sent by the second terminal when the first terminal is in a call with the second terminal, including: judging whether the sign language and voice conversion service is activated, and when at least one of the first terminal and the second terminal has activated the sign language and voice conversion service, obtaining the second voice sent by the second terminal when the first terminal is in a call with the second terminal.

[0048] In an exemplary embodiment, generating a corresponding digital human sign language action video according to the sign language logic includes: querying and determining the standard sign language action corresponding to the sign language logic based on a sign language coding library, and rendering the standard sign language action to obtain the digital human sign language action video; wherein, the rendering methods include: image rendering, expression rendering, action rendering, lip rendering and scene rendering.

[0049] In an exemplary embodiment, after converting the first sign language action in the first sign language action video into a first voice, it also includes: sending the first text to the first terminal through the data channel of the IMS based on the VoNR technology; and / or, after converting the second voice into a second text, it also includes: sending the second text to the second terminal through the data channel of the IMS based on the VoNR technology.

[0050] The first text generated by converting the first sign language action is sent to the first terminal, so that the user of the first terminal can check whether the conversion result is accurate.

[0051] Through the embodiment of the present disclosure, since the sign language action video of the first terminal is converted into voice and sent to the second terminal, the voice of the second terminal can also be converted into sign language action video and sent to the first terminal, so that users of the first terminal and the second terminal can respectively use their familiar expressions, that is, sign language and voice to communicate normally. Moreover, since the embodiment of the present disclosure obtains the sign language action video or voice to perform sign language and voice conversion while the first terminal is talking to the second terminal, there is no need to use other auxiliary equipment, and communication does not need to be limited to friend relationships. Therefore, it can solve the problem that users using different expressions (sign language and voice) cannot communicate efficiently, and achieve the effect that users using different expressions can communicate without obstacles using the expressions they are familiar with.

[0052] This embodiment also provides a device for converting speech to sign language. The device is configured to implement the above-mentioned embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0053] FIG4 is a structural block diagram of a device for converting speech to sign language according to an embodiment of the present disclosure. As shown in FIG4 , the device 400 for converting speech to sign language includes at least one of the following:

[0054] A first conversion module 10 is configured to obtain a first sign language action video sent by a first terminal when a first terminal is in a call with a second terminal, convert the first sign language action in the first sign language action video into a first voice, and send first interaction information to the second terminal, wherein the first interaction information includes the first voice;

[0055] The second conversion module 20 is configured to obtain a second voice sent by the second terminal when the first terminal and the second terminal are in a call, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0056] Figure 5 is a structural block diagram of a speech and sign language conversion device according to another embodiment of the present disclosure. As shown in Figure 5, the speech and sign language conversion device 500 includes: a first conversion unit 11, a first sending unit 12, a second conversion unit 21, and a second sending unit 22.

[0057] A first conversion unit 11 is configured to convert a first sign language action in the first sign language action video into a first voice, including: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text;

[0058] The first sending unit 12 is configured to send the first interaction information to the second terminal through the IMS data channel based on the VoNR technology;

[0059] The second conversion unit 21 is configured to convert the second speech into a second sign language action and generate a second sign language action video, including: converting the second speech into a second text, and converting the second text into a second sign language action video, wherein the second interaction information includes the second text;

[0060] The second sending unit 22 is configured to send the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

[0061] To facilitate understanding of the technical solutions provided by the embodiments of the present disclosure, the embodiments will be described in detail below in conjunction with specific scenarios.

[0062] Figure 6 is a schematic diagram of user data transmission according to an embodiment of the present disclosure. As shown in Figure 6, the embodiment of the present disclosure mainly uses VoNR+ technology to achieve real-time two-way translation of sign language and voice. VoNR+ uses IMS Data Channel technology to add a data channel in addition to the voice channel and video channel. The VoNR+ network side performs layered encoding and transmission for audio and video channel services, and provides different 5G service quality identifiers (5G QoS Identifier, 5QI) for service quality (QoS); for data channel services, it identifies different data packets and implements QoS control at a finer granularity, introducing new QoS parameters to support tactile data or sensor data transmission. The data channel can transmit location, pictures, text, and even richer interactive information such as auditory, visual, tactile, kinesthetic, and environmental information along with the call, upgrading the call from a single voice to a multimedia form. The transmission of user interactive data involved in this disclosure all uses DC data channels.

[0063] Figure 7 is a schematic diagram of the terminal display interface according to an embodiment of the present disclosure. As shown in Figure 7, the hearing-impaired person needs to expose his head and elbows in the hearing-impaired person's video window, and the system recognizes sign language and converts it into voice broadcast; the system collects the hearing person's voice, expression and other data in the hearing person's video window; the sign language person's digital window displays the sign language restored by the hearing person's voice conversion; the subtitle scrolling area presents the chat records of both parties on the same chat whiteboard, the hearing-impaired person can confirm the voice information broadcast by the system to the other party, and the hearing person can confirm the natural text of the sign language translation, and both parties can review the conversation context.

[0064] FIG8 is a flow chart of a method for converting speech and sign language according to yet another embodiment of the present disclosure. The specific process is as follows:

[0065] The user of terminal A actively calls and starts real-time sign language translation;

[0066] Terminal A requests the media interface to enable real-time sign language translation (taking the conversion of sign language to speech as an example);

[0067] After confirming that one of the two parties in the call has activated the sign language to voice conversion service, the real-time sign language translation system changes the video of terminal A to a four-grid mode;

[0068] The system replicates media streams (video streams) and performs AI sign language recognition on video streams;

[0069] The system converts the speech generated by the sign language interpreter into text and sends the sign language recognition subtitle stream to Terminal A and Terminal B;

[0070] Terminal B plays the speech generated by the sign language interpretation.

[0071] FIG9 is a flow chart of a method for converting speech to sign language according to another embodiment of the present disclosure. In contrast to the method for converting speech to sign language shown in FIG8 , the system converts speech to sign language. The specific process is as follows:

[0072] Terminal B requests the media interface to enable real-time sign language translation (taking voice to sign language conversion as an example);

[0073] After confirming that one of the two parties in the call has activated the sign language to voice conversion service, the real-time sign language translation system changes the video of terminal A to a four-grid mode;

[0074] The system replicates media streams (audio streams), converts speech into text, converts text into sign language codes, and renders digital humans based on the sign language codes;

[0075] The subtitle stream and the rendered digital human video stream are sent to terminal A and terminal B.

[0076] The key modules involved in the sign language and speech conversion method in Figures 8 and 9 include: a two-way translation module between sign language and natural language, a sign language digital human rendering center, and auxiliary subtitle synthesis.

[0077] The bidirectional translation module for sign language and natural language incorporates and encodes the national sign language standard gestures. Through data entry and computer deep learning, a parsing model for both speech and sign language is generated. When the bidirectional translation module is configured to convert spoken language into sign language, it uses a natural language processing model to convert the spoken language logic into the corresponding sign language logic, and then transmits the sign language logic encoding to the rendering center.

[0078] Sign language to spoken language conversion uses the opposite logical conversion process. The system receives a video of sign language movements as input and outputs spoken text or speech. Sign language extraction from the video is accomplished using image recognition technology. The corresponding sign language movement parsing model is used to parse the resulting sign language word sequence.

[0079] When the digital human rendering center receives rendering instructions, it calculates the corresponding sign language movements based on the sign language sequence. To do this, the digital human rendering center needs to synchronize the sign language encoding used by the translation module with the movement model library in advance. It then queries the movement library based on the encoding to generate the movement instructions for each frame, thereby obtaining the rendering movement parameters.

[0080] In addition, the system supports multi-dimensional overlay rendering of images, expressions, lip shapes, movements, scenes, etc., and pushes them to the user terminal in the form of real-time interactive video streaming.

[0081] Avatars: Supports both realistic and animated rendering. For realistic rendering, facial parameters are identified from the first few frames of the video, allowing for rapid model generation. For animated rendering, users select a system-provided digital human image.

[0082] Clothing: The system provides a clothing library that users can select during a call.

[0083] Expression: It is implemented based on the expression knowledge base and context and context judgment.

[0084] Action: Synchronize the sign language encoding library and find the corresponding gesture for the specified sign language encoding.

[0085] Scene: Realistic rendering extracts the background from the video frame, while animation rendering allows users to select the background provided by the system or upload their own background image for synthesis.

[0086] During a call, a unique session is generated, associating the caller and the called party. A session-level subtitle generation controller is established to control the timing of the subtitle streams returned by AI recognition, merge them into a new subtitle stream, and push it to the calling and called parties. Upon receiving the subtitles, the terminals play them in the designated display area on the screen. If the subtitles exceed the display area, the display area automatically scrolls up to cover them.

[0087] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0088] An embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0089] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0090] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0091] Currently, high-definition video calls under the VoNR+ technical architecture have been realized. With the combination of AI technology, video call scenarios under VoNR+ technology have unlimited development potential. At the same time, the emergence of high-bandwidth, low-latency deterministic networks in 5G technology has also brought high-performance processing potential to video call scenarios. The technical solution disclosed in the embodiments of this disclosure focuses on improving the user experience of video calls between deaf and mute people. Based on 5G multimedia calls, it carries an IMSDC channel to transmit user interaction information. A neural network model is used to achieve two-way translation of speech and sign language. During the call, the speech of the ordinary person is converted into text, then translated into sign language code, and restored to sign language numbers through real-time rendering technology; at the same time, the sign language of the deaf and mute is converted into text and voice and broadcast to the other party. This call method provides the most familiar communication method for both parties, and provides real-time auxiliary subtitles during the call. The system transmits communication information from multiple dimensions of text, voice, and sign language, making communication smoother.

[0092] Obviously, those skilled in the art should understand that the modules or steps of the present disclosure described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any particular combination of hardware and software.

[0093] The above description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will appreciate that various modifications and variations of the present disclosure are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present disclosure are intended to be within the scope of protection of the present disclosure.

Claims

1. A method for converting speech and sign language, comprising at least one of the following: A first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal is obtained, a first sign language action in the first sign language action video is converted into a first voice, and first interaction information is sent to the second terminal, wherein: The first interaction information includes the first voice; Acquire a second voice sent by the second terminal when the first terminal is in a call with the second terminal, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

2. The method according to claim 1, wherein: Converting the first sign language action in the first sign language action video into a first voice, including: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text; and / or, Converting the second speech into a second sign language action and generating a second sign language action video includes: converting the second speech into a second text, converting the second text into a second sign language action video, wherein the second interaction information includes the second text.

3. The method according to claim 1, wherein: Sending the first interaction information to the second terminal includes: based on the 5G ultra-high-definition audio and video call service VoNR, sending the first interaction information to the second terminal through a data channel of the Internet Protocol Multimedia Subsystem IMS; and / or, Sending the second interaction information to the first terminal includes: sending the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

4. The method according to claim 1, wherein: Acquiring the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, and converting the first sign language action in the first sign language action video into a first voice, including: The media stream copies the video stream of the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, identifies and extracts the first sign language action in the first sign language action video, converts the first sign language action into a sign language word sequence, converts the sign language word sequence into a first text, and converts the first text into the first speech.

5. The method according to claim 1, wherein: Acquiring a second voice sent by the second terminal when the first terminal and the second terminal are in a call, converting the second voice into a second sign language action and generating a second sign language action video, including: The media stream copies the audio stream of the second voice sent by the second terminal when the first terminal and the second terminal are in a call, converts the spoken logic of the second voice into the corresponding sign language logic, generates the corresponding digital human sign language action video according to the sign language logic, adds a digital human sign language action display window for the first terminal, and sends the digital human sign language action video to the digital human sign language action display window.

6. The method according to claim 5, wherein: Generating a corresponding digital human sign language action video according to the sign language logic includes: Based on the sign language coding library, query and determine the standard sign language action corresponding to the sign language logic, and render the standard sign language action to obtain the digital human sign language action video; Among them, rendering methods include: image rendering, expression rendering, action rendering, lip rendering and scene rendering.

7. The method according to claim 3, wherein: After converting the first sign language action in the first sign language action video into a first voice, the method further includes: sending a first text to the first terminal through a data channel of the IMS based on the VoNR technology; and / or, After converting the second voice into a second text, the method further includes: sending the second text to the second terminal through the data channel of the IMS based on the VoNR technology.

8. The method according to claim 1, wherein: Acquiring the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, comprising: determining whether a sign language and voice conversion service is activated, and acquiring the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal if at least one of the first terminal and the second terminal has activated the sign language and voice conversion service; and / or, Acquiring a second voice sent by the second terminal when the first terminal is in a call with the second terminal includes: determining whether a sign language and voice conversion service is activated, and acquiring a second voice sent by the second terminal when the first terminal is in a call with the second terminal if at least one of the first terminal and the second terminal has activated the sign language and voice conversion service.

9. A device for converting speech and sign language, comprising at least one of the following: The first conversion module is configured to obtain a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert the first sign language action in the first sign language action video into a first voice, and send the first interaction information to the second terminal, wherein: The first interaction information includes the first voice; The second conversion module is configured to obtain a second voice sent by the second terminal when the first terminal and the second terminal are in a call state, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

10. The device according to claim 9, wherein: The first conversion module also includes: The first conversion unit is configured to convert the first sign language action in the first sign language action video into a first voice, including: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text.

11. The device according to claim 9, wherein: The second conversion module also includes: The second conversion unit is configured to convert the second voice into a second sign language action and generate a second sign language action video, including: converting the second voice into a second text, converting the second text into a second sign language action video, wherein the second interaction information includes the second text.

12. The device according to claim 9, wherein: The first conversion module also includes: The first sending unit is configured to send the first interaction information to the second terminal through the data channel of the IMS based on the VoNR technology.

13. The device according to claim 9, wherein: The second conversion module also includes: The second sending unit is configured to send the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

14. A computer-readable storage medium having a computer program stored therein, wherein: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 8 are implemented.

Images