Voice and sign language conversion method and device

Abstract

The embodiment of the invention provides a voice and sign language conversion method and device, and the method comprises at least one of the following steps: obtaining a first sign language action video sent by a first terminal when the first terminal communicates with a second terminal, converting a first sign language action in the first sign language action video into a first voice, the first interaction information is sent to the second terminal; and acquiring a second voice sent by the second terminal in a state that the first terminal and the second terminal are in a call, converting the second voice into a second sign language action and generating a second sign language action video, and sending second interaction information to the first terminal. According to the method and the device, the problem that users using different expression modes (sign language and voice) cannot communicate efficiently is solved, and the effect that the users using different expression modes can communicate with each other without barriers by using the familiar expression modes is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method and device for converting speech and sign language. Background Art

[0002] Currently, when deaf-mute people have remote conversations with others, they usually use specific software, applets, professional equipment, service intermediaries (human intermediaries, artificial intelligence (AI) intermediaries), etc. to convert text to speech or speech to text to achieve remote conversations. These communication methods are either limited by expensive equipment, or limited by friend relationships, or are essentially text-to-speech communication, which is not the most familiar communication method for deaf-mute people. All of these have led to many limitations in the communication methods between the deaf-mute group and normal people, and they cannot really chat on their mobile phones like normal people.

[0003] At present, the main ways for deaf-mute people to communicate with healthy people are as follows:

[0004] Sign language is converted into speech through human intermediaries. The smartphone software arranges professional sign language translation customer service personnel in the background. Users can use the "dial phone" or "call customer service" function, and the sign language interpreter will complete the translation of spoken language into sign language or sign language into spoken language, so that hearing-impaired people and hearing people can make online calls;

[0005] The AI ​​sign language translator can translate sign language into text. The AI ​​sign language translator has the world's largest Chinese sign language recognition data set and can recognize most of the daily sign language expressions of the hearing-impaired. The hearing-impaired only need to face the camera to sign, and the translator screen can quickly convert the sign language into text after the background computer performs high-speed calculations.

[0006] Open the hearing-impaired escort applet through WeChat, open a video call and invite WeChat friends to join the video call. The deaf person sends text in the subtitles, and the system converts the normal person's speech into text and displays it in the subtitles, so that you can see the text chat history while watching the video.

[0007] Use offline translation media to translate sign language into speech, speech into sign language, and text into speech, but do not support direct remote video real-time call translation.

[0008] Summarizing the above methods, we can find that the current technological methods for deaf-mute people to communicate with the world mainly rely on speech-to-text, text-to-speech, intermediary sign language translation, device sign language translation and other technologies. The introduction of these technologies has improved the communication experience between deaf-mute people and others, but it still cannot solve the problem of barrier-free mobile phone conversations with more hearing people without auxiliary equipment. Summary of the invention

[0009] The embodiments of the present invention provide a method and device for converting speech and sign language, so as to at least solve the problem in the related art that it is impossible to convert sign language and speech during a call.

[0010] According to one embodiment of the present invention, a method for converting speech and sign language is provided, comprising at least one of the following: obtaining a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, converting a first sign language action in the first sign language action video into a first speech, and sending first interaction information to the second terminal, wherein the first interaction information includes the first speech; obtaining a second speech sent by the second terminal when the first terminal is in a call with the second terminal, converting the second speech into a second sign language action and generating a second sign language action video, and sending second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0011] In an exemplary embodiment, sending the first interaction information to the second terminal includes: based on the 5G ultra-high-definition audio and video call service (Voice over New Radio, VoNR), sending the first interaction information to the second terminal through the data channel of the Internet Protocol Multimedia Subsystem (IPMultimedia Subsystem, IMS); and / or sending the second interaction information to the first terminal includes: sending the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

[0012] According to another embodiment of the present invention, a device for converting speech and sign language is provided, including: a first conversion module, used to obtain a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert a first sign language action in the first sign language action video into a first speech, and send first interaction information to the second terminal, wherein the first interaction information includes the first speech; a second conversion module, used to obtain a second speech sent by the second terminal when the first terminal is in a call with the second terminal, convert the second speech into a second sign language action and generate a second sign language action video, and send the second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0013] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0014] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0015] Through the embodiments of the present invention, since the sign language action video of the first terminal is converted into voice and sent to the second terminal, the voice of the second terminal can also be converted into sign language action video and sent to the first terminal, so that users of the first terminal and the second terminal can respectively use their familiar expressions, that is, sign language and voice to communicate normally. In addition, since the embodiments of the present invention obtain the sign language action video or voice to perform sign language and voice conversion while the first terminal is talking with the second terminal, there is no need to use other auxiliary equipment, and communication does not need to be limited to friend relationships. Therefore, it can solve the problem that users using different expressions (sign language and voice) cannot communicate efficiently, and achieve the effect that users using different expressions can communicate without obstacles using their familiar expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a hardware structure block diagram of a computer terminal according to a method for converting speech and sign language according to an embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of a system for converting speech and sign language according to an embodiment of the present invention;

[0018] Figure 3 is a flow chart of a method for converting speech and sign language according to an embodiment of the present invention;

[0019] Figure 4 is a structural block diagram of a device for converting speech and sign language according to an embodiment of the present invention;

[0020] Figure 5 is a structural block diagram of a device for converting speech and sign language according to yet another embodiment of the present invention;

[0021] Figure 6 is a schematic diagram of user data transmission according to an embodiment of the present invention;

[0022] Figure 7 is a schematic diagram of a terminal display interface according to an embodiment of the present invention;

[0023] Figure 8 is a flow chart of a method for converting speech and sign language according to yet another embodiment of the present invention;

[0024] Fig. 9 is a flow chart of a method for converting speech and sign language according to yet another embodiment of the present invention. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer terminal of a method for converting speech and sign language according to an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for converting speech and sign language in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] The present application embodiment can be run on Figure 2 The speech and sign language conversion system shown in FIG. Figure 2 As shown, the system includes a new call operation management module, a new call sign language translation service application server (Application Server, AS), a VoNR capability network element, a VoNR+ media plane, a media capability platform, and a terminal.

[0031] New call operation management: responsible for the activation and setting of sign language interpretation service functions;

[0032] New call sign language interpretation service AS: responsible for the logical scheduling of sign language interpretation services;

[0033] VoNR capability network element: responsible for providing signaling forwarding and media capability interface;

[0034] VoNR+ media plane: provides media stream replication, user data interaction, a cloud rendering center based on a graphics processor to realize the generation of digital human in sign language, and provides the synthesis of real-time auxiliary subtitles;

[0035] Media capability platform: responsible for AI speech-to-text, AI text-to-sign language encoding, AI sign language action recognition, AI expression recognition, and AI lip shape recognition;

[0036] Terminal: Responsible for capturing audio and video in real time and streaming them to the media interface, interacting with users, submitting interactive data to the media interface, controlling terminal display, multi-grid expansion of video windows, and subtitle broadcasting.

[0037] In this embodiment, a method for converting speech and sign language is provided. Figure 3 is a flow chart of a method for converting speech and sign language according to an embodiment of the present invention, such as Figure 3 As shown, the process includes the following steps:

[0038] Step S302: obtaining a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, converting a first sign language action in the first sign language action video into a first voice, and sending first interaction information to the second terminal, wherein the first interaction information includes the first voice; and / or obtaining a second voice sent by the second terminal when the first terminal is in a call with the second terminal, converting the second voice into a second sign language action and generating a second sign language action video, and sending second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0039] In step S302, the first sign language action in the first sign language action video is converted into a first voice, including: converting the first sign language action in the first sign language action video into a first text, converting the first text into a first voice, wherein the first interaction information includes the first text; and / or, converting the second voice into a second sign language action and generating a second sign language action video, including: converting the second voice into a second text, converting the second text into a second sign language action video, wherein the second interaction information includes the second text.

[0040] By converting sign language movements into text and sending it to the second terminal, the user of the second terminal can refer to the text information while receiving the voice generated by the sign language conversion, thereby transmitting communication information in multiple dimensions and making communication more accurate and smooth.

[0041] In step S302, the first interaction information is sent to the second terminal, including: based on the 5G ultra-high-definition audio and video call service VoNR, the first interaction information is sent to the second terminal through the data channel of the Internet Protocol Multimedia Subsystem IMS; and / or, the second interaction information is sent to the first terminal, including: based on the VoNR technology, the second interaction information is sent to the first terminal through the data channel of the IMS.

[0042] Using the traditional 5G ultra-high-definition audio and video call service VoNR technology can realize high-quality calls with low connection delay. On the basis of VoNR technology, the interactive information generated in the embodiment of the present invention is sent to the first terminal and the second terminal via the IMS data channel, so that barrier-free communication of voice and sign language can be realized on the basis of ensuring the original call.

[0043] In step S302, the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal is obtained, and the first sign language action in the first sign language action video is converted into a first voice, including: media stream copying the video stream of the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, identifying and extracting the first sign language action in the first sign language action video, converting the first sign language action into a sign language word sequence, converting the sign language word sequence into a first text, and converting the first text into the first voice.

[0044] In step S302, a second voice sent by the second terminal when the first terminal is in a call with the second terminal is obtained, the second voice is converted into a second sign language action and a second sign language action video is generated, including: the media stream copies the audio stream of the second voice sent by the second terminal when the first terminal is in a call with the second terminal, the spoken language logic of the second voice is converted into the corresponding sign language logic, the corresponding digital human sign language action video is generated according to the sign language logic, a digital human sign language action display window is added to the first terminal, and the digital human sign language action video is sent to the digital human sign language action display window.

[0045] In the process of converting speech to sign language, the spoken language logic of the speech is first converted into the corresponding sign language logic, and then the conversion is performed. This can ensure the accuracy of the generated sign language action video, avoid the problem of incorrect conversion caused by the difference between spoken language logic and sign language logic, and improve the conversion quality of sign language and speech.

[0046] In step S302, the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal is obtained, including: judging whether the sign language and voice conversion service is activated, and when at least one of the first terminal and the second terminal has activated the sign language and voice conversion service, obtaining the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal; and / or, obtaining the second voice sent by the second terminal when the first terminal is in a call with the second terminal, including: judging whether the sign language and voice conversion service is activated, and when at least one of the first terminal and the second terminal has activated the sign language and voice conversion service, obtaining the second voice sent by the second terminal when the first terminal is in a call with the second terminal.

[0047] In an exemplary embodiment, generating a corresponding digital human sign language action video according to the sign language logic includes: querying and determining the standard sign language action corresponding to the sign language logic based on a sign language encoding library, and rendering the standard sign language action to obtain the digital human sign language action video; wherein the rendering methods include: image rendering, expression rendering, action rendering, lip rendering and scene rendering.

[0048] In an exemplary embodiment, after converting the first sign language action in the first sign language action video into a first voice, it also includes: sending the first text to the first terminal through the data channel of the IMS based on the VoNR technology; and / or, after converting the second voice into a second text, it also includes: sending the second text to the second terminal through the data channel of the IMS based on the VoNR technology.

[0049] The first text generated by converting the first sign language action is sent to the first terminal, so that the user of the first terminal can check whether the conversion result is accurate.

[0050] In this embodiment, a device for converting speech and sign language is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0051] Figure 4 is a structural block diagram of a device for converting speech and sign language according to an embodiment of the present invention, such as Figure 4 As shown, the speech and sign language conversion device 400 includes at least one of the following:

[0052] A first conversion module 10 is configured to obtain a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert the first sign language action in the first sign language action video into a first voice, and send first interaction information to the second terminal, wherein the first interaction information includes the first voice;

[0053] The second conversion module 20 is used to obtain a second voice sent by the second terminal when the first terminal and the second terminal are in a call state, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

[0054] Figure 5 is a structural block diagram of a device for converting speech and sign language according to yet another embodiment of the present invention. Figure 5 As shown, the speech and sign language conversion device 500 includes: a first conversion unit 11, a first sending unit 12, a second conversion unit 21, and a second sending unit 22.

[0055] The first conversion unit 11 is configured to convert the first sign language action in the first sign language action video into a first voice, including: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text;

[0056] A first sending unit 12 is configured to send the first interaction information to the second terminal through a data channel of the IMS based on the VoNR technology;

[0057] The second conversion unit 21 is used to convert the second speech into a second sign language action and generate a second sign language action video, including: converting the second speech into a second text, and converting the second text into a second sign language action video, wherein the second interaction information includes the second text;

[0058] The second sending unit 22 is configured to send the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

[0059] To facilitate the understanding of the technical solution provided by the present invention, a detailed description will be given below in conjunction with embodiments of specific scenarios.

[0060] Figure 6 is a schematic diagram of user data transmission according to an embodiment of the present invention, such as Figure 6 As shown, the embodiment of the present invention mainly uses VoNR+ technology to realize real-time two-way translation of sign language and voice. VoNR+ uses IMS Data Channel technology to add a data channel in addition to the voice channel and video channel. The VoNR+ network side performs layered encoding and transmission for audio and video channel services, and provides different 5G service quality identifiers (5G QoS Identifier, 5QI) for service quality (Quality of Service, QoS) guarantee; for data channel services, different data packets are identified and QoS control is implemented at a finer granularity, and new QoS parameters are introduced to support tactile data or sensor data transmission. The data channel can transmit location, pictures, text, and even richer interactive information such as auditory, visual, tactile, kinesthetic, and environmental information along with the call, upgrading the call from a single voice to a multimedia form. The transmission of user interaction data involved in the present invention uses a DC data channel.

[0061] Figure 7is a schematic diagram of a terminal display interface according to an embodiment of the present invention, such as Figure 7 As shown, the hearing-impaired person in the video window needs to show his head and elbows, and the system recognizes the sign language and converts it into voice broadcast; the system collects the voice, expression and other data of the hearing person in the video window; the sign language person's digital window displays the sign language restored from the hearing person's voice; the subtitle scrolling area presents the chat records of both parties on the same chat whiteboard, the hearing-impaired person can confirm the voice information broadcast by the system to the other party, and the hearing person can confirm the natural text of the sign language translation, and both parties can review the conversation context.

[0062] Figure 8 FIG. 1 is a flow chart of a method for converting speech and sign language according to another embodiment of the present invention, wherein the specific process is as follows:

[0063] The user of terminal A actively calls and starts real-time sign language translation;

[0064] Terminal A requests the media interface to start real-time sign language translation (taking the conversion of sign language to speech as an example)

[0065] After determining that one of the two parties in the call has activated the sign language and voice conversion service, the real-time sign language translation system changes the video of terminal A to a four-grid mode;

[0066] The system replicates media streams (video streams) and performs AI sign language recognition on video streams;

[0067] The system converts the speech generated by the sign language interpreter into text and sends the sign language recognition subtitle stream to terminal A and terminal B;

[0068] Terminal B plays the speech generated by the sign language translation.

[0069] Fig. 9 is a flow chart of a method for converting speech and sign language according to another embodiment of the present invention, and Figure 8 The method of converting speech to sign language is opposite to that of converting speech to sign language. The specific process is as follows:

[0070] Terminal B requests the media interface to start real-time sign language translation (taking the conversion of speech to sign language as an example);

[0071] After determining that one of the two parties in the call has activated the sign language and voice conversion service, the real-time sign language translation system changes the video of terminal A to a four-grid mode;

[0072] The system replicates media streams (audio streams), converts speech into text, converts text into sign language codes, and renders digital humans based on the sign language codes;

[0073] The subtitle stream and the rendered digital human video stream are sent to terminal A and terminal B.

[0074] Figure 8 and Fig. 9 The key modules involved in the conversion method between sign language and speech include: a two-way translation module between sign language and natural language, a sign language digital human rendering center, and auxiliary subtitle synthesis.

[0075] The two-way translation module between sign language and natural language collects and encodes the national sign language standard gestures. Through data entry and computer deep learning, the parsing model of self-language and sign language is generated. When the two-way translation module between sign language and natural language is used to convert spoken language into sign language, the spoken language logic is converted into the corresponding sign language logic through the natural language processing model, and the sign language logic code is passed to the rendering center.

[0076] The conversion of sign language to spoken language performs the opposite logical conversion process. The system receives the sign language action video as input and outputs the spoken text or voice. The sign language extraction in the sign language action video is completed by image recognition technology. The corresponding sign language action parsing model is parsed to obtain the sign language word sequence.

[0077] When the digital human rendering center receives the rendering instruction, it will calculate the corresponding sign language action according to the sign language sequence. Therefore, the digital human rendering center needs to synchronize the sign language coding used by the translation module with the action model library in advance, and query the action library according to the coding to generate the action instruction for each frame, so as to obtain the rendering action parameters.

[0078] In addition, the system supports multi-dimensional superposition rendering of images, expressions, lip shapes, actions, scenes, etc. It is pushed to the user terminal in the form of real-time interactive video streaming.

[0079] Avatar: supports both realistic and animated rendering. In realistic rendering, facial parameters are identified from the first few frames of the video and a rapid model is generated. In animated rendering, the user selects a digital human image provided by the system.

[0080] Clothing: The system provides a clothing library that users can select during a call.

[0081] Expression: Based on the expression knowledge base and context and context judgment

[0082] Action: Synchronize the sign language encoding library and find the corresponding gesture for the specified sign language encoding

[0083] Scene: Realistic rendering extracts the background from the video frame, while animation rendering is synthesized by users selecting the background provided by the system or uploading the background image themselves.

[0084] During the call, a unique session is generated to associate the caller and the called party. A session-level subtitle generation controller is established to perform timing control on the subtitle stream returned by AI recognition, merge it into a new subtitle stream, and push it to the calling and called terminals. After receiving the subtitle, the terminal plays it in the set subtitle display area on the screen. If the subtitle exceeds the display area, the display area automatically scrolls up to cover it.

[0085] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0086] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0087] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0088] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0089] At present, high-definition video calls under the VoNR+ technical architecture have been realized. With the combination with AI technology, the video call scene under VoNR+ technology has unlimited development potential. At the same time, the emergence of high-bandwidth, low-latency deterministic networks of 5G technology has also brought high-performance processing potential to video call scenes. The technical solution disclosed in the embodiment of the present invention focuses on improving the user experience of video calls between deaf-mute people and ordinary people. On the basis of 5G multimedia calls, IMS DC channels are carried to transmit user interaction information. A neural network model is used to achieve two-way translation of voice and sign language. During the call, the voice of ordinary people is converted into text, then translated into sign language code, and restored to sign language numbers through real-time rendering technology; at the same time, the sign language of the deaf-mute is converted into text and voice to broadcast to the other party. This way of calling brings the most familiar way of communication to both parties of the call, and provides real-time auxiliary subtitles during the call process. The system transmits communication information from multiple dimensions of text, voice, and sign language to make communication smoother.

[0090] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for converting speech and sign language, characterized in that: Include at least one of the following: Acquire a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert the first sign language action in the first sign language action video into a first voice, and send first interaction information to the second terminal, wherein the first interaction information includes the first voice; Acquire a second voice sent by the second terminal when the first terminal is in a call with the second terminal, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

2. The method according to claim 1, characterized in that Converting the first sign language action in the first sign language action video into a first voice, including: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text; and / or, Converting the second speech into a second sign language action and generating a second sign language action video includes: converting the second speech into a second text, converting the second text into a second sign language action video, wherein the second interaction information includes the second text.

3. The method according to claim 1, characterized in that: Sending the first interaction information to the second terminal includes: based on the 5G ultra-high-definition audio and video call service VoNR, sending the first interaction information to the second terminal through a data channel of the Internet Protocol Multimedia Subsystem IMS; and / or, Sending the second interaction information to the first terminal includes: sending the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

4. The method according to claim 1, characterized in that: Acquiring the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, and converting the first sign language action in the first sign language action video into a first voice, including: The media stream copies the video stream of the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, identifies and extracts the first sign language action in the first sign language action video, converts the first sign language action into a sign language word sequence, converts the sign language word sequence into a first text, and converts the first text into the first speech.

5. The method according to claim 1, characterized in that: Acquiring a second voice sent by the second terminal when the first terminal and the second terminal are in a call, converting the second voice into a second sign language action and generating a second sign language action video, including: The media stream copies the audio stream of the second voice sent by the second terminal when the first terminal and the second terminal are in a call, converts the spoken logic of the second voice into the corresponding sign language logic, generates the corresponding digital human sign language action video according to the sign language logic, adds a digital human sign language action display window for the first terminal, and sends the digital human sign language action video to the digital human sign language action display window.

6. The method according to claim 5, characterized in that Generating a corresponding digital human sign language action video according to the sign language logic includes: Based on the sign language coding library, query and determine the standard sign language action corresponding to the sign language logic, and render the standard sign language action to obtain the digital human sign language action video; Among them, rendering methods include: image rendering, expression rendering, action rendering, lip rendering and scene rendering.

7. The method according to claim 3, characterized in that After converting the first sign language action in the first sign language action video into a first voice, the method further includes: sending a first text to the first terminal through a data channel of the IMS based on the VoNR technology; and / or, After converting the second voice into a second text, the method further includes: sending the second text to the second terminal through the data channel of the IMS based on the VoNR technology.

8. The method according to claim 1, characterized in that: Acquiring the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, comprising: determining whether a sign language and voice conversion service is activated, and acquiring the first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal if at least one of the first terminal and the second terminal has activated the sign language and voice conversion service; and / or, Acquiring a second voice sent by the second terminal when the first terminal is in a call with the second terminal includes: determining whether a sign language and voice conversion service is activated, and acquiring a second voice sent by the second terminal when the first terminal is in a call with the second terminal if at least one of the first terminal and the second terminal has activated the sign language and voice conversion service.

9. A device for converting speech and sign language, characterized in that: Include at least one of the following: A first conversion module is configured to obtain a first sign language action video sent by the first terminal when the first terminal is in a call with the second terminal, convert the first sign language action in the first sign language action video into a first voice, and send first interaction information to the second terminal, wherein the first interaction information includes the first voice; The second conversion module is used to obtain a second voice sent by the second terminal when the first terminal and the second terminal are in a call state, convert the second voice into a second sign language action and generate a second sign language action video, and send second interaction information to the first terminal, wherein the second interaction information includes the second sign language action video.

10. The device according to claim 9, characterized in that The first conversion module also includes: A first conversion unit is used to convert the first sign language action in the first sign language action video into a first voice, including: converting the first sign language action in the first sign language action video into a first text, and converting the first text into a first voice, wherein the first interaction information includes the first text.

11. The device according to claim 9, characterized in that The second conversion module also includes: The second conversion unit is used to convert the second voice into a second sign language action and generate a second sign language action video, including: converting the second voice into a second text, converting the second text into a second sign language action video, wherein the second interaction information includes the second text.

12. The device according to claim 9, characterized in that The first conversion module also includes: The first sending unit is configured to send the first interaction information to the second terminal through a data channel of the IMS based on the VoNR technology.

13. The device according to claim 9, characterized in that The second conversion module also includes: The second sending unit is used to send the second interaction information to the first terminal through the data channel of the IMS based on the VoNR technology.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 8 when executed by a processor.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 8 are implemented.

Images