Display method, intelligent terminal and storage medium

Abstract

The present application proposes a display method, a smart terminal and a storage medium, the method comprising: S10: confirming or acquiring target information in response to receiving a trigger operation for a target interface; S20: displaying interactive text according to the target information. The present application determines and displays the interactive text corresponding to the target information by performing image recognition or voice recognition on the target information, thereby improving the friendliness of the voice and video functions in the communication software to special groups and improving the user experience of the product.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a display method, an intelligent terminal and a storage medium. Background Art

[0002] With the development of a large number of voice communication software, communication between people has become more convenient and faster. However, people with hearing impairments and speech impairments cannot communicate through voice communication software because they cannot hear or make sounds. For ordinary people, it is also difficult to communicate with people with hearing impairments and speech impairments through sign language without learning sign language.

[0003] In the process of conceiving and implementing this application, the inventor found that there are at least the following problems:

[0004] For ordinary people, sign language is difficult to learn. On smart devices, if users communicate through sign language, ordinary people cannot understand the meaning of the sign language, resulting in the inability to use the video call and voice functions in the communication software.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] In response to the above technical problems, the present application provides a display method, an intelligent terminal and a storage medium, which effectively improve the friendliness of voice and video functions to special groups of people, enhance the convenience and diversity of the product, and make the operation simple and convenient.

[0007] In order to solve the above technical problems, the present application provides a display method, comprising:

[0008] S10: in response to receiving a trigger operation for a target interface, confirming or acquiring target information;

[0009] S20: Displaying the interactive text according to the target information.

[0010] Optionally, the step S20 includes:

[0011] S21: Acquire emotional information and text information corresponding to the target information;

[0012] S22: Generate interactive text according to the emotion information and text information.

[0013] Optionally, the target information includes image information and / or audio information, and the step S21 includes at least one of the following:

[0014] In response to obtaining at least one of expression information, limb movement amplitude, and environmental information from the image information, determining or obtaining the emotion information;

[0015] In response to obtaining at least one of speech speed, voice volume, and preset words from the audio information, the emotion information is determined or obtained.

[0016] Optionally, the target information includes image information, and the step S21 includes:

[0017] performing gesture recognition according to the image information;

[0018] The text information is generated according to the gesture recognition result.

[0019] Optionally, performing gesture recognition according to the image information includes:

[0020] Identify the motion trajectory of the key points of the hand in the image information, and obtain the gesture recognition result according to the motion trajectory of the key points of the hand;

[0021] and / or,

[0022] The generating the text information according to the gesture recognition result includes: generating a word segmentation result according to the gesture recognition result; and generating text information according to the word segmentation result.

[0023] Optionally, the step S22 includes:

[0024] Generate or determine an emotion characteristic symbol according to the emotion information;

[0025] Generate or determine text based on text information;

[0026] The interactive text is determined according to the emotional characteristic symbols and words.

[0027] Optionally, the emotion characteristic symbols include emoticons and / or modal particles.

[0028] Optionally, the step S10 includes at least one of the following:

[0029] In response to obtaining the preset instruction, displaying a target interface;

[0030] In response to the current user interface belonging to a preset interface, a target interface is displayed.

[0031] The present application also provides an intelligent terminal, comprising: a memory and a processor, wherein a display program is stored in the memory, and when the display program is executed by the processor, the steps of the above-mentioned display method are implemented.

[0032] The present application also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0033] As mentioned above, the display method of the present application is applied to a smart terminal, comprising the steps of: S10: confirming or acquiring target information in response to receiving a trigger operation for a target interface; S20: displaying interactive text according to the target information. Through the above technical solution, the friendliness of the voice and video functions in the communication software to special groups is improved, and the user experience of the product is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0035] Figure 1 A schematic diagram of the hardware structure of a smart terminal for implementing each embodiment of the present application;

[0036] Figure 2 A communication network system architecture diagram provided for an embodiment of the present application;

[0037] Figure 3 is a flow chart of a display method according to the first embodiment of the present application;

[0038] Figure 4 is a flow chart of a display method according to the second embodiment of the present application;

[0039] Figure 5 is a flow chart of a display method according to the third embodiment of the present application;

[0040] Figure 6 It is a flowchart of a display method according to the fourth embodiment of the present application.

[0041] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings have shown clear embodiments of this application, which will be described in more detail later. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0044] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate that there are the described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or", "and / or", "including at least one of the following" etc. used in this application can be interpreted as inclusive, or mean any one or any combination. For example, “comprising at least one of the following: A, B, C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”, and for another example, “A, B or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”. An exception to this definition will only occur when a combination of elements, functions, steps or operations are inherently mutually exclusive in some manner.

[0045] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0046] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] It should be noted that in this article, step codes such as S10 and S20 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S20 first and then S10, etc., but these should all be within the scope of protection of this application.

[0048] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0050] The smart terminal may be implemented in various forms. For example, the smart terminal described in this application may include smart terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0051] The following description will be made by taking a mobile terminal as an example, and those skilled in the art will understand that, in addition to components specifically used for mobile purposes, the construction according to the embodiments of the present application can also be applied to fixed-type terminals.

[0052] See also Figure 1 , which is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present application, the mobile terminal 100 may include: RF (Radio Frequency) unit 101, WiFi module 102, audio output unit 103, A / V (audio / video) input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, processor 110, and power supply 111 and other components. Those skilled in the art can understand that Figure 1 The structure of the mobile terminal shown in the figure does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] Combine the following Figure 1 The following is a detailed introduction to the various components of the mobile terminal:

[0054] The radio frequency unit 101 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving the downlink information of the base station, it is sent to the processor 110 for processing; in addition, the uplink data is sent to the base station. Generally, the radio frequency unit 101 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 101 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution) and 5G, etc.

[0055] WiFi is a short-range wireless transmission technology. Mobile terminals can help users send and receive emails, browse web pages, and access streaming media through WiFi module 102. It provides users with wireless broadband Internet access. Figure 1 The WiFi module 102 is shown, but it is understandable that it is not an essential component of the mobile terminal and can be omitted as needed without changing the essence of the invention.

[0056] The audio output unit 103 can convert the audio data received by the RF unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output it as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, etc. Moreover, the audio output unit 103 can also provide audio output related to a specific function performed by the mobile terminal 100 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0057] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processor (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 106. The image frame processed by the graphics processor 1041 can be stored in the memory 109 (or other storage medium) or sent via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a telephone call mode, a recording mode, a voice recognition mode, and other operating modes, and can process such sound into audio data. The processed audio (voice) data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode. The microphone 1042 can implement various types of noise elimination (or suppression) algorithms to eliminate (or suppress) noise or interference generated in the process of receiving and sending audio signals.

[0058] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and / or the backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured on the mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0059] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0060] The user input unit 107 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile terminal. Optionally, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1071 or near the touch panel 1071 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 110, and can receive and execute the command sent by the processor 110. In addition, the touch panel 1071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may further include other input devices 1072. Optionally, the other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, etc., which are not specifically limited here.

[0061] Optionally, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Then, the processor 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Figure 1 In the figure, the touch panel 1071 and the display panel 1061 are used as two independent components to implement the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to implement the input and output functions of the mobile terminal, which is not limited here.

[0062] The interface unit 108 serves as an interface through which at least one external device can be connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.

[0063] The memory 109 can be used to store software programs and various data. The memory 109 can mainly include a program storage area and a data storage area. Optionally, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 109 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0064] The processor 110 is the control center of the mobile terminal. It uses various interfaces and lines to connect various parts of the entire mobile terminal. It executes various functions of the mobile terminal and processes data by running or executing software programs and / or modules stored in the memory 109, and calling data stored in the memory 109, so as to monitor the mobile terminal as a whole. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.

[0065] The mobile terminal 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 may be logically connected to the processor 110 via a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0066] although Figure 1 Not shown, the mobile terminal 100 may further include a Bluetooth module, etc., which will not be described in detail here.

[0067] In order to facilitate understanding of the embodiments of the present application, the communication network system on which the mobile terminal of the present application is based is described below.

[0068] See also Figure 2 , Figure 2 A communication network system architecture diagram is provided for an embodiment of the present application, wherein the communication network system is an LTE system of universal mobile communication technology, and the LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203 and an operator's IP service 204, which are connected in sequence for communication.

[0069] Optionally, UE201 may be the above-mentioned terminal 100, which will not be described in detail here.

[0070] E-UTRAN 202 includes eNodeB 2021 and other eNodeBs 2022 , etc. Optionally, eNodeB 2021 may be connected to other eNodeBs 2022 via a backhaul (eg, an X2 interface), and eNodeB 2021 is connected to EPC 203 , and eNodeB 2021 may provide UE 201 with access to EPC 203 .

[0071] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gate Way) 2034, PGW (PDN Gate Way) 2035 and PCRF (Policy and Charging Rules Function) 2036, etc. Optionally, MME 2031 is a control node that processes signaling between UE 201 and EPC 203, and provides bearer and connection management. HSS 2032 is used to provide some registers to manage functions such as home location register (not shown in the figure), and store some user-specific information such as service features and data rates. All user data can be sent through SGW2034, PGW2035 can provide IP address allocation and other functions for UE 201, PCRF2036 is the policy and charging control policy decision point for service data flow and IP bearer resources, which selects and provides available policy and charging control decisions for the policy and charging execution function unit (not shown in the figure).

[0072] The IP service 204 may include the Internet, an intranet, an IMS (IP Multimedia Subsystem) or other IP services.

[0073] Although the above introduction takes the LTE system as an example, those skilled in the art should know that the present application is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA and future new network systems (such as 5G), etc., without limitation here.

[0074] Based on the above-mentioned mobile terminal hardware structure and communication network system, various embodiments of the present application are proposed.

[0075] First embodiment

[0076] Reference Figure 3 , Figure 3 The first embodiment of the display method of the present invention comprises the following steps:

[0077] The target interface is displayed.

[0078] Optionally, the AI ​​(Artificial Intelligence) auxiliary system includes an interactive auxiliary function. The target interface in this embodiment uses an interactive auxiliary function selection interface. Based on the target interface, the input mode and output mode of the interactive auxiliary function can be selected. Optionally, the input mode can be voice input, gesture input, or text input, among which voice input can be the collection of audio information, gesture input can be the collection of image information, and text input can be the collection of text information.

[0079] Optionally, the output method may be voice output, gesture output or text output, etc., wherein voice output may be output of audio information, gesture output may be output of image information, and text output may be output of text information.

[0080] Optionally, the speech output is implemented by a speech synthesis framework of the Tacotron model plus Hifigan, wherein the Tacotron model is an end-to-end speech synthesis model and Hifigan is an efficient vocoder with multi-scale and multi-cycle discriminators. Optionally, the gesture output can be output by the virtual human moving key points in sequence according to the given gesture information.

[0081] Optionally, if voice input and voice output are selected, the language type corresponding to the voice input is converted into the language type corresponding to the voice input. Optionally, the language type can be English, German, etc., and the voice type can also be a dialect, such as Cantonese.

[0082] Optionally, if voice input and gesture output are selected, gesture information is generated and output according to the voice information of the voice input. Optionally, when chatting with a person with hearing impairment, voice information can be generated into gesture information to facilitate communication with the person with hearing impairment.

[0083] Optionally, if voice input and text output are selected, text information is generated and output based on the voice information of the voice input, that is, voice recognition is performed on the input voice information.

[0084] Optionally, if gesture input and voice output are selected, voice information is generated and output according to the gesture information of the gesture input. Optionally, the gesture information is converted into voice information according to the preset correspondence between gesture information and voice information, for example, if the input gesture information is an ok gesture, the output voice information is "OK".

[0085] Optionally, if gesture input and gesture output are selected, the gesture information of the gesture output is corrected and output according to the gesture information of the gesture input. Optionally, the input gesture information is corrected according to the sign language rules, and the corrected gesture information is output.

[0086] Optionally, if gesture input and text output are selected, text information is generated and output according to the gesture information of the gesture input, so as to realize the gesture translation function. Optionally, according to the preset correspondence between gesture information and voice information, the voice information is converted into gesture information, for example, if the input voice information is "OK", the output gesture information is the OK gesture.

[0087] Optionally, if text input and voice output are selected, voice information is generated and output based on the text information of the text input, that is, voice synthesis is performed on the input text information.

[0088] Optionally, if text input and gesture output are selected, gesture information is generated and output according to the text information of the text input. When chatting with a person with hearing impairment, the text information can be generated into gesture information to facilitate communication with the person with hearing impairment.

[0089] Optionally, if text input and text output are selected, the text information of the text input is corrected or filtered, and the corrected or filtered text information is generated and output. Optionally, the text information input is corrected according to a preset template. For example, the text information of the text input is "pick up the courier", and the text information of the corrected text output is "please help pick up the courier". Optionally, the input text information is filtered according to a preset keyword. For example, when the keyword is a letter, the text information of the text input is "pick up the courier and the letter", and the text information of the filtered text output is "pick up the courier".

[0090] Optionally, in response to obtaining a preset instruction, a target interface is displayed. Optionally, the preset instruction may be an instruction to enable an interactive function; and / or, in response to the current user interface belonging to a preset interface, a target interface is displayed. Optionally, the preset interface is an interface of a preset application, such as an emergency call application.

[0091] Optionally, the target interface includes a function switch button, a target information selection button, an interactive text selection button, etc.

[0092] Optionally, when the function switch button is turned on, step S20 is performed. When the function switch button is turned off, the collected target information is not processed.

[0093] Optionally, the target information selection button can be used to select the type of target information to be collected. Optionally, the target information can be at least one of image information and audio information.

[0094] Optionally, the type of interactive text can be selected through the interactive text selection button. Optionally, the type of interactive text can be different languages ​​or different dialects.

[0095] S10, in response to receiving a trigger operation for the target interface, generating an interactive text according to the collected target information.

[0096] Optionally, the target information is collected information, and the target information includes at least one of image information and audio information.

[0097] Optionally, when the target information includes image information, images or videos such as gesture information and / or body information of the person can be obtained; when the target information includes audio information, the person's speaking speed, voice volume, preset words, etc. can be obtained.

[0098] Optionally, the target information may also be text information, and the text information is converted into interactive text in a preset language.

[0099] Step S20: display the interactive text.

[0100] Optionally, the interactive text is displayed in a preset area. Optionally, for example, in a video chat scenario, the preset area may be a current chat interface.

[0101] Optionally, in scenarios such as video chat, when hearing and hearing impaired person A is chatting with friend B, hearing and hearing impaired person A expresses his or her words by inputting gesture information. Through the method in this embodiment, the gesture information is translated into interactive text or voice information and transmitted to friend B's terminal for display. Optionally, friend B can reply via text message or reply to handicapped person A via voice. Through the method in this embodiment, the voice content can be converted into text and displayed on the terminal interface of handicapped person A. Optionally, the voice content can also be converted into gesture information and displayed on the terminal interface of handicapped person A, so that hearing and hearing impaired people can communicate with family and friends through video or voice like ordinary people, while solving the problem of friends not understanding sign language and improving the convenience of communication.

[0102] Optionally, in scenarios such as emergency calls and phone calls, the front camera is activated in response to the user's trigger operation on the target interface to obtain image information containing the user's body movements and gesture information, and the body movements and gesture information (sign language) are converted into voice information or interactive text through image recognition technology and transmitted to the receiving end.

[0103] Optionally, when a person with hearing or speech impairment communicates with an ordinary person, in a scenario where the ordinary person uses voice or video to communicate, the ordinary person's voice can be converted into text or gestures to facilitate the person with hearing or speech impairment to understand what the ordinary person wants to express.

[0104] Optionally, when a person with hearing or speech impairment uses sign language to communicate with an ordinary person via video, the sign language is converted into text or speech so that an ordinary person who does not understand sign language can understand what the person with hearing or speech impairment wants to express.

[0105] In the technical solution of this embodiment, the target interface is displayed; in response to receiving a trigger operation on the target interface, an interactive text is generated according to the collected target information, and the interactive text is displayed. By performing image recognition or voice recognition on the target information, the interactive text corresponding to the target information is determined, which improves the friendliness of the voice and video functions in the communication software to special groups and improves the convenience of the product.

[0106] Second embodiment

[0107] Reference Figure 4 , Figure 4 This is a second embodiment of the display method of the present invention. Based on the first embodiment, the step S20 includes:

[0108] S21, acquiring emotion information and text information corresponding to the target information;

[0109] S22, generating an interactive text according to the emotion information and the text information.

[0110] Optionally, the emotional information corresponding to the target information is identified according to the application scenario. Optionally, when the application scenario is a chat scenario, the emotional information includes information such as anger, anxiety, worry, happiness, and relief. Optionally, when the application scenario is a service evaluation, the emotional information includes information such as satisfaction and dissatisfaction with the service.

[0111] Optionally, the target information includes image information and / or audio information.

[0112] Optionally, when the target information includes image information, such as image information of a hearing-impaired person doing sign language, the emotional information and text information are obtained according to at least one of the user's facial expression information, body movement amplitude, and environmental information in the target information. Optionally, the emotional information corresponding to at least one of the user's facial expression information, body movement amplitude, and environmental information is determined or obtained respectively, and each piece of emotional information is aggregated to obtain the emotional information corresponding to the target information.

[0113] Optionally, the expression information includes information on the activities of the five sense organs, and the emotion information is determined or obtained based on the expression information. For example, if the expression information is an open mouth and wide eyes, the determined or obtained emotion information is surprise, etc.

[0114] Optionally, when the amplitude of the limb movement is greater than a preset amplitude value, the emotional information is determined or obtained as excitement or anxiety, etc.; when the amplitude of the limb movement is less than or equal to the preset amplitude value, the emotional information is determined or obtained as relaxation or calmness, etc.

[0115] Optionally, when the environment information is in a chaotic state, for example, the items in the room are placed in a disorderly manner, the emotional information is determined or obtained as anxious, etc. When the environment information is in a tidy state, the emotional information is determined or obtained as calm, etc.

[0116] Optionally, when the target information includes audio information, the emotion information is obtained according to at least one of the user's speech speed, voice volume, and preset words in the target information. Optionally, the text information contained in the audio information is obtained through speech recognition technology.

[0117] Optionally, corresponding emotion information is obtained through at least one of the user's speaking speed, voice volume, and preset words.

[0118] Optionally, when the speech speed is greater than a preset speed, the emotional information is determined or obtained as excitement, etc., and when the speech speed is less than or equal to the preset speed, the emotional information is determined or obtained as calmness, etc.

[0119] Optionally, when the voice volume is greater than a preset volume value, the emotional information is determined or obtained as anger, etc., and when the voice volume is less than or equal to the preset volume value, the emotional information is determined or obtained as calm, etc.

[0120] Optionally, when the user's audio information includes a preset first word, the preset first word may be a positive emotional word, such as expectation, and the emotional information is determined to be positive emotional information, such as a happy mood, etc. When the user's audio information includes a preset second word, the preset second word may be a negative emotional word, such as wanting to cry, and the emotional information is determined to be negative emotional information, such as sadness.

[0121] Optionally, when the target information includes audio and image information, such as a video call, the emotional information is obtained based on at least one of the user's facial expression information, body movement amplitude, and environmental information in the target information; or the emotional information is obtained based on at least one of the user's speech speed, voice volume, and preset words in the target information; and the text information is determined through voice recognition or gesture recognition. Then, the interactive text is determined based on the emotional information and the text information.

[0122] Optionally, an interactive text is generated based on the emotional information and text information. For example, the text information is generally recognized as "The weather is great today", but if the user's emotional information is sensed, such as the emotional information is a happy mood, the interactive text displayed can be "The weather is great today", with a smiley face expression, making the expression more vivid and accurate.

[0123] In the technical solution of this embodiment, in response to receiving a trigger operation on a target interface, target information is obtained; emotional information and text information corresponding to the target information are obtained; and interactive text is generated based on the emotional information and text information. By identifying the emotional information corresponding to the target information and generating interactive text based on the emotional information and the target information, the accuracy of the interactive text is improved, and the text information is combined with the user's emotions, making the final interactive text more vivid.

[0124] Third embodiment

[0125] Reference Figure 5 , Figure 5 This is a third embodiment of the display method of the present invention, based on the first or second embodiment, the step S21 includes:

[0126] S211, performing gesture recognition according to the image information;

[0127] S212: Generate the text information according to the gesture recognition result.

[0128] Optionally, when the target information includes image information, gesture recognition is performed according to the image information to obtain a gesture recognition result, and text information is generated according to the gesture recognition result.

[0129] Optionally, the gesture recognition may be sign language recognition, which converts image information into text information according to sign language rules. Optionally, the gesture recognition may be a preset gesture recognition, for example, the interactive text corresponding to the OK gesture is "agree", the interactive text corresponding to the waving gesture is "reject", etc.

[0130] Optionally, step S211 includes: identifying the key points of the hand in the image information, as well as identifying the movement trajectory of the key points of the hand, and obtaining a gesture recognition result according to the movement trajectory of the key points of the hand; wherein, different combinations of the key points of the hand and the movement trajectory of the key points of the hand correspond to different gesture recognition results. For example, when the key point of the hand is the index finger and the movement trajectory of the index finger is swaying left and right, the gesture recognition result is "no"; when the key point of the hand is the index finger and the movement trajectory of the index finger is pointing to oneself, the gesture recognition result is the person who makes the gesture himself / herself.

[0131] Optionally, step S212 includes: generating a word segmentation result according to the gesture recognition result; generating an interaction text according to the word segmentation result.

[0132] In the technical solution of this embodiment, in response to receiving a trigger operation on the target interface, image information is obtained; gesture recognition is performed according to the image information; and text information is generated according to the gesture recognition result. By performing image recognition on the target information and displaying the interaction text corresponding to the target information, the convenience and diversity of communication are improved.

[0133] Fourth Embodiment

[0134] Refer to Figure 6 , Figure 6 This is the fourth embodiment of the display method of the present invention. Based on any one of the first to third embodiments, step S22 includes:

[0135] S221, generating or determining an emotional feature symbol according to the emotional information;

[0136] S222, generating or determining text according to the text information;

[0137] S223, determining the interaction text according to the emotional feature symbol and the text.

[0138] Optionally, the emotional feature symbol includes an emoji and / or an interjection, etc.

[0139] Optionally, when the emotional feature symbol is an emoji, an emoji is generated or determined according to the emotional information. Optionally, when the emotional information is anger, the determined emoji is an angry expression; when the emotional information is a pleasant mood, the determined emoji is a laughing expression.

[0140] Optionally, when the emotional feature symbol is an interjection, an interjection is generated or determined according to the emotional information. Optionally, when the emotional information is confusion, the determined interjection is "ma"; when the emotional information is surprise, the determined interjection is "wa".

[0141] Optionally, the interactive text is determined based on the emotional characteristic symbol and text. For example, the text information is generally recognized as "The weather is great today", but if the user's emotional information is sensed, such as the emotional information is a happy mood, the emotional characteristic symbol is generated or determined based on the emotional information, and the emotional characteristic symbol includes a smiley face emoticon and an interjection particle "ah", and the interactive text displayed can be "The weather is great today" with a smiley face emoticon.

[0142] In the technical solution of this embodiment, the emotion characteristic symbol is generated or determined according to the emotion information; the text is generated or determined according to the text information; and the interactive text is determined according to the emotion characteristic symbol and the text. By identifying the emotion information corresponding to the target information and generating the interactive text according to the emotion information and the target information, the accuracy of the interactive text determination is improved, and the text information is combined with the user's emotion, so that the finally formed interactive text is more vivid.

[0143] The present application also provides an intelligent terminal, which includes a memory and a processor. The memory stores a display program, and when the display program is executed by the processor, the steps of the display method in any of the above embodiments are implemented.

[0144] The present application also provides a storage medium, on which a display method program is stored. When the display method program is executed by a processor, the steps of the display method in any of the above embodiments are implemented.

[0145] In the embodiments of the smart terminal and storage medium provided in the present application, all technical features of any of the above-mentioned display method embodiments may be included, and the expanded and explained contents of the specification are basically the same as those of the embodiments of the above-mentioned methods, and will not be repeated here.

[0146] The embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above various possible implementation modes.

[0147] An embodiment of the present application also provides a chip, including a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device equipped with the chip executes the methods in various possible implementation modes as described above.

[0148] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0150] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0151] The units in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0152] In the present application, the same or similar terminology concepts, technical solutions and / or application scenario descriptions are generally described in detail only the first time they appear. When they appear again later, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of the present application, for the same or similar terminology concepts, technical solutions and / or application scenario descriptions that are not described in detail later, reference can be made to the previous related detailed descriptions.

[0153] In the present application, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] The various technical features of the technical solution of the present application can be arbitrarily combined. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0156] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a storage medium, or transmitted from one storage medium to another storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, storage disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state storage disk Solid State Disk (SSD)), etc.

[0157] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A display method, characterized in that: The method comprises the following steps: S10: in response to receiving a trigger operation for a target interface, confirming or acquiring target information; S20: displaying interactive text according to the target information; The step S20 includes: S21: Acquire emotional information and text information corresponding to the target information; S22: Generate interactive text according to the emotion information and text information; The target information includes image information, and the step S21 includes: In response to acquiring environmental information from the image information, the emotion information is determined or acquired.

2. The display method according to claim 1, characterized in that: The target information includes image information, and the step S21 further includes at least one of the following: In response to acquiring the expression information and / or the amplitude of body movement from the image information, determining or acquiring the emotion information; and / or The target information also includes audio information, and step S21 further includes: In response to obtaining at least one of speech speed, voice volume, and preset words from the audio information, the emotion information is determined or obtained.

3. The display method according to claim 1, characterized in that: The target information includes image information, and the step S21 includes: performing gesture recognition according to the image information; The text information is generated according to the gesture recognition result.

4. The display method according to claim 3, characterized in that: The performing gesture recognition according to the image information comprises: Identify the motion trajectory of the key points of the hand in the image information, and obtain the gesture recognition result according to the motion trajectory of the key points of the hand; and / or, The generating the text information according to the gesture recognition result includes: generating a word segmentation result according to the gesture recognition result; and generating text information according to the word segmentation result.

5. The display method according to any one of claims 1 to 4, characterized in that: The step S22 includes: Generate or determine an emotion characteristic symbol according to the emotion information; Generate or determine text based on text information; The interactive text is determined according to the emotional characteristic symbols and words.

6. The display method according to claim 5, characterized in that: The emotion characteristic symbols include emoticons and / or modal particles.

7. The display method according to any one of claims 1 to 4, characterized in that: The step S10 includes at least one of the following: In response to obtaining the preset instruction, displaying a target interface; In response to the current user interface belonging to a preset interface, a target interface is displayed.

8. An intelligent terminal, characterized in that: The intelligent terminal comprises: a memory and a processor, wherein a display program is stored in the memory, and when the display program is executed by the processor, the steps of the display method according to any one of claims 1 to 7 are implemented.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display method according to any one of claims 1 to 7 are implemented.

Images