Interaction method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an interaction method and device, electronic equipment and a computer readable storage medium. The interaction method comprises the following steps: responding to a first operation of a user, entering a first voice mode, and maintaining the first voice mode without manual operation of the user; performing an instant call with the user in the first voice mode; in the first voice mode, receiving visual information and voice information input by the user; and generating reply information according to the visual information and the voice information input by the user.

Description

[0001] The present disclosure relates to the field of computer technology, and in particular to an interaction method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] A voice assistant is an artificial intelligence application or smart device based on voice interaction technology. It can recognize and understand human voice commands, process them with the help of predefined rules or artificial intelligence algorithms, and make corresponding feedback or perform related operations in voice or other ways to provide users with various services and assistance.

[0003] Voice assistants can recognize the user's voice content, understand the intention, and respond in a natural and fluent voice, achieving communication similar to that between people. Summary of the invention

[0004] According to some embodiments of the present disclosure, an interaction method is provided, including: entering a first voice mode in response to a first operation of a user, wherein maintaining the first voice mode does not require manual operation of the user; conducting an instant call with the user in the first voice mode; receiving visual information and voice information input by the user in the first voice mode; and generating reply information based on the visual information and the voice information input by the user.

[0005] According to some other embodiments of the present disclosure, an interactive device is provided, including: a first module, configured to enter a first voice mode in response to a first operation of a user, wherein maintaining the first voice mode does not require manual operation of the user; a second module, configured to conduct an instant call with the user in the first voice mode; a third module, configured to receive visual information and voice information input by the user in the first voice mode; and a fourth module, configured to generate reply information based on the visual information and the voice information input by the user.

[0006] According to some other embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the interaction method according to some embodiments of the present disclosure based on instructions stored in the memory.

[0007] According to some other embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the instructions are executed by a processor, the interaction method according to some embodiments of the present disclosure is implemented.

[0008] According to some other embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the interaction method according to some embodiments of the present disclosure.

[0009] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The following is an explanation of the embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation to the present disclosure. In the accompanying drawings:

[0011] Figure 1 A flowchart showing an interaction method according to some embodiments of the present disclosure;

[0012] Figure 2a-2c A schematic diagram showing entering a first voice mode according to some embodiments of the present disclosure;

[0013] Figure 3a-3d A schematic diagram showing a call interface according to some embodiments of the present disclosure;

[0014] Figure 4a-4c A schematic diagram showing input visual information according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic diagram showing a third operation according to some embodiments of the present disclosure;

[0016] Figure 6 A schematic diagram showing a first voice mode running in the background according to some embodiments of the present disclosure;

[0017] Figure 7 A schematic diagram showing a third voice mode according to some embodiments of the present disclosure;

[0018] Figure 8 A block diagram showing an interaction device according to some embodiments of the present disclosure;

[0019] Fig. 9 A block diagram showing an electronic device according to some embodiments of the present disclosure;

[0020] Fig.10 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0021] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein.

[0023] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0024] The term “including” and its variations used in the present disclosure are intended to be open terms that include at least the following elements / features but do not exclude other elements / features, that is, “including but not limited to.” The term “based on” means “at least partly based on.”

[0025] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0029] It should be understood that the present disclosure does not limit how to obtain the image to be applied / processed. In some embodiments of the present disclosure, it can be obtained from a storage device, such as an internal memory or an external storage device. In other embodiments of the present disclosure, a photographic component can be mobilized to shoot. It should be noted that the acquired image can be a captured image or a frame of an image in a captured video, and is not particularly limited to this.

[0030] In the context of the present disclosure, an image may refer to any of a variety of images, such as a color image, a grayscale image, etc. It should be noted that in the context of the present specification, the type of image is not specifically limited. In addition, the image may be any appropriate image, such as an original image obtained by a camera device, or an image that has been subjected to specific processing, such as preliminary filtering, anti-aliasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that the preprocessing operation may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0031] In the related art, voice assistants only process voice information and have limited processing capabilities for user needs. The interactive method and device, electronic device, and computer-readable storage medium provided by the present disclosure achieve more convenient and flexible interaction.

[0032] Figure 1 A flowchart of an interaction method according to some embodiments of the present disclosure is shown.

[0033] like Figure 1 As shown, the interaction method includes: step S1, entering a first voice mode in response to a first operation of a user, wherein maintaining the first voice mode does not require manual operation of the user; step S2, conducting an instant call with the user in the first voice mode; step S3, receiving visual information and voice information input by the user in the first voice mode; step S4, generating reply information based on the visual information and voice information input by the user.

[0034] For example, in response to the user performing a first operation, the first voice mode is entered. After entering the first voice mode, the voice assistant can automatically maintain the first voice mode without the user manually operating the terminal.

[0035] In the first voice mode, the voice assistant and the user have an instant conversation. For example, the voice assistant can have multiple rounds of voice conversations with the user. The user does not need to continue manual operation. The user only needs to ask a question and the voice assistant can reply by voice. The voice information is received by sensors such as microphones.

[0036] Visual information includes, for example, images, videos, etc. For example, a user can speak a voice command and input visual information by operating on the screen. The voice assistant can respond to the user based on the command spoken by the user and the command on the screen. The response information may include visual information or voice information.

[0037] In the first voice mode, the voice assistant can receive both voice information and visual information, and can respond based on the voice information and visual information. This voice mode realizes multimodal interaction, and users can interact with the system through a combination of voice and image, making the interaction more convenient and flexible.

[0038] The interactive method of this embodiment may be executed on the client side, or may be partially executed on the server side.

[0039] Figure 2a-2c A schematic diagram showing entering a first voice mode according to some embodiments of the present disclosure is shown.

[0040] Combine the following Figure 2a-2c , introduces a method of entering the first voice mode according to some embodiments of the present disclosure.

[0041] like Figure 2a As shown, in the interactive interface, a first control may be displayed. The first control is used for voice input, for example, an identifier for triggering voice input. The first control is, for example, Figure 2a When the user triggers "Press and Talk", the following is displayed: Figure 2b The second area is, for example, Figure 2b The area where "continuous speech" is located. Figure 2b The white circle in the shows where the user touched.

[0042] In response to the user's hand moving from the first area to the second area, the first voice mode is entered. When the user's hand moves to the second area, the following is displayed: Figure 2c The interface shown. You can see that Figure 2c The user's finger has moved to the area where "Continuous Speech" is located.

[0043] When the user's finger moves from the first area ( Figure 2b middle strip area) to the second area ( Figure 2b The first voice mode can be entered after the user's finger slides to the second area and leaves the screen. The first voice mode can also be entered directly after the user's finger slides to the second area.

[0044] After entering the first voice mode, the user can let go and the first voice mode can be automatically maintained.

[0045] In the first voice mode, the call interface is as follows: Figure 3a-3d shown.

[0046] Users can ask further questions about previous conversations. For example, if they have previously asked “What fun places are there in place A”, they can continue to ask “What other fun places are there in place A”.

[0047] like Figure 3a As shown, when the user is speaking, "Listening" can be displayed on the call interface, so that the user can intuitively understand that the first voice mode is running normally and the system is receiving and processing his voice information.

[0048] After detecting that the user's voice input is completed, the user's voice input can be converted into text and displayed on the screen, for example Figure 3b shown.

[0049] While waiting for the system to generate a response, you can Figure 3c As shown, ellipsis etc. are displayed, indicating that the voice assistant is processing the user's question and guiding the user to wait patiently.

[0050] The generated response can be displayed as text on the screen, e.g. Figure 3d In the example, "Then I recommend city B" is displayed. In addition, a reply voice message may be played to the user.

[0051] In the first voice mode, an exit button may be displayed on the call interface, for example Figure 3a-3d Click the "X" on the right side of the screen.

[0052] In the first voice mode, an option to start the shooting function is displayed on the call interface. In response to the user starting the shooting function, a picture shot by the user is determined as visual information.

[0053] The options to start the shooting function are, for example, Figure 3a-3d The camera icon in the . Users can use the camera to take photos and videos by selecting the option to start the shooting function. The voice assistant can use the pictures taken by the user as visual information.

[0054] In other words, the user can enter the shooting function directly from the first voice mode without having to exit the first voice mode and then enter the shooting function, thereby reducing the user's operation steps and improving interaction efficiency.

[0055] In some embodiments, in response to the user starting the shooting function, the interface switches to the shooting interface; in response to the user completing the shooting in the shooting interface, the interface switches back to the call interface and continues the instant call with the user.

[0056] For example, when a user clicks Figure 3a-3dIn the camera pattern, enter Figure 4a The shooting interface shown in the figure. In the shooting interface, the user can click the shooting button to shoot. When the user clicks the shooting button and completes the shooting, it will directly return to the Figure 4b The call interface shown in the figure continues to have an instant call with the user. Figure 4b The call interface shown can display visual information input by the user.

[0057] That is to say, after the shooting is completed, the interface automatically switches back to the call interface and continues to talk with the user in the first voice mode. The user does not need to manually exit the shooting function and then re-enter the first voice mode, thereby reducing the user's operation steps and improving the interaction efficiency.

[0058] During the user's shooting process, the voice assistant can also maintain an instant conversation with the user.

[0059] For example, in the display Figure 4a While the shooting interface is displayed, the voice assistant still receives the user's voice input or outputs a voice reply. In other words, the call with the user is not interrupted by the user's shooting action. The user can freely switch to the camera application to take photos or videos without hanging up the call, reducing the inconvenience caused by frequent switching operations. In addition, the call and shooting are carried out at the same time, which improves the efficiency of multitasking.

[0060] After receiving the visual information, the voice assistant can wait for the user to input voice information related to the visual information, or actively guide the user to input voice. The voice assistant can first call the visual recognition model to determine the type and content of the visual information. Then, the voice assistant can generate guidance information based on the type and content of the visual information. For example Figure 4c As shown in the figure, the user inputs a picture of a book, and the text in the picture is different from the text used in the current conversation. The voice assistant outputs guidance information to the user in voice or text form: "Are you reading this book? This is a fragment of a novel. Do you need me to translate it for you?"

[0061] Users can also select visual information from a resource library (such as a gallery) and send it to the voice assistant. The following is an introduction to how to select visual information from a resource library.

[0062] In some embodiments, in the first voice mode, an option to launch a resource library is displayed on the call interface; in response to the user launching the resource library, a visual resource selected by the user in the resource library is determined as visual information.

[0063] For example, a gallery logo is displayed on the call interface. The user can enter the gallery by selecting the gallery logo and browse the images and videos in the gallery. The voice assistant can use the images and videos selected by the user in the gallery as visual information.

[0064] Users can enter the resource library directly from the first voice mode without having to exit the first voice mode and then enter the resource library, thereby reducing the user's operation steps and improving interaction efficiency.

[0065] In some embodiments, in response to the user launching the resource library, the display interface of the resource library is switched to; in response to the user completing the selection of visual resources on the display interface of the resource library, the call interface is switched back to, and the instant call with the user is continued.

[0066] That is to say, after the user completes the selection in the resource library, it automatically switches back to the call interface and continues to have an instant call with the user in the first voice mode. The user does not need to manually exit the resource library and re-enter the first voice mode, thereby reducing the user's operation steps and improving interaction efficiency.

[0067] While the user is operating in the resource library, the voice assistant can also maintain an instant conversation with the user.

[0068] That is to say, the call with the user is not interrupted by entering the resource library. The user can switch to the resource library freely without hanging up the call, reducing the inconvenience caused by frequent switching operations. In addition, the call and shooting are carried out at the same time, which improves the efficiency of multitasking.

[0069] In the call interface, you can display both the option to start the resource library and the option to start the shooting function, or you can display only one of them. In the shooting interface, you can also display the option to start the resource library, for example Figure 4a The library icon on the lower left is shown. The option to start the shooting function can also be displayed in the browsing interface of the resource library. In other words, the camera application and the resource library can be switched back and forth.

[0070] In some embodiments, based on the voice information, the user's processing instructions for the visual information are determined; based on the processing instructions, the visual information is processed to obtain reply information.

[0071] For example, if the visual information input by the user is a picture, and the voice information input by the user is a processing instruction for the picture, the picture is processed according to the processing instruction. The processing instruction for the picture can be an editing operation, such as cropping, modifying brightness, contrast, saturation, rotation, flipping, etc. The processing instruction for the picture can also be adding props to the picture, such as text, stickers, etc.

[0072] The processing instructions for the visual information may also be to analyze the content in the visual information, summarize the text in the visual information, translate the text in the visual information, generate new visual information based on the input visual information, etc.

[0073] In some embodiments, the reply message is generated based on the voice information, visual information, and the context of the user conversation.

[0074] For example, if a user wants to get professional advice on clothing matching, he or she may issue a voice command: "Please give me some suggestions on clothing matching suitable for summer." The voice assistant responds: "It is recommended that you choose a short-sleeved or three-quarter-sleeved shirt or sweater as an inner layer under the coat, and pair it with high-waisted straight-leg pants or pencil skirts."

[0075] The user continued, "Any other recommendations?" The voice assistant determined that the user might not like the previous recommendation and guided the user to enter more information: "In order to give better suggestions, please tell me what style you usually like?" Then the user uploaded a photo of what he usually wears.

[0076] Based on the previous voice command, the smart assistant determines that the user still needs summer outfit suggestions, analyzes the picture, and generates a response: "From your photos, it seems that you like casual and comfortable styles. For summer, you can try to choose a light and breathable dress."

[0077] In some embodiments, in the first voice mode, a second control is displayed on the call interface; when a reply message is played, in response to a user triggering operation on the second control, the current playback of the reply message is interrupted, and the user's voice input is continued to be detected.

[0078] For example Figure 3d As shown, a "click to interrupt" button can be displayed on the call interface.

[0079] For example, the voice assistant is explaining a complex problem in detail, but the user has found the information he needs and wants to ask a new question immediately. The user clicks the "Click to Interrupt" button on the screen, the voice assistant stops the sound, and can inform the user through visual feedback (such as the button state changes) that the interruption action has been completed.

[0080] In some embodiments, in the first voice mode, a text input area is displayed; and text information input by a user through the text input area is received.

[0081] For example, users can not only input visual information through the screen, but also input text information through the screen. Users can paste the copied text into the input box and input it to the voice assistant. By displaying the text input area in the first voice mode, users can freely choose which information to input through text and which information to input through voice, providing a more flexible input method.

[0082] In some embodiments, call information can also be displayed on the call interface, wherein the call information includes voice information input by the user, text information corresponding to the voice information input by the user, reply information, and at least one item of visual information input by the user; in response to the user's third operation on the call information, the call information is processed.

[0083] The voice information input by the user can be directly converted into text and displayed on the call interface. It can also display a mark for each voice message input by the user. When the user triggers the mark, the corresponding voice message is converted into text and displayed.

[0084] like Figure 5 As shown, the user can select a call message by long pressing and operate the call message. If the user selects a text message, the selected text range can also be adjusted.

[0085] In some embodiments, in response to the user's third operation on the call information, at least one of copying, deleting, saving, voice reading, translating, sharing, and image processing is performed on the call information.

[0086] For example Figure 5 As shown, in response to the user selecting a call message, options such as copy, read aloud, and translate are displayed, and the user can perform corresponding operations by triggering these options.

[0087] In some embodiments, in response to a fifth operation of the user, the first voice mode is switched to run in the background.

[0088] For example Figure 6 As shown in the figure, if the user locks the screen, the prompt information of the first voice mode is displayed in the locked screen state. In the locked screen state, if the user issues a voice command, the voice assistant can respond quickly. If the user switches to other applications, the voice assistant can also be switched to run in the background while using other applications. The user can continue to interact with the voice assistant without interrupting the conversation due to switching applications.

[0089] In some embodiments, the interaction method further includes: in response to not receiving the user's voice input within the first specified time period, exiting the first voice mode.

[0090] For example, by analyzing the sound intensity, frequency distribution and other features in the audio signal, the system can automatically identify the sound and silence sections. When a period of continuous silence is detected, the system assumes that the user has ended the conversation and exits the first voice mode.

[0091] According to some embodiments of the present disclosure, the voice assistant also provides a second voice mode, which is introduced below.

[0092] In response to the user's second operation, entering the second voice mode, wherein maintaining the second voice mode requires the user to continuously trigger the first control in the first area; receiving voice information input by the user in the second voice mode; and exiting the second voice mode in response to the user ending the triggering of the first control.

[0093] For example Figure 2b As shown, the user presses the first control, and the device starts recording the user's voice. As long as the user keeps the button pressed, the recording will continue. When the user finishes speaking, releases the button, the device stops recording and converts the recorded voice clip into a voice message and sends it to the voice assistant.

[0094] exist Figure 2b In the interface, after the user triggers the first control, if the user keeps triggering the first control in the first area, the second voice mode is maintained, and if the user slides from the first area to the second area, the first voice mode is entered. In other words, the user can switch from the second voice mode to the first voice mode conveniently and quickly.

[0095] According to some embodiments of the present disclosure, the voice assistant also provides a third voice mode, which is introduced below.

[0096] In the first voice mode, in response to a fourth operation of the user, switching to the third voice mode includes: in the first voice mode, in response to detecting a sliding operation of the user moving from the third area to the fourth area, switching to the third voice mode.

[0097] For example, in the first voice mode, the user slides his finger upward from the bottom of the screen to enter the third voice mode.

[0098] Among them, maintaining the third voice mode does not require manual operation of the user; in the third voice mode, a virtual image is displayed and an instant conversation is made with the user; in response to not receiving the user's voice input within a second specified time period, the third voice mode is exited, wherein the second specified time period is greater than the first specified time period.

[0099] For example, in the third voice mode, the user does not need to continue hand operation to maintain the mode. In the third voice mode, the virtual image of the voice assistant can also be displayed. When the voice assistant speaks, the lip movements, facial expressions and body language of the virtual image can be synchronized with the voice, providing a more natural, smooth and immersive conversation experience.

[0100] Furthermore, in the third voice mode, if the user does not speak for a long time, the third voice mode can be exited. Compared with the third voice mode, the user silence time required to exit the first voice mode is shorter. Entering the first voice mode requires fewer operations than entering the third voice mode, and the user can enter faster, that is, the cost of entering and exiting the first voice mode is lower.

[0101] The first voice mode can be used for quick and efficient conversations and timely exit. The third voice mode can be used for long conversations, giving users more time to think.

[0102] Figure 8 A block diagram of an interaction device according to some embodiments of the present disclosure is shown.

[0103] like Figure 8 As shown, the interactive device 8 includes: a first module 81 is configured to enter a first voice mode in response to a first operation of the user, wherein the first voice mode is maintained without manual operation of the user; a second module 82 is configured to have an instant conversation with the user in the first voice mode; a third module 83 is configured to receive visual information and voice information input by the user in the first voice mode; and a fourth module 84 is configured to generate reply information based on the visual information and voice information input by the user.

[0104] The first module 81 of the interactive device 8 can be used to perform Figure 1 The second module 82 can be used to perform Figure 1 The third module 83 can be used to perform Figure 1 Step S3. The fourth module 84 can be used to perform Figure 1 Step S4.

[0105] In some embodiments, the interaction device also includes: a fifth module, configured to enter a second voice mode in response to a second operation of the user, wherein maintaining the second voice mode requires the user to continuously trigger the first control in the first area; receiving voice information input by the user in the second voice mode; and exiting the second voice mode in response to the user ending the triggering of the first control.

[0106] In some embodiments, the interactive device also includes: a sixth module, which is configured to display a second control on the call interface in the first voice mode; when playing a reply message, in response to the user's triggering operation on the second control, interrupt the current playback of the reply message and continue to detect the user's voice input.

[0107] In some embodiments, the interactive device also includes: a seventh module, configured to display call information, wherein the call information includes voice information input by the user, text information corresponding to the voice information input by the user, reply information and at least one item of visual information input by the user; in response to the user's third operation on the call information, the call information is processed.

[0108] In some embodiments, the interaction device further includes: an eighth module configured to exit the first voice mode in response to not receiving a user's voice input within a first specified time period.

[0109] In some embodiments, the interactive device also includes: a ninth module, configured to, in the first voice mode, switch to a third voice mode in response to a fourth operation of the user, wherein maintaining the third voice mode does not require manual operation of the user; in the third voice mode, display a virtual image and conduct an instant conversation with the user; in response to not receiving the user's voice input within a second specified time period, exit the third voice mode, wherein the second specified time period is greater than the first specified time period.

[0110] In some embodiments, the interactive device also includes: a tenth module, configured to display call information, wherein the call information includes at least one item of voice information input by the user, text information corresponding to the voice information input by the user, reply information and visual information input by the user; in response to the user's third operation on the call information, the call information is processed.

[0111] In some embodiments, the interaction device further includes: an eleventh module configured to switch to running the first voice mode in the background in response to a fifth operation of the user.

[0112] Fig. 9 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0113] The memory 91 is used to store one or more computer-readable instructions. The memory 91 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. The memory 91 may store, for example, an operating system, an application, a boot loader (BootLoader), a database, and other programs, and may also store various applications and various data.

[0114] The processor 92 is used to run computer-readable instructions to implement the song screening method of any of the above embodiments or the method of any of the above embodiments. The specific implementation of each step of the method can be referred to the above embodiments, and the repeated parts are not repeated here.

[0115] The processor 92 may be configured to execute Figure 1 The processor 92 may be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc.

[0116] The processor 92 and the memory 91 may communicate with each other directly or indirectly. For example, the processor 92 and the memory 91 may communicate with each other via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 92 and the memory 91 may also communicate with each other via a system bus, which is not limited in the present disclosure.

[0117] It should be noted that Fig. 9 The components of the electronic device 9 shown are only exemplary and non-restrictive. The electronic device 9 may also have other components according to actual application requirements. The processor 92 may control other components in the electronic device 9 to perform desired functions.

[0118] The electronic device 9 may be implemented by software, firmware and / or hardware, and may be integrated into a device installed with relevant application programs.

[0119] Fig.10 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0120] Fig.10 The electronic device 10 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0121] Electronic devices include, but are not limited to, mobile terminals such as smart phones, laptops, personal digital assistants (PDA), tablet personal computers (Tablet PC), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital televisions, desktop computers, etc.

[0122] like Fig.10As shown, a central processing unit (CPU) 101 performs various processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage section 108 to a random access memory (RAM) 103. In the RAM 103, data required when the CPU 101 performs various processes, etc., is stored as needed. The central processing unit is merely exemplary, and it may also be other types of processors, such as the various processors described above. The ROM 102, the RAM 103, and the storage section 108 may be various forms of computer-readable storage media. It should be noted that although Fig.10 ROM 102, RAM 103 and storage section 108 are shown separately in FIG. 1 , but one or more of them may be combined or located in the same or different memory or storage modules.

[0123] The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. To the bus 104, an input / output interface 105 is also connected.

[0124] The following components are connected to the input / output interface 105: an input section 106, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output section 107, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage section 108, including a hard disk, a magnetic tape, etc.; and a communication section 109, including a network interface card such as a LAN card, a modem, etc. The communication section 109 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Fig.10 Parts of the electronic device 10 are shown to communicate via the bus 104, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless networks and wired networks.

[0125] A drive 1010 is also connected to the input / output interface 105 as needed. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 108 as needed.

[0126] When the above-described series of processing is realized by software, a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1011 .

[0127] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which, when the computer program product is run on a computer, enables the computer to implement the method described in any of the aforementioned embodiments. The computer program product includes a computer instruction carried on a computer-readable medium, containing a program code for executing the method shown in the flowchart. In such an embodiment, the computer instruction can be downloaded and installed from the network through the communication part 109, or installed from the storage part 108, or installed from the ROM 102. When the computer program is executed by the CPU 101, the method of the embodiment of the present disclosure is executed.

[0128] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0129] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0130] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device. Computer instructions are stored on a computer-readable storage medium, and when the instructions are executed by a processor, the method described in any of the foregoing embodiments is implemented.

[0131] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0133] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method described in any of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0134] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0137] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An interactive method, comprising: In response to a first operation of a user, entering a first voice mode, wherein maintaining the first voice mode does not require manual operation of the user; In the first voice mode, having an instant conversation with the user; In the first voice mode, receiving visual information and voice information input by the user; Reply information is generated according to the visual information and the voice information input by the user.

2. The interactive method according to claim 1, wherein: The step of receiving the visual information and voice information input by the user in the first voice mode includes: In the first voice mode, an option to start the shooting function is displayed on the call interface; In response to the user starting the shooting function, a picture shot by the user is determined as the visual information.

3. The interactive method according to claim 2, wherein: In response to the user starting the shooting function, determining the picture shot by the user as the visual information includes: In response to the user starting the shooting function, switching to a shooting interface; In response to the user completing the shooting in the shooting interface, switching back to the call interface and continuing the instant call with the user.

4. The interactive method according to claim 2, wherein: The step of receiving the visual information and voice information input by the user in the first voice mode includes: During the shooting process of the user, a real-time conversation is maintained with the user.

5. The interactive method according to claim 1, wherein: The step of receiving the visual information and voice information input by the user in the first voice mode includes: In the first voice mode, an option to start the resource library is displayed on the call interface; In response to the user launching the resource library, a visual resource selected by the user in the resource library is determined as the visual information.

6. The interactive method according to claim 5, wherein: In response to the user starting the resource library, determining the visual resource selected by the user in the resource library as the visual information includes: In response to the user starting the resource library, switching to a display interface of the resource library; In response to the user completing the selection of the visual resource on the display interface of the resource library, switching back to the call interface and continuing the instant call with the user.

7. The interactive method according to claim 5, wherein: The step of receiving the visual information and voice information input by the user in the first voice mode includes: During the operation of the user in the resource library, an instant conversation is maintained with the user.

8. The interactive method according to claim 1, wherein: The step of generating reply information according to the visual information and the voice information input by the user comprises: Determining, according to the voice information, a processing instruction of the user for the visual information; The visual information is processed according to the processing instruction to obtain the reply information.

9. The interactive method according to claim 1, wherein: The step of generating reply information according to the visual information and the voice information input by the user comprises: The reply information is generated according to the voice information, the visual information and the context of the user conversation.

10. The interactive method according to claim 1, wherein: The step of entering a first voice mode in response to a first operation by the user includes: Display a first control, wherein the first control is used for voice input; In response to the user's hand moving from the first area to the second area, the first voice mode is entered.

11. The interactive method according to claim 10, further comprising: In response to a second operation of the user, entering a second voice mode, wherein maintaining the second voice mode requires the user to continuously trigger the first control in the first area; receiving voice information input by the user in the second voice mode; In response to the user ending the triggering of the first control, the second voice mode is exited.

12. The interactive method according to claim 1, further comprising: In the first voice mode, displaying a second control on the call interface; In the case of playing the reply information, in response to the user's triggering operation on the second control, the current playing of the reply information is interrupted, and the detection of the user's voice input continues.

13. The interactive method according to claim 1, wherein: The step of receiving the visual information and voice information input by the user in the first voice mode includes: In the first voice mode, displaying a text input area; Receive text information input by the user through the text input area.

14. The interactive method according to claim 1, further comprising: Displaying call information, wherein the call information includes at least one of the voice information input by the user, text information corresponding to the voice information input by the user, the reply information, and the visual information input by the user; In response to a third operation performed by the user on the call information, the call information is processed.

15. The interactive method according to claim 14, wherein: The processing of the call information in response to the user's third operation on the call information includes: In response to the third operation of the user on the call information, at least one of copying, deleting, saving, voice reading, translating, sharing, and image processing is performed on the call information.

16. The interactive method according to claim 1, further comprising: In response to not receiving the user's voice input within a first specified time period, exiting the first voice mode.

17. The interactive method according to claim 1, further comprising: In the first voice mode, in response to a fourth operation of the user, switching to a third voice mode, wherein maintaining the third voice mode does not require manual operation of the user; In the third voice mode, a virtual image is displayed and an instant conversation is conducted with the user; In response to not receiving the user's voice input within a second specified time period, exiting the third voice mode, wherein the second specified time period is greater than the first specified time period.

18. The interactive method according to claim 16, wherein: The method of switching to a third voice mode in response to a fourth operation of the user in the first voice mode includes: In the first voice mode, in response to detecting a sliding operation of the user moving from the third area to the fourth area, switching to the third voice mode.

19. The interactive method according to claim 1, further comprising: In response to a fifth operation of the user, switching to running the first voice mode in the background.

20. An interactive device, comprising: A first module is configured to enter a first voice mode in response to a first operation of a user, wherein the first voice mode is maintained without manual operation of the user; A second module is configured to conduct an instant conversation with the user in the first voice mode; A third module is configured to receive visual information and voice information input by the user in the first voice mode; The fourth module is configured to generate reply information according to the visual information and the voice information input by the user.

21. An electronic device comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the interaction method according to any one of claims 1 to 19 based on instructions stored in the memory.

22. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the interaction method according to any one of claims 1 to 19.

23. A computer program product, comprising computer program instructions, which, when executed by a processor, implement the interaction method according to any one of claims 1 to 19.

Images