A session processing method and device, electronic equipment and readable storage medium

Abstract

The application provides a conversation processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: presenting a conversation editing area; in response to an input operation in the conversation editing area, obtaining conversation content formed by the input operation; and in response to a sending operation on the conversation content, presenting a special effect generated based on a virtual image of a target object in the conversation area, wherein the special effect is used to represent the conversation content. Through the application, the conversation content or message can be effectively and fully displayed.

Description

Technical Field

[0001] This application relates to Internet technology, and more particularly to a session processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of internet technology, instant messaging applications are becoming increasingly widespread. Instant messaging applications provide users with internet-based instant messaging services, allowing two or more people to instantly exchange text, voice, and video messages over the network. As instant messaging applications have developed, they have permeated people's lives, and more and more people are using them for communication.

[0003] When engaging in conversations through instant messaging applications, it's difficult to send appropriate video messages in unfavorable environments, such as when commuting, in dim lighting, or in noisy public spaces. In such situations, users are often forced to use text or voice messages, which struggle to convey the true context of the conversation, leading to low conversation efficiency. Summary of the Invention

[0004] This application provides a session processing method, apparatus, electronic device, and computer-readable storage medium, which can fully and effectively display session scenarios and improve session efficiency.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a session processing method, including:

[0007] Present the conversation editing area;

[0008] In response to an input operation based on the session editing area, the session content formed by the input operation is obtained;

[0009] In response to a sending operation for the session content, special effects generated based on the virtual avatar of the target object are presented in the session area, wherein the special effects are used to represent the session content.

[0010] In the above technical solution, before presenting the special effects generated based on the virtual image of the target object in the session area, the method further includes:

[0011] Based on the virtual image of the target object and the conversation content, special effects data for representing the conversation content are determined;

[0012] Based on the conversation content, determine the voice data that matches the voice of the target object;

[0013] The special effects data and the voice data are synthesized to obtain special effects generated based on the virtual image of the target object.

[0014] In the above technical solution, before determining the special effects data used to characterize the session content, the method further includes:

[0015] Obtain the real image of the target object;

[0016] The virtual image of the target object is obtained by calling the virtual image generation model based on the real image.

[0017] In the above technical solution, the step of calling the virtual image generation model based on the real image to obtain the virtual image of the target object includes:

[0018] The following processing is performed using the virtual avatar generation model:

[0019] Principal component analysis is performed on the real image to obtain the geometric information distribution and texture information distribution corresponding to the target object;

[0020] Based on the geometric information distribution and the texture information distribution, deformation processing is performed to obtain the virtual image of the target object.

[0021] In the above technical solution, determining the voice data that matches the target object's voice based on the conversation content includes:

[0022] Text analysis processing is performed on the text information corresponding to the conversation content to obtain the contextual features of the conversation content;

[0023] Based on the target object's voice, the contextual features of the conversation content are used for speech parameter prediction processing to obtain multiple speech parameters corresponding to the conversation content and conforming to the target object's voice;

[0024] The multiple speech parameters are synthesized to obtain speech data that matches the voice of the target object.

[0025] In the above technical solution, when the session content is a text message, determining the special effects data to characterize the session content based on the virtual image of the target object and the session content includes:

[0026] The text message is subjected to text feature extraction processing to obtain the text features of the text message;

[0027] Based on the text features, multiple candidate effect data contained in the database are matched, and the matched candidate effect data is used as effect data to characterize the conversation content.

[0028] The text features include at least one of the following: text, symbols, and emoticons.

[0029] In the above technical solution, when the conversation content is a voice message, determining the special effects data to characterize the conversation content based on the virtual image of the target object and the conversation content includes:

[0030] The voice message is converted to a text message to obtain the corresponding voice message.

[0031] The text message is subjected to text feature extraction processing to obtain the text features of the text message;

[0032] Based on the text features, multiple candidate effect data contained in the database are matched, and the matched candidate effect data is used as effect data to characterize the session content.

[0033] This application provides a session processing apparatus, including:

[0034] The first display module is used to present the session editing area;

[0035] The acquisition module is used to acquire the session content formed by the input operation in response to the input operation based on the session editing area;

[0036] The second display module is used to respond to the sending operation of the session content by presenting special effects generated based on the virtual image of the target object in the session area, wherein the special effects are used to represent the session content.

[0037] This application provides an electronic device for session processing, the electronic device comprising:

[0038] Memory, used to store executable instructions;

[0039] The processor, when executing executable instructions stored in the memory, implements the session processing method provided in the embodiments of this application.

[0040] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the session processing method provided in this application.

[0041] The embodiments of this application have the following beneficial effects:

[0042] By presenting special effects generated from virtual avatars of target objects in the conversation area to represent the conversation content, the conversation content can be displayed in a more efficient and effective manner. Compared with conversation solutions that directly display text or voice messages in the conversation area, this improves the interaction efficiency and conversation effect, thereby saving related communication and computing resources. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the architecture of the session system provided in the embodiments of this application;

[0044] Figure 2 This is a schematic diagram of the structure of an electronic device for session processing provided in an embodiment of this application;

[0045] Figure 3A This is a flowchart illustrating the session processing method provided in an embodiment of this application;

[0046] Figure 3B This is a schematic diagram of the session editing area provided in an embodiment of this application;

[0047] Figures 3C-3D This is a schematic diagram of the sending animation provided in an embodiment of this application;

[0048] Figure 4A This is a schematic diagram illustrating the generation of a virtual image provided in an embodiment of this application;

[0049] Figure 4B This is a schematic diagram of sound acquisition provided in an embodiment of this application;

[0050] Figure 5-6 This is a schematic diagram of the text publication provided in the embodiments of this application;

[0051] Figure 7 This is a schematic diagram of voice publishing provided in an embodiment of this application;

[0052] Figures 8A-8D This is a schematic diagram of a mobile terminal receiving surprise emojis provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of a personal computer (PC) receiving a surprise emoticon, as provided in an embodiment of this application.

[0054] Figure 10 This is a schematic diagram of a mobile terminal receiving video messages according to an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of a personal computer (PC) receiving video signals according to an embodiment of this application;

[0056] Figures 12A-12B This is a flowchart illustrating the session processing method provided in an embodiment of this application;

[0057] Figure 13 This is a schematic diagram illustrating the collection of user image features and voice features provided in an embodiment of this application;

[0058] Figure 14 This is a schematic flowchart of the speech synthesis method provided in the embodiments of this application;

[0059] Figure 15 This is a schematic diagram of the speech-to-text conversion process provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0063] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0064] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0065] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0066] 2) Emoticons: Emoticons are a popular culture that emerged after the rise of social applications. They are used to express specific emotions, such as the emotions displayed on a user's face or in their posture. In practical applications, emoticons can be divided into symbol emoticons, static image emoticons, dynamic image emoticons, video emoticons, etc. For example, emoticons can use faces that express various user emotions as material, or use currently popular celebrities, anime, movie screenshots, etc. as material, and then add a series of matching text.

[0067] 3) Surprise emoji: Also known as emoji effects, it can be obtained by synthesizing the user's voice through the text message sent by the user, and combining the text content with the user's image data in advance to synthesize an emoji image, or by taking the user's voice message as the voice, and combining the corresponding text content with the user's image data in advance to synthesize an emoji image.

[0068] 4) Video message: Also known as video effects, it can be obtained by synthesizing audio from text messages sent by users, and combining the text content with pre-recorded user image data to create a synthesized video frame, or by using voice messages sent by users as audio, and combining the corresponding text content with pre-recorded user image data to create a synthesized video frame.

[0069] 5) Virtual Avatar: The image of various people and objects that can interact in a virtual scene, such as virtual characters, virtual animals, and anime characters. This virtual avatar can be a virtual image representing the user within the virtual scene.

[0070] This application provides a session processing method, apparatus, electronic device, and computer-readable storage medium, which can fully and effectively display session scenarios and improve session efficiency.

[0071] The session processing method provided in this application embodiment can be implemented by the terminal alone; or it can be implemented collaboratively by the terminal and the server. For example, the terminal can undertake the session processing method described below alone, or the terminal of the session sender can send the input session content to the server. The server can obtain special effects generated based on the virtual image of the target object (e.g., real user, virtual object) to represent the session content according to the received session content, and send the special effects to the terminal of the session receiver to display the special effects in the session area of ​​the session receiver, so as to conveniently transmit information, make the session more vivid and interesting, and improve the stickiness of the session.

[0072] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of user terminals, such as laptops, tablets, desktop computers, set-top boxes, smart TVs, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, and in-vehicle devices). Exemplary applications of the electronic devices when implemented as terminals will be described below.

[0073] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the session system 100 provided in the embodiments of this application. The terminal (terminal 200-1 and terminal 200-2 are shown as examples) connects to the server 100 through the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0074] In some embodiments, taking the electronic device as a terminal as an example, the session processing method provided in this application embodiment can be implemented by the terminal. For example, terminal 200-1 presents a session editing area, and in response to an input operation based on the session editing area, obtains the session content formed by the input. In response to a sending operation for the session content, it presents special effects generated based on the virtual image of the target object in the session area to represent the session content, and sends the special effects to terminal 200-2 so that the special effects can be displayed in the session area of ​​terminal 200-2, so as to conveniently transmit information, make the session more vivid and interesting, and improve the stickiness of the session.

[0075] In some embodiments, the session processing method provided in this application can also be implemented collaboratively by a server and a terminal. For example, terminal 200-1 presents a session editing area, responds to input operations based on the session editing area, obtains the session content formed by the input, and sends the session content to server 100. Server 100, based on the received session content, obtains special effects generated based on the virtual image of the target object to represent the session content, and sends the special effects to terminals 200-1 and 200-2 to display the special effects in the session areas of terminals 200-1 and 200-2, so as to conveniently transmit information, make the session more vivid and interesting, and improve the stickiness of the session.

[0076] In some embodiments, the terminal or server can implement the session processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an instant messaging application; it can also be a small program, that is, a program that only needs to be downloaded to the browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0077] In some embodiments, server 100 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be live streaming processing services that can be invoked by terminals.

[0078] In some embodiments, multiple servers can form a blockchain, and server 100 is a node on the blockchain. Information connections can exist between each node in the blockchain, and information can be transmitted between nodes through these connections. Data related to the session processing method provided in this application embodiment (e.g., session processing logic and effects) can be stored on the blockchain.

[0079] The structure of the electronic device for session processing provided in the embodiments of this application is described below. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for session processing provided in an embodiment of this application. The explanation will take an example where the electronic device 500 is a terminal. Figure 2 The illustrated electronic device 500 for session processing includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0080] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0082] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0083] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0084] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0085] In some embodiments, the session processing apparatus provided in this application can be implemented in software. The session processing apparatus provided in this application can be provided in various software embodiments, including various forms such as applications, software, software modules, scripts or code.

[0086] Figure 2 A session processing device 555 stored in memory 550 is shown. It may be software in the form of programs and plug-ins, and includes a series of modules, including a first display module 5551, an acquisition module 5552, a second display module 5553, and a processing module 5554. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0087] As mentioned above, the session processing method provided in this application embodiment can be implemented by various types of electronic devices. See also Figure 3A , Figure 3A This is a flowchart illustrating the session processing method provided in the embodiments of this application, combined with... Figure 3A The steps shown are explained.

[0088] In step 101, the session editing area is presented.

[0089] like Figure 3B As shown, during conversational interaction, a conversation editing area 301 is presented in the human-computer interaction interface. The conversation editing area 301 is used to input conversation content. The conversation editing area 301 includes a text input box, a virtual keyboard, etc., for displaying the input text.

[0090] In step 102, in response to an input operation based on the session editing area, the session content formed by the input operation is obtained.

[0091] like Figure 3B As shown, when the conversation content is a text message, the user can enter text, characters, emoticons, etc., in the text input box. After inputting, the entered text message can be obtained. When the conversation content is a voice message, the user can respond to the trigger operation of the recording entry in the conversation editing area to obtain the voice message formed by the trigger operation. In this embodiment, the trigger operation is not limited, such as a click or long press operation.

[0092] In step 103, in response to the sending operation for the session content, special effects generated based on the virtual image of the target object are presented in the session area, wherein the special effects are used to represent the session content.

[0093] like Figure 4A As shown, in response to a sending operation for the session content, a sending effect is executed. When the sending is successful, an effect generated based on the virtual image of the target object (surprise emoticon 502 and video message 504 shown in the figure) is displayed in the session area used to display the session message record. When the effect fails to send, a prompt message indicating that the effect failed to send is displayed on the session interface, such as an exclamation mark icon indicating that the effect failed to send.

[0094] In some embodiments, presenting special effects generated based on the virtual avatar of the target object in the conversation area includes: playing facial expression effects generated based on the virtual avatar of the target object, wherein the facial expression effects include at least one facial expression image representing the conversation content based on the virtual avatar, and speech representing the conversation content based on the voice of the target object.

[0095] For example, special effects include facial expression effects, which consist of multiple static facial expression images representing the conversation content based on a virtual avatar. These images are played sequentially to create dynamic facial expression effects. It should be noted that facial expression effects can include not only multiple facial expression images but also speech representing at least a portion of the text of the conversation content based on the target object's voice. That is, the speech must match the target object's voice and be able to represent at least a portion of the text of the conversation content; for example, the speech must be able to capture keywords from the conversation content.

[0096] In some embodiments, emoticons are used to represent conversation content from at least one of the following dimensions: keywords of the conversation content, emotional information carried by the conversation content, and the topic to which the conversation content belongs.

[0097] For example, emoticons can represent the content of a conversation through keywords (e.g., images containing keywords). For instance, if the conversation contains the keyword "bye-bye," the emoticon can represent the content through the text "bye-bye" or the "bye-bye" gesture. Emoticons can also represent the content through emotional information carried by the conversation (e.g., happiness or sadness, represented by facial expressions or body movements). For instance, if the conversation carries the emotion of happiness, the emoticon can represent that emotion through the facial expressions or body movements of the virtual avatar. Emoticons can also represent the content through the theme of the conversation. For instance, if the theme of the conversation is tourism, the emoticon can represent the tourism theme by placing the virtual avatar in a background corresponding to the theme (e.g., a tourist attraction). Alternatively, the emoticon can represent the tourism theme by matching the virtual avatar's expressions and body movements (e.g., viewing a tourist attraction) with the theme.

[0098] In some embodiments, the emoticon effect is set to a playback mode, which includes: automatically playing any emoticon image when the emoticon effect is received, and switching to play multiple emoticon images when the emoticon effect is triggered to play, while simultaneously playing audio.

[0099] like Figure 8A As shown, when the recipient of the conversation message receives the emoticon effect, any one of the emoticon images in the emoticon effect 801 can be displayed in the conversation area of ​​the recipient. If the play button 802 is triggered, the emoticon effect will be played, and multiple emoticon images in the emoticon effect will be switched and played simultaneously with the audio.

[0100] For example, when the recipient of a conversation message receives an emoji effect, they can directly switch between multiple emoji images in the emoji effect in their conversation area, and simultaneously play audio.

[0101] In some embodiments, presenting special effects generated based on the virtual avatar of the target object in the session area includes: playing video special effects generated based on the virtual avatar of the target object, wherein the video special effects include multiple video frames representing session content based on the virtual avatar, and multiple audio frames representing the session content based on the voice of the target object.

[0102] For example, special effects include video effects, which consist of multiple video frames representing the conversation content based on a virtual avatar. These multiple video frames are played sequentially to form dynamic video effects. It should be noted that video effects can include not only multiple video frames, but also multiple audio frames representing at least a portion of the text of the conversation content based on the target object's voice. That is, the multiple audio frames must match the target object's voice and be able to represent at least a portion of the text of the conversation content; for example, the multiple audio frames must ensure the presence of keywords in the conversation content.

[0103] It should be noted that video effects carry more information than facial expression effects. Therefore, the animation effects of video effects are better than those of facial expression effects. However, video effects also take up more storage space than facial expression effects.

[0104] In some embodiments, the video effect is set to a playback mode, which includes: automatically playing any one video frame when the video effect is received, playing multiple video frames when the video effect is triggered to play, and synchronously playing multiple audio frames.

[0105] like Figure 10 As shown, when the session message receiver receives the video effect, it can display any one of the video images in the video effect 1001 in the session area of ​​the session message receiver. If the play button 1002 is triggered, the video effect will be played, switching between multiple video frames in the video effect and simultaneously switching between multiple audio frames.

[0106] For example, after the session message recipient receives a video effect, it can directly switch between playing multiple video frames in the video effect in the session area of ​​the session message recipient, and simultaneously switch between playing multiple audio frames.

[0107] In some embodiments, in response to a sending operation for the session content, before presenting special effects generated based on the virtual avatar of the target object in the session area, the session content is presented in the session editing area, and a preview screen representing the special effects of the session content is played; a sending entry for the session content is presented; in response to a triggering operation for the preview screen, and after the preview screen finishes playing, special effects generated based on the virtual avatar of the target object are presented in the session area. The session content can be a text message or a voice message.

[0108] The preview screen includes at least some of the facial expression images in the facial expression effects (and can also play at least some of the voice simultaneously) or at least some of the video frames in the video effects (and can also play at least some of the audio frames simultaneously). For example... Figure 3C As shown, the conversation content is presented in the conversation editing area, and a preview screen 302 representing the conversation content is played. In response to a trigger operation on the preview screen and the preview screen finishes playing (i.e., at least some of the emoticon images in the emoticon effects finish playing or at least some of the video frames in the video effects finish playing), the sending effect is executed, the effect 303 is presented in the conversation area, and the status of whether the sending was successful is displayed synchronously (i.e., if the effect fails to send, a prompt message indicating that the effect failed to send is displayed on the conversation interface). In this embodiment, the trigger operation is not limited, such as a click or long press operation.

[0109] In some embodiments, the types of special effects include facial expression effects and video effects; in response to a sending operation for session content, special effects generated based on the virtual image of the target object are presented in the session area, including: presenting an facial expression effect sending entry for facial expression effects, and in response to a triggering operation for the facial expression effect sending entry, executing the sending of facial expression effects and presenting the facial expression effects generated based on the virtual image of the target object in the session area; presenting a video effect sending entry for video effects, and in response to a triggering operation for the video effect sending entry, executing the sending of video effects and presenting the video effects generated based on the virtual image of the target object in the session area.

[0110] like Figure 3D As shown, the conversation content is presented in the conversation editing area, and an expression effect sending entry 304 for expression effects is also presented in the conversation editing area. In response to the trigger operation of the expression effect sending entry 304, the expression effect is sent, and the expression effect 303 generated based on the virtual image of the target object is presented in the conversation area. The conversation content is also presented in the conversation editing area, and a video effect sending entry 305 for video effects is also presented. In response to the trigger operation of the video effect sending entry 305, the video effect is sent, and the video effect generated based on the virtual image of the target object is presented in the conversation area. Thus, by presenting the secondary function entry of the effect, the type of effect to be sent can be quickly and intuitively selected.

[0111] In some embodiments, in response to a sending operation for session content, special effects generated based on a virtual avatar of a target object are presented in the session area, including: presenting a sending entry for the session content; in response to a sending operation for triggering the sending entry, playing a preview screen of special effects representing the session content; and in response to the completion of the preview screen playback, executing the sending special effects and presenting special effects generated based on a virtual avatar of the target object in the session area.

[0112] For example, the sending operation used to trigger the sending entry is a first trigger operation. The first trigger operation is distinct from the second trigger operation, which is used to execute the sending of session content. For instance, the first trigger operation is a single click, and the second trigger operation is a long press. In response to a single click on the sending entry, the session content is sent, and a special effect generated based on the virtual image of the target object is displayed in the session area, simultaneously showing whether the sending was successful. In response to a long press on the sending entry, a preview of the special effect representing the session content is played. After the preview finishes playing, the sending effect is executed, and the effect is displayed in the session area, simultaneously showing whether the sending was successful. This reuses the sending entry to achieve the function of playing special effects, saving space occupied by function entry points in the human-computer interaction interface and improving interaction efficiency compared to searching for secondary function entry points. However, the embodiments of this application are not limited to the specific forms of the first and second trigger operations.

[0113] In some embodiments, the types of special effects include facial expression effects and video effects; when the preview of the special effect used to represent the content of the session is from a first type of special effect, in response to the switching operation for the first type of special effect, the first type of special effect is switched to a second type of special effect, and the preview of the second type of special effect is played; wherein, the first type of special effect is either a facial expression effect or a video effect, and the second type of special effect is the other one.

[0114] For example, a preview of the emoji effect can be displayed in the chat area first, and in response to a switching operation, the emoji effect can be switched to a preview of the video effect. Alternatively, a preview of the video effect can be displayed in the chat area first, and in response to a switching operation, the video effect can be switched to a preview of the emoji effect, thus allowing users to select the appropriate effect to send.

[0115] In some embodiments, in response to a switching operation for a first type of effect, switching the first type of effect to a second type of effect includes: presenting a switching entry for a preview screen of the first type of effect; and in response to a triggering operation for the switching entry, switching the presented first type of effect to the second type of effect.

[0116] like Figure 5 As shown, when a preview screen for the first type of special effect (i.e., facial expression special effect) is presented, a switching entry 503 for the preview screen of the first type of special effect is also presented. In response to the trigger operation of the switching entry, the presented first type of special effect is switched to the second type of special effect (i.e., video special effect), so that the appropriate special effect can be selected and sent through the switching entry.

[0117] As an example, when a preview of the first type of effect (i.e., emoticon effect) is displayed, the first type of effect can be switched to the second type of effect (i.e., video effect) in response to a trigger operation on the sending entry, thereby allowing the user to select the appropriate effect for sending via the switching entry. In this embodiment, the trigger operation is not limited; for example, it can be a click or long press operation.

[0118] In some embodiments, the types of special effects include facial expression effects and video effects; in response to triggering a sending operation for a sending entry, playing a preview screen of special effects used to represent the content of the session includes: in response to triggering a triggering operation for a sending entry, determining the parameters corresponding to the triggering operation, and playing a preview screen of special effects of the type corresponding to the parameters; wherein, the triggering operation includes different parameters, and different parameters correspond to different types of special effects, and the parameters included in the triggering operation include at least one of the following: triggering time, triggering action method.

[0119] For example, the trigger operation includes parameters such as trigger time, and the types of effects include facial expression effects and video effects. In response to pressing the send entry and holding the press state for a first set duration before releasing the press state, a preview of the facial expression effect is played. In response to pressing the send entry and holding the press state for a second set duration before releasing the press state, a preview of the video effect is played. The first set duration is different from the second set duration.

[0120] For example, the parameters of the trigger operation include the trigger action method (such as swiping, double-tapping, etc.), and the types of effects include emoticon effects and video effects. In response to a double-tapping operation on the sending entry, a preview of the emoticon effect is played, and in response to a swiping operation on the sending entry, a preview of the video effect is played.

[0121] In some embodiments, when playing a preview of an effect used to represent session content, the method further includes: in response to a stop sending operation for the effect, stopping the playback of the preview of the effect and presenting the session content in the session area.

[0122] For example, if a user does not want to send effects while the preview is playing, the preview of the effects can be stopped in response to the stop sending operation for the effects, the session content can be sent, and the session content can be displayed in the session area, simultaneously showing whether the session content was successfully sent.

[0123] In some embodiments, in response to a stop sending operation for an effect, stopping the playback of the effect's preview screen includes: displaying a playback countdown control of the preview screen in the sending entry; and stopping the playback of the effect's preview screen in response to a triggering operation of the playback countdown control.

[0124] like Figure 5As shown, during the playback of the preview screen, if the user does not want to send special effects, a playback countdown control for the preview screen is displayed in the sending entry. In response to the trigger operation of the playback countdown control, a stop sending operation is triggered to stop the playback of the preview screen of the special effects, execute the sending of session content, and display the session content in the session area, and simultaneously display the status of whether the session content was successfully sent.

[0125] In some embodiments, in response to a stop sending operation for an effect, stopping the playback of the effect preview screen includes: in response to pressing the send entry and holding the pressed state for a set duration without releasing the pressed state, stopping the playback of the effect preview screen.

[0126] For example, if the send entry remains pressed until a set duration (e.g., 5 seconds) has elapsed, a stop sending operation is triggered to stop the preview of the effects, send the session content, and display the session content in the session area, simultaneously showing whether the session content was successfully sent.

[0127] In some embodiments, when the conversation content is a voice message, in response to an input operation based on the conversation editing area, the conversation content formed by the input operation is acquired, including: presenting a recording entry in the conversation editing area; in response to pressing the recording entry and maintaining the pressed state, acquiring the voice message collected by the voice input operation in the pressed state; in response to a sending operation for the conversation content, presenting a special effect generated based on the virtual image of the target object in the conversation area, including: presenting a special effect sending entry; in response to moving to the special effect sending entry in the pressed state and releasing the pressed state, presenting a special effect generated based on the virtual image of the target object in the conversation area.

[0128] like Figure 7 As shown, the types of special effects include facial expression effects and video effects. The special effect sending entry includes facial expression effect sending entry and video effect sending entry. In response to moving to the facial expression effect sending entry 702 while in a pressed state and then releasing the pressed state, facial expression effect 703 generated based on the virtual image of the target object is displayed in the conversation area. In response to moving to the video effect sending entry 704 while in a pressed state and then releasing the pressed state, video effect 705 generated based on the virtual image of the target object is displayed in the conversation area. In response to moving to an empty position while in a pressed state and then releasing the pressed state, content is displayed in the conversation area.

[0129] In some embodiments, before presenting special effects generated based on the virtual avatar of the target object in the conversation area, special effects data for representing the conversation content are determined based on the virtual avatar of the target object and the conversation content; speech data that matches the voice of the target object is determined based on the conversation content; and the special effects data and speech data are synthesized to obtain special effects generated based on the virtual avatar of the target object.

[0130] For example, special effects data includes facial expression images or video frames, and voice data includes voice files or audio frames. Based on the virtual avatar of the target object and the conversation content, multiple facial expression images are determined to represent the conversation content. Based on the conversation content, a voice file matching the target object's voice is determined. The multiple facial expression images and the voice file are then combined to obtain facial expression effects generated based on the target object's virtual avatar. Similarly, based on the virtual avatar of the target object and the conversation content, multiple video frames are determined to represent the conversation content. Based on the conversation content, multiple audio frames matching the target object's voice are determined. The multiple video frames and multiple audio frames are then combined to obtain video effects generated based on the target object's virtual avatar. It should be noted that when the conversation content is a voice message, the voice message can be directly used as voice data.

[0131] In some embodiments, before determining the special effects data used to characterize the session content, a real image of the target object is obtained; based on the real image, a virtual avatar generation model is invoked to obtain a virtual avatar of the target object.

[0132] like Figure 4A As shown, the user's personal image information (i.e., real image) is pre-entered. Clicking the virtual avatar generation entry 401 enters the product function module, prompting that at least one photo needs to be uploaded to generate a virtual avatar. Clicking the "Upload Photo" button 402 allows the user to select at least one photo from the album, or take at least one photo in real time. Clicking the "Upload Photo" button 403 uploads at least one photo. Based on the uploaded photo, the virtual avatar generation model is invoked to obtain the user's virtual avatar 404 (i.e., the virtual avatar of the target object).

[0133] In some embodiments, a virtual image generation model is invoked based on a real image to obtain a virtual image of the target object, including: performing the following processing through the virtual image generation model: performing principal component analysis on the real image to obtain the geometric information distribution and texture information distribution of the corresponding target object; performing deformation processing based on the geometric information distribution and texture information distribution to obtain the virtual image of the target object.

[0134] For example, multiple real images (including 3D shape and color data) can be aligned using a virtual avatar generation model (such as a 3D deformation model). Principal Component Analysis (PCA) is then used to extract lower-dimensional subspaces (i.e., the geometric and texture information distributions corresponding to the target object) from the 3D shape and color data. These subspaces are then combined and deformed to generate new virtual avatars.

[0135] It should be noted that before generating a virtual avatar of the target object, you can choose the style of the virtual avatar (such as anime, ancient style, etc.). In response to the selection of the virtual avatar style, a virtual avatar that matches the selected virtual avatar style is generated. Different virtual avatar styles correspond to different virtual avatar generation models.

[0136] In some embodiments, determining speech data that matches the voice of a target object based on the conversation content includes: performing text analysis processing on the text information corresponding to the conversation content to obtain the context features of the conversation content; performing speech parameter prediction processing on the context features of the conversation content based on the voice of the target object to obtain multiple speech parameters corresponding to the conversation content and matching the voice of the target object; and performing synthesis processing on the multiple speech parameters to obtain speech data that matches the voice of the target object.

[0137] like Figure 4B As shown, the user's real voice (i.e., the target object's voice) is pre-recorded. Clicking the "Start Recording" button 406 records the user's real voice. A Hidden Markov Model is trained using the target object's voice. When the input conversation content is a text message, the text information of the conversation message is processed by text analysis to obtain the context features of the text information. The trained Hidden Markov Model is then used to predict the speech parameters of the context features of the text information to obtain multiple speech parameters corresponding to the conversation content and conforming to the target object's voice. These multiple speech parameters are then synthesized to obtain speech data that conforms to the target object's voice.

[0138] In some embodiments, when the session content is a text message, based on the virtual image of the target object and the session content, determining the special effects data to represent the session content includes: performing text feature extraction processing on the text message to obtain the text features of the text message; matching multiple candidate special effects data contained in the database based on the text features, and using the matched candidate special effects data as the special effects data to represent the session content; wherein, the text features include at least one of the following: text, symbols, and emoticons.

[0139] The database includes various types, such as action libraries and emoji libraries. Action libraries contain various action data, such as emoji images containing the "bye-bye" gesture (i.e.,...). The text feature represents the video frame of "bye-bye" (i.e., a video frame containing a virtual avatar making a "bye-bye" gesture). The emoji library includes various emoji data, such as emoji images containing "smiley faces" and video frames representing "smiley faces." For example, if the text feature includes the word "bye-bye," then the matched emoji image containing "bye-bye" or the video frame representing "bye-bye" will be used as the special effect data representing the conversation content; if the text feature includes the character ":), then the matched emoji image containing "smiley faces" or the video frame representing "smiley faces" will be used as the special effect data representing the conversation content; if the text feature includes " "For emoticons, the matched images containing smiley faces or video frames representing smiley faces will be used as special effects data to represent the content of the conversation."

[0140] In some embodiments, when the conversation content is a voice message, based on the virtual image of the target object and the conversation content, special effects data for representing the conversation content is determined, including: performing format conversion processing on the voice message to obtain the text message corresponding to the voice message; performing text feature extraction processing on the text message to obtain the text features of the text message; matching multiple candidate special effects data contained in the database based on the text features, and using the matched candidate special effects data as special effects data for representing the conversation content.

[0141] For example, when the conversation content is a voice message, the voice message needs to be converted into text, and features need to be extracted based on the converted text to obtain text features. Based on the text features, multiple candidate effect data contained in the database are matched, and the matched candidate effect data is used as effect data to represent the conversation content.

[0142] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0143] In related technologies, functions such as text, emoticons, pure voice, pure audio, and pure video are used to realize instant messaging conversations.

[0144] However, compared to text input, conveying information through well-recorded videos has a higher barrier to entry. This is especially true when users are in unfavorable environments, such as when rushing, in dimly lit, or crowded public spaces, making it difficult to use suitable audio or video for interaction. In such situations, users are forced to use text to output information, but text struggles to convey the true context and the user's emotions.

[0145] To address the aforementioned issues, this application proposes a communication method based on virtual avatars (i.e., a conversation processing method). This method allows users to convey information through text and voice input without being limited by scene or time. For example, by first recording the user's image and voice data, surprise emoticons and video messaging functions appropriate to the scene can be triggered during chat communication. This helps users conveniently convey emotional information, integrate virtual avatars into the user's social scenarios, help users present themselves, and better connect with friends.

[0146] The following describes the communication method based on virtual avatars proposed in the embodiments of this application in detail, according to different scenarios:

[0147] Scenario 1: Collecting user voice and image information

[0148] like Figure 4A As shown, the user's personal image information is pre-entered. Clicking the virtual avatar generation entry 401 enters the product function module, prompting that at least one photo needs to be uploaded to generate a virtual avatar. Clicking the "Upload Photo" button 402 allows the user to select at least one photo from their album, or take at least one photo in real time. Clicking the "Upload Photo" button 403 uploads at least one photo, and the user's virtual avatar 404 is automatically generated.

[0149] Pre-record the user's voice information. For example... Figure 4A As shown, after automatically generating the user's virtual avatar (404 error), clicking the "Next" button (405) will result in the following... Figure 4B The recording interface shown allows users to click the "Start Recording" button 406 to record their real voice, and then click the "Sound Synthesis" button 407 to obtain a synthesized audio effect similar to their personal voice. Users can listen to the synthesized audio and display the generated virtual avatar of the user at the virtual avatar generation entry.

[0150] Scenario 2: Message publishing scenario (text publishing)

[0151] like Figure 5As shown in one method of sending, when text editing is completed in the conversation editing area 501, pressing and holding the "Send" button (i.e., the sending entry) for 2 seconds triggers the surprise emoticon function. The surprise emoticon 502 is previewed in the conversation interface, and a 5-second countdown begins, accompanied by sequential phone vibrations. When the 5-second countdown ends, the surprise emoticon 502 is sent. If the 5-second countdown has not ended, clicking the toggle button 503 of the surprise emoticon 502 switches the sending method, changing the surprise emoticon 502 to a video message. The video message 504 is previewed in the conversation interface, and a 5-second countdown also begins. When the 5-second countdown ends, the video message 504 is sent. If the 5-second countdown has not ended, clicking the countdown control 505 cancels the virtual avatar sending method and sends the text information 506 from the conversation editing area 501.

[0152] like Figure 6 Another method of sending text is shown below. When text editing is completed in the conversation editing area 501, pressing and holding the "Send" button for 2 seconds triggers the virtual avatar surprise emoji function. Releasing the "Send" button previews the surprise emoji 502 in the conversation interface, accompanied by a series of phone vibrations. When the preview animation of the surprise emoji 502 ends, the surprise emoji 502 is sent. If the virtual avatar surprise emoji function is triggered and the "Send" button is not released, the timer continues. After 2 seconds (cumulative time is 4 seconds), the surprise emoji animation disappears. When the long press time reaches a cumulative 5 seconds, the virtual avatar video message function is triggered. Releasing the "Send" button previews the video message 504 in the conversation interface, accompanied by a series of phone vibrations. When the preview animation of the video message 504 ends, the video message 504 is sent. If the virtual avatar video message function is triggered and the "Send" button is not released until the preview animation of the video message 504 ends, the virtual avatar sending method is canceled, and the text information in the conversation editing area is sent instead.

[0153] Scenario 3: Message publishing scenario (voice publishing)

[0154] like Figure 7As shown, pressing and holding the voice button 701 (i.e., the recording entry) triggers the voice input function. Once voice input is complete, the user moves their finger to the left icon to display a function prompt. For example, if the user keeps their finger pressed and moves it to the surprise emoji icon 702 (i.e., the emoji effect sending entry), and then releases their finger on the surprise emoji icon 702, the surprise emoji sending command is triggered, and the surprise emoji 703 is displayed in the conversation interface. If the user keeps their finger pressed and moves it to the video message icon 704, and then releases their finger on the video message icon 704 (i.e., the video effect sending entry), the video message sending command is triggered, and the video message 705 is displayed in the conversation interface. If the user keeps their finger pressed and moves it to the blank space 706, and then releases their finger on the blank space 706, the virtual avatar sending method is canceled, and the input voice message is sent.

[0155] Scenario 4: Mobile message receiving scenario (surprise emoji)

[0156] like Figure 8A As shown, after the message sender triggers the command to send a surprise emoji, when the message recipient on the mobile device opens the chat window, the static effect 801 of the surprise emoji is displayed by default. Clicking the play button 802 will play the message sender's sound and trigger the emoji animation.

[0157] like Figure 8B As shown, when the input text message includes an action type emoticon 803, the corresponding action tag 804 is matched in the action library and synthesized with the virtual avatar to display an emoticon animation; for example... Figure 8C As shown, when the input text message includes an object-type emoticon 805, the corresponding object's tag 806 is matched in the object library and synthesized with the virtual avatar to display an emoticon animation; for example... Figure 8D As shown, when the input text message includes a weather-related emoticon 807, the corresponding weather tag 808 is matched in the weather database and synthesized with the virtual avatar to display the emoticon animation.

[0158] Scenario 5: Receiving messages on a PC (with a surprise emoji)

[0159] like Figure 9 As shown, after the message sender triggers the command to send a surprise emoticon, when the message receiver on the PC opens the chat window, the static effect 901 of the surprise emoticon is displayed by default. Clicking the play button 802 will play the message sender's sound and trigger the surprise emoticon.

[0160] Scenario 6: Mobile Message Receiving Scenario (Video Message)

[0161] like Figure 10As shown, after the message sender triggers the command to send a video message, when the message receiver on the mobile device opens the chat window, the video message 1001 is displayed by default. Clicking the play button 1002 will play the video content 1003, and at the same time, the message sender's voice will be played.

[0162] Scenario 7: PC-side message receiving scenario (video message)

[0163] like Figure 11 As shown, after the message sender triggers the command to send a video message, when the message receiver on the PC opens the chat window, the video message 1101 is displayed by default. Clicking the play button 1102 will play the video content 1103, and at the same time, the message sender's voice will be played.

[0164] The following is combined with, for example Figures 12A-12B The flowchart shown illustrates the communication method based on virtual avatars proposed in this application:

[0165] like Figure 12A As shown, the front end uploads a text message and selects the sending type (i.e., a surprise emoji or a video message). After receiving the text message, the back end performs feature extraction processing to obtain text features, and synthesizes audio information based on the text features. When the text features include an emoji, the emoji's label information (i.e., type) is identified, and the database (e.g., a virtual avatar emoji library, a virtual avatar action library, etc.) is matched based on the emoji to match the first special effect data. When the text features do not include an emoji, the database (e.g., a virtual avatar emoji library, a virtual avatar action library, etc.) is matched based on the text in the text features to match the second special effect data. The video information of the virtual avatar is synthesized based on the first and second special effect data, and the final animation effect (i.e., a surprise emoji or a video message) is synthesized based on the audio and video information to present the final effect of the virtual avatar on the front end.

[0166] like Figure 12B As shown, the front end uploads a voice message and selects the sending type (i.e., a surprise emoji or a video message). After receiving the voice message, the back end converts the voice message into text and performs feature extraction processing on the text to obtain text features. Based on the text features, the back end matches the text with a database (such as a virtual avatar emoji library, a virtual avatar action library, etc.) to synthesize the video information of the virtual avatar based on the matched special effects data. Based on the voice message and video information, the final animation effect (i.e., a surprise emoji or a video message) is synthesized to present the final effect of the virtual avatar on the front end.

[0167] It should be noted that it is necessary to obtain the user's visual and vocal characteristics, and based on this, generate a personalized virtual avatar and voice sample for the user. For example... Figure 13As shown, the user uploads a personal photo on the front end (the photo can be taken in real time or obtained from the album). After receiving the personal photo, the back end identifies the image features based on the personal photo and synthesizes the user's virtual image based on the image features. The user also uploads a sound sample on the front end (the sound sample can be taken in real time or obtained from a pre-stored audio file). After receiving the sound sample, the back end identifies the sound features based on the sound sample and synthesizes the user's sound sample based on the sound features.

[0168] The following describes the communication method based on virtual avatars proposed in the embodiments of this application, based on specific algorithms:

[0169] 1) Generating virtual avatars based on users' personal photos

[0170] This application employs 3D Morphable Models (3DMM) technology to generate 3D virtual avatars from 2D photos. A 3D face model database is stored in the background to generate deformable face models. The input 2D image is analyzed and matched to the corresponding 3D face deformable model, and then adjusted to generate the 3D virtual avatar.

[0171] 3DMM can be applied to fields such as face analysis, model fitting, and image synthesis. In face synthesis, multiple sets of 3D face data (including 3D shape and color data) are first scanned simultaneously with high precision and aligned. Then, Principal Component Analysis (PCA) is used to extract lower-dimensional subspaces (i.e., geometric and texture information of the face) from these 3D shape and color data. These PCA subspaces are then combined and deformed to transfer the characteristics of one face to another, generating a new virtual face.

[0172] In this embodiment, the geometric information of the face is represented by a shape vector, and the texture information of the face is represented by a texture vector. The three-dimensional deformed face model established in this application consists of multiple face models. The multiple face models in the dataset are weighted and combined to obtain a new face model.

[0173] The matching process between 3DMM and face images adopts the analysis-by-synthesis technique. First, a preliminary 3D reconstruction of the face is performed based on the current model parameters, which is then mapped to a 2D image. This image is continuously compared with the input image, and the parameters are updated based on the residual information to make the generated 2D image as similar as possible to the input image. Based on the final result, some attributes of the face can be adjusted to output a virtual image similar to the user's input image.

[0174] 2) Synthesizing user voice samples based on user voice

[0175] By learning the acoustic features of user audio characteristics using speech synthesis technology, the synthesized user voice information is suitable for scenarios involving uploading text messages.

[0176] This application employs a parameter-based speech synthesis method. This method uses a statistical model to generate speech parameters in real time and converts the parameters into sound waveforms. The process involves abstracting text into phonetic features, learning the correspondence between phonetic and acoustic features using a statistical model, and then restoring the predicted acoustic features into waveforms. Specifically, a vocoder is used to generate the waveforms, thus realizing the feature-to-waveform conversion.

[0177] like Figure 14 As shown, a Hidden Markov Model (HMM) is trained based on user voices from a speech database to obtain a context-dependent HMM model. Text analysis is performed on the input text message to obtain context features. The context-dependent HMM model is then combined with these context features to generate a state sequence, yielding speech parameters. These parameters are then synthesized using a parametric synthesizer to obtain a synthesized user voice sample (i.e., audio representing the input text message). In this embodiment, the synthesized user voice sample can be stored for later direct retrieval.

[0178] It should be noted that after obtaining the speech signal from the speech library, audio feature extraction is required to obtain audio features (i.e., melspectrogram). Mel-scale frequency cepstral coefficients (MFCC) are an audio feature, i.e., a one-dimensional time-domain signal. It is difficult to intuitively see the frequency domain variation. By using Fourier transform, the corresponding frequency domain information is obtained, but the time domain information is lost, and the frequency domain variation with time cannot be seen. Thus, it is not possible to describe the sound well. To solve this problem, the embodiments of this application use short-time Fourier transform.

[0179] The Short-Time Fourier Transform (SFT) is performed on a short-time signal (obtained by dividing a long-time signal into frames). A long signal is divided into frames, windowed, and then a Fourier Transform (FFT) is performed on each frame. Finally, the results of each frame are stacked to obtain a two-dimensional signal (i.e., a spectrogram). The spectrogram is often a very large image. To obtain appropriately sized sound features, the spectrogram is transformed into a Mel-scale spectrum using Mel-scale filter banks. Cepstral analysis (i.e., taking the logarithm and performing a DCT transform) is then performed on the Mel-scale spectrum to obtain the Mel-cepstral spectrum.

[0180] During the training of HMM, the output state of HMM is represented by a single Gaussian function or a Gaussian mixture function (GMM). The goal of its parameter generation algorithm is to calculate the speech parameter sequence with the maximum likelihood function given a Gaussian distribution sequence.

[0181] 3) Voice messages are converted to text.

[0182] If the user inputs audio (i.e., voice message), in order to extract the user's emotional information from the voice, it is necessary to extract text features from the text. Therefore, it is necessary to convert the audio into text, that is, to upload the audio to the server. The server uses Automatic Speech Recognition (ASR) technology to convert the audio into text information.

[0183] like Figure 15 As shown, features are extracted from the input speech message to obtain speech features, and the speech features are decoded based on the acoustic model and the language model to obtain the text information corresponding to the speech message.

[0184] Feature extraction involves extracting important information that reflects speech characteristics from the speech waveform, removing irrelevant information (such as background noise), and converting this information into a set of discrete parameter vectors.

[0185] If multiple sound features exist in a voice message, and the speaker's voice is separated simultaneously, the sound feature with the highest proportion is determined. First, the voice message is preprocessed by performing silence suppression (VAD, Voice Activity Detection) endpoint detection and framing on the audio content to obtain a sound waveform. Then, Fourier transform is used to complete the conversion from the time domain to the frequency domain. That is, Fourier transform is performed on each frame and the feature parameters (i.e., Mel-scale Frequency Cepstral Coefficients (MFCC)) are used to represent it, so as to obtain the spectrum of each frame to remove background noise, irrelevant human voices, etc. in the audio.

[0186] After feature extraction, the speech is framed using an acoustic model, and pronunciation-related tasks are performed to obtain the basic phoneme states and probabilities of speech, identify the smallest phoneme in the speech, and then combine multiple phonemes to form words, and finally combine words to form text sentences. The language model is used to combine semantic scenes and context to generate coherent and accurate text.

[0187] 4) Text Information Extraction

[0188] If the user input is a text message, text feature extraction is performed directly to obtain text features. Text feature extraction includes steps such as automatic word segmentation, part-of-speech tagging, syntactic analysis, and phonological prediction. Syntactic analysis is used to determine the syntactic structure of the sentence or the dependency relationships between words in the sentence.

[0189] This application employs a statistical word segmentation method. A word is a stable combination of characters; therefore, the more frequently adjacent characters appear together in a context, the more likely they are to form a word. Thus, the frequency or probability of co-occurrence between adjacent characters can effectively reflect the feasibility of forming a word. The frequency of each combination of adjacent characters in the text can be statistically analyzed to calculate mutual occurrence information. For example, the mutual occurrence information of characters X and Y can be calculated as M(X,Y)=log(P(X,Y) / P(X)P(Y)). Here, P(X,Y) represents the probability of co-occurrence between characters X and Y, P(X) represents the frequency of character X in the text, and P(Y) represents the frequency of character Y in the text. Mutual occurrence information reflects the tightness of the relationship between characters. When the tightness exceeds a certain threshold, it can be considered that these adjacent characters (i.e., character groups) may constitute a word. This method only requires statistical analysis of the frequency of character groups in the text and does not require a segmentation dictionary.

[0190] Part-of-speech tagging refers to the process of assigning a correct part-of-speech tag to each word in the word segmentation result, that is, determining whether each word is a noun, verb, adjective, or other part of speech, to assist in syntactic analysis preprocessing. This application's embodiment employs a Hidden Markov Model-based statistical model, which can be trained using a large corpus of labeled data. Labeled data refers to text where each word has been correctly assigned a part-of-speech tag.

[0191] Meanwhile, the extracted text features are used to match databases such as action databases, expression databases, object databases, and weather databases to synthesize dynamic images of virtual avatars.

[0192] 5) Tag recognition of emoticons (i.e., emoticon images)

[0193] Emoji tag recognition is used to synthesize different virtual avatar actions. The following processing methods are used for different types of emojis:

[0194] 1. Action-related emoticons—Find similar action tags (such as action emoticons or action video frames) in the action library, and prioritize matching such actions based on text features extracted from text messages;

[0195] 2. Emotional Expression Emojis—Find similar emotion tags (such as emotional expressions or emotional video frames) in the emoji library, and prioritize matching such emojis based on text features extracted from text messages;

[0196] 3. Object-based emoticons—Create relevant object tags (such as item emoticons or item video frames) in the backend library. When creating special effects, the object tags appear in the background of the virtual character as needed, and can also be combined with matching action tags for display.

[0197] 4. Weather-related emojis - Create relevant weather tags (such as weather emojis or weather video frames) in the backend library. These weather tags will appear in the background of the virtual avatar as needed.

[0198] 6) Action matching

[0199] By parsing user-uploaded text or voice messages, text information can be obtained from text features, which can then be used to filter out the actions of virtual avatars in the corresponding action library. The actions are transmitted to the backend for confidence matching through image retrieval, and combined with the results of emoji tag recognition to determine the most suitable action, which is then bound to the user's virtual avatar for synthesis, resulting in virtual avatar actions (i.e., emoji images or video frames).

[0200] In summary, if the user inputs a text message, the synthesized audio and virtual avatar actions will be matched to synthesize a surprise emoji or video message; if the user inputs a voice message, the user input voice message and virtual avatar actions will be matched to synthesize a surprise emoji or video message.

[0201] Therefore, the embodiments of this application can solve the problem of users being unable to use language and video communication due to noisy environments or unattractive scenery, allowing users to obtain personalized virtual avatar video messages and voice expressions using low-cost text and voice input methods, and more conveniently enhance emotional connections between people.

[0202] The exemplary application and implementation of the electronic device provided in the embodiments of this application have been used to describe the session processing method provided in the embodiments of this application. The following will continue to describe the scheme of how the various modules (first display module 5551, acquisition module 5552, and second display module 5553) in the session processing device 555 provided in the embodiments of this application cooperate to implement session processing.

[0203] The first display module 5551 is used to present a session editing area; the acquisition module 5552 is used to acquire session content formed by the input operation based on the session editing area; the second display module 5553 is used to present special effects generated based on the virtual image of the target object in the session area in response to the sending operation of the session content, wherein the special effects are used to characterize the session content.

[0204] In some embodiments, the second display module 5553 is further configured to play facial expression effects generated based on the virtual image of the target object, wherein the facial expression effects include at least one facial expression image representing the conversation content based on the virtual image, and speech representing the conversation content based on the voice of the target object.

[0205] In some embodiments, the emoticon image is used to characterize the conversation content from at least one of the following dimensions: keywords of the conversation content, emotional information carried by the conversation content, and the topic to which the conversation content belongs.

[0206] In some embodiments, the facial expression effect is set to a playback mode, which includes: automatically playing any one of the facial expression images when the facial expression effect is received, and switching to play multiple facial expression images when the facial expression effect is triggered to play, while simultaneously playing the voice.

[0207] In some embodiments, the second display module 5553 is further configured to play video effects generated based on the virtual image of the target object, wherein the video effects include multiple video frames representing the conversation content based on the virtual image, and multiple audio frames representing the conversation content based on the voice of the target object.

[0208] In some embodiments, the video effect is configured with a playback mode, which includes: automatically playing any one of the video frames when the video effect is received, playing the plurality of video frames when the video effect is triggered to play, and synchronously playing the plurality of audio frames.

[0209] In some embodiments, the second display module 5553 is further configured to present a sending entry for the session content; in response to a sending operation for triggering the sending entry, a preview screen for special effects representing the session content is played, wherein the first triggering operation is different from the second triggering operation, the second triggering operation being used to send the session content; in response to the completion of the preview screen playback, the special effects are sent, and special effects generated based on the virtual image of the target object are presented in the session area.

[0210] In some embodiments, the types of special effects include facial expression effects and video effects; the second display module 5553 is further configured to, in response to a switching operation for the first type of special effect, switch the first type of special effect to a second type of special effect and play a preview of the second type of special effect when the preview of the special effect representing the session content is from the first type of special effect; wherein, the first type of special effect is any one of the facial expression effect and the video effect, and the second type of special effect is the other one of the facial expression effect and the video effect.

[0211] In some embodiments, the second display module 5553 is further configured to present a switching entry for a preview screen of the first type of special effect; in response to a trigger operation on the switching entry, the presented first type of special effect is switched to the second type of special effect.

[0212] In some embodiments, the types of special effects include facial expression effects and video effects; the second display module 5553 is further configured to, in response to a trigger operation for triggering the sending entry, determine the parameters corresponding to the trigger operation and play a preview of the special effect of the type corresponding to the parameters; wherein, the trigger operation includes different parameters and the different parameters correspond to different types of special effects, and the parameters included in the trigger operation include at least one of the following: trigger time and trigger action mode.

[0213] In some embodiments, when a preview of an effect representing the session content is played, the second display module 5553 is further configured to present the session content in the session area.

[0214] In some embodiments, the second display module 5553 is further configured to present a playback countdown control of the preview screen in the sending inlet; and to stop playing the preview screen of the special effect in response to a trigger operation on the playback countdown control.

[0215] In some embodiments, the second display module 5553 is further configured to stop playing the preview screen of the special effect in response to pressing the sending inlet and holding the pressed state for a set time without releasing the pressed state.

[0216] In some embodiments, when the conversation content is a voice message, the second display module 5553 is further configured to present a recording entry in the conversation editing area; in response to pressing the recording entry and maintaining the pressing state, acquire the voice message collected for the voice input operation in the pressing state; present an effect sending entry; in response to moving to the effect sending entry in the pressing state and releasing the pressing state, present an effect generated based on the virtual image of the target object in the conversation area.

[0217] In some embodiments, the types of special effects include facial expression effects and video effects, and the special effect sending entry includes facial expression effect sending entry and video effect sending entry; the second display module 5553 is further configured to, in response to moving to the facial expression effect sending entry in the pressed state and releasing the pressed state, present facial expression effects generated based on the virtual image of the target object in the session area; and in response to moving to the video effect sending entry in the pressed state and releasing the pressed state, present video effects generated based on the virtual image of the target object in the session area.

[0218] In some embodiments, before presenting the special effects generated based on the virtual image of the target object in the conversation area, the device further includes: a processing module 5554, configured to determine special effects data representing the conversation content based on the virtual image of the target object and the conversation content; determine voice data that matches the voice of the target object based on the conversation content; and synthesize the special effects data and the voice data to obtain special effects generated based on the virtual image of the target object.

[0219] In some embodiments, before determining the special effects data used to characterize the session content, the processing module 5554 is further configured to acquire a real image of the target object; and based on the real image, call the virtual image generation model to obtain a virtual image of the target object.

[0220] In some embodiments, the processing module 5554 is further configured to perform the following processing through the virtual image generation model: perform principal component analysis on the real image to obtain the geometric information distribution and texture information distribution corresponding to the target object; perform deformation processing based on the geometric information distribution and the texture information distribution to obtain the virtual image of the target object.

[0221] In some embodiments, the processing module 5554 is further configured to perform text analysis processing on the text information corresponding to the session content to obtain the context features of the session content; perform speech parameter prediction processing on the context features of the session content based on the target object's voice to obtain multiple speech parameters corresponding to the session content and conforming to the target object's voice; and perform synthesis processing on the multiple speech parameters to obtain speech data conforming to the target object's voice.

[0222] In some embodiments, when the session content is a text message, the processing module 5554 is further configured to perform text feature extraction processing on the text message to obtain the text features of the text message; match multiple candidate effect data contained in the database based on the text features, and use the matched candidate effect data as effect data to characterize the session content; wherein, the text features include at least one of the following: text, symbols, and emoticons.

[0223] In some embodiments, when the conversation content is a voice message, the processing module 5554 is further configured to perform format conversion processing on the voice message to obtain a text message corresponding to the voice message; perform text feature extraction processing on the text message to obtain the text features of the text message; and match multiple candidate effect data contained in the database based on the text features, and use the matched candidate effect data as effect data used to characterize the conversation content.

[0224] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the session processing method described above in this application.

[0225] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the session processing method provided in this application, for example... Figure 3A The session handling method is shown.

[0226] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0227] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0228] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0229] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0230] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A session processing method, characterized in that, Applied to a virtual scene, wherein the virtual scene includes a virtual avatar representing a target object, the method includes: Present the conversation editing area; In response to an input operation based on the session editing area, the session content formed by the input operation is obtained; Presents an entry point for sending the aforementioned session content; In response to a long press operation on the sending entry, a preview of the special effects representing the session content is played, a function countdown is started, and a function countdown control is displayed in the sending entry. In response to the end of the function countdown, the special effect is sent and the special effect generated based on the virtual image of the target object is displayed in the session area; In response to receiving a trigger operation from the function countdown control before the function countdown has ended, stop playing the preview screen of the special effect, execute the sending of the session content, and display the session content in the session area.

2. The method according to claim 1, characterized in that, The presentation of special effects generated based on the virtual avatar of the target object in the session area includes: Play facial expression effects generated based on the virtual image of the target object, wherein the facial expression effects include at least one facial expression image representing the conversation content based on the virtual image, and speech representing the conversation content based on the voice of the target object.

3. The method according to claim 2, characterized in that, The emoticons are used to represent the conversation content from at least one of the following dimensions: keywords of the conversation content, emotional information carried by the conversation content, and the topic to which the conversation content belongs.

4. The method according to claim 2, characterized in that, The facial expression effect is set with a playback mode, which includes: automatically playing any one of the facial expression images when the facial expression effect is received, and switching to play multiple facial expression images when the facial expression effect is triggered to play, while simultaneously playing the voice.

5. The method according to claim 1, characterized in that, The presentation of special effects generated based on the virtual avatar of the target object in the session area includes: Play video effects generated based on the virtual image of the target object, wherein the video effects include multiple video frames representing the content of the conversation based on the virtual image, and multiple audio frames representing the content of the conversation based on the voice of the target object.

6. The method according to claim 5, characterized in that, The video effect is set with a playback mode, which includes: automatically playing any one of the video frames when the video effect is received, playing multiple video frames when the video effect is triggered to play, and synchronously playing multiple audio frames.

7. The method according to claim 1, characterized in that, The types of special effects include facial effects and video effects; The method further includes: When the preview of the special effect used to represent the session content is from a first type of special effect, in response to the switching operation for the first type of special effect, the first type of special effect is switched to a second type of special effect, and the preview of the second type of special effect is played. Wherein, the first type of special effect is any one of the facial expression special effect and the video special effect, and the second type of special effect is the other one of the facial expression special effect and the video special effect.

8. The method according to claim 7, characterized in that, The step of switching the first type of effect to the second type of effect in response to the switching operation for the first type of effect includes: A switch to display the preview screen for the first type of special effects; In response to a trigger operation on the switching entry, the first type of effect is switched to the second type of effect.

9. The method according to claim 1, characterized in that, The types of special effects include facial effects and video effects; The triggering operation for the sending entry includes different parameters, and the different parameters correspond to different types of special effects. The parameters included in the triggering operation include at least one of the following: triggering time and triggering action method.

10. The method according to claim 1, characterized in that, When playing a preview of effects used to represent the session content, the method further includes: In response to the stop sending operation for the effect, the preview of the effect is stopped, and the session content is displayed in the session area.

11. The method according to claim 10, characterized in that, The method further includes: If the press and hold state of the send input is not released after a set duration, the preview screen of the special effect will stop playing.

12. The method according to claim 1, characterized in that, When the conversation content is a voice message, the step of responding to an input operation based on the conversation editing area and obtaining the conversation content formed by the input operation includes: The recording entry point is displayed in the session editing area; In response to pressing the recording inlet and holding the pressed state, the voice message collected in response to the voice input operation during the pressed state is acquired; The presentation of special effects generated based on the virtual avatar of the target object in the session area includes: The entry point for displaying special effects; In response to the release of the press state after moving to the effect sending entry point in the press state, an effect generated based on the virtual image of the target object is presented in the session area.

13. The method according to claim 12, characterized in that, The types of special effects include facial expression effects and video effects, and the special effects sending entry points include facial expression effect sending entry points and video effect sending entry points; The response, upon moving to the effect sending entry point in the pressed state and releasing the pressed state, presents an effect generated based on the virtual image of the target object in the session area, including: In response to the release of the pressing state after moving to the expression effect sending entry point in the pressing state, an expression effect generated based on the virtual image of the target object is presented in the chat area; In response to the release of the pressing state after moving to the video effect sending entry point in the pressing state, a video effect generated based on the virtual image of the target object is presented in the session area.

14. The method according to claim 1, characterized in that, Before presenting the special effects generated based on the virtual image of the target object in the session area, the method further includes: Based on the virtual image of the target object and the conversation content, special effects data for representing the conversation content are determined; Based on the conversation content, determine the voice data that matches the target object's voice; The special effects data and the voice data are synthesized to obtain special effects generated based on the virtual image of the target object.

15. The method according to claim 14, characterized in that, Before determining the special effects data used to characterize the session content, the method further includes: Obtain the real image of the target object; The virtual image of the target object is obtained by calling the virtual image generation model based on the real image.

16. A session processing apparatus, characterized in that, Applied to a virtual scene, the virtual scene including a virtual image representing a target object, the device includes: The first display module is used to present the session editing area; The acquisition module is used to acquire the session content formed by the input operation in response to the input operation based on the session editing area; The second display module is used to present the sending entry point for the session content; In response to a long press operation on the sending entry, a preview of the special effects representing the session content is played, a function countdown is started, and a function countdown control is displayed in the sending entry. In response to the end of the function countdown, the special effect is sent and the special effect generated based on the virtual image of the target object is displayed in the session area; In response to receiving a trigger operation from the function countdown control before the function countdown has ended, stop playing the preview screen of the special effect, execute the sending of the session content, and display the session content in the session area.

17. The apparatus according to claim 16, characterized in that, The second display module is further configured to play facial expression effects generated based on the virtual image of the target object, wherein the facial expression effects include at least one facial expression image representing the conversation content based on the virtual image, and speech representing the conversation content based on the voice of the target object.

18. The apparatus according to claim 17, characterized in that, The emoticons are used to represent the conversation content from at least one of the following dimensions: keywords of the conversation content, emotional information carried by the conversation content, and the topic to which the conversation content belongs.

19. The apparatus according to claim 17, characterized in that, The facial expression effects are set to a playback mode. The second display module is also used to automatically play any one of the expression images when the expression effect is received, and to switch between playing multiple expression images when the expression effect is triggered to play, while simultaneously playing the voice.

20. The apparatus according to claim 16, characterized in that, The second display module is further configured to play video effects generated based on the virtual image of the target object, wherein the video effects include multiple video frames representing the conversation content based on the virtual image, and multiple audio frames representing the conversation content based on the voice of the target object.

21. The apparatus according to claim 20, characterized in that, The video effects are set to playback modes. The second display module is further configured to automatically play any one of the video frames when the video effect is received, play the multiple video frames when the video effect is triggered to play, and simultaneously play the multiple audio frames.

22. The apparatus according to claim 16, characterized in that, The types of special effects include facial effects and video effects; The second display module is further configured to, in response to a switching operation for the first type of special effect, switch the first type of special effect to a second type of special effect and play a preview of the second type of special effect when the preview of the special effect representing the session content is from the first type of special effect; Wherein, the first type of special effect is any one of the facial expression special effect and the video special effect, and the second type of special effect is the other one of the facial expression special effect and the video special effect.

23. The apparatus according to claim 22, characterized in that, The second display module is also used to present a switching entry for the preview screen of the first type of special effects; In response to a trigger operation on the switching entry, the first type of effect is switched to the second type of effect.

24. The apparatus according to claim 16, characterized in that, The types of special effects include facial effects and video effects; The triggering operation for the sending entry includes different parameters, and the different parameters correspond to different types of special effects. The parameters included in the triggering operation include at least one of the following: triggering time and triggering action method.

25. The apparatus according to claim 16, characterized in that, When the content of the conversation is a voice message The second display module is also used to present the recording entry point in the session editing area; In response to pressing the recording inlet and holding the pressed state, the voice message collected in response to the voice input operation during the pressed state is acquired; The second display module is also used to present the special effects sending entry point; In response to the release of the press state after moving to the effect sending entry point in the press state, an effect generated based on the virtual image of the target object is presented in the session area.

26. The apparatus according to claim 25, characterized in that, The types of special effects include facial expression effects and video effects, and the special effects sending entry points include facial expression effect sending entry points and video effect sending entry points; The second display module is further configured to release the pressing state after moving to the expression effect sending entry in the pressing state, and to present an expression effect generated based on the virtual image of the target object in the conversation area; In response to the release of the pressing state after moving to the video effect sending entry point in the pressing state, a video effect generated based on the virtual image of the target object is presented in the session area.

27. The apparatus of claim 16, further comprising: The processing module is used to determine special effects data to characterize the session content based on the virtual image of the target object and the session content. Based on the conversation content, determine the voice data that matches the target object's voice; The special effects data and the voice data are synthesized to obtain special effects generated based on the virtual image of the target object.

28. The apparatus according to claim 27, characterized in that, The processing module is also used to acquire a real image of the target object; The virtual image of the target object is obtained by calling the virtual image generation model based on the real image.

29. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the session processing method according to any one of claims 1 to 15.

30. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the session processing method according to any one of claims 1 to 15 when executed by a processor.

31. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the session processing method according to any one of claims 1 to 15 is implemented.

Images