Object interaction method, device and electronic device based on virtual image

Abstract

The present disclosure relates to a method, device and electronic device for object interaction based on a virtual image, and belongs to the field of multimedia technology. The method includes: generating a virtual image of the object in response to a trigger operation based on the object; displaying the virtual image and an interactive control; and controlling the virtual image to perform human-computer interaction in response to a trigger operation on the interactive control. The virtual image provided by the present disclosure can assume the function of an intelligent agent for an object in a visualized virtual interactive form, enriching the object interaction mode, improving the efficiency of obtaining object information, and improving the efficiency of human-computer interaction.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia technology, and in particular to a method, device and electronic device for object interaction based on a virtual image. Background Art

[0002] With the development of multimedia technology, users can browse items of interest anytime and anywhere through applications on their terminals.

[0003] For example, when a user is watching a live broadcast, the host puts an item on the shelf in the live broadcast room. The user clicks on the item to enter the item details interface and browse the item's preview picture, specifications, size and other descriptive information. If the user wants to know more about the item information, he or she usually needs to talk to the merchant's customer service before obtaining the item information.

[0004] At present, the object interaction methods supported by terminals are relatively simple, the user's efficiency in obtaining object information is low, and the human-computer interaction efficiency is low. Summary of the invention

[0005] The present disclosure provides a method, device and electronic device for object interaction based on a virtual image, so as to at least enrich the object interaction mode, improve the efficiency of obtaining object information, and improve the efficiency of human-computer interaction. The technical solution of the present disclosure is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a method for interacting with an object based on a virtual image is provided, comprising:

[0007] In response to a trigger operation based on an object, a virtual image of the object is generated, wherein the virtual image is used to represent the object, and the virtual image is an interactive object for realizing human-computer interaction;

[0008] Displaying the virtual image and an interactive control, wherein the interactive control is used to trigger interaction with the virtual image;

[0009] In response to a triggering operation on the interactive control, the virtual image is controlled to perform human-computer interaction.

[0010] In some embodiments, in response to a trigger operation based on an object, generating a virtual image of the object includes:

[0011] Displaying an item list, wherein the item list contains at least one item;

[0012] In response to a triggering operation based on any one of the items in the item list, a virtual image of the item is generated.

[0013] In some embodiments, the item list is used to provide items that support human-computer interaction based on the virtual image.

[0014] In some embodiments, displaying the virtual image and the interactive control comprises:

[0015] Jump from the current interface to the avatar display interface;

[0016] The virtual image and the interactive controls are displayed in the virtual image display interface.

[0017] In some embodiments, the virtual image and the interactive control are independent of each other; or, part or all of the virtual image is configured as the interactive control.

[0018] In some embodiments, the interactive control is a voice interactive control, and in response to a triggering operation on the interactive control, controlling the virtual image to perform human-computer interaction includes:

[0019] In response to a triggering operation on the voice interaction control, recording a question voice;

[0020] Based on the question voice, generating an answer voice for the question voice;

[0021] Play the answering voice.

[0022] In some embodiments, generating a response voice for the question voice based on the question voice includes:

[0023] Performing speech recognition on the question voice to obtain a question text that matches the semantics of the question voice;

[0024] Based on the question text, searching for enhanced text of the question text, wherein the enhanced text and the question text meet similarity conditions;

[0025] Generate an answer text based on the question text and the enhanced text;

[0026] The answer text is subjected to text-to-speech conversion to obtain the answer speech that matches the semantics of the answer text.

[0027] In some embodiments, searching for enhanced text of the question text based on the question text includes:

[0028] Extracting question text features of the question text;

[0029] Searching for an item text feature whose similarity with the question text feature meets the similarity condition from an item feature library, wherein the item feature library is used to store text features of item information of each item;

[0030] The enhanced text to which the searched item text feature belongs is obtained.

[0031] In some embodiments, the item information includes detailed information of the item, and the method further includes:

[0032] Performing text segmentation on the detailed information in the item information of each item to obtain multiple candidate texts;

[0033] The text features of each of the candidate texts are extracted, and each extracted text feature is stored in the object feature library.

[0034] In some embodiments, generating the answer text based on the question text and the enhanced text includes:

[0035] The question text and the enhanced text are input into a text generation model, and the answer text is generated by the text generation model. The text generation model is used to provide interactive text for question-and-answer dialogues related to items.

[0036] In some embodiments, playing the answering voice includes:

[0037] Based on the answer voice or the answer text, generating a broadcast animation of the virtual image, wherein the broadcast animation is an animation of the virtual image broadcasting the answer voice;

[0038] Play the animation and the answering voice.

[0039] In some embodiments, the interactive control is a text interactive control, and in response to a triggering operation on the interactive control, controlling the virtual image to perform human-computer interaction includes:

[0040] In response to a triggering operation on the text interaction control, obtaining an input question text;

[0041] Based on the question text, generate an answer text for the question text;

[0042] Display the answer text; or, perform text-to-speech conversion on the answer text to obtain an answer voice that matches the semantics of the answer text, and play the answer voice.

[0043] In some embodiments, the method further comprises:

[0044] a preview control showing said item;

[0045] In response to a triggering operation on the preview control, capturing a video frame through a camera;

[0046] In the video frame, a real-life image of the object is displayed.

[0047] In some embodiments, the real scene image is an augmented reality image of the object, and displaying the real scene image of the object in the video frame includes:

[0048] Acquire an augmented reality image of the object, and draw the augmented reality image on the video frame; or,

[0049] Based on the augmented reality image, an item preview model of the item is generated, and the item preview model is drawn on the video frame.

[0050] In some embodiments, the method further comprises:

[0051] Based on the item information, generating an introduction text associated with the virtual image, the introduction text being used to simulate the virtual image introducing the item;

[0052] Based on the virtual image, the introduction text is displayed.

[0053] In some embodiments, generating an introduction text associated with the virtual image based on the item information includes:

[0054] The item information is input into a text generation model, and the introduction text is generated by the text generation model, wherein the text generation model is used to provide interactive text for a question-and-answer dialogue related to the item.

[0055] In some embodiments, the method further comprises:

[0056] Based on the introduction text, generating an introduction voice matching the introduction text;

[0057] Play the introductory voice.

[0058] In some embodiments, the method further comprises:

[0059] The operation prompt information for the interactive control is displayed, where the operation prompt information is used to prompt the interaction method with the interactive control.

[0060] In some embodiments, in response to a trigger operation based on an object, generating a virtual image of the object includes:

[0061] In response to a trigger operation based on the object, a virtual image of the object is generated based on the object information of the object.

[0062] In some embodiments, the item information includes at least one of detailed information or a preview image of the item, and generating a virtual image of the item based on the item information includes:

[0063] generating the virtual image based on the detailed information; or,

[0064] generating the virtual image based on the preview image; or,

[0065] The virtual image is generated based on the detailed information and the preview image.

[0066] In some embodiments, generating a virtual image of the object based on the object information of the object includes:

[0067] Inputting the item information into an image generation model, and generating a virtual image of the item through the image generation model, wherein the image generation model is used to generate a virtual image of the item;

[0068] Based on the avatar image, the avatar is generated.

[0069] In some embodiments, generating the virtual avatar based on the virtual avatar image includes:

[0070] determining the avatar image as the avatar; or,

[0071] Three-dimensional modeling is performed based on the virtual image to obtain a three-dimensional model of the virtual image, and the three-dimensional model is determined as the virtual image.

[0072] According to another aspect of an embodiment of the present disclosure, there is provided an object interaction device based on a virtual image, comprising:

[0073] A generating unit, configured to generate a virtual image of the object in response to a triggering operation based on the object, wherein the virtual image is used to represent the object, and the virtual image is an interactive object for realizing human-computer interaction;

[0074] a display unit configured to display the virtual image and an interactive control, wherein the interactive control is used to trigger interaction with the virtual image;

[0075] The control unit is configured to execute, in response to a triggering operation on the interactive control, control the virtual image to perform human-computer interaction.

[0076] In some embodiments, the generating unit is further configured to perform:

[0077] Displaying an item list, wherein the item list contains at least one item;

[0078] In response to a triggering operation based on any one of the items in the item list, a virtual image of the item is generated.

[0079] In some embodiments, the item list is used to provide items that support human-computer interaction based on the virtual image.

[0080] In some embodiments, the display unit is further configured to perform:

[0081] Jump from the current interface to the avatar display interface;

[0082] The virtual image and the interactive controls are displayed in the virtual image display interface.

[0083] In some embodiments, the virtual image and the interactive control are independent of each other; or, part or all of the virtual image is configured as the interactive control.

[0084] In some embodiments, the interactive control is a voice interactive control, and the control unit includes:

[0085] A voice recording subunit, configured to record a question voice in response to a triggering operation on the voice interaction control;

[0086] A speech generation subunit is configured to generate a response speech for the question speech based on the question speech;

[0087] The playing subunit is configured to play the answering voice.

[0088] In some embodiments, the speech generation subunit includes:

[0089] A speech recognition sub-subunit is configured to perform speech recognition on the question speech to obtain a question text that matches the semantics of the question speech;

[0090] A text search sub-subunit is configured to search for an enhanced text of the question text based on the question text, wherein the enhanced text meets a similarity condition with the question text;

[0091] A text generation sub-subunit, configured to generate an answer text based on the question text and the enhanced text;

[0092] The text-to-speech conversion sub-subunit is configured to perform text-to-speech conversion on the answer text to obtain the answer speech that matches the semantics of the answer text.

[0093] In some embodiments, the text search sub-subunit is further configured to perform:

[0094] Extracting question text features of the question text;

[0095] Searching for an item text feature whose similarity with the question text feature meets the similarity condition from an item feature library, wherein the item feature library is used to store text features of item information of each item;

[0096] The enhanced text to which the searched item text feature belongs is obtained.

[0097] In some embodiments, the item information includes detailed information of the item, and the device further comprises:

[0098] A text segmentation unit is configured to perform text segmentation on the detail information in the item information of each item to obtain a plurality of candidate texts;

[0099] The extraction storage unit is configured to extract text features of each candidate text and store each extracted text feature in the object feature library.

[0100] In some embodiments, the text generation sub-subunit is further configured to perform:

[0101] The question text and the enhanced text are input into a text generation model, and the answer text is generated by the text generation model. The text generation model is used to provide interactive text for question-and-answer dialogues related to items.

[0102] In some embodiments, the apparatus further comprises:

[0103] An animation generating unit is configured to generate a broadcasting animation of the virtual image based on the answering voice or the answering text, wherein the broadcasting animation is an animation of the virtual image broadcasting the answering voice;

[0104] The playing subunit is further configured to play the playing animation and the answering voice.

[0105] In some embodiments, the interactive control is a text interactive control, and the control unit includes:

[0106] A text acquisition subunit, configured to execute a response to a trigger operation on the text interaction control to acquire an input question text;

[0107] A text generation subunit, configured to generate an answer text for the question text based on the question text;

[0108] The display unit is further configured to display the answer text; or,

[0109] The conversion and playback subunit is configured to perform text-to-speech conversion on the answer text, obtain an answer voice that matches the semantics of the answer text, and play the answer voice.

[0110] In some embodiments, the display unit is further configured to perform: displaying a preview control of the item;

[0111] The device further includes: a collection unit configured to collect video frames through a camera in response to a triggering operation on the preview control;

[0112] The display unit is further configured to execute: displaying a real-life image of the object in the video frame.

[0113] In some embodiments, the real scene image is an augmented reality image of the object, and the display unit is further configured to execute:

[0114] Acquire an augmented reality image of the object, and draw the augmented reality image on the video frame; or,

[0115] Based on the augmented reality image, an item preview model of the item is generated, and the item preview model is drawn on the video frame.

[0116] In some embodiments, the apparatus further comprises:

[0117] A text generation unit, configured to generate an introduction text associated with the virtual image based on the item information, wherein the introduction text is used to simulate the virtual image introducing the item;

[0118] The display unit is further configured to display the introduction text based on the virtual image.

[0119] In some embodiments, the text generation unit is further configured to perform:

[0120] The item information is input into a text generation model, and the introduction text is generated by the text generation model, wherein the text generation model is used to provide interactive text for a question-and-answer dialogue related to the item.

[0121] In some embodiments, the apparatus further comprises:

[0122] A speech generating unit, configured to generate an introduction speech matching the introduction text based on the introduction text;

[0123] The playing unit is configured to play the introduction voice.

[0124] In some embodiments, the display unit is further configured to perform:

[0125] The operation prompt information for the interactive control is displayed, where the operation prompt information is used to prompt the interaction method with the interactive control.

[0126] In some embodiments, the generating unit comprises:

[0127] The generating subunit is configured to generate a virtual image of the object based on the object information in response to a triggering operation based on the object.

[0128] In some embodiments, the item information includes at least one of detailed information or a preview image of the item, and the generating subunit is further configured to execute:

[0129] generating the virtual image based on the detailed information; or,

[0130] generating the virtual image based on the preview image; or,

[0131] The virtual image is generated based on the detailed information and the preview image.

[0132] In some embodiments, the generating subunit comprises:

[0133] An image generation sub-subunit is configured to input the object information into an image generation model, and generate a virtual image of the object through the image generation model, wherein the image generation model is used to generate a virtual image of the object;

[0134] The image generation sub-subunit is configured to generate the virtual image based on the virtual image image.

[0135] In some embodiments, the image generation sub-subunit is further configured to perform:

[0136] determining the avatar image as the avatar; or,

[0137] Three-dimensional modeling is performed based on the virtual image to obtain a three-dimensional model of the virtual image, and the three-dimensional model is determined as the virtual image.

[0138] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0139] one or more processors;

[0140] one or more memories for storing the one or more processor-executable instructions;

[0141] The one or more processors are configured to execute the object interaction method based on a virtual image in any possible implementation of the above aspect.

[0142] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When at least one instruction in the computer-readable storage medium is executed by one or more processors of an electronic device, the electronic device is enabled to execute the virtual image-based object interaction method in any possible implementation of the above-mentioned aspect.

[0143] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, comprising one or more instructions, wherein the one or more instructions can be executed by one or more processors of an electronic device, so that the electronic device can execute the virtual image-based object interaction method in any possible implementation of the above-mentioned aspect.

[0144] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0145] When a trigger operation based on an object is detected, a virtual image created for the object is generated, and the virtual image and interactive controls for the virtual image are displayed. By performing a trigger operation with the interactive controls, the virtual image can be further controlled to achieve human-computer interaction, so that the virtual image can assume the function of an intelligent agent for the object in a visualized virtual interactive form, enriching the object interaction method, improving the efficiency of obtaining object information, and improving the efficiency of human-computer interaction.

[0146] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0147] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0148] Figure 1 is a schematic diagram of an implementation environment of a method for interacting with an object based on a virtual image according to an exemplary embodiment;

[0149] Figure 2 is a flow chart showing a method for interacting with an object based on a virtual image according to an exemplary embodiment;

[0150] Figure 3 is a flow chart showing a method for interacting with an object based on a virtual image according to an exemplary embodiment;

[0151] Figure 4 is a schematic diagram of a virtual image display interface provided by an embodiment of the present disclosure;

[0152] Figure 5is a principle flow chart of a method for interacting with an object based on a virtual image provided by an embodiment of the present disclosure;

[0153] Figure 6 is a schematic diagram of a virtual image display interface provided by an embodiment of the present disclosure;

[0154] Figure 7 is a schematic diagram of another virtual image display interface provided by an embodiment of the present disclosure;

[0155] Figure 8 is a flow chart showing a method for interacting with an object based on a virtual image according to an exemplary embodiment;

[0156] Fig. 9 is a schematic diagram of a generation principle of a reply voice provided by an embodiment of the present disclosure;

[0157] Fig.10 is a flow chart showing a method for interacting with an object based on a virtual image according to an exemplary embodiment;

[0158] Fig.11 is a logical structure diagram of a virtual image-based object interaction device according to an exemplary embodiment;

[0159] Fig.12 A structural block diagram of a terminal provided by an exemplary embodiment of the present disclosure is shown;

[0160] Fig.13 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0161] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0162] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0163] The user information involved in this disclosure may be information authorized by the user or fully authorized by all parties. In other words, the information involved in this disclosure (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the item information involved in this disclosure is obtained with full authorization.

[0164] In some embodiments, the meanings of A and / or B include: A and B, A, B.

[0165] The following describes the terms involved in the embodiments of the present disclosure.

[0166] Augmented Reality (AR): AR technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0167] Virtual Reality (VR): also known as virtual reality or spiritual environment technology, is a computer simulation system that can create and experience a virtual environment. VR technology encompasses computer, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments of various high technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. With the help of computers and other equipment, a realistic three-dimensional visual, tactile, olfactory and other sensory experiences are generated in the virtual environment. By combining virtuality and reality, people in the virtual environment can have an immersive feeling.

[0168] Mixed Reality (MR): MR technology is a further development of VR technology. MR technology presents real scene information in virtual scenes, builds an interactive feedback information loop between the real world, the virtual world and the user, so as to enhance the realism of the user experience.

[0169] Extended Reality (XR): XR refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. At the same time, XR technology is also a general term for multiple technologies such as VR, AR, and MR. By integrating the visual interaction technologies of the three, it brings the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world.

[0170] Artificial Intelligence (AI): AI usually refers to the simulation of human intelligence through computer programs, so that computers can process, understand and learn information like humans. AI technology is mainly implemented by using algorithms such as machine learning, deep learning, natural language processing, computer vision, etc., so that computers can process large amounts of data, thereby realizing operations such as understanding, classification, and recognition of information. In other words, AI is a technology that simulates human intelligent thinking, which can realize human cognition and thinking activities. Through this technology, computers can simulate human thinking and intelligence, so that they can complete many complex tasks, such as image recognition, speech recognition, natural language processing, decision making, etc.

[0171] Machine Learning (ML): Machine Learning is the science of making computers learn and act like humans, by providing them with data and information in the form of observations and real-world interactions, to improve their learning in an autonomous way. Machine Learning research is a part of Artificial Intelligence research that aims to provide computers with knowledge through data, observations, and interactions with the world. The acquired knowledge allows computers to generalize correctly to new settings.

[0172] Natural Language Processing (NLP): NLP is an interdisciplinary subject in the fields of computer science, artificial intelligence, and linguistics. It mainly studies how to enable computers to understand, process, generate, and simulate human language, so as to achieve the ability to have natural conversations with humans. Through natural language processing technology, a variety of applications such as machine translation, question-answering systems, sentiment analysis, and text summarization can be achieved. With the development of deep learning technology, artificial neural networks and other machine learning methods have made important progress in the field of natural language processing. Future development directions include deeper semantic understanding, better dialogue systems, more extensive cross-language processing, and more powerful transfer learning technology.

[0173] Text-to-Speech (TTS): also known as text-to-speech technology. TTS belongs to speech synthesis, which refers to the technology of converting computer-generated or externally input text information into understandable and fluent Chinese spoken language (or other language speech) output.

[0174] Speech-to-Text (STT): STT refers to a way of converting spoken words (or Chinese characters) into text files for subsequent use. Typically, STT systems involve speech recognition or automatic speech recognition (ASR) technology, where ASR is a technology that converts human speech into text, with the goal of converting the vocabulary content in human speech into computer-readable input.

[0175] Large Language Model (LLM): Also known as large language model or large model. LLM is a super-large deep learning model pre-trained based on a large amount of data, with excellent natural language understanding capabilities. Generally, LLM is a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and are an important path to artificial intelligence.

[0176] Intelligent agent: refers to an entity with intelligence, that is, a software or hardware entity that can act autonomously. It is a very important concept in the field of artificial intelligence. Any independent entity that can think and interact with the environment can be abstracted as an intelligent agent. In the field of artificial intelligence, intelligent agent is also translated as intelligent body, agent, agent, intelligent subject, AI real body, etc.

[0177] Virtual image: In the embodiments of the present disclosure, a virtual image refers to an interactive object used to represent an object and to achieve human-computer interaction. In other words, a virtual image refers to a visualized image of an intelligent agent created for an object to represent the object. A virtual image is usually an anthropomorphic image, but it can also be a simulated robot image, a simulated animal image, a simulated cartoon image, etc. The embodiments of the present disclosure do not specifically limit this. A virtual image can be a two-dimensional virtual image image, or a three-dimensional virtual image model (for example, a three-dimensional character built based on three-dimensional human skeleton technology, such as a digital human, a virtual human or other three-dimensional models, etc.). The embodiments of the present disclosure do not specifically limit the dimensions of the virtual image.

[0178] With the development of artificial intelligence and deep learning technology, multimodal interactive systems have become a key technology to improve user experience. In the field of online shopping, it has become a trend to provide services to users through a combination of images, voice and text.

[0179] In view of this, in the embodiment of the present disclosure, a unique virtual image is generated for the item, the virtual image is used to represent the item, and the virtual image is an interactive object for realizing human-computer interaction. Human-computer interaction is realized by using the virtual image as a visual simulation form of an intelligent agent, which enriches the item interaction mode, can provide a more intuitive and personalized human-computer interaction experience, has high interactivity and intelligence, and improves the depth and breadth of human-computer interaction, and improves the efficiency of item information acquisition and human-computer interaction. In addition, it can provide a comprehensive service platform, combining knowledge base construction, NLP, TTS, STT and other technologies to create a complete product closed-loop system from the merchant listing items to the end of the final human-computer interaction. The system involves item information management, intelligent customer service, multimodal interaction and AR technology, and can learn and establish a knowledge base from the platform's massive item information, that is, automatically build a knowledge base, and provide a virtual image (such as an anthropomorphic image) and AR experience consistent with the item, to achieve highly intelligent and personalized customer service.

[0180] The following describes the system architecture of the embodiment of the present disclosure.

[0181] Figure 1 is a schematic diagram of an implementation environment of a method for interacting with an object based on a virtual image according to an exemplary embodiment, see Figure 1 In this implementation environment, at least one terminal 101 and a server 102 may be included. The terminal 101 is directly or indirectly connected to the server 102 via wired or wireless communication, which is not limited in the embodiments of the present disclosure. For example, the terminal 101 and the server 102 are both electronic devices.

[0182] The terminal 101 has installed and runs an application that supports browsing items, including but not limited to: at least one of a short video application, an audio and video application, a shopping application, a food delivery application, a travel application, a game application, or a social application. Optionally, the application also supports XR technology, such as an application that supports at least one of XR, VR, AR, or MR.

[0183] The user can start the application in the terminal 101 to browse items. For example, the user opens the item list and performs a trigger operation on any item in the item list to open the virtual image display interface, and in the virtual image display interface, display the virtual image created for the item and the interactive controls for the virtual image. Among them, the virtual image display interface refers to any interface for displaying a virtual image. The virtual image display interface can be an interface for displaying an item list, or it can be another interface jumped from the interface for displaying an item list. The embodiment of the present application does not specifically limit whether the item list and the virtual image are displayed in the same interface. Optionally, the item list is used to provide at least one item that supports human-computer interaction based on a virtual image. For example, the item list can be a shopping cart list, a list of items on the live broadcast shelf, a recommended item list (such as "Guess you like"), etc. The item list is not specifically limited here, and the item list is not necessarily presented in a visual list form. The item list refers to a collection containing multiple items.

[0184] The server 102 is used to provide background services for the application running on the terminal 101, and the server 102 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0185] In some embodiments, the server 102 includes: a business server, a virtual image server, and an item database. The business server is used to maintain and support the main business of the application running on the terminal 101. For example, when the application is a short video application, the business server is a video server, and when the application is a live broadcast application, the business server is a live broadcast server. The virtual image server is used to create and support virtual images of various items in the platform. For example, when a merchant puts any item on the shelf, he can request the virtual image server to create a unique associated virtual image for the item, and store the item and the virtual image in association. The virtual image can be a two-dimensional virtual image picture or a three-dimensional virtual image model, which is not specifically limited here. Among them, each virtual image created for each item can be stored in the virtual image server, or it can also be sent and stored in the item database for subsequent access or call. The item database is also called an item knowledge base. The item database is at least used to store the item information of each item. Optionally, the item database also includes an item feature library, which is used to store text features extracted from each item information. Further, the item database can also store virtual images of each item.

[0186] In an exemplary scenario, when a merchant puts an item on the shelf, the merchant uploads the item information to the virtual image server. The virtual image server generates multiple candidate images for the item based on the item information and returns the multiple candidate images to the merchant. The merchant selects the virtual image that the item is finally bound to from the multiple candidate images. The virtual image server stores the item identification (ID) and the virtual image in association with each other, or the virtual image sends the item identification, item information and the virtual image to the item database, and the item database stores the item identification, item information and the virtual image in association with each other. Alternatively, the virtual image server directly generates the most suitable virtual image for the item, stores the item identification and the virtual image in association with each other locally, or sends them to the item database for association storage. This eliminates the need for the merchant to make a selection, saves communication overhead, and simplifies the virtual image creation process.

[0187] In another exemplary scenario, in addition to uploading item information, merchants can also upload image control information. Image control information can control the candidate images created by the virtual image server for the item. For example, image control information includes but is not limited to: specifying the form (such as anthropomorphism, anthropomorphism, robot, etc.), style (such as cartoon style, realistic style, anime style, etc.), appearance (such as clothing, accessories, appearance, etc.) of the candidate image, and the type of image control information is not specifically limited here. At this time, the merchant uploads the item information and image control information to the virtual image server. The virtual image server generates multiple candidate images that match the image control information based on the item information and the image control information, and returns the multiple candidate images to the merchant, so that the candidate images returned by the virtual image server are more in line with the merchant's expectations or preferences, and the controllability of the virtual image is improved. The merchant selects the virtual image that is finally bound to the item from the multiple candidate images, and the virtual image server stores the item identifier and the virtual image in association, or the virtual image sends the item identifier, item information and the virtual image to the item database, and the item database stores the item identifier, item information and the virtual image in association. Alternatively, the virtual image server directly generates a most suitable virtual image for the item, associates the item identifier and the virtual image and stores them locally, or sends them to the item database for associated storage. This eliminates the need for merchant selection, saves communication overhead, and simplifies the virtual image creation process.

[0188] In some embodiments, the device type of terminal 101 includes: at least one of a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. For example, terminal 101 is a smart phone, or other handheld portable electronic device. The following embodiments are illustrated by taking the case where terminal 101 includes a smart phone. Optionally, if terminal 101 supports XR technology, terminal 101 can also be provided as a wearable electronic device, such as HMD (Head-Mounted Display), VR glasses, VR helmet, VR goggles, etc.

[0189] Those skilled in the art will appreciate that the number of the terminals 101 may be more or less. For example, there may be only one terminal 101, or there may be dozens or hundreds of terminals 101, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals 101.

[0190] Next, combine Figure 1 The implementation environment shown illustrates the basic process of the virtual image-based object interaction method provided by the present invention.

[0191] Figure 2 is a flowchart of a method for interacting with an object based on a virtual image according to an exemplary embodiment. Figure 2 The virtual image-based object interaction method is executed by an electronic device, and is described below using the electronic device as a terminal as an example. For example, the terminal may be the terminal 101 in the above implementation environment.

[0192] In step 201, the terminal generates a virtual image of the object in response to a trigger operation based on the object, where the virtual image is used to represent the object and is an interactive object for realizing human-computer interaction.

[0193] Among them, items refer to items that can be given, transferred, exchanged or traded. For example, items include commodities, services or virtual resources, etc. Items can be physical items or virtual items. Whether the items have physical entities is not specifically limited. For example, items can be physical daily necessities, electronic products, cosmetics, etc., or they can be non-physical knowledge products, services, electronic tickets, platform memberships, game resources, etc. In e-commerce scenarios, items can be listed on the platform by merchants, or they can be sold by the platform itself. Whether the items have third-party merchants is not specifically limited.

[0194] Among them, the virtual image is used to represent the item selected by the triggered operation, and the virtual image is an interactive object for realizing human-computer interaction, that is, the virtual image refers to a virtual interactive form for simulating the triggered item. In addition, the virtual image is also a visual image of an intelligent agent created for the item and used to represent the item. In other words, the virtual image is an intelligent agent of the item, which can simulate the item to realize human-computer interaction with the user. Optionally, the virtual image includes but is not limited to: an anthropomorphic image, a simulated robot image, a simulated animal image, a simulated cartoon image, etc., and the embodiments of the present disclosure do not specifically limit this. The virtual image can be a two-dimensional virtual image image, or a three-dimensional virtual image model (for example, a three-dimensional character built based on three-dimensional human skeleton technology, such as a digital human, a virtual human or other three-dimensional model, etc.), and the embodiments of the present disclosure do not specifically limit the dimension of the virtual image.

[0195] In some embodiments, an application supporting browsing items is installed and running on the terminal, and one or more items are displayed in the application. After the user performs a trigger operation on any item, the terminal generates a virtual image of the item in response to the user's trigger operation on the item. Optionally, the trigger operation includes but is not limited to: click operation, long press operation, double click operation, drag operation, press operation, slide operation, voice command, gesture command, etc. The embodiment of the present disclosure does not specifically limit the type of trigger operation.

[0196] In other embodiments, the user may also receive an item sharing link sent by other users in the application or other applications. In response to the user opening or copying the item sharing link, it may be determined that a trigger operation for the item is detected, a virtual image of the item is generated, and then the application directly jumps to the virtual image display interface of the item, or jumps to the application from other applications and automatically opens the virtual image display interface of the item to display the virtual image and interactive controls in the virtual image display interface. At this time, the user's operation of opening or copying the item sharing link is also a trigger operation for the item. The embodiment of the present disclosure does not specifically limit the method of triggering the operation. Among them, the virtual image display interface refers to any interface for displaying a virtual image. The virtual image display interface may be an interface for displaying items, or any other interface different from the interface for displaying items. The embodiment of the present application does not specifically limit whether the item and the virtual image are displayed in the same interface.

[0197] In an exemplary scenario, the terminal displays an item list in the user interface of the application, and the item list contains one or more items. The user performs a trigger operation on any item in the item list so that the terminal generates a virtual image of the item. Optionally, the above user interface can be a live shopping interface, a shopping cart interface, an item recommendation interface, an item search interface, etc., and the item list is correspondingly presented as a list of items on the live shelves, a shopping cart list, a recommended item list (such as "Guess you like"), a search result list, etc., and the user interface or the item list is not specifically limited here. It should be noted that the item list is used to provide at least one item that supports human-computer interaction based on a virtual image, but the item list is not necessarily presented in a visual list form. The item list refers to a collection of one or more items. It should also be noted that the item list can be in a folded state in the user interface, that is, only one item list control is displayed in the user interface. After the user performs a trigger operation on the item list control, the terminal responds to the trigger operation on the item list control, opens the item list in the user interface, displays one or more items in the item list, or jumps from the user interface to the item list interface, and displays one or more items in the item list in the item list interface.

[0198] In another exemplary scenario, the terminal plays an associated video of an item in the video interface of the application. The associated video is a video recommended in the video stream or a video searched by the user. The associated video may be a live video or a recorded video. In order to facilitate the user to view the item, an item link is provided in the video interface. In response to the user's trigger operation on the item link, it is determined that a trigger operation on the item is detected, thereby generating a virtual image of the item. The embodiment of the present disclosure does not specifically limit the scenario in which the user performs a trigger operation on the item.

[0199] In some embodiments, in response to an item-based trigger operation, the terminal first obtains item information of the item, and then locally generates a virtual image of the item based on the obtained item information. Alternatively, the terminal sends a virtual image creation request to the server, and the virtual image creation request carries the item ID and the obtained item information. Then, the server generates a virtual image of the item based on the item information, and returns the generated virtual image to the terminal. The terminal obtains the virtual image returned by the server. In this way, the virtual image is created in real time based on the latest item information, which can ensure that the virtual image is highly consistent with the item information. However, the embodiments of the present disclosure do not specifically limit whether the virtual image is generated locally by the terminal or remotely by the server.

[0200] In other embodiments, the virtual image is not created in real time, but is pre-created by the server when the product is put on the shelf, and is stored in association with the item ID. In this case, the terminal can send a virtual image pull request to the server, and the virtual image pull request is used to pull the virtual image of the item. The virtual image pull request carries at least the item ID. The server receives the virtual image pull request from the terminal, parses the item ID from the virtual image pull request, queries the virtual image stored in association with the item ID based on the item ID, and returns the virtual image obtained by the query to the terminal, and the terminal obtains the virtual image returned by the server.

[0201] In other embodiments, in response to a user's trigger operation on the item, the terminal requests the server to return the item information and virtual image of the item at one time when sending an item pull request to the server. In this way, after the server has queried the item information according to the item ID, it can continue to query whether the virtual image of the item is still stored according to the item ID. If a virtual image bound to the item ID is queried, it means that a virtual image has been created and stored for the item. The item information and virtual image are directly returned at one time, saving the communication overhead between the terminal and the server. If no virtual image bound to the item ID is queried, it means that for some reason a virtual image has not been created for the item or the virtual image that has been created has not been cached. In this way, a virtual image is created for the item in real time, the virtual image is cached, and the item information and virtual image are returned to the terminal at one time, which can also save the communication overhead between the terminal and the server.

[0202] It should be noted that no matter whether the terminal creates the virtual image locally or the server creates the virtual image remotely, and no matter whether the virtual image is created in real time or in advance, the method of creating the virtual image is the same, which will be described in detail in the next embodiment and will not be repeated here.

[0203] In step 202, the terminal displays the virtual image and an interactive control, where the interactive control is used to trigger interaction with the virtual image.

[0204] Among them, the interactive control refers to an interactive control for the virtual image, that is, the interactive control refers to an interactive entrance for triggering human-computer interaction with the virtual image. The interactive control can be at least one of a voice interactive control or a text interactive control. If it is a voice interactive control, the user can input voice to achieve voice interaction. If it is a text interactive control, the user can input text to achieve text interaction. The human-computer interaction here refers to intelligent question and answer between the user and the virtual image.

[0205] In some embodiments, after the terminal generates a virtual image of the object in response to a trigger operation based on the object, it jumps from the current interface to the virtual image display interface, wherein the virtual image display interface refers to any interface for displaying a virtual image, and then displays the virtual image generated in step 201 and the interactive controls for the virtual image in the virtual image display interface. In this way, when the user wants to view the object, the user can perform a trigger operation on the object so that the terminal opens the virtual image display interface and displays the virtual image created for the object and the interactive controls for the virtual image. The user can then use the interactive controls to perform intelligent question and answer with the virtual image, providing an immersive question and answer experience between the user and the virtual image. Moreover, as an intelligent agent of the object, the virtual image can interact with the user, such as explaining the object to the user, answering the user's questions, and providing parameter information of the object, etc., which improves the user's efficiency in obtaining object information and improves the efficiency of human-computer interaction.

[0206] In some embodiments, when the terminal displays the virtual image and the interactive control, the virtual image and the interactive control are independent of each other. For example, the terminal displays a virtual image and an interactive control in the virtual image display interface. In this way, the user can intuitively and conveniently perform a trigger operation with the interactive control to start the human-computer interaction with the virtual image with one click, so that more information about the item represented by the virtual image can be obtained through human-computer interaction with the virtual image, such as obtaining information such as the price and performance of the item through human-computer interaction with the virtual image, thereby improving the efficiency of obtaining item information, enriching the item interaction method, and improving the efficiency of human-computer interaction.

[0207] It should also be noted that when the terminal displays a virtual image and explicit interactive controls, since the interactive controls include voice interactive controls and text interactive controls, the terminal can display only voice interactive controls, or only text interactive controls, or both voice interactive controls and text interactive controls at the same time. The embodiments of the present application do not specifically limit the number of interactive controls displayed.

[0208] In other embodiments, when displaying a virtual image and an interactive control, the terminal configures part or all of the virtual image as an interactive control, which is equivalent to the interactive control being not an explicit interactive control but an implicit interactive control. For example, the terminal provides a virtual image in the virtual image display interface, and the virtual image itself is configured as an interactive control. The user directly triggers the virtual image to start the human-computer interaction process with the virtual image, thereby simplifying the operation method of starting human-computer interaction and improving the success rate and interaction efficiency of the user's human-computer interaction. For another example, the terminal provides a virtual image in the virtual image display interface, and specific parts of the virtual image are configured as interactive controls. At this time, only when the user performs a trigger operation on the specific part can the human-computer interaction process with the virtual image be started. When the user performs a trigger operation on a non-specific part, no interactive feedback will be provided, and the human-computer interaction process with the virtual image cannot be started. In an example, taking the specific part as the head of the virtual image as an example, a virtual image is provided in the virtual image display interface. If the user performs a trigger operation on the head of the virtual image, the human-computer interaction process with the virtual image can be started (such as human-computer dialogue, intelligent question and answer, etc. with the virtual image). However, if the user performs a trigger operation on the torso of the virtual image, no interactive feedback will be provided, and the human-computer interaction process with the virtual image cannot be started. In this way, since the human-computer interaction can only be started by locating a specific part, it is possible to avoid the user from starting the human-computer interaction by accidental touch, thereby improving the accuracy of starting the human-computer interaction and optimizing the user experience.

[0209] In step 203, the terminal controls the virtual image to perform human-computer interaction in response to the triggering operation on the interactive control.

[0210] In some embodiments, the user can perform a trigger operation on the interactive control at any time, so that when the terminal detects the user's trigger operation on the interactive control, it responds to the trigger operation on the interactive control and controls the virtual image to interact with the user. The form of human-computer interaction includes but is not limited to: human-computer dialogue, intelligent question and answer, intelligent customer service, intelligent agent, voice assistant, etc., which is not specifically limited in the embodiments of this application. Optionally, the trigger operation includes but is not limited to: click operation, long press operation, double-click operation, drag operation, press operation, slide operation, voice command, gesture command, etc., and the embodiments of this disclosure do not specifically limit the type of trigger operation.

[0211] In some embodiments, since the interactive control displayed in step 202 can be at least one of a voice interaction control or a text interaction control, if a voice interaction control is displayed in step 202, the user can perform a trigger operation on the voice interaction control, so that the terminal controls the virtual image to perform voice interaction with the user; if a text interaction control is displayed in step 202, the user can perform a trigger operation on the text interaction control, so that the terminal controls the virtual image to perform text interaction with the user. The embodiments of the present application do not specifically limit the type of human-computer interaction.

[0212] In an exemplary scenario, the terminal displays a virtual image and displays an independent interactive control outside the virtual image, and the user performs a trigger operation on the interactive control to control the virtual image to perform human-computer interaction with the user; or, the terminal displays the virtual image, and the virtual image itself is configured as an implicit interactive control, and the user performs a trigger operation on the virtual image to control the virtual image to perform human-computer interaction with the user; or, the terminal displays the virtual image, and a specific part of the virtual image is configured as an implicit interactive control, and the user performs a trigger operation on a specific part of the virtual image to control the virtual image to perform human-computer interaction with the user. The specific part is pre-set by a technician, and the embodiments of the present application do not specifically limit this.

[0213] The method provided by the embodiment of the present disclosure generates a virtual image created for the object when a trigger operation based on the object is detected, and displays the virtual image and interactive controls for the virtual image. By performing a trigger operation with the interactive controls, the virtual image can be further controlled to achieve human-computer interaction, so that the virtual image can assume the function of an intelligent agent for the object in a visualized virtual interactive form, thereby enriching the object interaction mode, improving the efficiency of obtaining object information, and improving the efficiency of human-computer interaction.

[0214] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0215] The above embodiments are simple descriptions of the technical solutions provided by the embodiments of the present disclosure. The technical solutions provided by the embodiments of the present disclosure will be described in detail and clearly with reference to some examples. Figure 3 is a flow chart of a method for interacting with an object based on a virtual image according to an exemplary embodiment. Figure 3 As shown, the object interaction method based on the virtual image is executed by an electronic device, and is described by taking the electronic device as a terminal as an example. The embodiment includes the following steps.

[0216] In step 301, the terminal obtains item information of the item in response to a trigger operation based on the item.

[0217] Among them, item information refers to data or information related to the item. Optionally, item information may include at least one of the item's detailed information or preview images. Detailed information is used to introduce the details of the item, including but not limited to the item name, category label, description text, parameter information (such as specifications, dimensions, performance and other parameters), price information, discount information, etc., and the content of detailed information is not specifically limited here. Preview images are product images used to introduce items, including but not limited to three-view drawings, perspective drawings, design drawings, real shots, concept drawings, panoramic drawings, model drawings, cross-section drawings, structural drawings, schematic drawings, etc., and the types of preview images are not specifically limited here.

[0218] In some embodiments, the terminal may display an item list, wherein the item list includes at least one item. Then, after detecting that the user performs a trigger operation on any item in the item list, the terminal responds to the trigger operation based on any item in the item list, obtains the item information of the item, and then generates a virtual image of the item based on the item information of the item. In the above process, since one or more items are provided through the item list, the user can select an item of interest in the item list for interaction, which improves the efficiency of item screening and thus improves the efficiency of human-computer interaction.

[0219] It should be noted that the item list involved above is used to provide items that support human-computer interaction based on virtual images. The item list may contain one or more items. The embodiment of the present application does not specifically limit the number of items contained in the item list. In addition, the form of the item list includes but is not limited to: a list of items on the shelves for live broadcast, a shopping cart list, a list of recommended items (such as "Guess you like"), a search result list, a shopping mall homepage, etc. The form of the item list is not specifically limited here. In this way, users can quickly find out which items support human-computer interaction based on virtual images through the item list, thereby improving the user's information acquisition efficiency.

[0220] In some embodiments, when obtaining item information after detecting a trigger operation based on an item, the terminal sends an item pull request to the server in response to the trigger operation based on the item, the item pull request is used to pull the item information of the item, and the item pull request carries at least the item ID of the item. The server receives the item pull request from the terminal, parses the item ID from the item pull request, queries the item information stored in association with the item ID based on the item ID, returns the queried item information to the terminal, and the terminal obtains the item information returned by the server. Optionally, the item information may include at least one of the item details or preview image, and the disclosed embodiment does not specifically limit the content of the item information.

[0221] In some embodiments, after obtaining the item information of the item, the terminal can generate a virtual image of the item based on the item information. It should be noted that, on the one hand, the terminal can create a virtual image locally, or the server can remotely create the virtual image and return it to the terminal. On the other hand, the terminal can create a virtual image in real time when viewing the item, or directly pull the virtual image created in advance when the item is put on the shelf. The embodiments of the present disclosure do not specifically limit whether the virtual image is created locally, and do not specifically limit whether the virtual image is created in real time.

[0222] In the embodiment of the present disclosure, the following steps 302-303 will be taken as an example to illustrate a possible implementation method for the terminal to create a virtual image locally. The way the server remotely creates a virtual image is the same as the way the terminal locally creates a virtual image, which will not be repeated.

[0223] In step 302, the terminal inputs the object information into an image generation model, and generates a virtual image of the object through the image generation model, where the image generation model is used to generate a virtual image of the object.

[0224] In some embodiments, after the terminal obtains the item information of the triggered item based on step 301, it calls the image generation model to generate a virtual image for the item under the guidance of the item information, so that the virtual image has a certain degree of correlation with the item information, that is, the virtual image has an association relationship or a matching relationship with the item information.

[0225] In some embodiments, the server side trains the image generation model, and after the training is completed, sends the parameter set of the trained image generation model to the terminal, so that the terminal stores the parameter set of the image generation model locally. In this way, the terminal inputs the object information into the image generation model, uses the parameter set to generate the image, and obtains a virtual image. Alternatively, the terminal can also train the image generation model locally, and after the training is completed, store the parameter set of the trained image generation model locally, and then directly call the image generation model locally. The embodiment of the present disclosure does not specifically limit whether the image generation model is trained locally.

[0226] In other embodiments, the terminal side does not need to store the parameter set of the image generation model locally, but the server side stores the parameter set of the image generation model and provides remote image generation services for the terminal, so that the server can provide an image generation interface (Application Programming Interface, API) of the image generation model, and the terminal passes the object information obtained in step 301 to the image generation API of the image generation model, and the image generation API remotely calls the image generation model on the server side, so that the server side uses the parameter set of the image generation model to generate an image, obtains a virtual image, and returns the generated virtual image to the terminal through the image generation API. In this way, the terminal side does not need to spend storage resources to store the parameter set of the image generation model, saving the storage overhead of the terminal.

[0227] In some embodiments, the above-mentioned image generation model refers to any machine learning model with image generation function. Optionally, the image generation model can be a generative large model with AI painting function, such as a StableDiffusion model, a Waifu-Diffusion model, a NovelAI model, etc. The embodiment of the present disclosure does not specifically limit the type of image generation model.

[0228] In some embodiments, since the item information obtained in step 301 includes at least one of the item's detailed information or preview image, image generation models with different functions can be called to synthesize a virtual image according to the different contents contained in the item information, which is discussed below in a classified manner.

[0229] (1) Item information only contains item details

[0230] In some embodiments, the detail information is used to introduce the details of the item, including but not limited to the item name, category label, description text, parameter information (such as specifications, size, performance and other parameters), price information, discount information, etc., and the content of the detail information is not specifically limited here. Since the detail information is item information belonging to the text modality, in this case, an image generation model with a text-to-image function (referred to as a text-generated image model) can be called to input the detail information into the text-generated image model, which can include a text encoder and an image generator. Then, the encoding features of the detail information are extracted by the text encoder, and the extracted encoding features are input into the image generator. The image generator is used to synthesize a virtual image under the guidance of the encoding features to ensure that the virtual image image and the detail information have an associated relationship or a matching relationship. For example, if the name of the item described in the detail information is a red skirt, then the synthesized virtual image image can also allow the virtual image to wear a red skirt.

[0231] In the above process, through the text-image model, a virtual image can be directly synthesized using detailed information, which only requires item information in text mode, and does not require item information in image mode. For some items that lack preview images or have no entity and cannot provide preview images (such as services, knowledge products, etc.), the synthesis of virtual images can be completed efficiently, which improves the efficiency of image synthesis and ensures the image synthesis effect.

[0232] (2) Item information only includes a preview image of the item

[0233] In some embodiments, the preview image is used to introduce the product image of the item, including but not limited to three-view, perspective, design, real shot, concept, panorama, model, cross-section, structure, principle, etc., and the types of preview images are not specifically limited here. Since the preview image is item information belonging to the image modality, in this case, an image generation model (referred to as an image generation model) with an image-to-image conversion function, i.e., an image redrawing function, can be called, and the preview image is input into the image generation model, which can include an image encoder and an image generator, and then the image encoder is used to extract the coding features of the preview image, and then the extracted coding features are input into the image generator, and the image generator is used to synthesize a virtual image under the guidance of the coding features, so as to ensure that the virtual image image and the preview image have an associated relationship or a matching relationship, for example, a real shot of a blue and white water cup is provided in the preview image, then an anthropomorphic virtual image is created for the water cup in the synthesized virtual image image, and the virtual image is made to wear a blue and white suit.

[0234] In the above process, through the image-to-image model, the preview image can be used to directly synthesize a virtual image. The preview image will be completely deconstructed and then reconstructed to obtain the virtual image. This algorithm can produce a virtual image that is similar in appearance to the preview image but has different details and composition, and only requires item information in image mode, not text mode. For some items that lack detailed information or have a lot of missing detailed information (possibly omitted by the merchant when listing), the synthesis of virtual images can be completed efficiently, thereby improving the efficiency of image synthesis and ensuring the image synthesis effect.

[0235] (3) Item information includes item details and preview images

[0236] In some embodiments, the item information includes both the detail information in (1) and the preview image in (2), and thus has item information in two modes, text mode and image mode. Therefore, in this case, a dual-modal image generation model can be called to input the detail information and the preview image into the dual-modal image generation model. The dual-modal image generation model can include a text encoder, an image encoder and an image generator. Then, the text encoder is used to extract the encoding features of the detail information, and the image encoder is used to extract the encoding features of the preview image. The two extracted encoding features are then input into the image generator together, or the two extracted encoding features are fused and input into the image generator. The image generator is used to synthesize a virtual image to ensure that the virtual image and the detail information and the preview image are all associated or matched.

[0237] In the above process, the dual-modal image generation model can make full use of the two modal item information of detail information and preview image to generate a virtual image that is more compatible with the item itself. This is equivalent to redrawing the image based on the preview image, and additionally introducing detail information to provide text modality constraints. For example, keywords are extracted from the detail information as prompts for AI painting, or users are asked to input image control information as prompts for AI painting, thereby guiding the dual-modal image generation model to efficiently complete the synthesis of virtual image images, thereby improving the efficiency of image synthesis and ensuring the image synthesis effect.

[0238] In step 303, the terminal generates a virtual image of the object based on the virtual image image, wherein the virtual image is used to represent the object and is an interactive object for realizing human-computer interaction.

[0239] In some embodiments, after the terminal obtains the virtual image in step 302, it can use the virtual image to generate the virtual image of the object. Since the virtual image of the object can be a two-dimensional image or a three-dimensional model, the following will discuss the virtual image acquisition methods in two different situations.

[0240] Case 1: The avatar is a two-dimensional image

[0241] When the virtual image is a two-dimensional image, the virtual image image obtained in step 302 can be directly determined as the virtual image. In this way, it is equivalent to directly generating a virtual image of the object in step 302 without spending computing resources to synthesize a three-dimensional model, thereby simplifying the virtual image generation process, improving the virtual image generation efficiency, and saving computing resources of the terminal.

[0242] Case 2: The avatar is a 3D model

[0243] In the case where the virtual image is a three-dimensional model, three-dimensional modeling can be performed based on the virtual image image obtained in step 302 to obtain a three-dimensional model of the virtual image, and the three-dimensional model is determined as the virtual image. In this way, 3D modeling can be further performed on the basis of the two-dimensional virtual image image to obtain a 3D model as the virtual image. Since the 3D model has a better simulation effect and higher realism than the 2D image, the generated virtual image has a higher degree of realism, which helps to provide an immersive human-computer interaction experience and improve the efficiency of human-computer interaction based on the virtual image.

[0244] In some embodiments, when three-dimensional modeling is performed for a virtual image, an initial model matching the image type can be selected from multiple candidate models according to the image type of the virtual image. For example, a virtual human candidate model, a robot candidate model, a virtual cat candidate model, a virtual dog candidate model, etc. are provided. The image types matched by the above candidate models are anthropomorphic images, simulated robot images, simulated cat images, simulated dog images, etc., respectively. The embodiments of the present disclosure do not specifically limit this. Among them, the image type can be extracted from the detailed information by semantic analysis or intent recognition, or it can be extracted from the image control information input by the user by semantic analysis or intent recognition, or it can be an image type automatically matched according to the category label of the item, or it can be a default selected image type. The embodiments of the present disclosure do not specifically limit the source of the image type. Then, after obtaining the initial model, the virtual image image is used to extract the posture presented by the virtual image, and the posture of the initial model is adjusted according to the extracted posture, so that the posture of the initial model after adjustment is matched with the posture of the virtual image in the virtual image image, and then the virtual image image is texture mapped to map the appearance of the virtual image presented in the virtual image image to the initial model after the posture is adjusted. The model obtained after the texture mapping is completed is called the 3D model of the virtual image, so that a 3D model with an image type, posture and appearance that are highly adapted to the virtual image image can be obtained, thereby improving the realism of the virtual image, helping to provide an immersive human-computer interaction experience, and improving the efficiency of human-computer interaction based on the virtual image. It should be noted that the above step of selecting the initial model according to the image type is an optional step, that is, only one initial model can be configured, and all virtual images reuse the same initial model, without first determining the image type, and then selecting the initial model from the candidate models according to the image type. The embodiment of the present disclosure does not specifically limit this.

[0245] In the above steps 302 to 303, a possible implementation method for generating a virtual image of the object based on the object information of the object is provided, that is, using an image generation model to generate a virtual image based on the object information, and then obtaining a virtual image for final use based on the virtual image. In this way, in the process of generating the virtual image, the object information can be fully utilized as a guiding signal, so that the virtual image can have a high degree of adaptability with the object information, thereby better simulating the virtual interaction form of the object itself, so that the virtual image itself can reflect a part of the object information, and it is helpful to provide an immersive human-computer interaction experience and improve the human-computer interaction efficiency based on the virtual image; in addition, this implementation method does not limit the modality of the object information itself. The object information can be a text modality, an image modality, or a dual modality of a text modality and an image modality, and has wide applicability and extensibility.

[0246] In some other embodiments, step 303 may not be executed, and the virtual image obtained in step 302 may be directly used as the virtual image itself. This is equivalent to selecting a virtual image in the form of a two-dimensional image by default, which simplifies the virtual image generation process and improves the virtual image generation effect. Alternatively, multiple virtual image images under different perspectives may be generated in step 302. In this way, multiple virtual image images under different perspectives can be used to quickly implement 3D modeling, which can improve the 3D modeling efficiency for the virtual image. The embodiments of the present disclosure do not specifically limit the number of virtual image images used in 3D modeling.

[0247] In other embodiments, when the item information contains detailed information, the virtual image can be generated based on the detailed information. Optionally, as described in step 302, the detailed information is input into the text image model, and the virtual image image of the item is generated by the text image model. Then, the virtual image of the item is obtained based on the virtual image image, and the acquisition method is the same as that described in step 303, which will not be repeated. In this way, a virtual image image is directly synthesized using the detailed information, and only the item information in the text mode is required, and the item information in the image mode is not required. For some items that lack preview images or do not have entities and cannot provide preview images (such as services, knowledge products, etc.), the synthesis of virtual image images can be completed efficiently, thereby improving the efficiency of image synthesis and ensuring the image synthesis effect.

[0248] In other embodiments, when the item information includes a preview image, the virtual image can be generated based on the preview image. Optionally, as described in step 302, the preview image is input into a graph-generated image model, and a virtual image image of the item is generated by the graph-generated image model. Then, the virtual image of the item is obtained based on the virtual image image, and the acquisition method is the same as that described in step 303, which will not be repeated. In this way, a virtual image image is directly synthesized using the preview image, and the preview image will be completely deconstructed and then reconstructed to obtain the virtual image image. This algorithm can generate a virtual image image that is similar in appearance to the preview image but has different details and composition, and only requires item information in the image mode, and does not require item information in the text mode. For some items that lack detailed information or have a lot of missing detailed information (it is possible that the merchant missed it when listing), the synthesis of the virtual image image can be completed efficiently, which improves the efficiency of image synthesis and ensures the image synthesis effect.

[0249] In other embodiments, when the item information includes detail information and a preview image, the virtual image can be generated based on the detail information and the preview image. Optionally, as described in step 302, the detail information and the preview image are input into a dual-modal image generation model, and a virtual image image of the item is generated by the dual-modal image generation model. Then, the virtual image of the item is obtained according to the virtual image image, and the acquisition method is the same as that described in step 303, which will not be repeated. In this way, the item information of the two modes of detail information and preview image can be fully utilized to generate a virtual image image with a higher degree of adaptability to the item itself, which is equivalent to redrawing the image based on the preview image, and additionally introducing detail information to provide text modality constraints, such as extracting keywords from the detail information as prompts for AI painting, or allowing users to input image control information as prompts for AI painting, guiding the dual-modal image generation model to efficiently complete the synthesis of the virtual image image, improving the image synthesis efficiency, and ensuring the image synthesis effect.

[0250] It should also be noted that when the item information contains detailed information and a preview image, it is not necessary to use both the detailed information and the preview image to generate a virtual image. For example, even if the item information contains detailed information and a preview image, if the text quality of the item information is low or the amount of information is small, or the image quality of the preview image is low, then only single-modal item information can still be used to generate a virtual image. The embodiments of the present disclosure do not specifically limit this.

[0251] In step 304, the terminal displays the virtual image and an interactive control in the virtual image display interface, where the interactive control is used to trigger interaction with the virtual image.

[0252] Among them, the virtual image display interface refers to any interface used to display a virtual image. The virtual image display interface can be the current interface for displaying the item list, or it can be another interface jumped from the current interface. The embodiment of the present application does not specifically limit whether the item list and the virtual image are displayed on the same interface.

[0253] In some embodiments, if the item list and the virtual image are displayed in the same interface, the terminal displays the item list in the current interface, and after detecting that the user performs a trigger operation on any item in the item list, generates a virtual image of the item, and displays the virtual image and interactive controls in the current interface, that is, the current interface itself is the virtual image display interface, and there is no need to perform an interface jump operation. Optionally, in order to simplify the interface layout, the item list can be automatically closed when the virtual image and interactive controls are displayed in the current interface. For example, in a live shopping scenario, the current interface and the virtual image display interface both refer to the live broadcast room interface. Taking the item list as a shopping cart list as an example, a shopping cart control is provided in the live broadcast room interface. The user clicks the shopping cart control to open the shopping cart list. The user performs a trigger operation on any item in the shopping cart list, thereby generating a virtual image of the item, and folding the shopping cart list in the live broadcast room interface (that is, the shopping cart list is folded into the shopping cart control), and then the virtual image and interactive controls are displayed in the live broadcast room interface. At this time, there is no need to jump through the interface, and only the virtual image and interactive controls need to be loaded in the current interface, which can save the computing resources of the terminal.

[0254] In other embodiments, if the item list and the virtual image are displayed in different interfaces, the terminal displays the item list in the current interface, and generates a virtual image of the item after detecting that the user performs a trigger operation on any item in the item list. Then, the terminal jumps from the current interface to the virtual image display interface, and displays the virtual image and the interactive control in the virtual image display interface after the jump. That is, the current interface and the virtual image display interface are different interfaces. This can avoid jamming caused by carrying too much information in the same interface, and can focus on the triggered item in the virtual image display interface, and provide a human-computer interaction method for the item, thereby improving the user's efficiency in obtaining information on specific items. For example, in the mall recommendation scenario, the current interface refers to the mall homepage, and the list of items provided in the mall homepage refers to "Guess you like", and multiple items with personalized recommendations are displayed. The user performs a trigger operation on any item recommended in "Guess you like", generates a virtual image of the item, and jumps from the mall homepage to the virtual image display interface (such as the item details page), and then displays the virtual image and interactive controls in the virtual image display interface. At this time, the virtual image display interface can focus on the triggered item and provide a human-computer interaction method for the item, thereby improving the user's efficiency in obtaining information on specific items.

[0255] In this step 304, when the user wants to view the item, the user can perform a trigger operation on the item to cause the terminal to open the virtual image display interface and display the virtual image created for the item and the interactive controls for the virtual image. The user can then use the interactive controls to implement intelligent question and answer with the virtual image, providing an immersive question and answer experience between the user and the virtual image. Moreover, the virtual image, as an intelligent agent of the item, can interact with the user. For example, the virtual image can explain the item to the user, answer the user's questions, provide parameter information of the item, etc., thereby improving the user's efficiency in obtaining item information and improving the efficiency of human-computer interaction.

[0256] In some embodiments, when the terminal displays the virtual image and the interactive control, the virtual image and the interactive control are independent of each other. For example, the terminal displays a virtual image and an interactive control in the virtual image display interface. In this way, the user can intuitively and conveniently perform a trigger operation with the interactive control to start the human-computer interaction with the virtual image with one click, so that more information about the item represented by the virtual image can be obtained through human-computer interaction with the virtual image, such as obtaining information such as the price and performance of the item through human-computer interaction with the virtual image, thereby improving the efficiency of obtaining item information, enriching the item interaction method, and improving the efficiency of human-computer interaction.

[0257] It should also be noted that when the terminal displays a virtual image and explicit interactive controls, since the interactive controls include voice interactive controls and text interactive controls, the terminal can display only voice interactive controls, or only text interactive controls, or both voice interactive controls and text interactive controls at the same time. The embodiments of the present application do not specifically limit the number of interactive controls displayed.

[0258] In other embodiments, when displaying a virtual image and an interactive control, the terminal configures part or all of the virtual image as an interactive control, which is equivalent to the interactive control being not an explicit interactive control but an implicit interactive control. For example, the terminal provides a virtual image in the virtual image display interface, and the virtual image itself is configured as an interactive control. The user directly triggers the virtual image to start the human-computer interaction process with the virtual image, thereby simplifying the operation method of starting human-computer interaction and improving the success rate and interaction efficiency of the user's human-computer interaction. For another example, the terminal provides a virtual image in the virtual image display interface, and specific parts of the virtual image are configured as interactive controls. At this time, only when the user performs a trigger operation on the specific part can the human-computer interaction process with the virtual image be started. When the user performs a trigger operation on a non-specific part, no interactive feedback will be provided, and the human-computer interaction process with the virtual image cannot be started. In an example, taking the specific part as the head of the virtual image as an example, a virtual image is provided in the virtual image display interface. If the user performs a trigger operation on the head of the virtual image, the human-computer interaction process with the virtual image can be started (such as human-computer dialogue, intelligent question and answer, etc. with the virtual image). However, if the user performs a trigger operation on the torso of the virtual image, no interactive feedback will be provided, and the human-computer interaction process with the virtual image cannot be started. In this way, since the human-computer interaction can only be started by locating a specific part, it is possible to avoid the user from starting the human-computer interaction by accidental touch, thereby improving the accuracy of starting the human-computer interaction and optimizing the user experience.

[0259] Next, combine Figure 4 Let's take an example to illustrate the virtual image display interface. Figure 4 is a schematic diagram of a virtual image display interface provided by an embodiment of the present disclosure, such as Figure 4 As shown, taking a laptop as an example, in the avatar display interface 400, a avatar 401 and an interactive control 402 are displayed, and the user can trigger the interactive control 402 to achieve immersive human-computer interaction with the avatar 401. The avatar 401 generated for the laptop is in the form of a robot, and has a certain correlation with the category label of the laptop itself, and can simulate the laptop to communicate and interact with the user.

[0260] In some embodiments, in addition to the virtual image and interactive controls involved in step 304, at least one of the following information may also be displayed in the virtual image display interface:

[0261] 1. Operation prompt information

[0262] In some embodiments, the terminal displays operation prompt information for the interactive control in the virtual image display interface, and the operation prompt information is used to prompt the interaction method with the interactive control. Optionally, the operation prompt information can be text, animation, image, dynamic effect, magic expression, sticker, pop-up window, voice, floating window, bubble, etc. The embodiment of the present disclosure does not specifically limit the type of operation prompt information.

[0263] By displaying operation prompt information in the virtual image display interface, users can quickly and easily learn how to interact with interactive controls, so that users can intuitively understand the interaction methods of interactive controls without blindly groping in trial and error, thereby reducing the user's human-computer interaction cost and improving the user's information acquisition efficiency and human-computer interaction efficiency.

[0264] In some embodiments, the terminal can automatically display the operation prompt information while entering the virtual image display interface, and automatically cancel the display of the operation prompt information when the display duration of the operation prompt information reaches a first duration, wherein the first duration is a pre-set value greater than 0. In this way, it is equivalent to prompting the user how to operate the interactive control when the virtual image display interface is just opened, and the operation prompt information automatically disappears after being displayed for a certain period of time, thereby avoiding the operation prompt information from blocking the virtual image display interface for too long, and optimizing the display logic of the operation prompt information. Optionally, the operation prompt information may not be automatically canceled when the first duration is reached, but the operation prompt information will be removed from the virtual image display interface only after the user manually closes the operation prompt information. This can avoid the user from missing or forgetting to read the operation prompt information, and improve the user's information acquisition efficiency.

[0265] In other embodiments, in the virtual image display interface, the operation prompt information can also be hidden by default. When the user hovers on the interactive control for more than a second time period, or when the user issues an inquiry command (such as a voice command or a gesture command), the operation prompt information is automatically popped up, wherein the second time period is a pre-set value greater than 0. In this way, the operation prompt information can be automatically popped up when it is detected that the user has a high interactive intention, otherwise it will be folded or hidden by default, which can optimize the display logic of the operation prompt information and avoid blocking other information in the virtual image display interface. Optionally, a prompt control (such as a help button, a help button, an inquiry button, etc.) can also be displayed in the virtual image display interface. After the user performs a trigger operation on the prompt control, the operation prompt information will pop up in response to the trigger operation of the prompt control. In this way, although the operation prompt information is folded in the virtual image display interface, it still retains or provides a visual entrance to open the operation prompt information, which facilitates the user to view the operation prompt information at any time, prompts the user to control the content displayed in the virtual image display interface, and improves the efficiency of human-computer interaction.

[0266] It should be noted that displaying the operation prompt information is an optional step, and the operation prompt information may not be displayed. The embodiment of the present disclosure does not specifically limit whether to display the operation prompt information in the virtual image display interface.

[0267] 2. Introduction Text

[0268] In some embodiments, the terminal displays an introductory text for the item in the virtual image display interface, so that when the user enters the virtual image display interface, the user can initially understand the basic information of the item through the introductory text, and then if there are other questions or needs, the user can interact with the virtual image through interactive controls. The virtual image will act as an intelligent agent for the item and provide users with comprehensive pre-sales consultation and after-sales support.

[0269] Next, a method for generating and displaying an introduction text will be introduced by taking steps A1 to A2 as an example.

[0270] In step A1, the terminal generates an introduction text associated with the virtual image based on the item information, and the introduction text is used to simulate the virtual image introducing the item.

[0271] In some embodiments, the terminal generates an introduction text associated with the virtual image based on the item information obtained in step 301, so that the introduction text and the virtual image can be organically combined to create an interactive experience of the virtual image publishing the introduction text, allowing users to interact immersively with the virtual image, enriching the human-computer interaction method and improving the user's information acquisition efficiency.

[0272] In some embodiments, the terminal inputs the item information into a text generation model, and generates the introduction text through the text generation model, wherein the text generation model is used to provide interactive text for question-and-answer dialogues related to items. In this way, since the text generation model is a generative NLP model, it can automatically generate an introduction text based on the input item information, and the introduction text will introduce the item in the tone of the virtual image, so that the introduction text and the virtual image are more closely related, and the introduction text and the virtual image are organically combined, which optimizes the user's human-computer interaction experience, is conducive to immersive human-computer interaction, and improves the efficiency of human-computer interaction.

[0273] In some embodiments, the server side trains the text generation model, and after the training is completed, sends the parameter set of the trained text generation model to the terminal, so that the terminal stores the parameter set of the text generation model locally. In this way, the terminal inputs the item information into the text generation model, uses the parameter set to generate text, and obtains an introduction text. Alternatively, the terminal can also train the text generation model locally, and after the training is completed, store the parameter set of the trained text generation model locally, and then directly call the text generation model locally. The embodiments of the present disclosure do not specifically limit whether the text generation model is trained locally.

[0274] In other embodiments, since the text generation model may be a large model with many parameters, the terminal side does not need to store the parameter set of the text generation model locally, but the parameter set of the text generation model is stored by the server side, and a remote human-computer question-and-answer service is provided to the terminal, so that the server can provide a human-computer question-and-answer API with a text generation model, and the terminal passes the item information obtained in step 301 into the human-computer question-and-answer API of the text generation model, and the human-computer question-and-answer API remotely calls the text generation model on the server side, so that the server side uses the parameter set of the text generation model to generate text, obtains a section of introduction text, and returns the generated introduction text to the terminal through the human-computer question-and-answer API. In this way, the terminal side does not need to spend storage resources to store the parameter set of the text generation model, saving the storage overhead of the terminal.

[0275] In some embodiments, the above-mentioned text generation model refers to any machine learning model with text generation function. Optionally, the text generation model can be a generative large model with NLP function, for example, Qwen-7b model, ELMo model, BERT model, GPT model, etc. The embodiment of the present disclosure does not specifically limit the type of text generation model.

[0276] In other embodiments, the introductory text can also be extracted directly from the item information. For example, in the item details, the item name, category label, description text, parameter information, price information and discount information are extracted in sequence, and the above extracted information is spliced ​​using preset rules to generate an introductory text for the item. This eliminates the need to use a large generative model for text synthesis, which can save terminal computing resources.

[0277] In step A2, the terminal displays the introduction text based on the virtual image in the virtual image display interface.

[0278] In some embodiments, after the introduction text is generated locally or remotely by the server and returned in step A1, the terminal can display the introduction text in the virtual image display interface. Optionally, when displaying the introduction text, the introduction text is displayed based on the virtual image, presenting a visual effect of the virtual image publishing the introduction text, which can optimize the user's interactive experience with the virtual image.

[0279] In some embodiments, the introduction text is displayed within a preset range of the virtual image, and the preset range refers to the upper layer of the virtual image (referring to the part that is overlapped and displayed on the upper layer of the virtual image, covering or transparently covering the part of the virtual image), top, bottom, left, and right of the virtual image. The embodiments of the present disclosure do not specifically limit the preset range. For example, the introduction text is displayed in the form of a floating layer on the virtual image, and the floating layer can be set to have transparency (or opaque), or a dialog box is displayed around the virtual image, and the introduction text is displayed in the dialog box, presenting a visual effect of the virtual image and the user having a conversation. The subsequent dialog box can also support human-computer interaction. The user can perform human-computer interaction based on the dialog box or based on the interactive control, providing a variety of human-computer interaction methods with the virtual image. The embodiments of the present disclosure do not specifically limit the display method of the introduction text.

[0280] In some embodiments, the introduction text may not be displayed based on the virtual image, but the introduction text may be displayed at a fixed position of the virtual image display interface, for example, the introduction text may be displayed at the center, top, bottom, left or right side of the virtual image display interface. In this way, no matter where the virtual image is drawn on the virtual image display interface, it will not affect the display position of the introduction text. The embodiments of the present disclosure do not specifically limit whether the display position of the introduction text depends on the display position of the virtual image.

[0281] In some embodiments, when displaying the introductory text, the appearance animation of the introductory text can be played, which can make the display of the introductory text more rounded and natural. Alternatively, the appearance animation of the introductory text can be omitted, which can save the processing resources of the terminal. The embodiments of the present disclosure do not specifically limit this.

[0282] In some embodiments, the introductory text may automatically appear when the virtual image display interface is opened, and the introductory text may be automatically canceled when the display time of the introductory text reaches a third time, wherein the third time is a pre-set value greater than 0. In this way, it is equivalent to popping up the introductory text to briefly introduce the basic information of the item when the virtual image display interface is just opened, and automatically disappearing after the introductory text is displayed for a certain time, thereby avoiding the introductory text from blocking the virtual image display interface for too long, and optimizing the display logic of the introductory text. Optionally, the introductory text may not be automatically canceled when the third time is reached, but the introductory text may be removed from the virtual image display interface only after the user manually closes the introductory text. This can avoid the user from missing or forgetting to read the introductory text, and improve the user's information acquisition efficiency.

[0283] In other embodiments, in the virtual image display interface, the introduction text may be hidden by default. When the user hovers over the virtual image for more than a fourth time period, or when the user issues an inquiry command (such as a voice command, a text command, or a gesture command), the introduction text is automatically popped up, wherein the fourth time period is a pre-set value greater than 0. In this way, the introduction text will be automatically popped up when it is detected that the user has a high inquiry intention, otherwise it will be folded or hidden by default, which can optimize the display logic of the introduction text and avoid blocking other information in the virtual image display interface. Optionally, a dialogue control (such as unread dialogues, red dot buttons, etc.) can also be displayed based on the virtual image in the virtual image display interface. After the user performs a trigger operation on the dialogue control, the introduction text will pop up in response to the trigger operation on the dialogue control. In this way, although the introduction text is folded in the virtual image display interface, it still retains or provides a visual entrance to open the introduction text, which facilitates the user to view the introduction text at any time, prompts the user to control the content displayed in the virtual image display interface, and improves the efficiency of human-computer interaction.

[0284] Still Figure 4 Let's continue with the explanation, for example Figure 4 As shown, in the virtual image display interface 400, an introduction text 403 is displayed in a floating layer on the upper layer of the body of the virtual image 401, "Hello, I am a mobile notebook produced by XX company in 2023, equipped with a YY chip developed by XX company, and my name is AAA. I have powerful performance,... (performance and other parameter descriptions omitted)", and the introduction text 403 can be displayed in the form of a dialog box, presenting an immersive interactive experience of human-computer dialogue between the virtual image 401 and the user. The floating layer can be opaque or translucent, and the embodiments of the present disclosure do not specifically limit this.

[0285] In steps A1 to A2, a possible implementation method is provided for generating an introduction text that is highly consistent with the virtual image and displaying the introduction text based on the virtual image. Since the introduction text itself matches the virtual image, the introduction text and the virtual image are organically combined. Furthermore, since the introduction text is displayed around the virtual image, a visual effect of the virtual image publishing the introduction text is presented, allowing users to interact with the virtual image in an immersive manner, enriching the human-computer interaction method, improving the user's information acquisition efficiency, and optimizing the user's interaction experience with the virtual image.

[0286] It should be noted that the process of displaying the introduction text in steps A1 to A2 is an optional step, that is, displaying the introduction text in the virtual image display interface is not a necessary step, and the introduction text may not pop up in the virtual image display interface, but the user can interact with the virtual image through interactive controls, so that the introduction text can also be viewed. The embodiment of the present disclosure does not specifically limit whether the introduction text is displayed automatically or after the user asks a question. For example, if the introduction text is not automatically displayed in the virtual image display interface, the user can ask the virtual image through the interactive controls "Can you briefly introduce yourself?" (such as Figure 4 The question sentence shown above the introduction text 403 in the figure) causes the virtual image to generate an introduction text to answer this question. The specific interaction method based on the interactive control will be described in detail in the next embodiment and will not be expanded here.

[0287] In some embodiments, when the introduction text is displayed in the virtual image display interface, in order to further improve the human-computer interaction efficiency, the following steps A3 to A4 can also be performed to configure the corresponding introduction voice for the introduction text, so that a sound and text introduction of the item can be achieved.

[0288] In step A3, the terminal generates an introduction voice matching the introduction text based on the introduction text.

[0289] In some embodiments, based on the introduction text generated in step A1, TTS technology can be used to synthesize an introductory voice that reads the introduction text. Based on a given introduction text, TTS technology can use machine speech synthesis to dub the introduction text and generate an introductory voice that reads the introduction text. The sound source of the introduction text can be a machine-synthesized sound or a human voice with full authorization from the sound source provider. The embodiments of the present disclosure do not specifically limit this.

[0290] In some embodiments, the terminal inputs the introduction text into a speech generation model, parses the introduction text through the speech generation model, segments the introduction text into a character sequence, and then converts the character sequence into a phoneme sequence, wherein the phoneme is the basic unit of speech and the smallest sound segment with speech meaning in pronunciation. Then, the phoneme sequence is converted into acoustic features (such as pitch, volume, intonation, etc.), and then the acoustic features are converted into the introduction speech in the form of a digital signal. The speech generation model can be an end-to-end TTS model, or a combined model consisting of a language model and an acoustic model, wherein the language model is used to parse the introduction text and the acoustic model is used to synthesize the introduction speech. The disclosed embodiment does not specifically limit the structure of the speech generation model.

[0291] In some embodiments, the server side trains the speech generation model, and after the training is completed, sends the parameter set of the trained speech generation model to the terminal, so that the terminal stores the parameter set of the speech generation model locally. In this way, the terminal inputs the introduction text into the speech generation model, uses the parameter set to generate speech, and obtains an introduction speech. Alternatively, the terminal can also train the speech generation model locally, and after the training is completed, store the parameter set of the trained speech generation model locally, and then directly call the speech generation model locally. The embodiment of the present disclosure does not specifically limit whether the speech generation model is trained locally.

[0292] In other embodiments, the terminal side does not need to store the parameter set of the speech generation model locally, but the server side stores the parameter set of the speech generation model and provides a remote speech generation service for the terminal, so that the server can provide a speech generation API of the speech generation model, and the terminal passes the introduction text generated in step A1 into the speech generation API of the speech generation model, and the speech generation API remotely calls the speech generation model on the server side, so that the server side uses the parameter set of the speech generation model to generate speech, obtains an introduction speech, and returns the generated introduction speech to the terminal through the speech generation API. In this way, the terminal side does not need to spend storage resources to store the parameter set of the speech generation model, saving the storage overhead of the terminal.

[0293] In some other embodiments, the above-mentioned voice generation API and human-computer question and answer API can be integrated into the same intelligent agent API, that is, the human-computer question and answer service will not only provide an introduction text, but also provide an adapted introduction voice. In this way, it is only necessary to enter the item information in step A1 and call the intelligent agent API once. The server side will use the text generation model to generate the introduction text, and then use the voice generation model to generate the introduction voice, and return the introduction text and the introduction voice at one time, thereby saving the communication rounds between the terminal and the server and saving the communication overhead of the terminal. The embodiments of the present disclosure do not specifically limit this.

[0294] In step A4, the terminal plays the introduction voice.

[0295] In some embodiments, the terminal plays the introductory voice generated in step A3. For example, the introductory voice is input into the audio codec, and the introductory voice in the form of a digital signal can be output as a voice signal that the user can understand. Optionally, when playing the introductory voice, a broadcast animation can also be generated for the virtual image, and the broadcast animation can be played at the same time, so that the virtual image can broadcast the introductory voice and match the introduction text as a subtitle. The visual effect achieves a sound and text introduction or explanation of the item, which is conducive to the immersive interaction between the user and the virtual image. The synthesis method of the broadcast animation is similar to the synthesis method of the broadcast animation in the next embodiment. It will not be expanded here. Please refer to the next embodiment for details.

[0296] In some embodiments, the timbre of the introductory voice, that is, the timbre of the virtual image, can also be freely configured by the user to achieve personalized timbre configuration of the virtual image. For example, when a merchant puts an item on the shelf, he can not only select a virtual image for the item, but also specify the timbre of the virtual image. Alternatively, the merchant does not need to specify the timbre, but binds a timbre that matches the appearance to the virtual image when creating the virtual image, or binds a timbre that matches the image type to the virtual image, or directly randomly selects a timbre for the virtual image from a variety of timbres, or, when the virtual image is broadcast for the first time, a timbre selection pop-up window pops up, and the user can independently select the timbre they want to hear. The disclosed embodiments do not specifically limit the timbre selection method.

[0297] In steps A3 to A4, based on a given introduction text, TTS technology is used to dub the introduction text, generate an introduction voice, and play the introduction voice while displaying the introduction text, so as to achieve an audio-visual introduction or explanation of the item, which is conducive to the immersive interaction between the user and the virtual image, and further improves the efficiency of human-computer interaction. It should be noted that steps A3 to A4 are optional steps. For example, when the user does not have the authorized speaker permission or the user does not want to listen to the voice, only the introduction text can be displayed without playing the introduction voice. The embodiment of the present disclosure does not specifically limit whether to play the introduction voice.

[0298] 3. Preview Controls

[0299] In some embodiments, the terminal displays a preview control for the item in the virtual image display interface, so that the user can directly preview the item through the preview control when entering the virtual image display interface, so as to understand the actual product form of the item and fully understand the item information, thereby improving the efficiency of obtaining information about the item.

[0300] Below, taking steps B1 to B2 as an example, a possible implementation method of performing real scene preview based on the preview image is introduced.

[0301] In step B1, the terminal collects video frames through a camera in response to a triggering operation on the preview control.

[0302] In some embodiments, when a user wants to preview an item, he can perform a trigger operation on the preview control provided in the virtual image display interface. In response to the trigger operation on the preview control, the terminal applies for camera permission, and after obtaining full authorization from the user, calls the camera to collect real-time video frames. Optionally, the trigger operation includes but is not limited to: click operation, double-click operation, long press operation, slide operation, drag operation, voice command, gesture command, etc. The embodiment of the present disclosure does not specifically limit the type of trigger operation.

[0303] In step B2, the terminal displays a real-life image of the object in the video frame.

[0304] In some embodiments, the terminal obtains a real-life image of an object and displays the real-life image in a video frame taken by a camera. This can provide users with a better object preview experience and present an interactive effect of opening the camera to view the real scene of the object, enriching the object preview method and improving human-computer interaction efficiency.

[0305] In some embodiments, the real scene image may be obtained from the item information in step 301. For example, the item information includes detailed information and preview images. There may be multiple preview images. When the preview image includes the real scene image, the real scene image is directly obtained from the item information and drawn on the video frame. Alternatively, the real scene image may also be synthesized based on the preview image, for example, a 3D real scene image is synthesized based on a 2D preview image. In this case, the terminal may synthesize the real scene image locally, or request the server to synthesize the real scene image remotely and then send it to the terminal. The disclosed embodiment does not specifically limit the method for obtaining the real scene image.

[0306] In some embodiments, the real scene image is an augmented reality AR image of the object. If the terminal supports AR technology, the terminal can directly obtain the AR image of the object and draw the AR image on the video frame. For example, if the preview image of the object information contains an AR image, then the AR image is obtained from the object information and drawn on the video frame. If the preview image of the object information does not contain an AR image, then the AR image is generated based on the preview image. The AR image can be generated locally in the terminal or remotely generated by the server and then sent to the terminal. The disclosed embodiment does not specifically limit this. By drawing the AR image directly on the video frame, the process of real scene preview can be simplified, AR preview of objects can be realized, the object preview experience can be optimized, and the efficiency of human-computer interaction can be improved.

[0307] Still Figure 4 Take this as an example to illustrate: Figure 4 As shown, a preview control 404 of the object is also displayed in the virtual image display interface 400, and a prompt "Show real object AR" is displayed on the preview control 404, so that the user can perform a trigger operation on the preview control 404. In response to the trigger operation on the preview control 404, the camera is called to capture real-time video frames, and an AR image of the object is drawn on the video frame.

[0308] In other embodiments, when the terminal supports AR technology, after the terminal obtains the AR image, it can also generate an item preview model of the item based on the AR image, and draw the item preview model on the video frame. The item preview model here refers to a 3D item model that creates a real-scene preview for the item. Compared with the AR image, the item preview model has a better preview effect. When the user rotates the camera or rotates the item preview model, the item preview model can be observed from different perspectives, providing a more efficient item preview method, further optimizing the item preview experience, and improving the efficiency of human-computer interaction.

[0309] The above only takes the case where the real scene image is an AR image as an example to illustrate the method of AR previewing objects, but the real scene image can also be a VR image, an MR image or an XR image. The embodiments of the present disclosure do not specifically limit the type of the real scene image.

[0310] In steps B1 to B2, in view of the situation where a preview control of the item is also displayed in the virtual image display interface, an implementation method of interacting with the preview control to view the real-scene image is provided. Since the real-scene image has a better display effect than the product photo, it can enable the user to better understand the product form of the item, thereby improving the user's information acquisition efficiency and improving the efficiency of human-computer interaction.

[0311] In other embodiments, in response to the user's triggering operation on the preview control, the real-scene image may not be displayed, but one or more non-real-scene preview images of the object may be popped up directly in the virtual image display interface. This does not require requesting camera permissions and can save terminal processing resources. The embodiments of the present disclosure do not specifically limit whether real-scene preview of objects is supported.

[0312] In step 305, the terminal controls the virtual image to perform human-computer interaction in response to the triggering operation on the interactive control.

[0313] In some embodiments, the user can perform a trigger operation on the interactive control at any time, so that when the terminal detects the user's trigger operation on the interactive control, it responds to the trigger operation on the interactive control and controls the virtual image to interact with the user. The form of human-computer interaction includes but is not limited to: human-computer dialogue, intelligent question and answer, intelligent customer service, intelligent agent, voice assistant, etc., which is not specifically limited in the embodiments of this application. Optionally, the trigger operation includes but is not limited to: click operation, long press operation, double-click operation, drag operation, press operation, slide operation, voice command, gesture command, etc., and the embodiments of this disclosure do not specifically limit the type of trigger operation.

[0314] In some embodiments, since the interactive control displayed in step 304 can be at least one of a voice interaction control or a text interaction control, if a voice interaction control is displayed in step 304, the user can perform a trigger operation on the voice interaction control, so that the terminal controls the virtual image to perform voice interaction with the user; if a text interaction control is displayed in step 304, the user can perform a trigger operation on the text interaction control, so that the terminal controls the virtual image to perform text interaction with the user. The embodiments of the present application do not specifically limit the type of human-computer interaction.

[0315] In an exemplary scenario, the terminal displays a virtual image and displays an independent interactive control outside the virtual image, and the user performs a trigger operation on the interactive control to control the virtual image to perform human-computer interaction with the user; or, the terminal displays the virtual image, and the virtual image itself is configured as an implicit interactive control, and the user performs a trigger operation on the virtual image to control the virtual image to perform human-computer interaction with the user; or, the terminal displays the virtual image, and a specific part of the virtual image is configured as an implicit interactive control, and the user performs a trigger operation on a specific part of the virtual image to control the virtual image to perform human-computer interaction with the user. The specific part is pre-set by a technician, and the embodiments of the present application do not specifically limit this.

[0316] The method provided by the embodiment of the present disclosure generates a virtual image created for the object when a trigger operation based on the object is detected, and displays the virtual image and interactive controls for the virtual image. By performing a trigger operation with the interactive controls, the virtual image can be further controlled to achieve human-computer interaction, so that the virtual image can assume the function of an intelligent agent for the object in a visualized virtual interactive form, thereby enriching the object interaction mode, improving the efficiency of obtaining object information, and improving the efficiency of human-computer interaction.

[0317] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0318] In the previous embodiment, how to open the virtual image display interface and the various controls that may be displayed in the virtual image display interface are described in detail. The user can perform trigger operations on the controls provided in the virtual image display interface to achieve human-computer interaction of various functions. Figures 5 to 7 , to give an overall example of the object scene display process.

[0319] Figure 5 is a principle flow chart of a method for interacting with an object based on a virtual image provided by an embodiment of the present disclosure, such as Figure 5 As shown, when a merchant puts an item on the shelf, the merchant uploads the item information of the item to be put on the shelf to the server. The item information includes detailed information and a preview image. Then, the server inputs the detailed information and the preview image into the image generation model, and combines at least one of the text-to-image or image-to-image technologies to create an associated virtual image for the item to be put on the shelf, in preparation for the subsequent call of the item scene display.

[0320] Figure 6 is a schematic diagram of a virtual image display interface provided by an embodiment of the present disclosure, such as Figure 6As shown, after the user performs a trigger operation on the item, the terminal responds to the trigger operation based on the item and provides a virtual image display interface 600. The virtual image display interface 600 provides a virtual image 601 created for the item, an interactive control 602 of the virtual image, and an operation prompt information 603 of the interactive control "Hello, you can click the button below to talk to me", so that the user can perform a trigger operation on the interactive control 602 under the guidance of the operation prompt information 603, thereby realizing a human-computer question and answer with the virtual image 601. The virtual image 601 can use a large language model in the background to provide an intelligent agent, provide users with comprehensive pre-sales consultation and after-sales support, and provide an intelligent service platform based on the virtual image as an external expression.

[0321] Figure 7 is a schematic diagram of another virtual image display interface provided by an embodiment of the present disclosure, such as Figure 7 As shown, after the user performs a trigger operation on the item, the terminal responds to the trigger operation on the item and provides a virtual image display interface 700. The virtual image display interface 700 provides a virtual image 701 created for the item, an interactive control 702 of the virtual image, an introduction text 703 for the item, and a preview control 704 for the item. In the introduction text 703, the item is introduced in the voice of the virtual image 701, for example, "Hello, I am XXX, a blue and white porcelain from the Ming and Qing Dynasties. I have a unique underglaze coloring process, which makes my appearance more exquisite. I am widely used to make tableware, decorations, etc., and I am deeply loved by people." In the preview control 704, the function of AR previewing the item can be provided, so the preview control 704 prompts "Show physical object AR". In this way, when browsing the virtual image display interface 700, the user can simply browse the introduction text 703 to have a preliminary understanding of the item. While displaying the introduction text 703, the matching introduction voice can also be played at the same time to achieve a sound and text explanation of the item. Furthermore, the user can perform a trigger operation on the interactive control 702 to achieve a human-computer question and answer with the virtual image 701; or, the user can also perform a trigger operation on the preview control 704 to open the camera to shoot a video frame, and draw an AR image of the object on the video frame to achieve a real-life preview of the object.

[0322] The above virtual image display interface adaptively generates virtual images for items according to the characteristics of the items, merchant preferences or user preferences, and dubs the virtual images. The virtual images and dubbing enhance the personalized expression of different items, and enhance the user's immersion and interactive experience. Furthermore, combined with text, voice and visual AR technology, a full range of item displays are provided, and users can also ask questions or interact with the personified virtual images of items through voice or text dialogues, and obtain personalized recommendations and information through the intelligent agent in the virtual image background, realizing a multimodal and personalized interactive experience, enriching the item interaction method, and improving the efficiency of information acquisition and human-computer interaction.

[0323] In the above embodiments, how to trigger the display of the virtual image display interface and what information or controls will be displayed in the virtual image display interface are described in detail, and in the following two embodiments, how the user can interact with the virtual image through the interactive controls will be described in detail. Since the interactive controls can be voice interactive controls or text interactive controls, and both voice interactive controls and text interactive controls can be provided in the virtual image display interface at the same time, two embodiments will be used to illustrate them respectively.

[0324] In the embodiments of the present disclosure, it will be described how a user can perform human-computer interaction with a virtual image through a voice interaction control. Figure 8 is a flow chart of a method for interacting with an object based on a virtual image according to an exemplary embodiment. Figure 8 As shown, the interactive control provided in the virtual image display interface is a voice interactive control as an example for description, and how to implement the voice interaction mode with the virtual image based on the voice interactive control is introduced. This embodiment is executed by an electronic device, and the electronic device is described as a terminal as an example.

[0325] In step 801, the terminal records a question voice in response to a triggering operation on a voice interaction control in a virtual image display interface.

[0326] In some embodiments, after the terminal opens the virtual image display interface through the previous embodiment, at least a virtual image and an interactive control with the virtual image are provided in the virtual image display interface. In the case where the interactive control is a voice interactive control, the user can perform a trigger operation on the voice interactive control, and the terminal responds to the user's trigger operation on the voice interactive control, applies for microphone permission, and after soliciting the user's full authorization, calls the microphone to collect the user's question voice. Optionally, the trigger operation includes but is not limited to: click operation, long press operation, double-click operation, drag operation, press operation, slide operation, voice command, gesture command, etc. The embodiment of the present disclosure does not specifically limit the type of trigger operation.

[0327] In an exemplary scenario, the user presses and holds the voice interaction control to start recording, and stops recording after releasing it. The terminal will call the microphone to record the question voice spoken by the user during the period when the user presses and holds the voice interaction control. Alternatively, the user starts recording after clicking the voice interaction control for the first time, and then clicks the voice interaction control again to end recording after starting recording. The terminal will call the microphone to record the question voice spoken by the user during the two clicks of the voice interaction control. The embodiments of the present disclosure do not specifically limit the operation method of the voice interaction control.

[0328] In step 802, the terminal generates an answer voice for the question voice based on the question voice.

[0329] In some embodiments, the terminal generates a voice answer for the question voice based on the question voice recorded in step 801, so that the machine and the user can realize voice question and answer, which can optimize the user's voice interaction experience when browsing items, and does not require the user to type questions, thereby improving the efficiency of human-computer interaction.

[0330] In some embodiments, an end-to-end voice question-answering model can be trained, the input signal of which is the question voice, and the output signal is the answer voice, so that an end-to-end voice question-answering service can be realized. Optionally, the voice question-answering model can be trained locally on the terminal, or it can be trained on the cloud by the server and then sent to the terminal. This embodiment of the present disclosure does not specifically limit this. Optionally, the terminal may not store the voice question-answering model locally, but instead remotely call the end-to-end voice question-answering model on the server side through the voice question-answering API. This embodiment of the present disclosure does not specifically limit this.

[0331] In other embodiments, there is no need to build an end-to-end voice question-answering model. Instead, the technology of search-enhanced text is used to improve the accuracy of intelligent voice question-answering. The following will use steps C1 to C4 to illustrate the method of synthesizing answering speech using the technology of search-enhanced text.

[0332] In step C1, the terminal performs speech recognition on the question voice to obtain a question text that matches the semantics of the question voice.

[0333] In some embodiments, the terminal performs speech recognition on the question voice recorded in step 801, and extracts a question text that semantically matches the question voice, that is, based on the STT technology, speech recognition is performed on the question voice to obtain a question text that is phoneme-aligned and semantically matches the question voice. The STT technology can use a machine to perform speech recognition based on a given segment of question voice to identify a matching question text for the question voice.

[0334] In some embodiments, the terminal inputs the question speech into an ASR model, parses the question speech through the ASR model, and extracts a feature sequence from the question speech. For example, the Mel-scale Frequency Cepstral Coefficients (MFCC) features of the question speech can be extracted, and then the feature sequence is decoded into a phoneme sequence, and then the phoneme sequence is used to translate the character sequence of the question text, so that the ASR model outputs the final question text. The ASR model can also be an end-to-end model, or the ASR model can also be a combined model consisting of an acoustic model and a speech model. The disclosed embodiments do not specifically limit the structure of the ASR model.

[0335] In some embodiments, the server side trains the ASR model, and after the training is completed, sends the parameter set of the trained ASR model to the terminal, so that the terminal stores the parameter set of the ASR model locally. In this way, the terminal inputs the question voice into the ASR model, uses the parameter set to perform ASR, and obtains a question text. Alternatively, the terminal can also train the ASR model locally, and after the training is completed, store the parameter set of the trained ASR model locally, and then directly call the ASR model locally. The embodiment of the present disclosure does not specifically limit whether the ASR model is trained locally.

[0336] In other embodiments, the terminal side does not need to store the parameter set of the ASR model locally, but the server side stores the parameter set of the ASR model and provides remote speech recognition services for the terminal. In this way, the server can provide a speech recognition API of the ASR model, and the terminal passes the question speech generated in step 801 to the speech recognition API of the ASR model, and the speech recognition API remotely calls the ASR model on the server side, so that the server side uses the parameter set of the ASR model to perform speech recognition, obtain a question text, and return the generated question text to the terminal through the speech recognition API. In this way, the terminal side does not need to spend storage resources to store the parameter set of the ASR model, saving the storage overhead of the terminal.

[0337] In some embodiments, after the terminal recognizes the question text based on step C1, the question text can also be displayed on the virtual image display interface, presenting a visual interaction effect that the virtual image has understood the question voice and converted it into question text, thereby improving the efficiency of human-computer interaction. However, the display of the question text is an optional step and the embodiments of the present disclosure do not specifically limit this.

[0338] In step C2, the terminal searches for an enhanced text of the question text based on the question text, and the enhanced text meets a similar condition with the question text.

[0339] In some embodiments, the terminal searches for an enhanced text that meets similar conditions to the question text based on the question text obtained in step C1. The role of the enhanced text is to enhance the important information worthy of attention in the question text. By searching for the enhanced text, the question text and the enhanced text can be used together to generate an answer text. Under the condition of the important information emphasized by the enhanced text, the answer text can give an answer that is more in line with the intention of the question text, thereby improving the accuracy of the machine-generated answer text.

[0340] Below, taking steps C21 to C23 as an example, a possible implementation method of using an object feature library to search for enhanced text is introduced. In an object interaction scenario, the user is likely to focus on or pay attention to the features of the object itself in the question text. Therefore, using the object feature library to support enhanced text allows the enhanced text to better highlight or emphasize important information worthy of attention in the question text, thereby improving the search efficiency of the enhanced text.

[0341] In step C21, the terminal extracts question text features of the question text.

[0342] In some embodiments, the terminal performs feature extraction on the question text obtained in step C1 to obtain question text features of the question text. Optionally, the terminal inputs the question text into a feature extraction model to extract the question text features through the feature extraction model. The feature extraction model can be trained locally on the terminal or sent to the terminal after training on the server cloud. This is not specifically limited in the embodiments of the present disclosure. The terminal may not store the feature extraction model locally, but may remotely call the feature extraction model on the server side through the feature extraction API. This is not specifically limited in the embodiments of the present disclosure.

[0343] It should be noted that the feature extraction model can be any machine learning model that can extract feature information of the text modality. For example, the feature extraction model is an embedding model, a word vector model or a one-hot encoding model, etc. The embodiment of the present disclosure does not specifically limit the structure of the feature extraction model.

[0344] In an exemplary scenario, the feature extraction model is taken as an example, and the M3E model is trained on a sentence pair data set by contrastive learning using in-batch negative sampling. The terminal stores the M3E model locally, inputs the question text obtained in step C1 into the M3E model, extracts the text semantic vector of the question text through the M3E model, determines the text semantic vector as the extracted text feature, and implements the text-to-quantity conversion process.

[0345] In step C22, the terminal searches for an item text feature whose similarity with the question text feature meets the similarity condition from an item feature library, where the item feature library is used to store text features of item information of each item.

[0346] In some embodiments, the item feature library can be a database of text features of item information of each item that has been put on the shelf in the storage platform, and the question text and the enhanced text meet the similarity condition, which means that the question text features extracted from the question text in step C21 and the item text features of the enhanced text itself meet the similarity condition, that is, the similarity between a pair of text features is used to measure the similarity between a pair of texts. On this basis, the terminal uses the question text features extracted in step C21 to perform a similarity search in a full item feature library, and the item feature library returns an item text feature with the highest feature similarity to the question text feature.

[0347] In some embodiments, the above-mentioned item feature library is a part of the item database remotely maintained and supported by the server, so that the terminal can send an enhanced text search request to the server, and the enhanced text search request carries the question text feature. After receiving the enhanced text search request, the server parses the enhanced text search request to obtain the question text feature, and then uses the question text feature to perform a similarity search in the item feature library to find the item text feature with the highest feature similarity with the question text feature, and then enters step C23 to find the enhanced text to which the item text feature belongs, so as to ensure that the server finally returns an enhanced text to the server, wherein the item text feature of the enhanced text meets the similarity condition with the question text feature provided by the terminal.

[0348] In some embodiments, the feature similarity mentioned above may refer to the cosine distance between two text features. The larger the cosine distance, the higher the feature similarity. Alternatively, the similarity mentioned above may refer to the Euclidean distance between two text features. The smaller the Euclidean distance, the higher the feature similarity. The embodiment of the present disclosure does not specifically limit the measurement index of feature similarity.

[0349] In an exemplary scenario, the question text feature extracted in step C21 refers to the text semantic vector, and the item feature library stores the item semantic vector of the item information of each item. In this way, the item feature library can be regarded as a vector database. Furthermore, given the text semantic vector, a similarity search is performed on the vector database to find the item semantic vector with the highest feature similarity with the text semantic vector. The found item semantic vector is the item text feature that meets the similarity condition.

[0350] Since the item feature library provides similarity search functionality from a textual modality to support search for enhanced text, an example of a possible way to construct the item feature library will be given below.

[0351] In some embodiments, for each item listed on the platform, since each item has item information entered when it is listed, the item information may include detailed information and / or preview images. In this way, for all items whose item information contains detailed information, the detailed information of these items can be used to build an item feature library.

[0352] In some embodiments, the detailed information in the item information of each item is segmented to obtain multiple candidate texts; then, the text features of each candidate text are extracted, and the extracted text features are stored in the item feature library. In other words, for any item containing detailed information in the item information, the detailed information of the item is segmented to obtain multiple candidate texts formed after segmentation, and then, for each candidate text, the text features of each candidate text are extracted in the same manner as step C21, and the text features of all the candidate texts obtained in advance are stored in the item feature library.

[0353] In the above method of constructing the item feature library, since the candidate text after the detail information is segmented will use the feature extraction model to extract text features, the candidate text can be converted into a feature space, and the same feature extraction model will be used to extract text features for the question text in step C21, thereby ensuring that the question text is converted into the same feature space, so that when calculating the feature similarity in the same feature space, the best item text feature that meets the similarity conditions can be found, thereby improving the similarity search efficiency and search accuracy of the item text features.

[0354] In step C23, the terminal obtains the enhanced text to which the searched text feature of the object belongs.

[0355] In some embodiments, since each text feature in the item feature library has a unique corresponding candidate text, a unique corresponding candidate text can be retrieved based on the item text feature searched in step C22, and the retrieved candidate text is determined as the enhanced text. Since the feature similarity between the item text feature and the question text feature is the highest, the enhanced text meets the similarity condition with the question text.

[0356] In steps C21 to C23, a possible implementation method of using an object feature library to search for enhanced text is provided. In an object interaction scenario, the user is likely to focus on or pay attention to the features of the object itself in the question text. Therefore, the object feature library is used to support the enhanced text, so that the enhanced text can better highlight or emphasize important information worthy of attention in the question text, thereby improving the search efficiency of the enhanced text.

[0357] It should be noted that steps C21 to C23 are optional steps, that is, using the object feature library to search for enhanced text, which can quickly and efficiently search for enhanced text, and the performance overhead of the search process is low. In some other embodiments, attention extraction can also be directly performed on the question text to extract enhanced fields worthy of attention as enhanced text. The embodiments of the present disclosure do not specifically limit the method for obtaining enhanced text.

[0358] In step C3, the terminal generates an answer text based on the question text and the enhanced text.

[0359] In some embodiments, after the terminal obtains the enhanced text based on step C2, it uses both the question text and the enhanced text to generate the answer text. This text generation method combining the question text and the enhanced text can be called Retrieval-augmented Generation.

[0360] In some embodiments, a text generation model is used to implement a retrieval enhancement generation process. Optionally, the question text and the enhanced text are input into a text generation model, and the answer text is generated by the text generation model. The text generation model is used to provide interactive text for item-related question-and-answer dialogues.

[0361] In some embodiments, the above-mentioned text generation model refers to any machine learning model with a text generation function. Optionally, the text generation model can be a large generative model with NLP functions, for example, a Qwen-7b model, an ELMo model, a BERT model, a GPT model, etc. The disclosed embodiment does not specifically limit the type of the text generation model. The text generation model here can be the same model as the text generation model of step A1 in the previous embodiment, or a model with the same structure but different parameters, or a model with a different structure, and the disclosed embodiment does not specifically limit this.

[0362] In some embodiments, the server side trains the text generation model, and after the training is completed, the parameter set of the trained text generation model is sent to the terminal, so that the terminal stores the parameter set of the text generation model locally. In this way, the terminal inputs the question text and the enhanced text into the text generation model together, or the terminal fuses the question text and the enhanced text, and inputs the fused fusion text into the text generation model. Then, the text generation model uses the parameter set to generate text according to the input information, and automatically outputs a section of answer text. The disclosed embodiment does not specifically limit this. Alternatively, the terminal can also train the text generation model locally, and after the training is completed, the parameter set of the trained text generation model is stored locally, and the text generation model is directly called locally later. The disclosed embodiment does not specifically limit whether the text generation model is trained locally.

[0363] In other embodiments, since the text generation model may be a large model with more parameters, the terminal side does not need to store the parameter set of the text generation model locally, but the server side stores the parameter set of the text generation model and provides a remote human-machine question-answering service for the terminal, so that the server can provide a human-machine question-answering API with a text generation model, and the terminal passes the question text and the enhanced text together into the human-machine question-answering API of the text generation model, or the terminal fuses the question text and the enhanced text, and passes the fused fusion text into the human-machine question-answering API of the text generation model, and the human-machine question-answering API remotely calls the text generation model on the server side, so that the server side uses the parameter set of the text generation model to generate text, obtains a section of answer text, and returns the generated answer text to the terminal through the human-machine question-answering API. In this way, the terminal side does not need to spend storage resources to store the parameter set of the text generation model, saving the storage overhead of the terminal.

[0364] In the above answer text generation process, the use of the text generation model can fully focus on the information provided by the original question text, as well as the information emphasized by the enhanced text, so as to generate an answer text that is more in line with the user's question intention and improve the accuracy of text generation.

[0365] In step C4, the terminal performs text-to-speech conversion on the answer text to obtain the answer speech that matches the semantics of the answer text.

[0366] In some embodiments, based on the answer text obtained in step C3, the terminal can use TTS technology to synthesize a reply voice that reads the answer text for the answer text. The TTS technology can use machine speech synthesis to dub the answer text and generate a reply voice that reads the answer text based on a given answer text. The sound source of the answer text can be a machine-synthesized sound or a human voice with full authorization from the sound source provider. The embodiments of the present disclosure do not specifically limit this.

[0367] In some embodiments, the terminal inputs the answer text into a speech generation model, parses the answer text through the speech generation model, segments the answer text into a character sequence, and then converts the character sequence into a phoneme sequence. Then, the phoneme sequence is converted into acoustic features (such as pitch, volume, intonation, etc.), and then the acoustic features are converted into answer speech in the form of digital signals. The speech generation model can be an end-to-end TTS model, or a combined model consisting of a language model and an acoustic model, wherein the language model is used to parse the answer text and the acoustic model is used to synthesize the answer speech. The disclosed embodiment does not specifically limit the structure of the speech generation model.

[0368] The speech generation model here can be the same model as the speech generation model in step A3 in the previous embodiment, or a model with the same structure but different parameters, or a model with a different structure, which is not specifically limited in the embodiments of the present disclosure.

[0369] In some embodiments, the server side trains the speech generation model, and after the training is completed, sends the parameter set of the trained speech generation model to the terminal, so that the terminal stores the parameter set of the speech generation model locally. In this way, the terminal inputs the answer text into the speech generation model, uses the parameter set to generate speech, and obtains a piece of answer speech. Alternatively, the terminal can also train the speech generation model locally, and after the training is completed, store the parameter set of the trained speech generation model locally, and then directly call the speech generation model locally. The embodiment of the present disclosure does not specifically limit whether the speech generation model is trained locally.

[0370] In other embodiments, the terminal side does not need to store the parameter set of the speech generation model locally, but the server side stores the parameter set of the speech generation model and provides a remote speech generation service for the terminal, so that the server can provide a speech generation API of the speech generation model, and the terminal passes the answer text generated in step C3 into the speech generation API of the speech generation model, and the speech generation API remotely calls the speech generation model on the server side, so that the server side uses the parameter set of the speech generation model to generate speech, obtains a piece of answer speech, and returns the generated answer speech to the terminal through the speech generation API. In this way, the terminal side does not need to spend storage resources to store the parameter set of the speech generation model, saving the storage overhead of the terminal.

[0371] In some other embodiments, the above-mentioned speech generation API and human-computer question and answer API can be integrated into the same intelligent agent API, that is, the human-computer question and answer service will not only provide answer text, but also provide adapted answer voice. In this way, it is only necessary to input the question text and enhanced text (or the fused text after the two are merged) in step C3, and call the intelligent agent API once. The server side will use the text generation model to generate the answer text, and then use the speech generation model to generate the answer voice, and return the answer text and answer voice at one time, thereby saving communication rounds between the terminal and the server and saving the communication overhead of the terminal. The embodiments of the present disclosure do not specifically limit this.

[0372] In the above steps C1 to C4, the technology of searching for enhanced text is used to first extract the enhanced text to highlight or emphasize the important information worthy of attention in the question text, and then introduce the enhanced text and the question text together to generate the answer text, thereby improving the accuracy of the machine-generated answer text, and naturally improving the accuracy of the answer voice. Therefore, the overall accuracy of intelligent voice question and answer is improved, and it is possible to achieve virtual image interaction with users with both voice and text.

[0373] Fig. 9 is a schematic diagram of a generation principle of a reply voice provided by an embodiment of the present disclosure, such as Fig. 9As shown, after the user performs a trigger operation on the voice interaction control, the microphone is called to record the question voice, and then the question voice is converted into text to obtain the question text, and the question text is converted into text to obtain the question text feature (such as text semantic vector). Here, the feature extraction model is the M3E model as an example; then, the question text feature is used to perform a similarity search, and the item text feature with the highest feature similarity is searched from the item feature library (such as a vector database), and then the enhanced text to which the item text feature belongs is reversely checked, wherein the features stored in the item feature library are text features obtained after text segmentation of the item details information and text conversion using the M3E model; then, the retrieval enhancement generation technology is used to generate text based on the question text and the enhanced text, and the text generation model is taken as the Qwen-7b model as an example, a reply text is output, and then the reply text is converted into speech using the TTS technology to obtain a reply voice.

[0374] It should be noted that steps C1 to C4 are optional steps. Instead of going through the process of first generating the answer text and then synthesizing the answer speech, an end-to-end voice question-answering model can be directly trained. The embodiments of the present disclosure do not specifically limit this.

[0375] In step 803, the terminal plays the answering voice.

[0376] In some embodiments, the terminal plays the answering voice generated in step 802, for example, inputs the answering voice into an audio codec, and can output the answering voice in the form of a digital signal as a voice signal that the user can understand.

[0377] In some embodiments, the timbre of the answering voice, that is, the timbre of the virtual image, can also be freely configured by the user to realize the personalized timbre configuration of the virtual image. For example, when a merchant puts an item on the shelf, he can not only select a virtual image for the item, but also specify the timbre of the virtual image. Alternatively, the merchant does not need to specify the timbre, but binds a timbre that matches the appearance to the virtual image when creating the virtual image, or binds a timbre that matches the image type to the virtual image, or directly randomly selects a timbre for the virtual image from a variety of timbres, or, when the virtual image is broadcast for the first time, a timbre selection pop-up window pops up, and the user can independently select the timbre they want to hear. The disclosed embodiments do not specifically limit the timbre selection method.

[0378] In some embodiments, the terminal can display the answer text while playing the answer voice, so as to present a visual effect of providing subtitles for the answer voice, thereby achieving an interactive effect of both sound and text between the virtual image and the user, thereby improving the user's information acquisition efficiency and human-computer interaction efficiency.

[0379] In other embodiments, the terminal can also generate a broadcast animation of the virtual image based on the answer voice or the answer text, and the broadcast animation is an animation of the virtual image broadcasting the answer voice; then, the broadcast animation and the answer voice are played. In this way, by generating a broadcast animation for the virtual image, the broadcast animation can be played at the same time as the answer voice is played, so that the visual effect of the virtual image broadcasting the answer voice and matching the answer text as subtitles can be created, achieving a visual and auditory dual interactive effect with sound and text and animation effects, which is conducive to immersive interaction between users and virtual images.

[0380] In some embodiments, when generating a broadcast animation, only the answer voice can be used to generate the broadcast animation, or only the answer text can be used to generate the broadcast animation, or the information of the two modes of the answer voice and the answer text can be used to generate the broadcast animation, and the embodiment of the present disclosure does not specifically limit this. Optionally, based on at least one of the answer voice or the answer text, the virtual image is driven by the face to generate the facial broadcast animation of the virtual image, and then, based on at least one of the answer voice or the answer text, the virtual image is driven by the body to generate the body broadcast animation of the virtual image, and then the facial broadcast animation and the body broadcast animation are combined to align the time axis and synthesize a broadcast animation of the virtual image. Optionally, only the facial broadcast animation can be considered without synthesizing the body broadcast animation, or only the body broadcast animation can be considered without synthesizing the facial broadcast animation to save the processing resources of the terminal, and the embodiment of the present disclosure does not specifically limit this. Optionally, the broadcast animation can also be remotely generated by the server, and the terminal only needs to request the server to obtain the broadcast animation, and the embodiment of the present disclosure does not specifically limit this.

[0381] In steps 801 to 803, a human-computer interaction method based on a voice interaction control is provided. The user provides a question voice, and the virtual image will feedback a reply voice, so that human-computer interaction can be achieved by voice. Moreover, since the virtual image itself is a simulated object interaction form, it can create an immersive interactive experience in which the user talks with the object and the anthropomorphic image of the object answers questions, thereby improving the efficiency of human-computer interaction.

[0382] It should be noted that when some users have not authorized the speaker permission, or the user does not want to listen to the voice, only the answer text can be displayed without playing the answer voice. The embodiment of the present disclosure does not specifically limit whether to play the answer voice.

[0383] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0384] In the previous embodiment, the interactive control provided in the virtual image display interface is a voice interactive control as an example, and how to implement the voice interaction mode with the virtual image based on the voice interactive control is introduced. In the embodiment of the present disclosure, the interactive control provided in the virtual image display interface is a text interactive control as an example, and how to implement the text interaction mode with the virtual image based on the text interactive control is introduced.

[0385] Fig.10 is a flow chart of a method for interacting with an object based on a virtual image according to an exemplary embodiment. Fig.10 As shown, this embodiment is executed by an electronic device, and is described by taking the electronic device as a terminal as an example.

[0386] In step 1001, the terminal obtains input question text in response to a triggering operation on the text interaction control.

[0387] In some embodiments, after the terminal opens the virtual image display interface, at least a virtual image and an interactive control with the virtual image are provided in the virtual image display interface. In the case where the interactive control is a text interactive control, the user can perform a trigger operation on the text interactive control, and the terminal responds to the user's trigger operation on the text interactive control and provides a text input box, and the user can type the question text through the text input box. The question text can be input through a virtual keyboard, an external keyboard, or a voice-to-text method. The embodiment of the present disclosure does not specifically limit the input method of the question text.

[0388] In some embodiments, trigger operations include but are not limited to: click operation, long press operation, double-click operation, drag operation, press operation, slide operation, voice command, gesture command, etc. The embodiments of the present disclosure do not specifically limit the type of trigger operation.

[0389] In step 1002, the terminal generates an answer text for the question text based on the question text.

[0390] In some embodiments, the terminal only uses the question text to generate the answer text. At this time, it is only necessary to input the question text into the text generation model and use the text generation model to generate the answer text. The text generation model is described in step C3 of the previous embodiment and will not be repeated here.

[0391] In other embodiments, the terminal may also obtain an enhanced text for the question text in a manner similar to step C2 in the previous embodiment, and then, in a manner similar to step C3, use both the question text and the enhanced text to generate an answer text. Please refer to the introduction of steps C2 to C3 in the previous embodiment and will not repeat them here.

[0392] In step 1003, the terminal displays the answer text; or, performs text-to-speech conversion on the answer text to obtain an answer voice that matches the semantics of the answer text, and plays the answer voice.

[0393] In some embodiments, the terminal directly displays the answer text generated in step 1002 in the virtual image display interface, so as to realize text question and answer between the virtual image and the user in text form. This can be applicable to scenarios where it is inconvenient for some users to turn on the microphone or speaker, thereby expanding the applicable scenarios of human-computer interaction between the virtual image and the user.

[0394] In some embodiments, the terminal may also convert the answer text into an answer voice based on the same method as step C4 in the previous embodiment, and play the answer voice based on the same method as step 803 in the previous embodiment, which will not be repeated here.

[0395] In steps 1001 to 1003, taking the interactive control provided in the virtual image display interface as a text interactive control as an example, a text interaction method with the virtual image based on the text interactive control is provided, and text question and answer between the virtual image and the user is realized in text form. It can be applied to scenarios where it is inconvenient for some users to turn on the microphone or speaker, thereby expanding the applicable scenarios of the human-computer interaction method between the virtual image and the user.

[0396] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0397] In the above embodiments, the provided item interaction solutions based on virtual images and the human-computer interaction solutions between virtual images and users based on interactive controls can use machine learning and NLP technology to build an item knowledge base as detailed as possible based on the item information uploaded by the merchant on the server side, that is, an item feature library, or an item database containing more extensive information, so that the intelligent agent in the virtual image background can provide more accurate customer consulting services. Moreover, the item knowledge base can be automatically updated when the item information is updated, so that the text features are extracted based on the latest item information, ensuring that the answer text provided to the user will not be outdated.

[0398] Furthermore, since the intelligent agent in the background of the avatar can perform data calculation based on the big model to realize text or voice Q&A with the user, it can handle various complex Q&A tasks. The big model can perform deep text understanding, accurately realize user intent recognition and natural language generation, so that the avatar has the ability to interact with the user in a highly natural way. It can not only provide some basic answers to item information, but also assist in: (a) pre-sales consultation, including user shopping decision support, item comparison, inventory query, item reservation, etc.; (b) after-sales service, such as order processing, logistics tracking, return and exchange assistance, user feedback survey, etc. For example, when a user buys a laptop from Company X, he can ask the avatar questions about the laptop, such as asking "What is the screen size of the laptop?" After that, the avatar will explain the detailed parameters of the laptop in a vivid and emotional way, and actively prompt the user to use the AR preview function to view the appearance and size of the laptop in the actual use environment.

[0399] Furthermore, since the virtual image can provide personalized intelligent agent services according to the user's specific situation and business needs, a highly customized user experience is achieved. Users can also more easily obtain the information they want to know about the items, and can quickly get after-sales support, which improves the efficiency of human-computer interaction and information acquisition. In addition, it reduces the user's dependence on manual customer service, significantly reduces operating costs, and also provides service efficiency and response speed for item consultation. Of course, if the intelligent agent cannot understand the user's very complex business needs, it can still transfer the user to manual customer service to assist the user in achieving business needs, ensuring the continuity and service quality of consulting services.

[0400] Furthermore, from merchants uploading item information, to automatically building a knowledge base, and then to users interacting with virtual images, a closed-loop product service system is formed, covering the complete closed-loop process of item listing, knowledge base updating, human-computer interaction and feedback loop. After the platform solves the user's question through the virtual image, it can also collect feedback from the user on this question and answer, so as to optimize the knowledge base and service quality, so as to continuously improve the interactive performance of the virtual image.

[0401] Fig.11 is a logical structure diagram of a virtual image-based object interaction device according to an exemplary embodiment. Fig.11 The device includes a generating unit 1101, a display unit 1102 and a control unit 1103.

[0402] A generating unit 1101 is configured to generate a virtual image of the object in response to a trigger operation based on the object, wherein the virtual image is used to represent the object, and the virtual image is an interactive object for realizing human-computer interaction;

[0403] A display unit 1102 is configured to display the virtual image and an interactive control, where the interactive control is used to trigger interaction with the virtual image;

[0404] The control unit 1103 is configured to execute a human-computer interaction in response to a triggering operation on the interactive control.

[0405] The device provided by the embodiment of the present disclosure generates a virtual image created for the object when a trigger operation based on the object is detected, and displays the virtual image and interactive controls for the virtual image. By performing a trigger operation with the interactive controls, the virtual image can be further controlled to achieve human-computer interaction, so that the virtual image can assume the function of an intelligent agent for the object in a visualized virtual interactive form, thereby enriching the object interaction mode, improving the efficiency of obtaining object information, and improving the efficiency of human-computer interaction.

[0406] In some embodiments, the generating unit 1101 is further configured to execute:

[0407] Displaying an item list, wherein the item list contains at least one item;

[0408] In response to a triggering operation based on any one of the items in the item list, a virtual image of the item is generated.

[0409] In some embodiments, the item list is used to provide items that support human-computer interaction based on the virtual image.

[0410] In some embodiments, the display unit 1102 is further configured to perform:

[0411] Jump from the current interface to the avatar display interface;

[0412] The virtual image and the interactive control are displayed in the virtual image display interface.

[0413] In some embodiments, the virtual image and the interactive control are independent of each other; or, part or all of the virtual image is configured as the interactive control.

[0414] In some embodiments, the interactive control is a voice interactive control based on Fig.11 The control unit 1103 comprises:

[0415] A voice recording subunit is configured to execute a recording of a question voice in response to a triggering operation on the voice interaction control;

[0416] The speech generation subunit is configured to generate a response speech for the question speech based on the question speech;

[0417] The playing subunit is configured to play the answering voice.

[0418] In some embodiments, based on Fig.11 The speech generation subunit comprises:

[0419] The speech recognition sub-subunit is configured to perform speech recognition on the question speech to obtain a question text that matches the semantics of the question speech;

[0420] A text search sub-subunit is configured to search for an enhanced text of the question text based on the question text, wherein the enhanced text meets a similarity condition with the question text;

[0421] A text generation sub-subunit is configured to generate an answer text based on the question text and the enhanced text;

[0422] The text-to-speech conversion sub-subunit is configured to perform text-to-speech conversion on the answer text to obtain the answer speech that matches the semantics of the answer text.

[0423] In some embodiments, the text search sub-subunit is further configured to perform:

[0424] Extracting question text features of the question text;

[0425] Searching for an item text feature whose similarity with the question text feature meets the similarity condition from an item feature library, wherein the item feature library is used to store text features of item information of each item;

[0426] Get the enhanced text to which the searched text feature of the item belongs.

[0427] In some embodiments, the item information includes detailed information of the item, based on Fig.11 The device is composed of:

[0428] A text segmentation unit is configured to perform text segmentation on the detail information in the item information of each item to obtain a plurality of candidate texts;

[0429] The extraction storage unit is configured to extract the text features of each candidate text and store the extracted text features in the object feature library.

[0430] In some embodiments, the text generation sub-subunit is further configured to perform:

[0431] The question text and the enhanced text are input into a text generation model, and the answer text is generated by the text generation model. The text generation model is used to provide interactive text for question-and-answer dialogues related to items.

[0432] In some embodiments, based on Fig.11 The device is composed of:

[0433] An animation generating unit is configured to generate a broadcasting animation of the virtual image based on the answering voice or the answering text, wherein the broadcasting animation is an animation of the virtual image broadcasting the answering voice;

[0434] The playing subunit is also configured to play the playing animation and the answering voice.

[0435] In some embodiments, the interactive control is a text interactive control based on Fig.11 The control unit 1103 comprises:

[0436] A text acquisition subunit is configured to execute a trigger operation on the text interaction control to acquire an input question text;

[0437] A text generation subunit is configured to generate an answer text for the question text based on the question text;

[0438] The display unit 1102 is further configured to display the answer text; or,

[0439] The conversion and playback subunit is configured to perform text-to-speech conversion on the answer text, obtain an answer voice that matches the semantics of the answer text, and play the answer voice.

[0440] In some embodiments, the display unit 1102 is further configured to perform: displaying a preview control of the item;

[0441] based on Fig.11 The device further comprises: a collection unit configured to collect video frames through a camera in response to a trigger operation on the preview control;

[0442] The display unit 1102 is further configured to execute: displaying a real-life image of the object in the video frame.

[0443] In some embodiments, the real scene image is an augmented reality image of the object, based on Fig.11 The display unit 1102 is further configured to perform:

[0444] Acquire an augmented reality image of the object, and draw the augmented reality image on the video frame; or,

[0445] Based on the augmented reality image, an item preview model of the item is generated, and the item preview model is drawn on the video frame.

[0446] In some embodiments, based on Fig.11 The device is composed of:

[0447] A text generation unit is configured to generate an introduction text associated with the virtual image based on the item information, the introduction text being used to simulate the virtual image introducing the item;

[0448] The display unit 1102 is further configured to display the introduction text based on the virtual image.

[0449] In some embodiments, the text generation unit is further configured to perform:

[0450] The item information is input into a text generation model, and the introduction text is generated by the text generation model. The text generation model is used to provide interactive text for question-and-answer dialogues related to the item.

[0451] In some embodiments, based on Fig.11 The device is composed of:

[0452] A speech generating unit is configured to generate an introduction speech matching the introduction text based on the introduction text;

[0453] The playing unit is configured to play the introduction voice.

[0454] In some embodiments, the display unit 1102 is further configured to perform:

[0455] Displays operation prompt information for the interactive control, where the operation prompt information is used to prompt the interaction method with the interactive control.

[0456] In some embodiments, based on Fig.11 The generating unit 1101 comprises:

[0457] The generating subunit is configured to generate a virtual image of the object based on the object information in response to a triggering operation based on the object.

[0458] In some embodiments, the item information includes at least one of detailed information or a preview image of the item, and the generating subunit is further configured to execute:

[0459] Based on the detailed information, generate the virtual image; or,

[0460] Based on the preview image, generate the virtual image; or,

[0461] Based on the detailed information and the preview image, the virtual image is generated.

[0462] In some embodiments, based on Fig.11 The generating subunit comprises:

[0463] An image generation sub-subunit is configured to input the object information into an image generation model, and generate a virtual image of the object through the image generation model, wherein the image generation model is used to generate a virtual image of the object;

[0464] The image generation sub-subunit is configured to generate the virtual image based on the virtual image image.

[0465] In some embodiments, the image generation sub-subunit is further configured to perform:

[0466] determining the avatar image as the avatar; or,

[0467] Three-dimensional modeling is performed based on the virtual image to obtain a three-dimensional model of the virtual image, and the three-dimensional model is determined as the virtual image.

[0468] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0469] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the virtual image-based object interaction method, and will not be elaborated here.

[0470] Fig.12 A structural block diagram of a terminal provided by an exemplary embodiment of the present disclosure is shown. Fig.12 As shown, terminal 1200 is an exemplary description of an electronic device. The terminal 1200 may be: a smart phone, a tablet computer, a Moving Picture Experts Group Audio Layer III (MP3) player, a Moving Picture Experts Group Audio Layer IV (MP4) player, a laptop computer or a desktop computer. The terminal 1200 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0471] Typically, the terminal 1200 includes: a processor 1201 and a memory 1202 .

[0472] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0473] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction, which is used to be executed by the processor 1201 to implement the virtual image-based item interaction method provided in various embodiments of the present disclosure.

[0474] In some embodiments, the terminal 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a touch display screen 1205, a camera assembly 1206, an audio circuit 1207 and a power supply 1208.

[0475] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0476] The radio frequency circuit 1204 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a wireless fidelity (Wireless Fidelity, WiFi) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to near field communication (NFC), which is not limited in the present disclosure.

[0477] The display screen 1205 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal can be input to the processor 1201 as a control signal for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1205 can be one, and the front panel of the terminal 1200 is set; in other embodiments, the display screen 1205 can be at least two, which are respectively set on different surfaces of the terminal 1200 or are folded; in some other embodiments, the display screen 1205 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1200. Even, the display screen 1205 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1205 can be made of materials such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0478] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and virtual reality (VR) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0479] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1200. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0480] The power supply 1208 is used to power various components in the terminal 1200. The power supply 1208 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1208 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0481] In some embodiments, the terminal 1200 further includes one or more sensors 1210 . The one or more sensors 1210 include, but are not limited to: an acceleration sensor 1211 , a gyroscope sensor 1212 , a pressure sensor 1213 , an optical sensor 1214 , and a proximity sensor 1215 .

[0482] The acceleration sensor 1211 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1200. For example, the acceleration sensor 1211 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1201 can control the touch display screen 1205 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1211. The acceleration sensor 1211 can also be used for collecting game or user motion data.

[0483] The gyro sensor 1212 can detect the body direction and rotation angle of the terminal 1200, and the gyro sensor 1212 can cooperate with the acceleration sensor 1211 to collect the user's 3D actions on the terminal 1200. The processor 1201 can implement the following functions based on the data collected by the gyro sensor 1212: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0484] The pressure sensor 1213 can be set on the side frame of the terminal 1200 and / or the lower layer of the touch display screen 1205. When the pressure sensor 1213 is set on the side frame of the terminal 1200, it can detect the user's holding signal of the terminal 1200, and the processor 1201 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1213. When the pressure sensor 1213 is set on the lower layer of the touch display screen 1205, the processor 1201 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1205. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0485] The optical sensor 1214 is used to collect the ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the touch display screen 1205 according to the ambient light intensity collected by the optical sensor 1214. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1205 is reduced. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1214.

[0486] The proximity sensor 1215, also called a distance sensor, is usually arranged on the front panel of the terminal 1200. The proximity sensor 1215 is used to collect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1215 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the touch display screen 1205 to switch from the screen-on state to the screen-off state; when the proximity sensor 1215 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the touch display screen 1205 to switch from the screen-off state to the screen-on state.

[0487] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0488] Fig.131 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. The electronic device 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 stores at least one program code, and the at least one program code is loaded and executed by the processor 1301 to implement the virtual image-based object interaction method provided by the above-mentioned various embodiments. Of course, the electronic device 1300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 1300 may also include other components for implementing device functions, which will not be repeated here.

[0489] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction, and the at least one instruction can be executed by a processor in an electronic device to complete the virtual image-based object interaction method in the above embodiment. Optionally, the computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0490] In an exemplary embodiment, a computer program product is also provided, including one or more instructions, which can be executed by a processor of an electronic device to complete the virtual image-based object interaction method provided in the above-mentioned embodiments.

[0491] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0492] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for object interaction based on a virtual image, characterized in that: include: In response to a trigger operation based on an object, a virtual image of the object is generated, wherein the virtual image is used to represent the object, and the virtual image is an interactive object for realizing human-computer interaction; Displaying the virtual image and an interactive control, wherein the interactive control is used to trigger interaction with the virtual image, and the interactive control is a voice interactive control; In response to a triggering operation on the voice interaction control, recording a question voice; Performing speech recognition on the question voice to obtain a question text that matches the semantics of the question voice; Extracting question text features of the question text; Searching for item text features whose similarity with the question text features meets similarity conditions from an item feature library, wherein the item feature library is used to store text features of item information of each item; the item text features are obtained by performing text feature extraction on candidate texts, wherein the candidate texts are obtained by performing text segmentation on the stored detailed information of each item, and the item text features correspond to the candidate texts one by one; Determine the candidate text corresponding to the searched item text feature as the enhanced text; Generate an answer text based on the question text and the enhanced text; Performing text-to-speech conversion on the answer text to obtain an answer voice that matches the semantics of the answer text; Play the answering voice.

2. The method for object interaction based on a virtual image according to claim 1, characterized in that: The generating a virtual image of the object in response to a trigger operation based on the object comprises: Displaying an item list, wherein the item list contains at least one item; In response to a triggering operation based on any one of the items in the item list, a virtual image of the item is generated.

3. The method for object interaction based on virtual images according to claim 2, characterized in that: The item list is used to provide items that support human-computer interaction based on the virtual image.

4. The method for object interaction based on a virtual image according to any one of claims 1 to 3, characterized in that: The displaying of the virtual image and the interactive controls comprises: Jump from the current interface to the avatar display interface; The virtual image and the interactive controls are displayed in the virtual image display interface.

5. The method for object interaction based on virtual images according to claim 4, characterized in that: The virtual image and the interactive control are independent of each other; or part or all of the virtual image is configured as the interactive control.

6. The method for object interaction based on a virtual image according to claim 1, characterized in that: The generating of the answer text based on the question text and the enhanced text comprises: The question text and the enhanced text are input into a text generation model, and the answer text is generated by the text generation model. The text generation model is used to provide interactive text for question-and-answer dialogues related to items.

7. The method for object interaction based on virtual images according to claim 6, characterized in that: The playing of the answering voice comprises: Based on the answer voice or the answer text, generating a broadcast animation of the virtual image, wherein the broadcast animation is an animation of the virtual image broadcasting the answer voice; Play the animation and the answering voice.

8. The method for object interaction based on a virtual image according to claim 1, characterized in that: The method further comprises: a preview control showing said item; In response to a triggering operation on the preview control, capturing a video frame through a camera; In the video frame, a real-life image of the object is displayed.

9. The method for object interaction based on a virtual image according to claim 1, characterized in that: The method further comprises: The operation prompt information for the interactive control is displayed, where the operation prompt information is used to prompt the interaction method with the interactive control.

10. The method for object interaction based on a virtual image according to claim 1, characterized in that: The generating a virtual image of the object in response to a trigger operation based on the object comprises: In response to a trigger operation based on the object, a virtual image of the object is generated based on the object information of the object.

11. The method for object interaction based on virtual images according to claim 10, characterized in that: The item information includes at least one of detailed information or a preview image of the item, and generating a virtual image of the item based on the item information includes: generating the virtual image based on the detailed information; or, generating the virtual image based on the preview image; or, The virtual image is generated based on the detailed information and the preview image.

12. The method for object interaction based on a virtual image according to claim 10 or 11, characterized in that: The step of generating a virtual image of the object based on the object information of the object includes: Inputting the item information into an image generation model, and generating a virtual image of the item through the image generation model, wherein the image generation model is used to generate a virtual image of the item; Based on the avatar image, the avatar is generated.

13. The method for object interaction based on virtual images according to claim 12, characterized in that: The generating the virtual image based on the virtual image image comprises: determining the avatar image as the avatar; or, Three-dimensional modeling is performed based on the virtual image to obtain a three-dimensional model of the virtual image, and the three-dimensional model is determined as the virtual image.

14. An object interaction device based on a virtual image, characterized in that: include: A generating unit, configured to generate a virtual image of the object in response to a triggering operation based on the object, wherein the virtual image is used to represent the object, and the virtual image is an interactive object for realizing human-computer interaction; a display unit configured to display the virtual image and an interactive control, wherein the interactive control is used to trigger interaction with the virtual image, and the interactive control is a voice interactive control; A control unit is configured to execute, in response to a trigger operation on the voice interaction control, recording a question voice; performing voice recognition on the question voice to obtain a question text that matches the semantics of the question voice; extracting question text features of the question text; searching for item text features that meet similarity conditions with the question text features from an item feature library, the item feature library being used to store text features of item information of each item; the item text features are obtained by performing text feature extraction on candidate texts, the candidate texts are obtained by performing text segmentation on the stored detailed information of each item, and the item text features correspond to the candidate texts one by one; determining the candidate text corresponding to the searched item text features as enhanced text; and generating a reply text based on the question text and the enhanced text; Perform text-to-speech conversion on the answer text to obtain an answer voice that matches the semantics of the answer text; and play the answer voice.

15. An electronic device, characterized in that: include: one or more processors; one or more memories for storing the one or more processor-executable instructions; The one or more processors are configured to execute the instructions to implement the virtual image-based object interaction method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: When at least one instruction in the computer-readable storage medium is executed by one or more processors of an electronic device, the electronic device is enabled to execute the virtual image-based object interaction method as described in any one of claims 1 to 13.

Images