An interactive method and system based on virtual digital human

Abstract

The present application discloses an interactive method and system based on a virtual digital human, and belongs to the field of computer technology. Through the technical solution provided by the embodiment of the present application, in response to the call request of the target object to the virtual digital human, the first object information of the target object and the environmental information of the environment in which it is located are obtained, and the initial virtual digital human is generated and displayed using the first object information and the environmental information. The second object information collected when displaying the initial virtual digital human is obtained, and the initial virtual digital human is updated using the second object information to obtain the target virtual digital human, so as to use the target virtual digital human to interact with the target object, improve the target object's satisfaction with the target virtual digital human, and thus improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an interaction method and system based on a virtual digital human. Background Art

[0002] With the development of science and technology, virtual digital humans have begun to be applied in many fields. Traditional virtual digital human interactions are often relatively static, that is, users select the virtual digital humans they want to interact with from multiple existing virtual digital humans for further interaction.

[0003] However, the number of existing virtual digital humans is limited, and users may not be able to select the virtual digital humans they need, resulting in poor interaction between users and virtual digital humans.

[0004] Therefore, how to provide users with more satisfactory virtual digital humans has become a hot topic of research. Summary of the invention

[0005] The embodiment of the present application provides an interactive method and system based on a virtual digital human, which can provide users with richer virtual digital humans. The technical solution is as follows:

[0006] On the one hand, a method for interaction based on a virtual digital human is provided, the method comprising:

[0007] In response to a virtual digital human call request of a target object, first object information of the target object and environmental information of the environment in which the target object is located are obtained, wherein the first object information includes object features, a first object image, first physiological data, and historical digital human preferences of the target object, wherein the historical digital human preferences are used to indicate the historical preferences of the target object for different types of virtual digital humans;

[0008] Based on the first object information of the target object and the environment information, generating and displaying an initial virtual digital human;

[0009] Acquire second object information of the target object, where the second object information includes a second object image, second physiological data, and target audio of the target object, where the target audio is audio collected during the display of the initial virtual digital human;

[0010] Based on the second object information, the initial virtual digital human is updated to obtain and display the target virtual digital human;

[0011] Control the target virtual digital human to interact with the target object.

[0012] In a possible implementation, the generating and displaying an initial virtual digital human based on the first object information of the target object and the environment information includes:

[0013] Based on the object characteristics and the historical digital human preferences, determining a reference virtual digital human from a plurality of candidate virtual digital humans, wherein different candidate virtual digital humans have different images;

[0014] determining a first object state of the target object based on the first object image and the first physiological data;

[0015] Determining a second object state of the target object based on the environmental information and the object feature;

[0016] The reference virtual digital human is updated based on the first object state and the second object state to obtain the initial virtual digital human;

[0017] The initial virtual digital human is displayed.

[0018] In a possible implementation, determining the first object state of the target object based on the first object image and the first physiological data includes:

[0019] Based on the first object image, determining a first expression, a first posture, and clothing information of the target object;

[0020] Determining a first initial object state of the target object based on a first expression, a first posture, and clothing information of the target object;

[0021] The first initial object state is modified based on the first physiological data to obtain the first object state.

[0022] In a possible implementation manner, the environmental information includes environmental weather, environmental temperature, and environmental type, and determining the second object state of the target object based on the environmental information and the object feature includes:

[0023] Determining a second initial object state of the target object based on the ambient weather and the ambient temperature;

[0024] Based on the environment type and the object characteristics, the second initial object state is modified to obtain the second object state.

[0025] In a possible implementation, the updating of the reference virtual digital human based on the first object state and the second object state to obtain the initial virtual digital human includes:

[0026] Determining a first target object state of the target object based on the first object state and the second object state;

[0027] Acquire first digital human description information of the reference virtual digital human, where the first digital human description information is used to describe the situation of the reference virtual digital human;

[0028] Based on the first target object state and the first digital human description information, generate first update information of the reference virtual digital human;

[0029] The reference virtual digital human is updated based on the first update information to obtain the initial virtual digital human.

[0030] In a possible implementation, the updating of the initial virtual digital human based on the second object information to obtain and display the target virtual digital human includes:

[0031] determining a third object state of the target object based on the second object image;

[0032] determining a fourth object state of the target object based on the second physiological data;

[0033] Based on the target audio, the third object state and the fourth object state, the initial virtual digital human is updated to obtain the target virtual digital human;

[0034] Display the target virtual digital human.

[0035] In a possible implementation, the updating of the initial virtual digital human based on the target audio, the third object state, and the fourth object state to obtain the target virtual digital human includes:

[0036] Performing speech recognition and emotion recognition on the target audio to obtain a target text and a target emotion corresponding to the target audio;

[0037] determining a second target object state of the target object based on the third object state and the fourth object state;

[0038] Acquire second digital human description information of the initial virtual digital human, where the second digital human description information is used to describe the situation of the initial virtual digital human;

[0039] Generate second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information;

[0040] The initial virtual digital human is updated based on the second update information to obtain the target virtual digital human.

[0041] In a possible implementation, the generating the second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information includes:

[0042] Based on the target emotion and the state of the second target object, determining the target object's liking for the initial virtual digital person;

[0043] Generate a digital human update guide based on the target text and the second digital human description information;

[0044] Based on the target object's liking for the initial virtual digital human and the digital human update guide, second update information of the initial virtual digital human is generated.

[0045] In a possible implementation, the updating of the initial virtual digital human based on the second update information to obtain the target virtual digital human includes:

[0046] Splitting the second update information to obtain a plurality of sub-update information, wherein different sub-update information is used to update different initial virtual components of the initial virtual digital human, and the initial virtual digital human is composed of a plurality of initial virtual components;

[0047] Based on the plurality of sub-update information, a plurality of target virtual components are generated, wherein one of the target virtual components corresponds to one of the initial virtual components;

[0048] Each of the target virtual components is used to replace the corresponding virtual initial virtual components in the initial virtual digital human to obtain the target virtual digital human.

[0049] On the one hand, an interactive device based on a virtual digital human is provided, the device comprising:

[0050] an acquisition module, configured to respond to a virtual digital human call request of a target object and acquire first object information of the target object and environmental information of the environment in which the target object is located, wherein the first object information includes object features, a first object image, first physiological data, and historical digital human preferences of the target object, wherein the historical digital human preferences are used to indicate the historical preferences of the target object for different types of virtual digital humans;

[0051] A generating module, configured to generate and display an initial virtual digital human based on the first object information of the target object and the environment information;

[0052] The acquisition module is further used to acquire second object information of the target object, wherein the second object information includes a second object image, second physiological data and target audio of the target object, and the target audio is the audio collected during the display of the initial virtual digital human;

[0053] An updating module, configured to update the initial virtual digital human based on the second object information, and obtain and display a target virtual digital human;

[0054] The control module is used to control the target virtual digital human to interact with the target object.

[0055] In a possible implementation, the generation module is used to determine a reference virtual digital human from multiple candidate virtual digital humans based on the object features and the historical digital human preferences, where different candidate virtual digital humans have different images; determine a first object state of the target object based on the first object image and the first physiological data; determine a second object state of the target object based on the environmental information and the object features; update the reference virtual digital human based on the first object state and the second object state to obtain the initial virtual digital human; and display the initial virtual digital human.

[0056] In one possible implementation, the generation module is used to determine a first expression, a first posture, and dressing information of the target object based on the first object image; determine a first initial object state of the target object based on the first expression, the first posture, and dressing information of the target object; and correct the first initial object state based on the first physiological data to obtain the first object state.

[0057] In a possible implementation, the environmental information includes ambient weather, ambient temperature, and ambient type, and the generation module is used to determine the second initial object state of the target object based on the ambient weather and the ambient temperature; based on the ambient type and the object characteristics, the second initial object state is corrected to obtain the second object state.

[0058] In a possible implementation, the generation module is used to determine a first target object state of the target object based on the first object state and the second object state; obtain first digital human description information of the reference virtual digital human, wherein the first digital human description information is used to describe the situation of the reference virtual digital human; generate first update information of the reference virtual digital human based on the first target object state and the first digital human description information; and update the reference virtual digital human based on the first update information to obtain the initial virtual digital human.

[0059] In a possible implementation, the updating of the initial virtual digital human based on the second object information to obtain and display the target virtual digital human includes:

[0060] determining a third object state of the target object based on the second object image;

[0061] determining a fourth object state of the target object based on the second physiological data;

[0062] Based on the target audio, the third object state and the fourth object state, the initial virtual digital human is updated to obtain the target virtual digital human;

[0063] Display the target virtual digital human.

[0064] In a possible implementation, the update module is used to perform speech recognition and emotion recognition on the target audio to obtain a target text and a target emotion corresponding to the target audio; determine a second target object state of the target object based on the third object state and the fourth object state; obtain second digital human description information of the initial virtual digital human, the second digital human description information being used to describe the situation of the initial virtual digital human; generate second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information; and update the initial virtual digital human based on the second update information to obtain the target virtual digital human.

[0065] In a possible implementation, the update module is used to determine the target object's degree of liking for the initial virtual digital human based on the target emotion and the second target object state; generate a digital human update guide based on the target text and the second digital human description information; and generate second update information for the initial virtual digital human based on the target object's degree of liking for the initial virtual digital human and the digital human update guide.

[0066] In a possible implementation, the update module is used to split the second update information to obtain multiple sub-update information, where different sub-update information is used to update different initial virtual components of the initial virtual digital person, and the initial virtual digital person is composed of multiple initial virtual components; based on the multiple sub-update information, multiple target virtual components are generated, and one target virtual component corresponds to one initial virtual component; and each target virtual component is used to replace the corresponding virtual initial virtual component in the initial virtual digital person to obtain the target virtual digital person.

[0067] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the interaction method based on virtual digital humans.

[0068] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the interaction method based on a virtual digital human.

[0069] On the one hand, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device executes the above-mentioned interaction method based on a virtual digital human.

[0070] Through the technical solution provided by the embodiment of the present application, in response to the target object's call request for the virtual digital human, the first object information of the target object and the environmental information of the environment in which it is located are obtained, and the initial virtual digital human is generated and displayed using the first object information and the environmental information. The second object information collected when displaying the initial virtual digital human is obtained, and the initial virtual digital human is updated using the second object information to obtain the target virtual digital human, so that the target virtual digital human is used to interact with the target object, and the target object's satisfaction with the target virtual digital human is improved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0072] Figure 1 It is a schematic diagram of an implementation environment of an interaction method based on a virtual digital human provided in an embodiment of the present application;

[0073] Figure 2 It is a flow chart of an interaction method based on a virtual digital human provided in an embodiment of the present application;

[0074] Figure 3 is a flow chart of another interactive method based on a virtual digital human provided in an embodiment of the present application;

[0075] Figure 4It is a structural schematic diagram of an interactive device based on a virtual digital human provided in an embodiment of the present application;

[0076] Figure 5 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0077] Figure 6 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0079] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0080] Virtual digital human: Digital human is a virtual character image generated by computer technology. It integrates information science and life science and aims to simulate the human body at different levels. As an emerging technological product, digital human has multiple human characteristics, including appearance, human performance ability and interaction ability.

[0081] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0082] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0083] Semantic features: features used to represent the semantics expressed by text. Different texts can correspond to the same semantic features. For example, the text "How is the weather today" and the text "How is the weather today" can correspond to the same semantic feature. Computer devices can map characters in text to character vectors, and combine and operate on character vectors based on the relationship between characters to obtain the semantic features of the text. For example, computer devices can use bidirectional encoder representations from transformers (BERT).

[0084] Normalization: Mapping sequences with different value ranges to the interval (0, 1) facilitates data processing. In some cases, the normalized values ​​can be directly implemented as probabilities.

[0085] Embedded Coding: Embedded coding mathematically represents a correspondence, that is, mapping data in X space to Y space through a function F, where the function F is an injective function, and the mapping result is structural preservation. The injective function indicates that the data after mapping is uniquely corresponding to the data before mapping, and structural preservation indicates that the size relationship of the data before mapping is the same as the size relationship of the data after mapping. For example, there are data X1 and X2 before mapping, and after mapping, Y1 corresponding to X1 and Y2 corresponding to X2 are obtained. If the data X1 before mapping is greater than X2, then correspondingly, the data Y1 after mapping is greater than Y2. For words, it means mapping the words to another space to facilitate subsequent machine learning and processing.

[0086] Attention weight: It can indicate the importance of a certain data in the training or prediction process. Importance indicates the influence of input data on output data. Data with high importance has a higher corresponding attention weight value, and data with low importance has a lower corresponding attention weight value. In different scenarios, the importance of data is different. The process of training the attention weight of the model is also the process of determining the importance of data.

[0087] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the object information and environmental information involved in this application are obtained with full authorization.

[0088] Figure 1 is a schematic diagram of an implementation environment of an interactive method based on a virtual digital human provided in an embodiment of the present application, see Figure 1 , the implementation environment may include a terminal 110 and a server 140.

[0089] The terminal 110 is connected to the server 140 via a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 110 is installed and runs an application that supports the interaction of virtual digital humans.

[0090] Server 140 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0091] Those skilled in the art will appreciate that the number of the above terminals may be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or a greater number, and the above implementation environment may also include other terminals. The embodiment of the present application does not limit the number and device type of the terminals.

[0092] After introducing the implementation environment of the embodiment of the present application, the application scenario of the embodiment of the present application is introduced below. The technical solution provided by the embodiment of the present application can be used in scenarios where virtual digital people interact with users, for example, it can be applied in scenarios where virtual digital people are used as customer service to interact with users, or in games with virtual digital people as the theme, or in other scenarios where virtual digital people are used to replace real people, and the embodiment of the present application does not limit this.

[0093] After introducing the implementation environment and application scenarios of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below. Figure 2 is a flowchart of an interactive method based on a virtual digital human provided in an embodiment of the present application, see Figure 2 Taking the execution subject as a server as an example, the method includes the following steps.

[0094] 201. In response to a virtual digital human call request of a target object, the server obtains first object information of the target object and environmental information of the environment in which the target object is located. The first object information includes object features, a first object image, first physiological data, and historical digital human preferences of the target object. The historical digital human preferences are used to indicate the target object's historical preferences for different types of virtual digital humans.

[0095] Among them, the target object is a user who uses the virtual digital human related services. The virtual digital human is a virtual character image generated by computer technology. Different virtual digital humans have different appearances, sounds and actions. Accordingly, the virtual digital human call request is used to request to call the virtual digital human for interaction. Environmental information is used to represent the environmental state of the environment in which the target object is located. The object characteristics of the target object are used to represent the object characteristics of the target object. The object characteristics are important dimensions that can distinguish the target object from other objects. The first object image is the object image of the target object directly obtained after sending the virtual digital human call request. The first physiological data is the physiological data of the target object directly obtained after sending the virtual digital human call request, which is used to reflect the physiological state of the target object. The historical digital human preference is used to reflect the historical preference of the target object for different types of virtual digital humans. The type of virtual digital human is set by the technician according to the actual situation, and the embodiment of the present application does not limit this.

[0096] 202. The server generates and displays an initial virtual digital human based on the first object information of the target object and the environment information.

[0097] The initial virtual digital human is a virtual digital human that is generated using the first object information and the environment information and that may meet the current preferences of the target object. Displaying the initial virtual digital human refers to displaying the initial virtual digital human and controlling the initial virtual digital human to speak and perform actions.

[0098] 203. The server obtains second object information of the target object, where the second object information includes a second object image, second physiological data, and target audio of the target object, where the target audio is audio collected during the display of the initial virtual digital human.

[0099] The second object image is an object image of the target object collected during the display of the initial virtual digital human, which can reflect the target object's preference for the initial virtual digital human. The second physiological data is the physiological data of the target object collected during the display of the initial virtual digital human, that is, it can reflect the target object's preference for the initial virtual digital human. Since the target audio is the audio collected during the display of the initial virtual digital human, the target audio can directly reflect the target object's evaluation and expectations of the initial virtual digital human.

[0100] 204. The server updates the initial virtual digital human based on the second object information, and obtains and displays the target virtual digital human.

[0101] Among them, based on the fact that the second object information can reflect the target object's preferences, evaluations and expectations for the initial virtual digital person, therefore, by using the second object information to update the initial virtual digital person, the target object's preferences, evaluations and expectations for the initial virtual digital person can be updated in the initial virtual digital person, thereby obtaining a target virtual digital person that better meets the needs of the target object.

[0102] 205. The server controls the target virtual digital human to interact with the target object.

[0103] Through the technical solution provided by the embodiment of the present application, in response to the target object's call request for the virtual digital human, the first object information of the target object and the environmental information of the environment in which it is located are obtained, and the initial virtual digital human is generated and displayed using the first object information and the environmental information. The second object information collected when displaying the initial virtual digital human is obtained, and the initial virtual digital human is updated using the second object information to obtain the target virtual digital human, so that the target virtual digital human is used to interact with the target object, and the target object's satisfaction with the target virtual digital human is improved, thereby improving the user experience.

[0104] The above steps 201-205 are a brief introduction to the technical solution provided in the embodiment of the present application. The following will combine some examples to more clearly explain the technical solution provided in the embodiment of the present application. Figure 3 Taking the execution subject as a server as an example, the method includes the following steps.

[0105] 301. The server obtains a virtual digital human calling request of a target object.

[0106] The target object is a user who uses the virtual digital human related services. The virtual digital human is a virtual character image generated by computer technology. Different virtual digital humans have different appearances, voices and actions. Accordingly, the virtual digital human call request is used to request to call the virtual digital human for interaction.

[0107] In a possible implementation, the server obtains the virtual digital human calling request from a terminal, where the terminal is a terminal used by the target object.

[0108] Among them, a target application interface is displayed on the terminal, and the target application interface is used to indicate triggering a call request for a virtual digital human.

[0109] 302. In response to the virtual digital human call request of the target object, the server obtains the first object information of the target object and the environmental information of the environment in which it is located, wherein the first object information includes the object characteristics, the first object image, the first physiological data and the historical digital human preferences of the target object, and the historical digital human preferences are used to indicate the historical preferences of the target object for different types of virtual digital humans.

[0110] Among them, the environmental information is used to represent the environmental state of the environment in which the target object is located. The object features of the target object are used to represent the object characteristics of the target object. The object features are important dimensions that can distinguish the target object from other objects. The first object image is the object image of the target object directly obtained after sending the virtual digital human call request. The first physiological data is the physiological data of the target object directly obtained after sending the virtual digital human call request, which is used to reflect the physiological state of the target object. In some embodiments, the physiological data includes heart rate, blood pressure, and body temperature. The historical digital human preference is used to reflect the historical preferences of the target object for different types of virtual digital humans. The type of virtual digital human is set by the technician according to the actual situation, and the embodiment of the present application does not limit this. It should be noted that the above-mentioned first object information and environmental information are both obtained after the target object permits.

[0111] In a possible implementation, in response to a virtual digital human call request of a target object, the server sends a first information acquisition request to the terminal, where the first information acquisition request is used to request the acquisition of the first object information and the environmental information. The server acquires the first object information of the target object and the environmental information of the environment returned by the terminal in response to the first information acquisition request.

[0112] Among them, the environmental information includes the location information of the target object, the indoor environment information collected by the smart home device, and the collected environmental images. The location information of the target object can be used to determine the outdoor environment information of the target object's environment, and the environmental image can be used to classify the target object's environment.

[0113] 303. The server generates an initial virtual digital human based on the first object information of the target object and the environment information.

[0114] In a possible implementation, the server determines a reference virtual digital human from a plurality of candidate virtual digital humans based on the object feature and the historical digital human preference, and different candidate virtual digital humans have different images. The server determines a first object state of the target object based on the first object image and the first physiological data. The server determines a second object state of the target object based on the environmental information and the object feature. The server updates the reference virtual digital human based on the first object state and the second object state to obtain the initial virtual digital human.

[0115] Among them, multiple candidate virtual digital humans are virtual digital humans configured in advance by technicians, and the reference virtual digital human is a candidate virtual digital human selected from the multiple candidate virtual digital humans. In addition to different images, different candidate virtual digital humans also have different voices and actions. The first object state is obtained using the first object image and the first physiological data, and can reflect the baseline object state of the target object. The second object state is obtained by combining environmental information and object characteristics, and can reflect the object state of the target object under the influence of the environment.

[0116] In this implementation, the first object information and the environment information are fully mined and utilized to determine the reference virtual digital human, the first object state, and the second object state. The reference virtual digital human is updated using the first object state and the second object state to obtain an initial virtual digital human that is relatively matched with the current state of the target object.

[0117] In order to explain the above implementation more clearly, the above implementation is explained in several parts below.

[0118] In the first part, the server determines a reference virtual digital human from multiple candidate virtual digital humans based on the object features and the historical digital human preferences.

[0119] In a possible implementation, the server extracts features of the historical digital human preferences to obtain preference features corresponding to the historical digital human preferences. The server fuses the object features with the preference features to obtain digital human screening features of the target object. Based on the digital human screening features, the server determines a reference virtual digital human from the multiple candidate virtual digital humans.

[0120] For example, the server performs feature extraction on the historical digital human preference to obtain the preference feature corresponding to the historical digital human preference. The server performs weighted fusion of the object feature and the preference feature to obtain the digital human screening feature of the target object. The server obtains the candidate virtual digital human description information of the multiple candidate virtual digital humans. The server performs feature extraction on the candidate virtual digital human description information of each candidate virtual digital human to obtain the candidate virtual digital human feature of each candidate virtual digital human. The server determines the first feature similarity between the digital human screening feature and the candidate virtual digital human feature of each candidate virtual digital human. The server determines the candidate virtual digital human with the highest first feature similarity among the multiple candidate virtual digital humans as the reference virtual digital human.

[0121] In the second part, the server determines a first object state of the target object based on the first object image and the first physiological data.

[0122] Among them, in the embodiment of the present application, the object state is a hidden state. The concept of hidden state is similar to the hidden state h in the long short-term memory network (LSTM), which can reflect the high-dimensional characteristics of the data. The different object states of the target object can reflect the states of the target object in different dimensions. By using these states of different dimensions to correct the existing virtual digital human, the final target virtual digital human can be obtained. In step 303, the first object state and the second object state are used to generate an initial virtual digital human.

[0123] In a possible implementation, the server determines the first expression, first posture, and clothing information of the target object based on the first object image. The server determines the first initial object state of the target object based on the first expression, first posture, and clothing information of the target object. The server modifies the first initial object state based on the first physiological data to obtain the first object state.

[0124] The first expression is used to describe the facial condition of the target object, and the first posture is used to describe the limb condition of the target object.

[0125] For example, the server performs image recognition on the first object image to obtain the first expression, first posture and dressing information of the target object. The server performs feature extraction on the first expression, first posture and dressing information of the target object to obtain the first expression feature, first posture feature and dressing feature of the target object. The server fuses the first expression feature, first posture feature and dressing feature of the target object to obtain the first state feature of the target object. The server performs multiple full connections on the first state feature of the target object to obtain the first initial object state of the target object.

[0126] For example, the server inputs the first object image into an image recognition model, and extracts features of the first object image through the image recognition model to obtain first object image features of the first object image. The server processes the first object image features through the image recognition model to obtain the first expression, first posture and dressing information of the target object. The server extracts features of the first expression, first posture and dressing information of the target object to obtain the first expression features, first posture features and dressing features of the target object. The server performs weighted fusion of the first expression features, first posture features and dressing features of the target object to obtain the first state features of the target object. The server performs multiple full connections on the first state features of the target object to obtain the first initial object state of the target object.

[0127] Among them, the weights of weighted fusion are set by technical personnel according to actual conditions, and the embodiments of the present application do not limit this.

[0128] Part three: The server determines a second object state of the target object based on the environment information and the object feature.

[0129] In a possible implementation, the environmental information includes environmental weather, environmental temperature, and environmental type, and the server determines the second initial object state of the target object based on the environmental weather and the environmental temperature. The server modifies the second initial object state based on the environmental type and the object characteristics to obtain the second object state.

[0130] The second initial object state is determined based on the ambient weather and ambient temperature, and can reflect the object state of the target object under the influence of the ambient weather and ambient temperature.

[0131] For example, the server uses the ambient weather and the ambient temperature to query in the first relationship table to obtain the second initial object state of the target object, and the first relationship table stores the correspondence between multiple ambient weathers, multiple ambient temperatures, and multiple object states. The server embeds the environment type to obtain the environment type code of the environment type. The server fuses the environment type code of the environment type with the object feature to obtain the state correction feature. The server fuses the state correction feature with the second initial object state to obtain the second object state.

[0132] The fusion includes any one of weighted fusion, addition and concatenation.

[0133] Part 4: The server updates the reference virtual digital human based on the first object state and the second object state to obtain the initial virtual digital human.

[0134] In a possible implementation, the server determines a first target object state of the target object based on the first object state and the second object state. The server obtains first digital human description information of the reference virtual digital human, where the first digital human description information is used to describe the situation of the reference virtual digital human. The server generates first update information of the reference virtual digital human based on the first target object state and the first digital human description information. The server updates the reference virtual digital human based on the first update information to obtain the initial virtual digital human.

[0135] In order to explain the above implementation more clearly, the above implementation will be further explained in several parts below.

[0136] A. The server determines a first target object state of the target object based on the first object state and the second object state.

[0137] Among them, the first object state and the second object state are both hidden states of the target object, which can reflect the states of the target object in different dimensions. The first object state focuses on the dimension of the target object itself, and the second object state focuses on the environmental dimension. The first target object state obtained based on the first object state and the second object state can be regarded as the result of fusing the first object state and the second object state.

[0138] In a possible implementation, the server uses a first weight and a second weight to merge the first object state and the second object state to obtain a first target object state of the target object, where the first weight is a weight corresponding to the first object state, and the second weight is a weight corresponding to the second object state.

[0139] For example, the server performs full connection and normalization on the first object state to obtain the first weight. The server performs full connection and normalization on the second object state to obtain the second weight. The server uses the first weight and the second weight to fuse the first object state and the second object state to obtain the first target object state of the target object.

[0140] B. The server obtains the first digital human description information of the reference virtual digital human.

[0141] In a possible implementation, the server performs a query based on the digital human identification of the reference virtual digital human to obtain the first digital human description information of the reference virtual digital human.

[0142] The reference virtual digital person is a pre-configured virtual digital person, and the first digital person description information is the pre-configured digital person description information for the reference virtual digital person. The first digital person description information is used to describe the appearance, voice and action of the reference virtual digital person, and can be regarded as the text form of the reference virtual digital person.

[0143] C. The server generates first update information of the reference virtual digital human based on the first target object state and the first digital human description information.

[0144] In a possible implementation, the server extracts features from the first digital human description information to obtain first description information features of the first digital human description information. The server generates first update information of the reference virtual digital human based on the first target object state and the first description information features.

[0145] For example, the server encodes the first digital human description information based on the attention mechanism to obtain the first description information feature of the first digital human description information. The server fuses the first target object state with the first description information feature to obtain the first update feature. The server performs multiple rounds of iterative decoding on the first update feature based on the attention mechanism to obtain the first update information of the reference virtual digital human.

[0146] D. The server updates the reference virtual digital human based on the first update information to obtain the initial virtual digital human.

[0147] In a possible implementation, the server splits the first update information to obtain multiple sub-update information, and different sub-update information is used to update different reference virtual components of the reference virtual digital person, and the reference virtual digital person is composed of multiple reference virtual components. The server generates multiple reference virtual components based on the multiple sub-update information, and one reference virtual component corresponds to one reference virtual component. The server uses each reference virtual component to replace the corresponding virtual reference virtual component in the reference virtual digital person to obtain the initial virtual digital person.

[0148] For example, the server splits the first update information according to the dimension of the virtual component to obtain multiple sub-update information. The server extracts features from each sub-update information to obtain sub-update information features of each sub-update information. The server generates reference virtual components corresponding to each sub-update information based on the sub-update information features of each sub-update information. The server uses each of the reference virtual components to replace the corresponding virtual reference virtual components in the reference virtual digital human to obtain the initial virtual digital human.

[0149] Among them, the method of generating the reference virtual component corresponding to each sub-update information based on the sub-update information features of each sub-update information is equivalent to a sequence-to-sequence process, that is, the sub-update information is regarded as an input sequence, and the coordinates of multiple key points in the reference virtual component are regarded as an output sequence. The generation of the reference virtual component based on the sub-update information features can be achieved through a multimodal large language model. The embodiment of the present application does not limit the type and structure of the multimodal large language model.

[0150] 304. The server displays the initial virtual digital human.

[0151] The initial virtual digital human is a virtual digital human that is generated using the first object information and the environment information and that may meet the current preferences of the target object. Displaying the initial virtual digital human refers to displaying the initial virtual digital human and controlling the initial virtual digital human to speak and perform actions.

[0152] In a possible implementation, the server sends the digital human display information of the initial virtual digital human to the terminal, and the terminal displays the initial virtual digital human based on the digital human display information of the initial virtual digital human.

[0153] 305. The server obtains second object information of the target object, where the second object information includes a second object image, second physiological data, and target audio of the target object, where the target audio is audio collected during the display of the initial virtual digital human.

[0154] The second object image is an object image of the target object collected during the display of the initial virtual digital human, which can reflect the target object's preference for the initial virtual digital human. The second physiological data is the physiological data of the target object collected during the display of the initial virtual digital human, that is, it can reflect the target object's preference for the initial virtual digital human. Since the target audio is the audio collected during the display of the initial virtual digital human, the target audio can directly reflect the target object's evaluation and expectations of the initial virtual digital human.

[0155] In a possible implementation, the server sends a second information acquisition request to the terminal, the first information acquisition request being used to request acquisition of the second object information, and the server acquires the second object information of the target object returned by the terminal in response to the second information acquisition request.

[0156] 306. The server updates the initial virtual digital human based on the second object information to obtain a target virtual digital human.

[0157] Among them, the second object information can reflect the reaction of the target object after watching the display of the initial virtual digital human. By using the second object information to update the initial virtual digital human, a target virtual digital human that is closer to the needs of the target object can be obtained.

[0158] In a possible implementation, the server determines a third object state of the target object based on the second object image. The server determines a fourth object state of the target object based on the second physiological data. The server updates the initial virtual digital human based on the target audio, the third object state, and the fourth object state to obtain the target virtual digital human.

[0159] In order to explain the above implementation more clearly, the above implementation is explained in several parts below.

[0160] In the first part, the server determines a third object state of the target object based on the second object image.

[0161] In a possible implementation, the server determines the second expression and the second posture of the target object based on the second object image. The server determines the third object state of the target object based on the second expression and the second posture of the target object.

[0162] The second expression is the expression of the target object when viewing the displayed initial virtual digital human, and the second posture is the posture of the target object when viewing the displayed initial virtual digital human.

[0163] For example, the server performs image recognition on the second object image to obtain the second expression and second posture of the target object. The server performs feature extraction on the second expression and second posture of the target object to obtain the second expression feature and second posture feature of the target object. The server fuses the second expression feature and second posture feature of the target object to obtain the third object state of the target object.

[0164] In the second part, the server determines a fourth object state of the target object based on the second physiological data.

[0165] In a possible implementation, the server performs feature extraction on the second physiological data to obtain the physiological data feature of the second physiological data. The server performs multiple full connections on the physiological data feature to obtain the third object state of the target object.

[0166] Part three: The server updates the initial virtual digital human based on the target audio, the third object state and the fourth object state to obtain the target virtual digital human.

[0167] In a possible implementation, the server performs speech recognition and emotion recognition on the target audio to obtain a target text and a target emotion corresponding to the target audio. The server determines a second target object state of the target object based on the third object state and the fourth object state. The server obtains second digital human description information of the initial virtual digital human, and the second digital human description information is used to describe the situation of the initial virtual digital human. The server generates second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information. The server updates the initial virtual digital human based on the second update information to obtain the target virtual digital human.

[0168] In order to explain the above implementation more clearly, the above implementation is further explained in several parts below.

[0169] A. The server performs speech recognition and emotion recognition on the target audio to obtain the target text and target emotion corresponding to the target audio.

[0170] In a possible implementation, the server inputs the target audio into a speech recognition model, performs speech recognition on the target audio through the speech recognition model, and obtains a target text corresponding to the target audio. The server inputs the target audio into an emotion recognition model, performs speech recognition on the target audio through the emotion recognition model, and obtains a target emotion corresponding to the target audio.

[0171] Among them, both the speech recognition model and the emotion recognition model can adopt models in related technologies, and the embodiments of the present application are not limited to this.

[0172] B. The server determines a second target object state of the target object based on the third object state and the fourth object state.

[0173] In a possible implementation, the third object state and the fourth object state are both hidden states of the target object, which can reflect the states of the target object in different dimensions, the third object state focuses on the dimension of the expression and posture of the target object, and the fourth object state focuses on the physiological data of the target object. The second target object state obtained based on the third object state and the fourth object state can be regarded as the result of fusing the third object state and the fourth object state.

[0174] In one possible implementation, the server uses a third weight and a fourth weight to merge the third object state and the fourth object state to obtain a second target object state of the target object, the third weight being the weight corresponding to the third object state, and the fourth weight being the weight corresponding to the fourth object state.

[0175] For example, the server performs full connection and normalization on the third object state to obtain the third weight. The server performs full connection and normalization on the fourth object state to obtain the fourth weight. The server uses the third weight and the fourth weight to fuse the third object state and the fourth object state to obtain the second target object state of the target object.

[0176] C. The server obtains the second digital human description information of the initial virtual digital human.

[0177] In a possible implementation, the server generates second digital human description information of the initial virtual digital human based on multiple initial virtual components of the initial virtual digital human.

[0178] The initial virtual digital human is composed of the multiple initial virtual components, and different initial virtual components correspond to different parts. For example, there are initial virtual components corresponding to the eyes, initial virtual components corresponding to the mouth, initial virtual components corresponding to the arms, and so on.

[0179] For example, the server extracts features of multiple initial virtual components of the initial virtual digital person to obtain virtual component features of each initial virtual component. The server generates second digital person description information of the initial virtual digital person based on the virtual component features of each initial virtual component.

[0180] For example, the server extracts features of multiple initial virtual components of the initial virtual digital person to obtain virtual component features of each initial virtual component. The server fuses the virtual component features of each initial virtual component to obtain virtual digital human features of the initial virtual digital person. The server performs multiple rounds of iterative decoding of the virtual digital human features of the initial virtual digital person based on the attention mechanism to obtain second digital human description information of the initial virtual digital person.

[0181] D. The server generates second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information.

[0182] In a possible implementation, the server determines the target object's liking for the initial virtual digital person based on the target emotion and the second target object's state. The server generates a digital person update guide based on the target text and the second digital person description information. The server generates second update information for the initial virtual digital person based on the target object's liking for the initial virtual digital person and the digital person update guide.

[0183] For example, the server extracts features of the target emotion to obtain the emotional features of the target emotion. The server fuses the emotional features of the target emotion with the state of the second target object to obtain the likeness feature. The server fully connects and normalizes the likeness feature to obtain the likeness of the target object for the initial virtual digital person. The server encodes the target text and the description information of the second digital person based on the attention mechanism to obtain the target text features of the target text and the second description features of the description information of the second digital person. The server fuses the target text features of the target text and the second description features of the description information of the second digital person to obtain the update guidance feature. The server performs multiple rounds of iterative decoding on the update guidance feature based on the attention mechanism to obtain the digital person update guidance. The server fuses the likeness of the target object for the initial virtual digital person with the digital person update guidance to obtain the second update information of the initial virtual digital person.

[0184] E. The server updates the initial virtual digital human based on the second update information to obtain the target virtual digital human.

[0185] In a possible implementation, the server splits the second update information to obtain multiple sub-update information, and different sub-update information is used to update different initial virtual components of the initial virtual digital person, and the initial virtual digital person is composed of multiple initial virtual components. The server generates multiple target virtual components based on the multiple sub-update information, and one target virtual component corresponds to one initial virtual component. The server uses each of the target virtual components to replace the corresponding virtual initial virtual components in the initial virtual digital person to obtain the target virtual digital person.

[0186] For example, the server splits the second update information according to the dimension of the virtual component to obtain multiple sub-update information. The server extracts features from each sub-update information to obtain sub-update information features of each sub-update information. The server generates target virtual components corresponding to each sub-update information based on the sub-update information features of each sub-update information. The server replaces the corresponding initial virtual components in the initial virtual digital human with each target virtual component to obtain the target virtual digital human.

[0187] Among them, the method of generating the target virtual component corresponding to each sub-update information based on the sub-update information features is equivalent to a sequence-to-sequence process, that is, the sub-update information is regarded as an input sequence, and the coordinates of multiple key points in the target virtual component are regarded as an output sequence. Generating the target virtual component based on the sub-update information features can be achieved through a multimodal large language model. The embodiment of the present application does not limit the type and structure of the multimodal large language model.

[0188] 307. The server displays the target virtual digital human.

[0189] Among them, based on the fact that the second object information can reflect the target object's preferences, evaluations and expectations for the initial virtual digital person, therefore, by using the second object information to update the initial virtual digital person, the target object's preferences, evaluations and expectations for the initial virtual digital person can be updated in the initial virtual digital person, thereby obtaining a target virtual digital person that better meets the needs of the target object.

[0190] In a possible implementation, the server sends the digital human display information of the target virtual digital human to the terminal, and the terminal displays the target virtual digital human based on the digital human display information of the target virtual digital human.

[0191] 308. The server controls the target virtual digital human to interact with the target object.

[0192] In a possible implementation, the server controls the target virtual digital human to perform actions and / or make sounds based on the interaction information, and the interaction information is generated based on the interaction instructions of the target object.

[0193] For example, the server obtains the interaction instruction of the target object. The server extracts the semantic features of the interaction instruction to obtain the semantic features of the interaction instruction. The server generates the interaction information based on the semantic features of the interaction instruction. The server controls the target virtual digital human to perform actions and / or make sounds based on the interaction information.

[0194] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0195] Through the technical solution provided by the embodiment of the present application, in response to the target object's call request for the virtual digital human, the first object information of the target object and the environmental information of the environment in which it is located are obtained, and the initial virtual digital human is generated and displayed using the first object information and the environmental information. The second object information collected when displaying the initial virtual digital human is obtained, and the initial virtual digital human is updated using the second object information to obtain the target virtual digital human, so that the target virtual digital human is used to interact with the target object, and the target object's satisfaction with the target virtual digital human is improved, thereby improving the user experience.

[0196] Figure 4 is a structural diagram of an interactive device based on a virtual digital human provided in an embodiment of the present application, see Figure 4 The device includes: an acquisition module 401, a generation module 402, an update module 403 and a control module 404.

[0197] The acquisition module 401 is used to respond to the virtual digital human call request of the target object, obtain the first object information of the target object and the environmental information of the environment in which it is located, wherein the first object information includes the object characteristics, the first object image, the first physiological data and the historical digital human preferences of the target object, and the historical digital human preferences are used to indicate the historical preferences of the target object for different types of virtual digital humans.

[0198] The generation module 402 is used to generate and display an initial virtual digital human based on the first object information of the target object and the environment information.

[0199] The acquisition module 401 is also used to acquire the second object information of the target object, where the second object information includes the second object image, second physiological data and target audio of the target object, where the target audio is the audio collected during the display of the initial virtual digital human.

[0200] The updating module 403 is used to update the initial virtual digital human based on the second object information, and obtain and display the target virtual digital human.

[0201] The control module 404 is used to control the target virtual digital human to interact with the target object.

[0202] In a possible implementation, the generation module 402 is used to determine a reference virtual digital human from a plurality of candidate virtual digital humans based on the object feature and the historical digital human preference, wherein different candidate virtual digital humans have different images. Determine a first object state of the target object based on the first object image and the first physiological data. Determine a second object state of the target object based on the environmental information and the object feature. Update the reference virtual digital human based on the first object state and the second object state to obtain the initial virtual digital human. Display the initial virtual digital human.

[0203] In a possible implementation, the generating module 402 is configured to determine a first expression, a first posture, and dressing information of the target object based on the first object image. Determine a first initial object state of the target object based on the first expression, the first posture, and dressing information of the target object. Modify the first initial object state based on the first physiological data to obtain the first object state.

[0204] In a possible implementation, the environmental information includes environmental weather, environmental temperature, and environmental type, and the generating module 402 is used to determine the second initial object state of the target object based on the environmental weather and the environmental temperature, and to modify the second initial object state based on the environmental type and the object characteristics to obtain the second object state.

[0205] In a possible implementation, the generation module 402 is used to determine a first target object state of the target object based on the first object state and the second object state. Obtain first digital human description information of the reference virtual digital human, where the first digital human description information is used to describe the situation of the reference virtual digital human. Generate first update information of the reference virtual digital human based on the first target object state and the first digital human description information. Update the reference virtual digital human based on the first update information to obtain the initial virtual digital human.

[0206] In a possible implementation, the initial virtual digital human is updated based on the second object information to obtain and display the target virtual digital human, including:

[0207] Based on the second object image, a third object state of the target object is determined.

[0208] Based on the second physiological data, a fourth object state of the target object is determined.

[0209] Based on the target audio, the third object state and the fourth object state, the initial virtual digital human is updated to obtain the target virtual digital human.

[0210] Display the target virtual digital human.

[0211] In a possible implementation, the update module 403 is used to perform speech recognition and emotion recognition on the target audio to obtain a target text and a target emotion corresponding to the target audio. Based on the third object state and the fourth object state, a second target object state of the target object is determined. Second digital human description information of the initial virtual digital human is obtained, and the second digital human description information is used to describe the situation of the initial virtual digital human. Based on the target text, the target emotion, the second target object state, and the second digital human description information, second update information of the initial virtual digital human is generated. The initial virtual digital human is updated based on the second update information to obtain the target virtual digital human.

[0212] In a possible implementation, the update module 403 is used to determine the target object's liking for the initial virtual digital person based on the target emotion and the second target object's state. Generate a digital person update guide based on the target text and the second digital person description information. Generate second update information of the initial virtual digital person based on the target object's liking for the initial virtual digital person and the digital person update guide.

[0213] In a possible implementation, the update module 403 is used to split the second update information to obtain multiple sub-update information, and different sub-update information is used to update different initial virtual components of the initial virtual digital person, and the initial virtual digital person is composed of multiple initial virtual components. Based on the multiple sub-update information, multiple target virtual components are generated, and one target virtual component corresponds to one initial virtual component. Each of the target virtual components is used to replace the corresponding virtual initial virtual component in the initial virtual digital person to obtain the target virtual digital person.

[0214] It should be noted that: the interactive device based on virtual digital human provided in the above embodiment only uses the division of the above functional modules as an example when displaying multimedia resource information. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the interactive device based on virtual digital human provided in the above embodiment and the interactive method embodiment based on virtual digital human belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0215] Through the technical solution provided by the embodiment of the present application, in response to the target object's call request for the virtual digital human, the first object information of the target object and the environmental information of the environment in which it is located are obtained, and the initial virtual digital human is generated and displayed using the first object information and the environmental information. The second object information collected when displaying the initial virtual digital human is obtained, and the initial virtual digital human is updated using the second object information to obtain the target virtual digital human, so that the target virtual digital human is used to interact with the target object, and the target object's satisfaction with the target virtual digital human is improved, thereby improving the user experience.

[0216] The embodiment of the present application provides a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is first introduced below:

[0217] Figure 5 5 is a schematic diagram of a terminal structure provided in an embodiment of the present application. The terminal 500 may be: a smart phone, a tablet computer, a laptop computer or a desktop computer. The terminal 500 may also be called a user equipment, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0218] Typically, the terminal 500 includes: one or more processors 501 and one or more memories 502 .

[0219] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0220] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one computer program, which is used to be executed by the processor 501 to implement the interactive method based on the virtual digital human provided in the method embodiment of the present application.

[0221] In some embodiments, the terminal 500 may further optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507 and a power supply 508.

[0222] The peripheral device interface 503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0223] The radio frequency circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like.

[0224] The display screen 505 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to collect touch signals on the surface or above the surface of the display screen 505. The touch signal may be input to the processor 501 as a control signal for processing. At this point, the display screen 505 may also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0225] The camera assembly 506 is used to collect images or videos. Optionally, the camera assembly 506 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal.

[0226] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them to the processor 501 for processing, or input them to the radio frequency circuit 504 to achieve voice communication.

[0227] The power supply 508 is used to supply power to various components in the terminal 500. The power supply 508 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0228] In some embodiments, the terminal 500 further includes one or more sensors 509 , including but not limited to: an acceleration sensor 510 , a gyroscope sensor 511 , a pressure sensor 512 , an optical sensor 513 , and a proximity sensor 514 .

[0229] The acceleration sensor 510 can detect the magnitude of acceleration on three coordinate axes of the coordinate system established with the terminal 500 .

[0230] The gyro sensor 511 can detect the body direction and rotation angle of the terminal 500 . The gyro sensor 511 can cooperate with the acceleration sensor 510 to collect the user's 3D actions on the terminal 500 .

[0231] The pressure sensor 512 may be provided at the side frame of the terminal 500 and / or the lower layer of the display screen 505. When the pressure sensor 512 is provided at the side frame of the terminal 500, it may detect the user's holding signal of the terminal 500, and the processor 501 may perform left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 512. When the pressure sensor 512 is provided at the lower layer of the display screen 505, the processor 501 may control the operability controls on the UI interface according to the user's pressure operation on the display screen 505.

[0232] The optical sensor 513 is used to collect the ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the display screen 505 according to the ambient light intensity collected by the optical sensor 513.

[0233] The proximity sensor 514 is used to collect the distance between the user and the front of the terminal 500 .

[0234] Those skilled in the art will understand that Figure 5The structure shown in the figure does not constitute a limitation on the terminal 500, and the terminal 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0235] The above-mentioned computer device can also be implemented as a server. The structure of the server is introduced below:

[0236] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 601 and one or more memories 602, wherein at least one computer program is stored in the one or more memories 602, and the at least one computer program is loaded and executed by the one or more processors 601 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 600 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 600 may also include other components for implementing device functions, which will not be described in detail here.

[0237] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program, and the computer program can be executed by a processor to complete the interactive method based on a virtual digital human in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0238] In an exemplary embodiment, a computer program product or a computer program is also provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device executes the above-mentioned interaction method based on a virtual digital human.

[0239] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0240] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0241] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An interactive method based on virtual digital human, characterized in that: The method comprises: In response to a virtual digital human call request of a target object, first object information of the target object and environmental information of the environment in which the target object is located are obtained, wherein the first object information includes object features, a first object image, first physiological data, and historical digital human preferences of the target object, wherein the historical digital human preferences are used to indicate the historical preferences of the target object for different types of virtual digital humans; Based on the first object information of the target object and the environment information, generating and displaying an initial virtual digital human; Acquire second object information of the target object, where the second object information includes a second object image, second physiological data, and target audio of the target object, where the target audio is audio collected during the display of the initial virtual digital human; Based on the second object information, the initial virtual digital human is updated to obtain and display the target virtual digital human; Control the target virtual digital human to interact with the target object.

2. The method according to claim 1, characterized in that The generating and displaying an initial virtual digital human based on the first object information of the target object and the environment information includes: Based on the object characteristics and the historical digital human preferences, determining a reference virtual digital human from a plurality of candidate virtual digital humans, wherein different candidate virtual digital humans have different images; determining a first object state of the target object based on the first object image and the first physiological data; Determining a second object state of the target object based on the environmental information and the object feature; The reference virtual digital human is updated based on the first object state and the second object state to obtain the initial virtual digital human; The initial virtual digital human is displayed.

3. The method according to claim 2, characterized in that The determining a first object state of the target object based on the first object image and the first physiological data includes: Based on the first object image, determining a first expression, a first posture, and clothing information of the target object; Determining a first initial object state of the target object based on a first expression, a first posture, and clothing information of the target object; The first initial object state is modified based on the first physiological data to obtain the first object state.

4. The method according to claim 2, characterized in that: The environmental information includes environmental weather, environmental temperature, and environmental type, and determining the second object state of the target object based on the environmental information and the object feature includes: Determining a second initial object state of the target object based on the ambient weather and the ambient temperature; Based on the environment type and the object characteristics, the second initial object state is modified to obtain the second object state.

5. The method according to claim 2, characterized in that: The updating of the reference virtual digital human based on the first object state and the second object state to obtain the initial virtual digital human includes: Determining a first target object state of the target object based on the first object state and the second object state; Acquire first digital human description information of the reference virtual digital human, where the first digital human description information is used to describe the situation of the reference virtual digital human; Based on the first target object state and the first digital human description information, generate first update information of the reference virtual digital human; The reference virtual digital human is updated based on the first update information to obtain the initial virtual digital human.

6. The method according to claim 1, characterized in that The updating of the initial virtual digital human based on the second object information to obtain and display the target virtual digital human includes: determining a third object state of the target object based on the second object image; determining a fourth object state of the target object based on the second physiological data; Based on the target audio, the third object state and the fourth object state, the initial virtual digital human is updated to obtain the target virtual digital human; Display the target virtual digital human.

7. The method according to claim 6, characterized in that The updating of the initial virtual digital human based on the target audio, the third object state and the fourth object state to obtain the target virtual digital human includes: Performing speech recognition and emotion recognition on the target audio to obtain a target text and a target emotion corresponding to the target audio; determining a second target object state of the target object based on the third object state and the fourth object state; Acquire second digital human description information of the initial virtual digital human, where the second digital human description information is used to describe the situation of the initial virtual digital human; Generate second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information; The initial virtual digital human is updated based on the second update information to obtain the target virtual digital human.

8. The method according to claim 7, characterized in that The step of generating second update information of the initial virtual digital human based on the target text, the target emotion, the second target object state, and the second digital human description information includes: Based on the target emotion and the state of the second target object, determining the target object's liking for the initial virtual digital person; Generate a digital human update guide based on the target text and the second digital human description information; Based on the target object's liking for the initial virtual digital human and the digital human update guide, second update information of the initial virtual digital human is generated.

9. The method according to claim 7, characterized in that: The updating of the initial virtual digital human based on the second update information to obtain the target virtual digital human includes: Splitting the second update information to obtain a plurality of sub-update information, wherein different sub-update information is used to update different initial virtual components of the initial virtual digital human, and the initial virtual digital human is composed of a plurality of initial virtual components; Based on the plurality of sub-update information, a plurality of target virtual components are generated, wherein one of the target virtual components corresponds to one of the initial virtual components; Each of the target virtual components is used to replace the corresponding virtual initial virtual components in the initial virtual digital human to obtain the target virtual digital human.

10. An interactive system based on virtual digital human, characterized in that: The system comprises: an acquisition module, configured to respond to a virtual digital human call request of a target object and acquire first object information of the target object and environmental information of the environment in which the target object is located, wherein the first object information includes object features, a first object image, first physiological data, and historical digital human preferences of the target object, wherein the historical digital human preferences are used to indicate the historical preferences of the target object for different types of virtual digital humans; A generation module, used to generate and display an initial virtual digital human based on the first object information of the target object and the environment information; The acquisition module is further used to acquire second object information of the target object, wherein the second object information includes a second object image, second physiological data and target audio of the target object, and the target audio is the audio collected during the display of the initial virtual digital human; An updating module, used to update the initial virtual digital human based on the second object information, and obtain and display a target virtual digital human; A control module is used to control the target virtual digital human to interact with the target object.

Images