Virtual image generation method and device and storage medium

Abstract

The invention relates to a virtual image generation method and device and a storage medium. The virtual image generation method comprises the steps that in response to an object, detected by a terminal, of which a virtual image is to be generated exists, ecological information of the object is recognized, and the ecological information comprises at least one of the following information types of the object: emotion information, physical appearance information and action posture information; generating a virtual image reflecting the ecological information; and displaying the virtual image on the terminal. The personification of the virtual image is ensured, and the controllability of the generated virtual image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology, and in particular to a method, device and storage medium for generating a virtual image. Background Art

[0002] With the development of terminal technology, terminals support more and more functions. For example, terminals support generating virtual images such as artificial intelligence (AI) images on third-party applications to provide interaction with users.

[0003] However, the current generation of virtual images is mainly based on fixed templates and cannot truly interact with users. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a method, device and storage medium for generating a virtual image.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating a virtual image is provided, comprising: in response to a terminal detecting the presence of an object to be generated with a virtual image, identifying ecological information of the object, the ecological information including at least one of the following information types of the object: emotional information, physical appearance information, and motion posture information; generating a virtual image embodying the ecological information; and displaying the virtual image on the terminal.

[0006] In one embodiment, the generating of a virtual image reflecting the ecological information includes: determining a virtual image template that matches the object; based on the information type included in the ecological information, adding ecological information corresponding to the information type to the virtual image template to obtain a virtual image reflecting the ecological information.

[0007] In one embodiment, based on the information type included in the ecological information, adding ecological information corresponding to the information type to the virtual image template to obtain a virtual image embodying the ecological information includes: in response to image information included in the ecological information, facial key points and / or body key points in the image information representing the physical appearance information, generating a first image including body movements and a second image including facial expressions based on the facial key points and / or the body key points, and fusing the first image and the second image in the virtual image template to obtain a virtual image embodying the physical appearance information; and / or

[0008] In response to the ecological information including emotional information, determining a third image matching the emotional information from a gallery, fusing the third image into the virtual image template, and obtaining a virtual image embodying the emotional information; and / or

[0009] In response to the action posture information included in the ecological information, a video matching the key point changes is generated based on the key point change information representing the action posture changes in the action posture information, and the situation of the virtual image in the virtual image template is adjusted in the video corresponding to the key point changes to generate a virtual image reflecting the action posture changes.

[0010] In one embodiment, the ecological information includes emotional information, and the identifying of the ecological information of the object includes: acquiring image information and audio information of the object; obtaining an emotional state representing the emotional information of the object based on the emotional information respectively identified from the image information and audio information; determining a probability value for the emotional state to correspond to at least two types of emotional information, and in response to the probability value satisfying a condition, inputting the image information and audio information of the object into an emotional information classification model to obtain the situation information emotional information of the object; the emotional information classification model is used to identify the at least two types of emotional information.

[0011] In one embodiment, the virtual image template is generated in the following manner: identifying first scene information and converting the first scene information into text features; obtaining a base image provided by a user and determining an image style selected by the user; based on a text-to-image generation model, adding the first scene information to the base image, and performing stylization processing based on the image style selected by the user to obtain multiple initial virtual image templates; and using the virtual image template selected by the user from the multiple initial virtual image templates as the virtual image template finally used.

[0012] In one embodiment, the method further includes: identifying second scene information and converting the second scene information into text features; based on a text-to-image generation model, converting the text features into image information, and adding them to the virtual image to generate a virtual image including the second scene information.

[0013] In one implementation, the target scene information includes first scene information or second scene information:

[0014] Identifying target scene information and converting the target scene information into text features, including:

[0015] Identifying current target scene information based on one or more sensors of the terminal;

[0016] A text feature bound to the target scene information is selected from a text feature library to obtain the text feature of the target scene information, wherein the text feature library includes text features pre-bound to scene information.

[0017] According to a second aspect of an embodiment of the present disclosure, a virtual image generating device is provided, comprising: an identification unit for identifying ecological information of an object to be generated for a virtual image in response to a terminal detecting the presence of the object, wherein the ecological information includes at least one of the following information types of the object: emotional information, physical appearance information, and motion posture information; a processing unit for generating a virtual image embodying the ecological information; and a display unit for displaying the virtual image on the terminal.

[0018] In one embodiment, the processing unit generates a virtual image that embodies the ecological information in the following manner: determining a virtual image template that matches the object; based on the information type included in the ecological information, adding ecological information corresponding to the information type to the virtual image template to obtain a virtual image that embodies the ecological information.

[0019] In one embodiment, the processing unit adds ecological information corresponding to the information type on the virtual image template based on the information type included in the ecological information to obtain a virtual image embodying the ecological information in the following manner: in response to the ecological information including image information, facial key points and / or body key points in the image information representing the physical appearance information, based on the facial key points and / or the body key points, a first image including body movements and a second image including facial expressions are generated, and the first image and the second image are merged in the virtual image template to obtain a virtual image embodying the physical appearance information; and / or

[0020] In response to the ecological information including emotional information, determining a third image matching the emotional information from a gallery, fusing the third image into the virtual image template, and obtaining a virtual image embodying the emotional information; and / or

[0021] In response to the action posture information included in the ecological information, a video matching the key point changes is generated based on the key point change information representing the action posture changes in the action posture information, and the situation of the virtual image in the virtual image template is adjusted in the video corresponding to the key point changes to generate a virtual image reflecting the action posture changes.

[0022] In one embodiment, the ecological information includes emotional information, and the recognition unit identifies the ecological information of the object in the following manner: acquiring image information and audio information of the object; obtaining an emotional state representing the emotional information of the object based on the emotional information respectively identified from the image information and audio information; determining a probability value that the emotional state corresponds to at least two types of emotional information, and in response to the probability value satisfying a condition, inputting the image information and audio information of the object into an emotional information classification model to obtain the emotional information of the object; the emotional information classification model is used to identify the at least two types of emotional information.

[0023] In one embodiment, the processing unit generates a virtual image template in the following manner: identifying first scene information and converting the first scene information into text features; obtaining a base image provided by a user and determining an image style selected by the user; based on a text-to-image generation model, adding the first scene information to the base image, and performing stylization processing based on the image style selected by the user to obtain multiple initial virtual image templates; and using the virtual image template selected by the user from the multiple initial virtual image templates as the virtual image template finally used.

[0024] In one embodiment, the recognition unit is also used to: recognize second scene information and convert the second scene information into text features; based on a text-to-image generation model, convert the text features into image information and add them to the virtual image to generate a virtual image including the second scene information.

[0025] In one embodiment, the recognition unit adopts the following method: the target scene information includes the first scene information or the second scene information, identifies the target scene information, and converts the target scene information into text features, including: identifying the current target scene information based on one or more sensors of the terminal; selecting text features bound to the target scene information in a text feature library to obtain text features of the target scene information, wherein the text feature library includes text features pre-bound to scene information.

[0026] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: generating a virtual image based on the ecological information of the object of the virtual image to be generated, ensuring that the virtual image can be personified, showing the action and emotions of the object of the virtual image to be generated, and further improving the controllability of the generated virtual image.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] Figure 1 The figure is a flowchart of a method for generating a virtual image according to an exemplary embodiment.

[0030] Figure 2 The present invention is a flowchart showing a method of generating a virtual image embodying ecological information according to an exemplary embodiment.

[0031] Figure 3 The present invention is a flowchart showing a method of identifying emotion information of an object according to an exemplary embodiment.

[0032] Figure 4 The present invention is a flowchart showing a method of generating a first scene virtual image template according to an exemplary embodiment.

[0033] Figure 5 The present invention is a flowchart showing a method of generating a second-scenario virtual image template according to an exemplary embodiment.

[0034] Figure 6 The present invention is a flowchart showing a method of generating a virtual image according to an emotional image of an object according to an exemplary embodiment.

[0035] Figure 6a The invention is a flow chart for detecting the current motion state of a user.

[0036] Figure 7 The figure is a block diagram of a virtual image generating device according to an exemplary embodiment.

[0037] Figure 8 It is a block diagram of a virtual image generating device according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0039] The virtual image generation method provided in the embodiment of the present disclosure is applied to artificial intelligence scenarios. The virtual image generation method involved in the embodiment of the present disclosure is mainly applied to the scenario of virtual image generation in a terminal device.

[0040] In the related art, in a third-party application software, a specific virtual image corresponding to the user is generated according to the specific virtual image selected by the user. For example, when the user selects a virtual image that can display emotions in the third-party application software, when the user selects a happy virtual image, the third-party application software will generate a happy virtual image corresponding to the user, even if the user is in an angry state. Or, the third-party application software contains various emotions, and the user is specified to make corresponding emotions, and then a virtual image is generated. It is impossible to generate a virtual image that matches the user's current emotional information based on the recognized information.

[0041] Moreover, in the related art, when a user needs to have multiple virtual images at the same time, different virtual images are regenerated based on an existing virtual image to meet multiple different needs. Different virtual images cannot be generated based on the existing virtual image, resulting in excessive storage space being occupied on the terminal device.

[0042] In view of this, the present disclosure provides a method for generating a virtual image, which determines the user's current emotion based on the user's current environment or current state, and then generates a virtual image that matches the user's current environment and current state.

[0043] Figure 1 FIG. 1 is a flow chart of a method for generating a virtual image according to an exemplary embodiment. Figure 1 As shown, the virtual image generation method is used in a terminal, comprising the following steps:

[0044] In step S11 , in response to the terminal detecting the existence of an object for which a virtual image is to be generated, ecological information of the object is identified.

[0045] In the disclosed embodiment, the terminal device is supported to detect the existence of an object for which a virtual image is to be generated, and to identify ecological information of the object.

[0046] Among them, the object can be understood as the image of a person to be generated as a virtual image, or the image of a pet to be generated as a virtual image, etc., and the ecological information can be understood as including at least one of the following information types of the object: emotional information, physical appearance information, and action posture information.

[0047] In step S12, a virtual image embodying ecological information is generated.

[0048] In the disclosed embodiment, a virtual image that meets the expectations of the object is generated, such as a virtual image that reflects the ecological information of the object.

[0049] Among them, the virtual image that embodies ecological information can be understood as a virtual image that embodies at least one of the following information types of the object: emotional information, physical appearance information, and action posture information.

[0050] In step S13, the virtual image is displayed on the terminal.

[0051] In the disclosed embodiment, the virtual image is displayed according to the generated virtual image, for example, it is supported to be displayed on a terminal device, or it is displayed in a third-party application software in the terminal device.

[0052] In the disclosed embodiment, the actions and emotions of the virtual image can be displayed according to the ecological information of the object, so as to generate a virtual image reflecting the ecological information of the object, thereby ensuring that the virtual image is more humanized, improving the interaction between the user and the terminal device, and improving the user experience.

[0053] In the disclosed embodiment, a virtual image that meets the expectations of the subject is determined based on the generated virtual image that embodies the ecological information.

[0054] Figure 2 FIG. 1 is a flow chart showing a method of generating a virtual image embodying ecological information according to an exemplary embodiment. Figure 2 As shown, the following steps are included:

[0055] In step S21, a virtual image template of a matching object is determined.

[0056] In the disclosed embodiment, a basic virtual image, such as a virtual image template, is generated before generating the virtual image.

[0057] The virtual image template may be understood as a basic virtual image that is applicable to all virtual images of an object and has been adjusted.

[0058] In step S22, based on the information type included in the ecological information, the ecological information corresponding to the information type is added to the virtual image template to obtain a virtual image that reflects the ecological information.

[0059] In the disclosed embodiment, based on the obtained virtual image template, ecological information corresponding to at least one information type in the object's ecological information is obtained, the object's ecological information under the information type is added, and a virtual image that can reflect the ecological information is obtained based on the object's ecological information.

[0060] In the disclosed embodiment, according to the virtual image template, a virtual image is generated based on the technology of the virtual image template according to at least one item of ecological information corresponding to the information type included in the ecological information reflected by the object, ensuring that the virtual image can be generated based on the virtual image template in different scenarios.

[0061] In the disclosed embodiment, a virtual image is generated on a virtual image template in different ways according to the type of information included in the ecological information, thereby ensuring that virtual images of multiple different types of objects are generated based on a virtual template of an object, thereby reducing excessive storage space occupied on the terminal device.

[0062] In an embodiment of the present disclosure, a virtual image is generated based on key points in image information.

[0063] In the disclosed embodiment, ecological information containing an object is detected as image information, key points in the current image information are identified, and key points in the image information are determined to be facial key points and / or body key points. Based on the obtained facial key points and / or body key points, a first image including body movements and a second image including facial expressions are generated, and the first image and the second image are fused on the basis of the virtual image to obtain a virtual image that reflects the physical appearance information.

[0064] Among them, the image information can be understood as an image containing emotional information that can determine the object, the first image can be understood as an image that contains the object's body movements and the key points of the body movements can be identified based on the object's body movements, and the second image can be understood as an image that contains the object's facial expressions and the key points of the facial expressions can be identified based on the object's facial expressions.

[0065] In an embodiment of the present disclosure, a virtual image is generated according to emotional information.

[0066] In the disclosed embodiment, the ecological information containing the object is detected as emotional information, the obtained emotional information is converted into corresponding text information in the gallery, and based on the corresponding text information in the gallery, a third image in the gallery that matches the emotional information is determined, and the third image is integrated into the virtual image template to obtain a virtual image that embodies the emotional information.

[0067] The third image can be understood as a virtual image corresponding to the emotional information of the object in the gallery.

[0068] In an embodiment of the present disclosure, a virtual image is generated according to the object's motion posture information.

[0069] In the disclosed embodiments, motion posture information of an object is detected, key points in the motion posture information of the object are obtained, and a virtual image that can reflect the emotion information matched according to the motion posture of the object is determined in a gallery based on the key point change information representing the motion posture change in the motion posture information, and a video matching the key point changes is generated, or an image matching the key point changes is generated, and the virtual image in the virtual image template is adjusted according to the key point changes in the video or the picture, so as to generate a virtual image that reflects the emotion of the object according to the motion posture changes.

[0070] In the disclosed embodiments, a virtual image is generated according to different methods, such as obtaining the object's facial key points, and / or body key points, or current action posture, or emotional information to obtain the desired emotional state of the virtual image. This helps the resulting virtual image to better fit the object's current emotional characteristics. For example, actions that match the current emotion are added to the virtual image based on the body key points and current action posture, making the virtual image more humanized.

[0071] In the disclosed embodiment, ecological information includes emotional information.

[0072] Figure 3 FIG. 1 is a flow chart showing a method of identifying emotional information of an object according to an exemplary embodiment. Figure 3 As shown, the following steps are included:

[0073] In step S31, image information or audio information of the object is acquired.

[0074] In the embodiment of the present disclosure, the information that can obtain the emotional information of the object is obtained, such as obtaining the graphic information of the object or the audio information of the object.

[0075] The audio information may be understood as audio that can identify the emotional state of the subject.

[0076] In step S32, based on the emotion information respectively identified from the image information and the audio information, an emotion state representing the emotion information of the object is obtained.

[0077] In the embodiment of the present disclosure, emotional information of an object is identified based on its image information or audio information. When the acquired information is image information of the object, key point information contained in the graphic information is converted into text information and matched with text information corresponding to the emotional information in the gallery to obtain an emotional state that characterizes the emotional information of the object.

[0078] When the acquired information is audio information of the object, the information of the identifiable emotional state of the object contained in the audio information is converted into text information and matched with the text information of the corresponding emotional information in the gallery to obtain the emotional state representing the emotional information of the object.

[0079] In the case where the acquired information is the image information and audio information of the object, the emotional information obtained based on the user image information and the emotional information obtained based on the user audio information are combined to obtain the emotional state representing the emotional information of the object.

[0080] In step S33, the probability value that the emotional state corresponds to at least two kinds of emotional information is determined, and in response to the probability value satisfying a condition, the image information or audio information of the object is input into an emotional information classification model to obtain the emotional information of the object.

[0081] In an embodiment of the present disclosure, when it is detected that the user's emotion is in an easily confused emotional state, the probability value of the emotional state corresponding to at least two kinds of emotional information is determined. When the obtained probability values ​​of at least two kinds of emotional information meet certain conditions, the image information or audio information of the object is input into the emotional information classification model to obtain the emotional information of the object.

[0082] Among them, the emotional information classification model can be understood as a model for identifying at least two types of emotional information, and the probability value satisfaction condition can be understood as, for at least two emotional information that are difficult to distinguish in the obtained audio information, the probabilities of the two emotional information are obtained according to the training model of the distinguishable audio state. When the sum and difference of the probabilities of the two emotional information meet the corresponding thresholds, for example, the sum of the probabilities is greater than 0.8 and the difference is less than 0.2, the two emotional information are distinguished separately.

[0083] In the disclosed embodiment, the emotional state of the object is obtained according to the image information or audio information, which increases the method of obtaining the emotional state of the object, and facilitates the terminal device to be more accurate and detailed when generating a virtual image.

[0084] Figure 4 FIG. 1 is a flowchart showing a method of generating a first scene virtual image template according to an exemplary embodiment. Figure 4 As shown, the following steps are included:

[0085] In step S41, first scene information is identified and converted into text features.

[0086] In the disclosed embodiment, initial scene information is identified and named as first scene information, and the first scene information is converted into text features.

[0087] Among them, the first scene information can be understood as the scene information in which the object is located when the virtual image is pre-set. The scene information can be picture scene information generated according to the needs of the current object, or video scene information generated according to the action posture information of the current object.

[0088] In step S42, the base image provided by the user is obtained, and the image style selected by the user is determined.

[0089] In the disclosed embodiment, a base image of an object required by a terminal device provided by a user is obtained, and an image style that meets the user's expectations is determined, such as determining the image style according to the user's preferences.

[0090] Among them, the base image can be understood as a template of a virtual image initially generated based on the photo of the object provided by the user, and the image style can be understood as including images of multiple different styles, such as watercolor, oil painting, graffiti, illustration, Ukiyo-e, realism, Hayao Miyazaki and other styles.

[0091] In step S43, based on the text-to-image generation model, the first scene information is added to the base image, and stylization processing is performed based on the image style selected by the user to obtain multiple initial virtual image templates.

[0092] In the disclosed embodiment, the converted text features are converted into corresponding virtual scenes according to the text features converted from the first scene information, and stylized according to the image style selected by the user to obtain multiple initial virtual image templates under the same scene information.

[0093] In step S44, the avatar template selected by the user from the plurality of initial avatar templates is used as the avatar template to be finally used.

[0094] In the disclosed embodiment, based on the generated multiple initial virtual image templates that meet the requirements, the user selects at least one virtual image template from the multiple initial virtual image templates obtained, and uses the template as the virtual image template ultimately used in a display interface of the terminal device.

[0095] In the disclosed embodiments, first scene information is pre-set based on the environment in which the object for which the virtual image is to be generated is located, that is, the scene in which the user is located when setting the virtual image for the first time is intelligently identified. The first scene information can be used as default scene information when the terminal device displays the virtual image again, or, the current scene information, to ensure that the virtual image is generated for the first time based on the object, or, the object has no intention to change the scene characteristics, or, when the virtual image is generated again, the corresponding scene cannot be identified.

[0096] Figure 5 FIG. 1 is a flow chart showing a method of generating a second scene virtual image template according to an exemplary embodiment. Figure 5 As shown, the following steps are included:

[0097] In step S51, second scene information is identified and converted into text features.

[0098] In the disclosed embodiment, the scene information of the object for which the virtual image is to be generated is currently located is identified, and the current scene information is named as the second scene information, and the second scene information is converted into text features.

[0099] The second scene information may be understood as scene information generated according to the current scene. If the currently detected scene is the same as the pre-set scene information, the second scene information may be further understood as the first scene information.

[0100] In step S52, based on the text-to-image generation model, the text features are converted into image information and added to the virtual image to generate the virtual image including the second scene information.

[0101] In the disclosed embodiment, according to the text features converted from the second scene information, the converted text features are converted into corresponding virtual scenes, and the scene is added to the virtual image to obtain the virtual image under the second scene information.

[0102] In the disclosed embodiment, the first scene information is replaced with the current scene information according to the current scene of the object for which the virtual image is to be generated, thereby ensuring the controllability of the scene in which the virtual image is located, and automatically generating any image in combination with a specific scene, thereby ensuring that the virtual image is more vivid and expressive.

[0103] In the disclosed embodiment, the target scene information includes first scene information or second scene information.

[0104] Among them, identifying the target scene information and converting the target scene information into text features can be understood as identifying the current target scene information based on one or more sensors of the terminal; selecting text features bound to the target scene information in the text feature library to obtain the text features of the target scene information, and the text feature library includes text features pre-bound to the scene information.

[0105] In the disclosed embodiment, the scene information of the scene in which the object is located is converted into text information, and the text information is matched with the scene information corresponding to the text in the text feature library to ensure that the scene information when the object generates a virtual image is also virtual scene information.

[0106] Figure 6 FIG. 1 is a flow chart showing a method of generating a virtual image according to an emotional image of an object according to an exemplary embodiment. Figure 6 As shown, the following steps are included:

[0107] In step S61, the scene in which the user is located is identified, and relevant information of the current scene is obtained.

[0108] In one implementation, the raw data is preprocessed to generate various features or directly input into a neural network to determine the current scene. For example, the current scene is determined by sensor data, such as an accelerometer plus a gyroscope, GPS, a barometer, and audio, a camera, etc.

[0109] Among them, the relevant information of the current scene can be understood as the actions displayed by the user in scenes such as indoors and outdoors, walking, cycling, running, stairs, elevators, climbing, driving, buses, subways, high-speed trains, airplanes, pockets, etc., such as sitting still, lying down, sports including various ball games, swimming, dancing, etc.

[0110] Figure 6a is a flow chart for detecting the user's current motion status. Figure 6a As shown, the following steps are included:

[0111] In one embodiment, the pre-processing stage includes:

[0112] In step S6a1, accelerometer data is imported.

[0113] In step S6a2, gyroscope data is imported.

[0114] In one implementation, the neural network first preprocesses the imported accelerometer and gyroscope data, determines the posture of the terminal device through the gyroscope data, and thereby decomposes the constant gravitational acceleration into data in different dimensional coordinate axes, such as decomposing the constant gravitational acceleration of 9.8m / s^2 into data in the three axes of x, y, and z.

[0115] In step S6a3, the linear acceleration is obtained.

[0116] In one implementation, according to the decomposed constant gravitational acceleration, gravity is subtracted from the gravitational acceleration to obtain linear accelerations of different dimensions, such as obtaining linear accelerations of three axes.

[0117] Among them, linear acceleration can be understood as multiple data, such as 120 data, are input into the current neural network each time, and multiple zeros, such as 8 zeros, are added after the data. The purpose is to facilitate the Fourier transform calculation of frequency domain information.

[0118] In step S6a4, pre-processing is performed.

[0119] In one implementation, the imported accelerometer data and the imported gyroscope data are preprocessed to obtain data values ​​that are required to be applied later.

[0120] In step S6a5, the original data.

[0121] In one implementation, the current neural network uses the raw data of the linear acceleration and gyroscope before being fed into the next layer of the neural network.

[0122] In step S6a6, features are calculated.

[0123] In one implementation, the preprocessed data value is calculated to obtain two module values, and then the module values ​​are used to perform feature calculation to obtain the feature value of the data.

[0124] Among them, the characteristic values ​​of the data can be understood as 21*2, including 42 characteristic values ​​including mean, standard deviation, maximum value, minimum value, range, quartile, AC component (current value - average value), variance, kurtosis, skewness, root mean square, mean crossing rate, zero crossing rate, slope sign change, energy, spectral centroid, spectral spread, spectral flatness, spectral roll-off, spectral peak, and spectral kurtosis.

[0125] In one implementation, based on the user's current state, the data from the accelerometer and gyroscope are first preprocessed and imported into the neural network, the posture of the terminal device is determined by the data from the gyroscope, and the constant gravitational acceleration is decomposed into data of three coordinate axes based on the posture of the terminal device. Gravity is subtracted from the acceleration to obtain the linear acceleration of the three axes.

[0126] In one implementation, before importing into the next layer of neural network, the current neural network uses the raw data of the linear acceleration and the gyroscope to first calculate two modulus values, and then obtain the eigenvalues ​​of the data through the modulus values.

[0127] For the convenience of calculation, 8 zeros are added to the acquired data to facilitate the Fourier transform calculation of frequency domain information.

[0128] In one implementation, the DNN model 1 stage includes:

[0129] In step S6a7, DNN1.

[0130] In one embodiment, the architecture of the neural network is a nested model mechanism, which first puts the data feature values, such as the 42 feature values ​​obtained, into a one-layer deep neural network model to make a binary classification judgment on whether the current state is a relatively static state or a moving state.

[0131] Among them, the relatively static state includes static, bus, car, train, high-speed rail, airplane, and the moving state includes walking, running, bicycle, electric car, electric scooter, and motorcycle. Because the movement of some vehicles and moving states is closer to uniform speed, the accelerometer produces obvious characteristics, so this classification is made. After classification, each is imported into the next layer of neural network model to determine the specific movement mode.

[0132] Among them, DNN can be understood as a deep neural network, a multi-layer unsupervised neural network, and uses the output features of the previous layer as the input of the next layer for feature learning. After layer-by-layer feature mapping, the features of the existing spatial samples are mapped to another feature space, so as to learn better feature expression for the existing input.

[0133] In step S6a8, the model is moved.

[0134] In one implementation, it is determined that the current user is in motion based on a deep neural network model.

[0135] In step S6a9, DNN2.

[0136] In one implementation, if it is a motion state, the above 42 features are imported into a one-layer deep neural network model (DNN model 2). Since the acceleration and gyroscope data change greatly in the motion state, the current state can be judged based on the obtained feature values.

[0137] In step S6a10, the current motion state is obtained.

[0138] In one embodiment, the current motion state includes, but is not limited to, walking, running, riding a bicycle, an electric vehicle, an electric scooter, a motorcycle, etc.

[0139] In step S6a11, the model is stopped.

[0140] In one implementation, it is determined based on a deep neural network model that the current user is in a stationary state.

[0141] In one implementation, the CNN model 1 includes:

[0142] Among them, CNN can be understood as a convolutional neural network.

[0143] In step S6a12, convolutional layer.

[0144] In one implementation, the convolutional layer can be understood as a component of a convolutional neural network.

[0145] In step S6a13, a maximum pooling layer is used.

[0146] In one implementation, the maximum pooling layer can be understood as a component of a convolutional neural network.

[0147] In one implementation, the six-axis raw data is firstly imported into a convolutional neural network model as a labeled data, each layer of the model is composed of a convolutional layer and a maximum pooling layer, and there are three such layers. The model stops after the last layer, and finally obtains the characteristic value.

[0148] In step S6a14, features are added.

[0149] In one implementation, the eigenvalues ​​calculated in the convolutional neural network and the partial eigenvalues ​​obtained in the static model, such as 21 eigenvalues ​​in the static model, are added.

[0150] In step S6a15, the process is applied.

[0151] In one implementation, application processing can be understood as putting the features and some of the feature values ​​calculated previously, such as 21 feature values ​​representing static, into a layer-by-layer LSTM model, and then passing through a fully connected layer and a softmax layer to finally obtain the current state.

[0152] In step S6a16, the current stationary state is obtained.

[0153] In one implementation, the current stationary state includes, but is not limited to, a bus, a car, a train, a high-speed train, an airplane, etc.

[0154] In one implementation, after the terminal device is turned on, sensor data is acquired at a certain frequency, such as 20 Hz, and new preset value data, such as 20 pieces of data, replace the preset value data a certain time ago, such as 6 seconds, every certain time, such as 1 second, to ensure that the data in the model is the latest required data, such as 120 pieces of data, so as to calculate the current state.

[0155] In one implementation, the model only uses accelerometer and gyroscope data, so power consumption is low; and the model has many judgment scenarios, so as long as it is within the existing scenarios, the possibility of misjudgment is also reduced.

[0156] In step S62, a vocabulary model is designed.

[0157] In one implementation, a vocabulary model is designed that can establish a correspondence between scenes and texts.

[0158] In one implementation, keywords that are more suitable for the model are found, and the vocabulary is further correlated with the required keywords, the scenes and actions that can be recognized by the terminal device are included in the vocabulary, and each word is bound by correlation, so as to design a neural network.

[0159] Among them, the keywords can be understood as, for example, indoor and sofa, table, bed, chair and other furniture are bound together, riding and bicycle, gloves, helmet, outdoor scenes are bound together, and finally a model of the correlation between objects is obtained, which is the correlation between words.

[0160] In one embodiment, each time a scene and / or an action is obtained, if some other associated features can be identified, such as mountain climbing plus the date can obtain suitable plant species, and the time can obtain a suitable light background, the obtained associated features are added to the current keyword.

[0161] In one implementation, after some features (including scenes and actions) are calculated through the model, some other features are selected based on the obtained features. For example, relevant features are extracted for each clearly identified keyword, and the maximum and minimum numbers of extracted features with correlations greater than 0.5 and less than 0.5 are set respectively, and the sum of similarities needs to meet a certain threshold, and then features are randomly extracted to form a random keyword.

[0162] In step S63, an avatar model is generated.

[0163] In one embodiment, the method for generating a virtual image model is: let the user provide a photo of a person or pet as a base image, select a style that meets the user's expectations from a variety of styles, such as watercolor, oil painting, graffiti, illustration, Ukiyo-e, realism, Miyazaki style, etc., fine-tune the model according to the obtained base image and the selected style, and obtain the current scene features, generate multiple models that meet the user's requirements according to the current scene features, and then generate a virtual image.

[0164] Among them, model fine-tuning can be understood as the model fine-tuning method based on HyperDreamBooth and LoRA, such as using HyperDreamBooth to fine-tune the image generation base model so that it can generate the image of a fixed person, or a fixed pet, and using LoRA for stylized adjustment.

[0165] Acquiring the current scene feature can be understood as acquiring the current scene feature and obtaining key information of a text type.

[0166] In step S64, emotions, facial expressions, and character postures are added to the avatar model.

[0167] In one implementation, when image information of the current object is available, facial key points or body key points are attempted to be acquired based on the acquired image information of the current object.

[0168] In one embodiment, when the current facial key points of the object are obtained, the current emotional characteristics of the object are obtained based on the facial key point information of the object, and the current emotional characteristics of the object are used as text input, and the text is input into the fine-tuned base model to obtain a complete picture containing the object's body movements.

[0169] In one embodiment, when the current body key points of the object are obtained, the body key point information of the object is obtained by adopting a posture recognition algorithm, and the emotional characteristics of the virtual image obtained based on the body key points are filled into the fine-tuned base model, thereby obtaining a complete picture of the facial emotions obtained based on the object's body movements.

[0170] In one implementation, a virtual image obtained according to the current object image information is decoded, and the decoded virtual image is encoded again to obtain a complete virtual image including facial expressions.

[0171] In one embodiment, when the facial key points and body key points of the current object are obtained, the information corresponding to the facial key points and body key points of the current object is converted into text information, and the corresponding virtual image is selected from the emotion library according to the text information to generate the virtual image.

[0172] In one embodiment, when the image information of the current object cannot be obtained and only the emotional information of the current object is available, for each emotion, a matching emotion is selected from the emotion library and matched with the emotional information of the current object, and a virtual image matching the emotional information of the current object is randomly matched with a virtual image model to obtain a virtual image with emotion.

[0173] In step S65, if the user does not need dynamic, a picture is generated.

[0174] In one implementation, when the user does not need dynamics, a virtual image image that meets the user's expectations is generated.

[0175] In step S66, if the user requires dynamic video, a video is generated.

[0176] In one implementation, when the user requires dynamics, a virtual image video that meets the user's expectations is generated.

[0177] In one implementation, an encoding neural network is constructed that can calculate hidden variables that conform to a certain probability model, such as a Transformer neural network. This neural network is only used in the training process. A decoding neural network is constructed that can calculate image information through hidden variables, such as a Transformer neural network. Based on the constructed neural network, a Variational Auto-Encoder (VAE) network is trained.

[0178] In one implementation, the parameters of the VAE are fixed and a diffusion network is trained one by one.

[0179] In one implementation, z' is obtained by adding noise of a certain probability distribution to z at a certain threshold number of times, such as 2000 times.

[0180] The latent variable of the changed probability model is calculated by the neural network with the last added noise, and the noise is subtracted to get the z' of the second to last step, and so on. After a certain threshold number of times, such as 2000 times, the predicted value of the latent variable of the original probability model is obtained.

[0181] At this time, a diffusion model is obtained.

[0182] Among them, z can be understood as a latent variable that conforms to the probability model, and z' can be understood as a latent variable with a lot of noise added.

[0183] In one implementation, the parameters of the VAE are fixed and the diffusion network is adjusted to generate the video.

[0184] In one implementation, several consecutive frames of a video are used as training data. At this time, the latent variable of a certain probability model has a time dimension. At this time, the latent variable of a certain probability model has one more dimension, from the original z = [batch, channel, height, width] to z_time = [batch, channel, height, width, time].

[0185] Among them, z_time can be understood as a continuous latent variable on the time axis.

[0186] In one implementation, VAE generates multiple latent variables of a certain probability model for multiple frames of data, which are combined into z_time

[0187] In one implementation, z_time' is calculated by a neural network as the last added e_time, and e_time is subtracted to obtain the penultimate z_time', and so on. After a certain threshold number of times, such as 2000 times, the predicted value of the original z_time is obtained, and z_time is predicted by argmin(z_time'-z_time), and the video diffusion model is obtained.

[0188] Among them, z_time' can be understood as the predicted value of the continuous latent variable on the time axis obtained by calculating the prediction, and e_time can be understood as the noise calculated by multiple frames is spliced ​​in the time dimension to obtain the continuous noise prediction value on the time axis.

[0189] Based on the same concept, the embodiment of the present disclosure also provides a virtual image generating device.

[0190] It is understandable that in order to realize the above functions, the virtual image generation device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0191] Figure 7 is a block diagram of a virtual image generation device according to an exemplary embodiment. Figure 7 The device 100 includes a recognition unit 101, a processing unit 102 and a display unit 103.

[0192] The recognition unit 101 is used to identify ecological information of the object in response to the terminal detecting the existence of an object for which a virtual image is to be generated, the ecological information including at least one of the following information types of the object: emotional information, physical appearance information, and motion posture information; the processing unit 102 is used to generate a virtual image that embodies the ecological information; the display unit 103 is used to display the virtual image on the terminal.

[0193] In one embodiment, the processing unit 102 generates a virtual image embodying ecological information in the following manner: determining a virtual image template of a matching object; based on the information type included in the ecological information, adding ecological information corresponding to the information type to the virtual image template to obtain a virtual image embodying the ecological information.

[0194] In one embodiment, the processing unit 102 adds ecological information corresponding to the information type on the virtual image template based on the information type included in the ecological information to obtain a virtual image embodying the ecological information in the following manner: in response to the image information included in the ecological information, facial key points and / or body key points representing the physical appearance information in the image information, based on the facial key points and / or body key points, a first image including body movements and a second image including facial expressions are generated, and the first image and the second image are merged in the virtual image template to obtain a virtual image embodying the physical appearance information; and / or

[0195] In response to the ecological information including emotional information, determining a third image matching the emotional information from the image library, fusing the third image into the virtual image template, and obtaining a virtual image embodying the emotional information; and / or

[0196] In response to the action posture information included in the ecological information, a video matching the key point changes is generated based on the key point change information representing the action posture changes in the action posture information, and the situation of the virtual image in the virtual image template is adjusted corresponding to the key point changes in the video to generate a virtual image reflecting the action posture changes.

[0197] In one embodiment, the ecological information includes emotional information, and the recognition unit 101 recognizes the ecological information of the object in the following manner: acquiring image information and audio information of the object; obtaining an emotional state representing the emotional information of the object based on the emotional information respectively recognized from the image information and the audio information; determining a probability value corresponding to at least two emotional information of the emotional state, and in response to the probability value satisfying a condition, inputting the image information and audio information of the object into an emotional information classification model to obtain the emotional information of the object; the emotional information classification model is used to recognize at least two emotional information.

[0198] In one embodiment, the processing unit 102 generates a virtual image template in the following manner: identifying first scene information and converting the first scene information into text features; obtaining a base image provided by a user and determining an image style selected by the user; based on a text-to-image generation model, adding the first scene information to the base image, and performing stylization processing based on the image style selected by the user to obtain multiple initial virtual image templates; and using a virtual image template selected by the user from the multiple initial virtual image templates as the virtual image template finally used.

[0199] In one embodiment, the recognition unit 101 is further used to: recognize second scene information and convert the second scene information into text features; based on a text-to-image generation model, convert the text features into image information and add them to the virtual image to generate a virtual image including the second scene information.

[0200] In one implementation, the recognition unit 101 uses the following method: the target scene information includes the first scene information or the second scene information, identifies the target scene information, and converts the target scene information into text features, including: identifying the current target scene information based on one or more sensors of the terminal; selecting text features bound to the target scene information in a text feature library to obtain text features of the target scene information, wherein the text feature library includes text features pre-bound to scene information.

[0201] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0202] Figure 82 is a block diagram of a device 200 for generating a virtual image according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0203] Reference Figure 8 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0204] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 202 may include one or more modules to facilitate the interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.

[0205] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0206] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0207] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0208] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0209] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0210] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0211] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0212] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0213] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0214] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0215] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0216] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0217] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0218] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0219] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0220] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A method for generating a virtual image, characterized in that: include: In response to the terminal detecting the existence of an object for which a virtual image is to be generated, identifying ecological information of the object, the ecological information including at least one of the following information types of the object: emotional information, physical appearance information, and action posture information; generating a virtual image embodying the ecological information; The virtual image is displayed on the terminal.

2. The method according to claim 1, characterized in that The generating of a virtual image reflecting the ecological information includes: determining a virtual image template matching the object; Based on the information type included in the ecological information, the ecological information corresponding to the information type is added to the virtual image template to obtain a virtual image that embodies the ecological information.

3. The method according to claim 2, characterized in that The adding ecological information corresponding to the information type to the virtual image template based on the information type included in the ecological information to obtain a virtual image reflecting the ecological information includes: In response to the ecological information including image information, facial key points and / or body key points in the image information representing the physical appearance information, generating a first image including body movements and a second image including facial expressions based on the facial key points and / or the body key points, and fusing the first image and the second image in the virtual image template to obtain a virtual image that embodies the physical appearance information; and / or In response to the ecological information including emotional information, determining a third image matching the emotional information from a gallery, fusing the third image into the virtual image template, and obtaining a virtual image embodying the emotional information; and / or In response to the action posture information included in the ecological information, a video matching the key point changes is generated based on the key point change information representing the action posture changes in the action posture information, and the situation of the virtual image in the virtual image template is adjusted in the video corresponding to the key point changes to generate a virtual image reflecting the action posture changes.

4. The method according to any one of claims 1 to 3, characterized in that The ecological information includes emotional information, and the identifying ecological information of the object includes: Acquiring image information or audio information of the object; Based on the emotion information respectively identified from the image information or the audio information, obtaining an emotion state representing the emotion information of the object; Determine a probability value that the emotional state corresponds to at least two types of emotional information, and in response to the probability value satisfying a condition, input the image information or audio information of the object into an emotional information classification model to obtain the emotional information of the object; The emotion information classification model is used to identify the at least two emotion information.

5. The method according to claim 2, characterized in that: The virtual image template is generated in the following manner: Identifying first scene information, and converting the first scene information into text features; Obtain the base image provided by the user and determine the image style selected by the user; Based on the text-to-image generation model, the first scene information is added to the base image, and stylized based on the image style selected by the user to obtain multiple initial virtual image templates; The avatar template selected by the user from the plurality of initial avatar templates is used as the avatar template finally used.

6. The method according to claim 1, characterized in that The method further comprises: identifying second scene information, and converting the second scene information into text features; Based on a text-to-image generation model, the text features are converted into image information and added to the virtual image to generate a virtual image including the second scene information.

7. The method according to claim 5 or 6, characterized in that: The target scene information includes the first scene information or the second scene information: Identifying target scene information and converting the target scene information into text features, including: Identifying current target scene information based on one or more sensors of the terminal; A text feature bound to the target scene information is selected from a text feature library to obtain the text feature of the target scene information, wherein the text feature library includes text features pre-bound to scene information.

8. A virtual image generating device, characterized in that: include: an identification unit, configured to identify ecological information of an object to be generated as a virtual image in response to the terminal detecting that the object exists, wherein the ecological information includes at least one of the following information types of the object: emotional information, physical appearance information, and action posture information; A processing unit, configured to generate a virtual image embodying the ecological information; A display unit is used to display the virtual image on the terminal.

9. The virtual image generation device according to claim 8, characterized in that: The processing unit generates a virtual image that reflects the ecological information in the following manner: determining a virtual image template matching the object; Based on the information type included in the ecological information, the ecological information corresponding to the information type is added to the virtual image template to obtain a virtual image that embodies the ecological information.

10. The virtual image generation device according to claim 9, characterized in that: The processing unit adds the ecological information corresponding to the information type to the virtual image template based on the information type included in the ecological information to obtain a virtual image reflecting the ecological information in the following manner: In response to the ecological information including image information, facial key points and / or body key points in the image information representing the physical appearance information, generating a first image including body movements and a second image including facial expressions based on the facial key points and / or the body key points, and fusing the first image and the second image in the virtual image template to obtain a virtual image embodying the physical appearance information; and / or In response to the ecological information including emotional information, determining a third image matching the emotional information from a gallery, fusing the third image into the virtual image template, and obtaining a virtual image embodying the emotional information; and / or In response to the action posture information included in the ecological information, a video matching the key point changes is generated based on the key point change information representing the action posture changes in the action posture information, and the situation of the virtual image in the virtual image template is adjusted in the video corresponding to the key point changes to generate a virtual image reflecting the action posture changes.

11. The virtual image generation device according to any one of claims 8 to 10, characterized in that: The identification unit identifies the ecological information of the object in the following manner: Acquire image information and audio information of the object; obtain an emotional state representing the emotional information of the object based on the emotional information respectively identified from the image information and the audio information; Determine a probability value that the emotional state corresponds to at least two types of emotional information, and in response to the probability value satisfying a condition, input the image information and audio information of the object into an emotional information classification model to obtain the emotional information of the object; The emotion information classification model is used to identify the at least two emotion information.

12. The virtual image generation device according to claim 9, characterized in that: The processing unit generates a virtual image template in the following manner: Identifying first scene information, and converting the first scene information into text features; Obtain the base image provided by the user and determine the image style selected by the user; Based on the text-to-image generation model, the first scene information is added to the base image, and stylized based on the image style selected by the user to obtain multiple initial virtual image templates; The avatar template selected by the user from the plurality of initial avatar templates is used as the avatar template finally used.

13. The virtual image generating device according to claim 8, characterized in that: The identification unit is also used for: identifying second scene information, and converting the second scene information into text features; Based on a text-to-image generation model, the text features are converted into image information and added to the virtual image to generate a virtual image including the second scene information.

14. The virtual image generation device according to claim 12 or 13, characterized in that: The recognition unit adopts the following method: the target scene information includes the first scene information or the second scene information: Identifying target scene information and converting the target scene information into text features, including: Identifying current target scene information based on one or more sensors of the terminal; A text feature bound to the target scene information is selected from a text feature library to obtain the text feature of the target scene information, wherein the text feature library includes text features pre-bound to scene information.

15. A virtual image generating device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method described in any one of claims 1 to 7.

16. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method described in any one of claims 1 to 7.

Images