Image processing method and device of virtual human generation system

Abstract

The invention relates to the technical field of virtual human generation, and discloses an image processing method and device of a virtual human generation system. The image processing method of the virtual human generation system comprises the following steps: inputting fusion features of objects of a first object set and a second object set into an image generator to respectively generate a role virtual image and a scene virtual image; calculating semantic similarity between the user text features and specific features associated with the objects in the first object set and the second object set, and displaying a role virtual image and a scene virtual image corresponding to the first object and the second object with the maximum semantic similarity to the user; according to the method, the virtual human image representing the novel role is generated by combining design drawing design with novel text analysis, and the virtual human image and the virtual scene image can be adjusted according to the interactive text of the user, so that the sense of substitution of the user can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual human generation, and more specifically, to an image processing method and device for a virtual human generation system. Background Art

[0002] Virtual humans are widely used in various industries, including personal assistance, intelligent customer service, and online education. Anthropomorphic digital humans can quickly interact with people and enhance user experience in human-computer interaction.

[0003] Traditional virtual human video generation is mainly divided into 2D / 3D facial reconstruction, talking-head generation, body movement and human body movement. This method is essentially a kind of action control of a 3D model of a character. In the era of artificial intelligence, virtual humans need to provide users with more emotional value, which cannot be provided by general character models to simulate human expressions and movements. In the field of fiction character virtualization, it is necessary to build virtual characters according to the plot of the novel, and it is necessary to consider the growth and changes of the characters of the novel as the plot progresses, so as to improve the user's sense of substitution when interacting with the user. Furthermore, combining virtual scenes can further improve the user's sense of substitution. Summary of the invention

[0004] The present invention provides an image processing method and device for a virtual human generation system, which solves the technical problem that traditional virtual human videos cannot provide users with a sufficient sense of immersion.

[0005] The present invention provides an image processing method for a virtual human generation system, comprising the following steps:

[0006] Step 101, obtaining design drawings of characters and scenes of the novel;

[0007] Step 102, arranging the characters and scene texts described in the novel into character text sequences and scene text sequences according to the order of the plot, wherein one character text sequence unit includes one character text segment, and one scene text sequence unit includes one scene text segment;

[0008] Step 103, constructing a structural dimension of ordered information based on the character image sequence, the character text sequence, the scene image sequence and the scene text sequence, wherein the basic structural elements contained in this structural dimension are objects, and the objects include: character design drawings, character texts, scene design drawings, and scene texts;

[0009] Step 104, integrating specific features associated with objects from the order information, and then obtaining integrated features of the character text object and the scene text object, and then extracting the character text object integrated with the specific features of the character design drawing object to generate a first object set, and extracting the scene text object integrated with the specific features of the scene design drawing object to generate a second object set;

[0010] Step 105, inputting the fusion features of the objects of the first object set and the second object set into an image generator to respectively generate a character virtual image and a scene virtual image;

[0011] Step 106, identifying the text sequence input by the user to obtain user text features, calculating the semantic similarity between the user text features and specific features associated with objects in the first object set and the second object set, and displaying the character virtual images and scene virtual images corresponding to the first object and the second object with the greatest semantic similarity to the user.

[0012] Furthermore, the method for processing and obtaining the orderliness information includes:

[0013] Objects for extracting orderliness information;

[0014] Extract specific features associated with objects;

[0015] Establishing a mapping between the extracted objects and their associated specific features;

[0016] For any two objects, if there is a correlation between the associated specific features related to the virtual human generation task or there is a correlation between the objects related to the virtual human generation task, an object relationship is established between the two objects.

[0017] Further, the character design drawing object is associated with specific features: image features of the character design drawing;

[0018] Specific features associated with role text objects: semantic features of role text segments;

[0019] Specific features associated with scene design objects: Image features of the scene design;

[0020] Specific features associated with scene text objects: semantic features of scene text fragments.

[0021] Furthermore, the relevance related to the virtual human generation task includes:

[0022] Character design drawings and the character texts they referenced when designing;

[0023] The scene design drawings and the scene texts used as references during their design;

[0024] There are more than A identical setting words in two character texts;

[0025] There are more than B identical setting words in the two scene texts;

[0026] The default value of A and B is 5, which decreases as the language of the novel becomes more refined.

[0027] Furthermore, the feature similarity of the fusion features of the objects in the first object set is calculated, and one of the two objects whose feature similarity is greater than a set similarity threshold is deleted; and the same operation is performed on the second object set.

[0028] Furthermore, step 106 also includes the step of performing 3D modeling on the character virtual image and the scene virtual image to obtain a 3D virtual model.

[0029] Furthermore, the image generator uses GAN (Generative Adversarial Networks) or upsampled CNN (Convolutional Neural Networks) or VAE (Variational Autoencoder).

[0030] Furthermore, step 106 is executed again when the text sequence input by the user is updated.

[0031] The present invention provides an image processing device for a virtual human generation system, comprising: a design drawing acquisition module, which is used to acquire design drawings of characters and scenes in a novel;

[0032] A sequence generation module, which is used to organize the characters and scene texts described in the novel into character text sequences and scene text sequences according to the order of the plot, wherein a character text sequence unit includes a character text segment, and a scene text sequence unit includes a scene text segment;

[0033] A data sorting module, which constructs a structural dimension of ordered information based on the character image sequence, the character text sequence, the scene image sequence and the scene text sequence. The basic structural elements contained in this structural dimension are objects, including: character design drawings, character texts, scene design drawings, and scene texts;

[0034] A feature fusion module, which fuses specific features associated with objects from the order information, and then obtains fusion features of the character text object and the scene text object, and then extracts the character text object fused with the specific features of the character design drawing object to generate a first object set, and extracts the scene text object fused with the specific features of the scene design drawing object to generate a second object set;

[0035] A virtual image generation module, which inputs the fusion features of the objects of the first object set and the second object set into the image generator to generate a character virtual image and a scene virtual image respectively;

[0036] The image matching module is used to identify the text sequence input by the user to obtain the user text features, calculate the semantic similarity between the user text features and the specific features associated with the objects in the first object set and the second object set, and display the character virtual images and scene virtual images corresponding to the first object and the second object with the greatest semantic similarity to the user.

[0037] The present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method of the aforementioned virtual human generation system are implemented.

[0038] The beneficial effects of the present invention are:

[0039] In general, the present invention generates virtual human images representing novel characters through design drawing design combined with novel text analysis, and can adjust the virtual human images and virtual scene images according to the user's interactive text, which will greatly enhance the user's sense of immersion.

[0040] Text parsing and script construction: First, the novel text is deeply analyzed to identify and extract important elements such as characters, dialogues, action descriptions, and scene settings. These elements will be organized into data structures similar to storyboards and scene scripts for subsequent processing.

[0041] Design creation and association: For each key plot or scene, the invention creates a corresponding design and associates it with the relevant novel text fragment. This association is not a simple correspondence, but a selective mapping based on a deep understanding of the text, aiming to capture those details that are crucial to the development of the story.

[0042] Sparse processing: Since events and descriptions in novels are often distributed in different chapters and paragraphs, the directly extracted information sequence may be discontinuous. Therefore, the present invention needs to perform sparse processing on this information, that is, retain the text parts that best represent the characteristics of a specific character or scene, and ignore those that do not affect the overall understanding. For example, if a character's armor is mentioned early in the story, changes later, and the changed armor is used at a certain critical moment, then these three non-continuous text fragments will be specially selected as the basis for generating the character and its armor changes.

[0043] In order to avoid the overfitting problem that may be caused by traditional temporal models and attention mechanisms, the present invention adopts a rule-defined method to construct order information. This method allows the present invention to treat each text segment as an object and define the association between objects based on the relevance of their content.

[0044] The present invention realizes the time jump through the association between the character text and the scene text, not only considering the importance of individual text fragments, but also evaluating their role in the entire story framework. In this way, the present invention can ensure that the virtual people and scenes finally generated not only conform to the description of the novel in appearance, but also reflect the spiritual core of the original work in terms of behavior patterns and interaction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of the image processing method of the virtual human generation system of the present invention. DETAILED DESCRIPTION

[0046] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0047] At least one embodiment of the present invention discloses an image processing method for a virtual human generation system, such as Figure 1 As shown, the following steps are included:

[0048] Step 101, obtaining design drawings of characters and scenes of the novel;

[0049] The aforementioned design drawings refer to images that can describe characters and scenes, including hand-drawn paintings, photos of real scenes or characters, or images of other virtual images.

[0050] Step 102, the characters and scene texts described in the novel are sorted into character text sequences and scene text sequences according to the chronological order of the plot, wherein one character text sequence unit includes one character text segment, and one scene text sequence unit includes one scene text segment.

[0051] Step 103, constructing a structural dimension of ordered information based on the character image sequence, the character text sequence, the scene image sequence and the scene text sequence, wherein the basic structural elements contained in this structural dimension are objects, and the objects include: character design drawings, character texts, scene design drawings, and scene texts;

[0052] In some embodiments of the present invention, the method for processing and obtaining orderliness information includes:

[0053] Objects for extracting orderliness information;

[0054] Extract specific features associated with objects;

[0055] Establishing a mapping between the extracted objects and their associated specific features;

[0056] For any two objects, if there is a correlation between the associated specific features related to the virtual human generation task or there is a correlation between the objects related to the virtual human generation task, an object relationship is established between the two objects.

[0057] In some embodiments of the present invention, the specific features associated with the character design object are: image features of the character's design;

[0058] Specific features associated with role text objects: semantic features of role text segments;

[0059] Specific features associated with scene design objects: Image features of the scene design;

[0060] Specific features associated with scene text objects: semantic features of scene text fragments.

[0061] In some embodiments of the present invention, the associations related to the virtual human generation task include:

[0062] Character design drawings and the character texts they referenced when designing;

[0063] The scene design drawings and the scene texts used as references during their design;

[0064] There are more than A identical setting words in two character texts;

[0065] There are more than B identical setting words in the two scene texts;

[0066] The default value of A and B is 5, which decreases as the language of the novel becomes more refined.

[0067] The aforementioned setting words refer to the words stored in the setting word library. The setting words are artificially defined words and can be general keywords related to the definition or description of characters or scenes, such as "willow leaf eyebrows", "willow", etc.

[0068] The aforementioned objects all refer to objects of one type. For example, a character design drawing refers to all objects belonging to a character design drawing, not just one character design drawing. Of course, when describing the association, what is described is the association between a specific character design drawing and the character text.

[0069] Step 104, integrating specific features associated with objects from the order information, and then obtaining integrated features of the character text object and the scene text object, and then extracting the character text object integrated with the specific features of the character design drawing object to generate a first object set, and extracting the scene text object integrated with the specific features of the scene design drawing object to generate a second object set;

[0070] Step 105, inputting the fusion features of the objects of the first object set and the second object set into an image generator to respectively generate a character virtual image and a scene virtual image;

[0071] Step 106, identifying the text sequence input by the user to obtain user text features, calculating the semantic similarity between the user text features and specific features associated with objects in the first object set and the second object set, and displaying the character virtual images and scene virtual images corresponding to the first object and the second object with the greatest semantic similarity to the user.

[0072] In step 104, a duplication deletion operation is also performed. Specifically, the feature similarity of the fusion features of the objects in the first object set is calculated. If the feature similarity is greater than a set similarity threshold, one of the two objects whose feature similarity is greater than the set similarity threshold is deleted.

[0073] The same operation is performed for the second object set. For cosine similarity, the feature similarity threshold can be set to 0.80~0.95.

[0074] In some embodiments of the present invention, a sequence recognition network such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), or GRU (Gated Recurrent Unit) may be used to recognize a text sequence input by a user.

[0075] In some embodiments of the present invention, step 106 also includes the step of performing 3D modeling on the character virtual image and the scene virtual image to obtain a 3D virtual model, and what is finally displayed to the user is the result of the 3D modeling of the character virtual image and the scene virtual image.

[0076] In some embodiments of the present invention, the image generator adopts an existing image generation model, such as GAN (Generative Adversarial Networks) or upsampled CNN (Convolutional Neural Networks) or VAE (Variational Autoencoder), and the image generator can be generally pre-trained.

[0077] In some embodiments of the present invention, the step 104 of obtaining fusion features of the character text object and the scene text object is performed using an aggregation network.

[0078] In one embodiment of the present invention, the aggregation network is a multi-layer network. The expression of the layer is as follows:

[0079]

[0080] : Object In the The feature representation of the layer, hour , For object High-dimensional feature representation of linked volume data, hour , For object Characteristic features of is the total number of layers in the aggregation network, .

[0081] : Object In the The feature representation of the layer.

[0082] : From object To Object news;

[0083] ;

[0084] : Object In the The feature representation of the layer, hour , For object Specific features of an association.

[0085] : Message calculation function, which can be implemented in many ways, for example:

[0086] Simple linear transformation: , Indicates the message calculation weight.

[0087] Simple linear transformation: , Indicates splicing.

[0088] Use non-linear transformations, such as ReLU, to enhance expressiveness.

[0089] : Aggregation function used to aggregate messages from neighbors (e.g. sum, average, or maximum).

[0090] : Update function used to update object features, such as GRU (Gated Recurrent Unit) or simple MLP (Multilayer Perceptron).

[0091] : Object A collection of objects that have associations related to the virtual human generation task.

[0092] Step 106 will be executed again when the text sequence input by the user is updated, and the displayed character virtual image and scene virtual image will be updated when the user inputs new interactive text.

[0093] In some embodiments of the present invention, step 101 also includes the step of obtaining a design drawing of an object in the novel.

[0094] Step 102 also includes the step of generating an item text sequence, where an item text sequence unit includes an item text segment.

[0095] The order information constructed in step 103 also includes item text and item design drawings.

[0096] Step 104 also generates a third object set, where the objects in the third object set are item text objects that incorporate specific features of the item design drawing objects.

[0097] Step 105 also inputs the fusion features of the objects in the third object set into the image generator to generate a virtual image of the object;

[0098] Step 106 further calculates the semantic similarity between the user text feature and a specific feature associated with an object in the third object set, and displays the object virtual image corresponding to the third object with the greatest semantic similarity to the user.

[0099] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.

Claims

1. An image processing method for a virtual human generation system, characterized in that: The following steps are involved: Step 101, obtaining design drawings of characters and scenes of the novel; Step 102, arranging the characters and scene texts described in the novel into character text sequences and scene text sequences according to the order of the plot, wherein one character text sequence unit includes one character text segment, and one scene text sequence unit includes one scene text segment; Step 103, constructing a structural dimension of ordered information based on the character image sequence, the character text sequence, the scene image sequence and the scene text sequence, wherein the basic structural elements contained in this structural dimension are objects, and the objects include: character design drawings, character texts, scene design drawings, and scene texts; Step 104, integrating specific features associated with objects from the order information, and then obtaining integrated features of the character text object and the scene text object, and then extracting the character text object integrated with the specific features of the character design drawing object to generate a first object set, and extracting the scene text object integrated with the specific features of the scene design drawing object to generate a second object set; Step 105, inputting the fusion features of the objects of the first object set and the second object set into an image generator to respectively generate a character virtual image and a scene virtual image; Step 106, identifying the text sequence input by the user to obtain user text features, calculating the semantic similarity between the user text features and specific features associated with objects in the first object set and the second object set, and displaying the character virtual images and scene virtual images of the first object and the second object corresponding to the maximum semantic similarity to the user.

2. The image processing method of a virtual human generation system according to claim 1, characterized in that: Methods for processing and obtaining ordered information include: Objects for extracting orderliness information; Extract specific features associated with objects; Establishing a mapping between the extracted objects and their associated specific features; For any two objects, if there is a correlation between the associated specific features related to the virtual human generation task or there is a correlation between the objects related to the virtual human generation task, an object relationship is established between the two objects.

3. The image processing method of a virtual human generation system according to claim 2, characterized in that: Specific features associated with the character design object: image features of the character design; Specific features associated with role text objects: semantic features of role text segments; Specific features associated with scene design objects: Image features of the scene design; Specific features associated with scene text objects: semantic features of scene text fragments.

4. The image processing method of a virtual human generation system according to claim 2, characterized in that: Relevance to the task of virtual human generation includes: Character design drawings and the character texts they referenced when designing; The scene design drawings and the scene texts used as references during their design; There are more than A identical setting words in two character texts; There are B or more identical setting words in two scene texts.

5. The image processing method of a virtual human generation system according to claim 1, characterized in that: The feature similarity of the fusion features of the objects in the first object set is calculated, and one of the two objects whose feature similarity is greater than a set similarity threshold is deleted; and the same operation is performed on the second object set.

6. The image processing method of a virtual human generation system according to claim 1, characterized in that: Step 106 also includes the step of performing 3D modeling on the character virtual image and the scene virtual image to obtain a 3D virtual model.

7. The image processing method of a virtual human generation system according to claim 1, characterized in that: The image generator uses GAN or upsampled CNN or VAE.

8. The image processing method of a virtual human generation system according to claim 1, characterized in that: Step 106 is executed again when the text sequence input by the user is updated.

9. An image processing device for a virtual human generation system, characterized in that: include: A design drawing acquisition module, which is used to obtain design drawings of characters and scenes in the novel; A sequence generation module, which is used to organize the characters and scene texts described in the novel into character text sequences and scene text sequences according to the order of the plot, wherein a character text sequence unit includes a character text segment, and a scene text sequence unit includes a scene text segment; A data sorting module, which constructs a structural dimension of ordered information based on the character image sequence, the character text sequence, the scene image sequence and the scene text sequence. The basic structural elements contained in this structural dimension are objects, including: character design drawings, character texts, scene design drawings, and scene texts; A feature fusion module, which fuses specific features associated with objects from the order information, and then obtains fusion features of the character text object and the scene text object, and then extracts the character text object fused with the specific features of the character design drawing object to generate a first object set, and extracts the scene text object fused with the specific features of the scene design drawing object to generate a second object set; A virtual image generation module, which inputs the fusion features of the objects of the first object set and the second object set into the image generator to generate a character virtual image and a scene virtual image respectively; The image matching module is used to identify the text sequence input by the user to obtain the user text features, calculate the semantic similarity between the user text features and the specific features associated with the objects in the first object set and the second object set, and display the character virtual images and scene virtual images corresponding to the first object and the second object with the greatest semantic similarity to the user.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of an image processing method of a virtual human generation system as described in any one of claims 1-8 are implemented.

Images