System and method for virtual avatar simulation creation

Abstract

According to the system and the method for simulating creation of the virtual copy, drawing actions are obtained through an action capture technology to generate vectorized drawing action information, the vectorized drawing action information serves as training data of a generative artificial intelligence model so that a drawing action model can be formed after training is completed, and the virtual copy is generated through an imaging technology. Then, the cue word and the virtual duplicate are input into a drawing action model to generate the virtual duplicate with a drawing action, a drawing image is generated through a generator of a generative adversarial network according to the cue word, and a discriminator feeds back the authenticity of the drawing image to the generator to serve as a basis for adjusting the drawing image. And then, covering a shade on the adjusted drawing image, and removing the corresponding shade according to an overlapping region of a drawing tool held by the virtual copy and the shade until the action of the virtual copy is stopped or the shade is completely removed, so that the technical effect of improving the creation simulation of the virtual copy is achieved.

Description

Technical Field

[0001] The present invention relates to a system and method for simulation creation, in particular to a system and method for simulation creation of virtual avatar. Background Art

[0002] In recent years, with the popularity and vigorous development of virtual avatars, various applications of virtual avatars have sprung up. However, how to improve the realism of virtual avatars has always been one of the problems that manufacturers are eager to solve. Virtual avatars not only play an increasingly important role in social media, games and virtual reality, but also show great potential in remote conferencing, education and psychotherapy.

[0003] Generally speaking, traditional virtual avatars are usually based on two-dimensional images and basic animation technology, which enable virtual avatars to show basic human characteristics and movements to a certain extent. However, although this method can provide a certain degree of interactivity, it often lacks in expression, movement smoothness and detail realism, that is, the virtual avatar has the problem of insufficient realism.

[0004] In view of this, some manufacturers have proposed technical means to combine advanced image processing algorithms to make the virtual avatar's imaging more realistic, the appearance more delicate, the skin color more realistic, etc., in order to improve the realism of the virtual avatar. However, this method only improves the appearance of the virtual avatar itself, so the realism is still insufficient. Therefore, some manufacturers will pre-set the actions that the virtual avatar wants to present in the form of 3D modeling. However, even if the pre-set actions are smooth, in the absence of action diversity, it is inevitable that they will feel the same after watching for a long time, and eventually lead to boredom, so it still cannot effectively solve the problem of insufficient realism of the virtual avatar.

[0005] In summary, it can be seen that the existing technology has long had the problem of insufficient realism of virtual avatars, so it is necessary to propose improved technical means to solve this problem. Summary of the invention

[0006] The invention discloses a system and method for simulating creation of a virtual avatar.

[0007] First, the present invention discloses a system for simulating the creation of a virtual avatar, which includes: a motion capture device and a computer host. The motion capture device is used to continuously capture different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embed corresponding label information in each of the painting action information, and the label information includes the painting subject, painting style, painting size, painting tool name and painting location. Then, in the part of the computer host, it is connected to the motion capture device, and the computer host includes: a non-transitory computer-readable storage medium and a hardware processor. The non-transitory computer-readable storage medium is used to store multiple computer-readable instructions, and store the painting action information generated by the motion capture device as model training data; and the hardware processor is electrically connected to the non-transitory computer-readable storage medium to execute the multiple computer-readable instructions, so that the hardware processor is configured to: generate a virtual avatar through imaging technology at the initial stage, and load the model training data from the non-transitory computer-readable storage medium to input into the Generative Artificial Intelligence (GAI). The method comprises the steps of: receiving a prompt word that specifies a painting subject, a painting style, a painting size, a painting tool name, and a painting position, and inputting the prompt word and a virtual avatar into the painting action generation model, so that the painting action generation model outputs a virtual avatar having a painting action and holding painting tools, wherein the action of the virtual avatar is allowed to be continuously dynamically adjusted through different prompt words; generating a corresponding painting image according to the painting subject, painting style, painting size, and painting position in the prompt word by a generator of a generative adversarial network, and then feeding back the authenticity of the painting image to the generator by a discriminator as a basis for adjusting the painting image; and covering a mask on the adjusted painting image, identifying an overlapping area between the painting tools held by the virtual avatar and the mask, and removing the mask corresponding to the overlapping area when the virtual avatar moves, so as to present a painting image at a corresponding position of the bottom layer of the mask, until the action of the virtual avatar stops or all masks are completely removed.

[0008] In addition, the present invention also discloses a method for simulating the creation of a virtual avatar, the steps of which include: connecting a motion capture device to a computer host, wherein the motion capture device continuously captures different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embeds corresponding label information in each of the painting action information, the label information includes a painting subject, a painting style, a painting size, a painting tool name and a painting location, and the computer host is used to receive the painting action information to store it as model training data; the computer host initially generates a virtual avatar through imaging technology, and loads the model training data to input into a generative artificial intelligence model for training, and after the training is completed, a painting action generation model based on multimodal pre-training technology is formed; the computer host receives the specified painting subject, painting style, painting size, painting tool name, and painting location. and a prompt word indicating the position of the painting, and inputting the prompt word and the virtual avatar into a painting action generation model, so that the painting action generation model outputs a virtual avatar having a painting action and holding painting tools, wherein the action of the virtual avatar is allowed to be dynamically adjusted continuously through different prompt words; the computer host generates a corresponding painting image according to the painting subject, painting style, painting size and painting position in the prompt word through a generator of a generative adversarial network, and then the discriminator feeds back the authenticity of the painting image to the generator as a basis for adjusting the painting image; and the computer host covers a mask on the adjusted painting image, and identifies the overlapping area between the painting tools held by the virtual avatar and the mask, and at the same time, when the virtual avatar moves, removes the mask corresponding to the overlapping area to present the painting image at the corresponding position of the bottom layer of the mask, until the action of the virtual avatar stops or all masks are completely removed.

[0009] The system and method disclosed in the present invention are as described above. The difference from the prior art is that the present invention uses motion capture technology to obtain painting movements to generate vectorized painting movement information, and uses it as training data for a generative artificial intelligence model to form a painting movement model after training, and generates a virtual avatar through imaging technology. Then, the prompt words and the virtual avatar are input into the painting movement model to generate a virtual avatar with painting movements, and a painting image is generated according to the prompt words by a generator of a generative adversarial network, and then the authenticity of the painting image is fed back to the generator by the discriminator as a basis for adjusting the painting image. Then, a mask is covered on the adjusted painting image, and the corresponding mask is removed according to the overlapping area between the painting tools held by the virtual avatar and the mask, until the movement of the virtual avatar stops or the mask is completely removed.

[0010] Through the above-mentioned technical means, the present invention can achieve the technical effect of improving the realism of the virtual avatar. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A system block diagram of the system for virtual avatar simulation creation of the present invention.

[0012] Figure 2A and Figure 2B A flow chart of the method for simulating creation of a virtual avatar according to the present invention.

[0013] Figure 3 A schematic diagram of applying the present invention to generate a virtual avatar with painting actions and holding painting tools.

[0014] Figure 4 A schematic diagram showing the application of the present invention to simulate the creation of a virtual avatar.

[0015] Description of reference numerals:

[0016] 110: Motion Capture Device

[0017] 120: Computer host

[0018] 121: Non-transitory computer readable storage medium

[0019] 122: Hardware Processor

[0020] 310: Motion Capture Device

[0021] 311a~311n:Sensor

[0022] 320: User

[0023] 410: Virtual Avatar

[0024] 420: Painting Images

[0025] 430: Mask

[0026] 431: Overlapping area

[0027] Step 210: Connecting the motion capture device to a computer host, wherein the motion capture device continuously captures different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embeds corresponding label information in each of the painting action information, the label information includes the painting subject, painting style, painting size, painting tool name and painting location, and the computer host is used to receive the painting action information to store it as model training data

[0028] Step 220: The computer host initially generates a virtual avatar through imaging technology, and loads the model training data to input into the generative artificial intelligence model for training, and after the training is completed, a painting action generation model based on multimodal pre-training technology is formed.

[0029] Step 230: The computer host receives a prompt word that specifies the painting subject, the painting style, the painting size, the name of the painting tool, and the painting location, and inputs the prompt word and the virtual avatar into the painting action generation model, so that the painting action generation model outputs the virtual avatar having the painting action and holding the painting tool, wherein the action of the virtual avatar can be continuously dynamically adjusted by different prompt words

[0030] Step 240: The computer host generates a corresponding painting image according to the painting subject, the painting style, the painting size and the painting position in the prompt word through the generator of the generative adversarial network, and then the discriminator feeds back the authenticity of the painting image to the generator as a basis for adjusting the painting image

[0031] Step 250: The computer host covers the adjusted painting image with a mask, and identifies the overlapping area between the painting tools held by the virtual avatar and the mask. When the virtual avatar moves, the mask corresponding to the overlapping area is removed to present the painting image at the corresponding position of the bottom layer of the mask, until the movement of the virtual avatar stops or all the masks are completely removed. DETAILED DESCRIPTION

[0032] The following will describe the implementation methods of the present invention in detail with reference to the accompanying drawings and embodiments, so that the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0033] First, please refer to the Figure 1 ”, “ Figure 1” is a system block diagram of the system for virtual avatar simulation creation of the present invention, and the system includes: a motion capture device 110 and a computer host 120. The motion capture device 110 is used to continuously capture different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embed corresponding tag information in each of the painting action information. The tag information includes: painting subject, painting style, painting size, painting tool name and painting position. For example, the tag information can record the painting subject, such as: "animal", "character"; painting style, such as: "ink painting", "watercolor painting"; painting size, such as: "20 cm * 20 cm", "10 inches * 5 inches"; painting tool name, such as: "pencil", "watercolor pen"; painting position, such as: "horizontally placed", "vertically placed", "45 degree angle placed", etc. In actual implementation, the sensors can be worn on the user's joints, limbs, torso, head The drawing action information can be stored in a structured file format (such as JSON, XML, etc.) to accurately capture the user's drawing action in three-dimensional space, and can be instantly transmitted to the computer host 120 for storage and processing. In fact, the drawing action information can be stored in a structured file format (such as JSON, XML, etc.). In addition, the drawing action information can collect 3D motion data through a motion capture device, and embed a timestamp to make the drawing action information time series data to ensure that it can be consistent with the action of the virtual avatar, and allow the drawing sound to be recorded through a sound receiving device (such as a microphone) to be embedded in the drawing action information, so that the generated virtual avatar can not only have drawing actions, but also can be equipped with a drawing sound corresponding to the name of the drawing tool to enhance the realism of the drawing situation. Taking the drawing tool name as a pencil as an example, the corresponding drawing sound can be the sound of the pencil sketching on paper. In actual implementation, different drawing tool names can correspond to different drawing sounds.

[0034] Next, in the part of the computer host 120, it is connected to the motion capture device 110, and the computer host 120 includes: a non-transitory computer-readable storage medium 121 and a hardware processor 122. Among them, the non-transitory computer-readable storage medium 121 is used to store a plurality of computer-readable instructions, and store the painting action information generated by the motion capture device 110 as model training data. In actual implementation, the non-transitory computer-readable storage medium 121 may include: a hard disk, an optical disk, a flash memory or the like. In addition, the computer-readable instructions are executed by the computer host 120, and the computer-readable instructions may be assembly language instructions, instruction set architecture instructions, machine instructions, machine-related instructions, microinstructions, firmware instructions, or source code or object code (Object Code) written in any combination of one or more programming languages, and the programming language includes object-oriented programming languages, such as: Common Lisp, Python, C++, Objective-C, Smalltalk, Delphi, Java, Swift, C#, Perl, Ruby and PHP, etc., and conventional procedural programming languages, such as: C language or similar programming languages. In actual implementation, the painting action information used as model training data is usually vectorized data and can be preprocessed first, such as data standardization, feature extraction, etc. Taking feature extraction as an example, when painting sounds are embedded in the painting action information, preprocessing can be achieved by extracting Mel spectrograms or other acoustic features.

[0035] The hardware processor 122 is electrically connected to the non-transitory computer-readable storage medium 121 to execute the plurality of computer-readable instructions, so that the hardware processor 122 is configured to: generate a virtual avatar through imaging technology at the beginning, and load model training data from the non-transitory computer-readable storage medium 121 to input into the generative artificial intelligence model for training, and after the training is completed, form a painting action generation model based on multimodal pre-training technology; receive a prompt word that has specified a painting subject, a painting style, a painting size, a painting tool name, and a painting location, and input the prompt word and the virtual avatar into the painting action generation model, so that the painting action generation model outputs a painting action having a painting action. and a virtual avatar holding painting tools, wherein the action of the virtual avatar is allowed to be dynamically adjusted continuously through different prompt words; a corresponding painting image is generated by a generator of a generative adversarial network according to the painting subject, painting style, painting size and painting position in the prompt words, and then the authenticity of the painting image is fed back to the generator by the discriminator as a basis for adjusting the painting image; and a mask is covered on the adjusted painting image, and the overlapping area between the painting tools held by the virtual avatar and the mask is identified, and at the same time, when the virtual avatar moves, the mask corresponding to the overlapping area is removed to present the painting image at the corresponding position of the bottom layer of the mask, until the action of the virtual avatar stops or all masks are completely removed. It should be particularly noted that the so-called generative artificial intelligence model refers to an artificial intelligence technology that can generate new content based on input data. This type of model is usually trained using a large amount of data. In the present invention, the relationship between various painting actions and label information will be learned during the training process, and a virtual avatar with new painting actions can be generated based on the prompt words and the virtual avatar. In actual implementation, the computer host can receive the situation description sentence, and analyze it through natural language technology to perform word segmentation and sentence segmentation on the situation description sentence, and classify it into painting subject, painting style, painting size, painting tool name and painting location to generate prompt words. In addition, the discriminator can continuously feedback the authenticity of the painting image to the generator, so as to continuously adjust the parameters of the generator and the discriminator through the reinforcement learning algorithm to optimize the painting image. It should be particularly noted that the mask can be a solid color layer (i.e., a single color, such as off-white, light gray, etc.) covering the painting image, and when the overlapping area is identified, the boundary of the overlapping area is smoothed according to at least one of the painting style, painting size and painting tool name through image processing technology, and when the mask removal area has reached a preset threshold value, the mask is completely removed. For example, the threshold value is set to 95%: when the mask removal area has reached 95%, even if there is still 5% of the mask that has not been removed, the remaining 5% of the mask will be automatically removed completely.

[0036] It should be particularly noted that in actual implementation, the present invention can be partially or completely implemented based on hardware. For example, the hardware processor 122 in the system can be implemented by an integrated circuit chip, a system on chip (System on Chip, SoC), a complex programmable logic device (Complex Programmable Logic Device, CPLD), and a field programmable gate array (Field Programmable Gate Array, FPGA). The non-transitory computer-readable storage medium 121 described in the present invention carries computer-readable instructions (or computer program instructions) for enabling the hardware processor 122 to implement various aspects of the present invention. The non-transitory computer-readable storage medium 121 can be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of the non-transitory computer-readable storage medium 121 include: a hard disk, a random access memory, a read-only memory, a flash memory, an optical disk, a floppy disk, and any suitable combination of the above. The non-transitory computer-readable storage medium 121 used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical signals through fiber-optic cables), or electrical signals transmitted through wires. In addition, the computer-readable instructions described herein can be downloaded from the non-transitory computer-readable storage medium 121 to each computing / processing device, or downloaded to an external computer device or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber-optic transmission, wireless transmission, routers, firewalls, switches, hubs, and / or gateways. The network card or network interface in each computing / processing device receives the computer-readable instructions from the network and forwards the computer-readable instructions for storage in the non-transitory computer-readable storage medium 121 in each computing / processing device.

[0037] Please refer to " Figure 2A Figure 2A "and" Figure 2B ”, “ Figure 2A Figure 2A "and" Figure 2B” is a method flow chart of the method for simulating creation of a virtual avatar of the present invention, and the steps include: connecting the motion capture device 110 to the computer host 120, wherein the motion capture device 110 continuously captures different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embeds corresponding label information in each of the painting action information, and the label information includes the painting subject, painting style, painting size, painting tool name and painting location, and the computer host 120 is used to receive the painting action information to store as model training data (step 210); the computer host 120 generates a virtual avatar through imaging technology at the beginning. , and load the model training data to input into the generative artificial intelligence model for training, and after the training is completed, a painting action generation model based on the multimodal pre-training technology is formed (step 220); the computer host 120 receives the prompt words that have specified the painting subject, painting style, painting size, painting tool name and painting location, and inputs the prompt words and the virtual avatar into the painting action generation model, so that the painting action generation model outputs a virtual avatar with painting actions and holding painting tools, wherein the action of the virtual avatar is allowed to be dynamically adjusted continuously through different prompt words (step 230); the computer host 120 generates a generator of the adversarial network according to the prompt words The painting subject, painting style, painting size and painting position of the virtual avatar are used to generate a corresponding painting image, and the discriminator then feeds back the authenticity of the painting image to the generator as a basis for adjusting the painting image (step 240); and the computer host 120 covers the adjusted painting image with a mask, and identifies the overlapping area between the painting tools held by the virtual avatar and the mask. At the same time, when the virtual avatar moves, the mask corresponding to the overlapping area is removed to present the painting image at the corresponding position of the bottom layer of the mask, until the movement of the virtual avatar stops or all masks are completely removed (step 250). Through the above steps, the painting movement can be obtained through motion capture technology to generate vectorized The drawing action information is used as training data for the generative artificial intelligence model so as to form a drawing action model after the training is completed, and a virtual avatar is generated through imaging technology. Then, the prompt words and the virtual avatar are input into the drawing action model to generate a virtual avatar with drawing actions, and a drawing image is generated according to the prompt words by the generator of the generative adversarial network, and then the authenticity of the drawing image is fed back to the generator by the discriminator as a basis for adjusting the drawing image. Then, a mask is covered on the adjusted drawing image, and the corresponding mask is removed according to the overlapping area between the drawing tools held by the virtual avatar and the mask, until the action of the virtual avatar stops or the mask is completely removed.

[0038] The following matches " Figure 3 "and" Figure 4 ” is described as follows by way of example, such as “ Figure 3 " Figure 3” is a schematic diagram of applying the present invention to generate a virtual avatar with painting movements and holding painting tools. In actual implementation, first, the motion capture device 310 and the computer host 120 can be connected to each other by wire or wireless means. When the user 320 is painting, the user's painting movements can be continuously captured by the multiple sensors (311a~311n) of the motion capture device 310 and advanced motion capture technology. In actual implementation, the sensors (311a~311n) can be as shown in “ Figure 3 ” is shown as being worn on the user 320, and the painting action captured by the motion capture device 310 will form vectorized painting action information, which will be transmitted to the computer host 120 and stored as model training data. It should be particularly noted that painting actions in different situations will form different painting action information. For example, different painting styles, painting sizes, painting tools used or painting positions, such as vertical placement, horizontal placement, etc., will all form different painting action information. Therefore, the painting action information will also be embedded with corresponding label information to record the painting subject, painting style, painting size, painting tool name and painting position, etc., so that when training the artificial intelligence model (i.e., the painting action generation model), the characteristics of the painting actions in different situations can be identified according to the label information.

[0039] Next, the computer host 120 will generate a virtual avatar through imaging technology at the beginning, for example, using 3D modeling technology to generate a virtual avatar that can be displayed in virtual reality (VR), augmented reality (AR) or mixed reality (MR), and load model training data to input into the generative artificial intelligence model for training, and after the training is completed, a painting action generation model based on multimodal pre-training technology is formed. Among them, the multimodal pre-training technology is a machine learning method that aims to use information from multiple different data types (such as: images of virtual avatars, 3D modeling files, label information or painting sounds, etc.) from model training data to pre-train the model, and improve the model's understanding ability by learning the associations or contexts between each other, so as to be able to output a virtual avatar with painting actions and holding painting tools. Among them, holding painting tools is generated according to the name of the painting tools. For example, assuming that the name of the painting tool is a pencil, the holding painting tool will be a 3D image of a pencil. When the computer host 120 receives the prompt words for the user to specify the painting subject, painting style, painting size, painting tool name and painting position, the prompt words and the virtual avatar are input into the painting action generation model, so that the generation model outputs a virtual avatar with painting actions and holding painting tools, wherein the action of the virtual avatar is allowed to be dynamically adjusted continuously through different prompt words. In other words, the computer host 120 first generates a basic virtual avatar (which can be stored in the form of vectorized files or 3D modeling files) through imaging technology. This basic virtual avatar does not have any painting actions and does not hold painting tools. Only after the user inputs the prompt words and the virtual avatar into the painting action generation model, the painting action generation model outputs a virtual avatar with painting actions and holding painting tools, and the painting actions are dynamically adjusted by different prompt words. At this point, a virtual avatar with painting actions and holding painting tools is successfully output, and this virtual avatar can be presented by a head-mounted device, a naked-eye 3D display or the like.

[0040] like" Figure 4 " Figure 4” is a schematic diagram showing the application of the present invention to simulate the creation of a virtual avatar. In actual implementation, in addition to outputting a virtual avatar 410 with painting actions and holding painting tools, the computer host 120 also generates a corresponding painting image based on the prompt word to simulate the creation of the virtual avatar. In actual implementation, the computer host 120 generates a corresponding painting image based on the painting subject, painting style, painting size and painting position in the prompt word through the generator of the generative adversarial network, and then the discriminator feeds back the authenticity of the painting image as a basis for adjusting the painting image. This process is repeated multiple times. , each iteration will improve the quality of the painting images generated by the generator and the recognition ability of the discriminator, and the generator and the discriminator are interdependent. The goal of the discriminator is to correctly identify real images and generated painting images, while the generator hopes that the generated painting images can deceive the discriminator, that is, they are considered to be real images. The aforementioned generator and discriminator are both neural network models. The former is used to train to generate fake data, and the latter is used to train to identify the authenticity of data. As the training progresses, the generator will generate more and more realistic painting images, and the discriminator will become more sensitive. For example, suppose If the subject of the painting in the prompt is an animal, the painting style is impressionism, the painting size is 50 cm*50 cm, and the painting position is vertical, then the generator will generate an animal painting with impressionism style and send it to the discriminator for authenticity identification. The discriminator then feeds the identification result back to the generator for the generator to adjust the painting image. After repeated execution, the generator will adjust the animal painting with impressionism style that is difficult to identify the authenticity, that is, painting image 420. Next, a mask 430 is covered on the adjusted painting image 420, and the painting tools held by the virtual avatar 410 and the mask 430 are identified. The overlapping area 431, when the virtual avatar 410 moves, removes the mask corresponding to the overlapping area 431 to present the painting image 420 at the corresponding position of the bottom layer of the mask 430, until the movement of the virtual avatar 410 stops or the mask 430 is completely removed. When the mask 430 is completely removed, the painting image 420 can be fully presented, which also means that the virtual avatar 410 has completed the creation. In this way, the whole can present a virtual avatar that can create, and the creation process and the created painting image 420 are highly realistic, which is different from the traditional virtual avatar with only fixed movements.

[0041] In summary, it can be seen that the difference between the present invention and the prior art is that the painting action is obtained by motion capture technology to generate vectorized painting action information, and is used as training data for the generative artificial intelligence model to form a painting action model after training, and a virtual avatar is generated by imaging technology. Then, the prompt word and the virtual avatar are input into the painting action model to generate a virtual avatar with painting action, and the generator of the generative adversarial network generates a painting image according to the prompt word, and then the discriminator feeds back the authenticity of the painting image to the generator as a basis for adjusting the painting image. Then, a mask is covered on the adjusted painting image, and the corresponding mask is removed according to the overlapping area of ​​the painting tools held by the virtual avatar and the mask, until the action of the virtual avatar stops or the mask is completely removed. This technical means can solve the problems existing in the prior art, and thus achieve the technical effect of improving the realism of the virtual avatar.

[0042] Although the present invention is disclosed as above with the aforementioned embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of patent protection of the present invention shall be subject to the scope defined by the claims attached to this specification.

Claims

1. A system for simulating creation of a virtual avatar, the system comprising: A motion capture device, used to continuously capture different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embed corresponding tag information in each of the painting action information, the tag information including the painting subject, painting style, painting size, painting tool name and painting location; and A computer host, used to connect the motion capture device, the computer host comprising: A non-transitory computer-readable storage medium for storing a plurality of computer-readable instructions and storing the painting motion information generated by the motion capture device as model training data; and A hardware processor is electrically connected to the non-transitory computer-readable storage medium and is used to execute the plurality of computer-readable instructions, so that the hardware processor is configured to: Initially, a virtual avatar is generated by imaging technology, and the model training data is loaded from the non-transitory computer-readable storage medium to be input into the generative artificial intelligence model for training, and after the training is completed, a painting action generation model based on multimodal pre-training technology is formed; receiving a prompt word that specifies the painting subject, the painting style, the painting size, the painting tool name, and the painting location, and inputting the prompt word and the virtual avatar into the painting action generation model, so that the painting action generation model outputs the virtual avatar having a painting action and holding painting tools, wherein: The action of the virtual avatar can be dynamically adjusted continuously through different prompt words; Generate a corresponding painting image according to the painting subject, the painting style, the painting size and the painting position in the prompt word by a generator of a generative adversarial network, and then feed back the authenticity of the painting image to the generator by a discriminator as a basis for adjusting the painting image; as well as A mask is covered on the adjusted painting image, and the overlapping area between the painting tools held by the virtual avatar and the mask is identified. When the virtual avatar moves, the mask corresponding to the overlapping area is removed to present the painting image at the corresponding position of the bottom layer of the mask, until the movement of the virtual avatar stops or all the masks are completely removed.

2. A system for virtual avatar simulation creation as described in claim 1, wherein the painting action information is 3D motion data collected by the motion capture device, and a timestamp is embedded to make the painting action information time series data to ensure that it can be consistent with the movement of the virtual avatar, and allows painting sounds to be recorded through a sound receiving device to be embedded in the painting action information.

3. A system for virtual avatar simulation creation as described in claim 1, wherein the computer host receives a situation description sentence and parses it through natural language technology to perform word segmentation and sentence segmentation on the situation description sentence, and at the same time classifies it into the painting subject, the painting style, the painting size, the painting tool name and the painting location to generate the prompt word.

4. A system for virtual avatar simulation creation as described in claim 1, wherein the discriminator continuously feeds back the authenticity of the painting image to the generator, so as to continuously adjust the parameters of the generator and the discriminator through a reinforcement learning algorithm to optimize the painting image.

5. The system for virtual avatar simulation creation as described in claim 1, wherein the mask is a solid color layer covering the painting image, and when the overlapping area is identified, the boundary of the overlapping area is smoothed according to at least one of the painting style, the painting size and the name of the painting tool through image processing technology, and when the removal area of ​​the mask reaches a threshold value, the mask is completely removed.

6. A method for simulating creation of a virtual avatar, the steps comprising: Connecting a motion capture device to a computer host, wherein the motion capture device continuously captures different painting actions through multiple sensors and motion capture technology to generate multiple vectorized painting action information, and embeds corresponding label information in each of the painting action information, the label information includes the painting subject, painting style, painting size, painting tool name and painting location, and the computer host is used to receive the painting action information to store it as model training data; The computer host initially generates a virtual avatar through imaging technology, and loads the model training data to input into the generative artificial intelligence model for training, and after the training is completed, a painting action generation model based on multimodal pre-training technology is formed; The computer host receives a prompt word that specifies the painting subject, the painting style, the painting size, the name of the painting tool, and the painting location, and inputs the prompt word and the virtual avatar into the painting action generation model, so that the painting action generation model outputs the virtual avatar having a painting action and holding painting tools, wherein the action of the virtual avatar can be continuously dynamically adjusted by different prompt words; The computer host generates a corresponding painting image according to the painting subject, the painting style, the painting size and the painting position in the prompt word through a generator of a generative adversarial network, and then the discriminator feeds back the authenticity of the painting image as a basis for adjusting the painting image; and The computer host covers a mask on the adjusted painting image and identifies the overlapping area between the painting tools held by the virtual avatar and the mask. When the virtual avatar moves, the computer host removes the mask corresponding to the overlapping area to present the painting image at the corresponding position of the bottom layer of the mask until the movement of the virtual avatar stops or all the masks are completely removed.

7. A method for simulating creation of a virtual avatar as described in claim 6, wherein the painting action information is 3D motion data collected by the motion capture device, and a timestamp is embedded to make the painting action information time series data to ensure that it can be consistent with the movement of the virtual avatar, and allows painting sounds to be recorded through a sound receiving device to be embedded in the painting action information.

8. The method for simulating creation of a virtual avatar as described in claim 6, wherein the computer host receives a situation description sentence and parses it through natural language technology to perform word segmentation and sentence segmentation on the situation description sentence, and classifies it into the painting subject, the painting style, the painting size, the painting tool name and the painting location to generate the prompt word.

9. The method for virtual avatar simulation creation as described in claim 6, wherein the discriminator continuously feeds back the authenticity of the painting image to the generator, so as to continuously adjust the parameters of the generator and the discriminator through a reinforcement learning algorithm to optimize the painting image.

10. The method for virtual avatar simulation creation as described in claim 6, wherein the mask is a solid color layer covering the painting image, and when the overlapping area is identified, the boundary of the overlapping area is smoothed according to at least one of the painting style, the painting size and the name of the painting tool through image processing technology, and when the removal area of ​​the mask reaches a threshold value, the mask is completely removed.

Images