Method and system for generating and editing humanoid role in virtual scene

Abstract

The invention relates to a method and system for generating and editing a human-shaped role in a virtual scene, and the method comprises the steps: translating a natural language command of a user into a 1 * 6 instruction sequence through employing a large language model, and enabling the instruction sequence to be a 1 * 6 matrix; by analyzing the instruction sequence, performing operations of generation, deletion, body shape adjustment and posture adjustment on the virtual character; in the step of posture adjustment, a body part to be adjusted at present and the change trend of the body part are obtained by analyzing an instruction sequence, so that the rotation amount of the body part is discretely changed by a certain amplitude in the corresponding axial direction; and body shape adjustment: obtaining the body part to be adjusted at present and the change trend thereof by analyzing the instruction sequence, thereby realizing the change of body shape parameters or enabling the scaling amount of the corresponding part to be changed in the corresponding axial direction. A novel virtual reality interaction mode is provided, and the method has the outstanding advantages of being visual, efficient and easy to master.

Description

Technical Field

[0001] The invention belongs to the technical field of computer application, and relates to a method and a system for generating and editing a humanoid character in a virtual scene. Background Art

[0002] Virtual Reality (VR) technology is a human-computer interaction technology that allows users to immerse themselves in and interact with a three-dimensional environment generated by a computer. VR systems are usually composed of hardware devices, software platforms, and content resources. They allow users to have an immersive experience by simulating the real world or creating unique virtual scenes. The background development of VR technology can be traced back to the 1960s, but due to the limitations of computing power, display devices, and sensor technology at the time, the concept of VR remained only in the theoretical stage. With the improvement of computer technology, display resolution, sensor technology, and data processing capabilities, virtual reality has gradually moved towards practical applications and has become a research direction and technical hotspot that has attracted attention in many fields.

[0003] Virtual reality interaction technology is the core of VR experience, which determines the user's immersion and smooth operation in the virtual environment. VR interaction technology covers many types, including visual interaction, tactile interaction, somatosensory interaction and sound interaction, which rely on a variety of sensors and control devices. Visual interaction technology is the most basic means of interaction. Through display devices such as VR headsets, the system can capture the user's head movement and adjust the viewing angle in real time, so that the user can get an immersive visual effect. The realization of this interaction depends on precise head position and direction tracking, usually using sensors such as accelerometers and gyroscopes, or using external camera systems for positioning. In addition, with the development of eye tracking technology, some VR devices can track the direction of the user's eye line of sight, allowing the system to automatically focus on the area of ​​the user's attention, thereby improving the efficiency and immersion of picture rendering.

[0004] VR interactive technology allows users to interact with the virtual environment through input devices such as handles, gloves, and motion capture devices. These devices can capture the user's hand movements and positions, allowing users to grab objects and operate tools in the virtual environment. Motion capture usually uses infrared cameras, laser tracking and other technologies to accurately identify subtle movements of the hands and body, allowing users to move freely in the virtual space. Tactile interaction simulates the texture and touch of objects through devices such as force feedback gloves and vibration feedback handles, allowing users to experience the realism of touch and collision in virtual scenes. With the advancement of tactile technology, some VR systems can even provide force feedback, allowing users to feel the weight, elasticity or resistance of virtual objects, enhancing the authenticity of interaction. Sound interaction uses spatial audio technology to dynamically adjust the sound effect according to the user's position, allowing users to perceive the direction and distance of the sound, simulating a natural auditory environment.

[0005] As virtual reality (VR) technology continues to gain popularity and mature, consumers and businesses are seeing a sharp increase in demand for better interaction methods. Consumers expect to gain greater satisfaction through more intuitive and immersive experiences, which drives the need for innovative interaction methods. Different users have diverse preferences, such as gesture control and voice recognition. Although the existing voice recognition technology on the market has matured, if it is directly applied to VR scenarios, it faces the problem of "only being able to recognize and operate fixed sentences and their corresponding instructions, and being unable to correctly trigger sentences with the same semantics." Summary of the invention

[0006] The technical problem to be solved by the present invention is: to overcome the shortcomings of the prior art, to perform semantic analysis on the natural language commands input by the user in combination with a large language model and convert them into corresponding instruction sequences, so as to provide users with a more intuitive and lower-learning-cost VR interaction method, and to quickly realize the user's needs in the virtual scene in a more intuitive and efficient way.

[0007] Technical solution of the present invention:

[0008] A method for generating and editing a humanoid character in a virtual scene, comprising:

[0009] Step 1: Use a large language model to translate the user's natural language command into a 1×6 instruction sequence, which is a 1×6 matrix. Each column of the matrix represents: "operation type", "generated gender", "body part to adjust posture", "trend of adjusting body shape", "joint point corresponding to the adjustment command", and "motion trend of the corresponding joint point".

[0010] Step 2: by parsing the instruction sequence obtained from step 1, the virtual character is generated, deleted, and the body shape and posture are adjusted;

[0011] The body shape adjustment includes: using the mean square formula to calculate the error and using it as the loss function to fit and obtain the overall body shape change parameters of the skinned human mesh model SMPL, so as to adjust the overall body shape of the virtual character;

[0012] Posture adjustment includes: mapping the spatial rotation information of the user's handle to the spatial rotation of the root node of the virtual character's currently adjusted joint, so as to achieve further adjustment of the virtual character's posture.

[0013] A system for generating and editing a humanoid character in a virtual scene comprises: a large language model access and receiving module, a speech recognition and content copy module, an instruction parsing module, a posture adjustment module and a body shape adjustment module.

[0014] Large language model access and receiving module: access the trained fine-tuned large language model through the OpenAI interface, send the natural language commands input by the user to it through network communication, wait for and receive the command sequence returned by it, and give it to the command parsing module for parsing;

[0015] Speech recognition and content copy module: The speech recognition plug-in obtains the user's natural language command input in text form, accesses and transmits it to the large language model through network communication, and the large language model is responsible for translating the natural language command into the corresponding instruction sequence;

[0016] Instruction parsing module: receives the instruction sequence translated by the large language model through network communication, parses each bit, and performs corresponding operations based on the parsing results. It can be divided into directly performing corresponding operations or indirectly calling corresponding other modules to complete the corresponding operations;

[0017] Posture adjustment module: by parsing the instruction sequence, it obtains the body part to be adjusted and its change trend, so that the rotation amount of the part is discretely changed by a certain amplitude on the corresponding axis;

[0018] Body shape adjustment module: by parsing the instruction sequence, the body part to be adjusted and its changing trend are obtained, so as to change the body shape parameters or change the scaling amount of the corresponding part in the corresponding axis.

[0019] Beneficial effects:

[0020] Compared with the existing methods of directly estimating postures through neural networks (such as PyMAF, CycleAdapt, ROMP, VIBE, HyberIK, etc.), the posture editing method based on a large language model and rotation mapping proposed in the present invention has a higher upper limit of restoration accuracy. In addition, the method described in the present invention also has strong interactivity that traditional pure posture estimation methods do not have. To a certain extent, it has taken a small step towards the process of leaving the screen editor, and is more suitable for more immersive application scenarios based on head-mounted display devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of a method for generating and editing a humanoid character in a virtual scene according to the present invention;

[0022] Figure 2 A structural framework diagram of a system for generating and editing a humanoid character in a virtual scene according to the present invention;

[0023] Figure 3 The 17 key points of the human body involved in the posture editing part of the present invention;

[0024] Figure 4The detailed content and meaning of the custom instruction sequence involved in the present invention;

[0025] Figure 5 These are some examples of natural language commands and their corresponding instruction sequences mainly involved in the present invention;

[0026] Figure 6 This is an effect example of the method for obtaining SMPL body posture parameters mainly involved in the present invention;

[0027] FIG. 7 (a) and FIG. 7 (b) are demonstrations of a virtual character generation operation in an embodiment of the present invention;

[0028] FIG8 (a) and FIG8 (b) are demonstrations of body shape adjustment operations in one embodiment of the present invention;

[0029] FIG9(a), FIG9(b) and FIG9(c) are demonstrations of posture adjustment operations in one embodiment of the present invention;

[0030] Fig.10 This is a demonstration of the application of some posture restoration results and posture changes of the present invention. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0032] The present invention relates to a method for generating and editing a humanoid character in a virtual scene, comprising: after entering the virtual scene through a consumer-grade head-mounted display device, the user can select a coordinate point or a virtual character by using a right-hand handle button, and then long-press the left-hand handle trigger button to start voice input and issue a natural language command. This command is recorded and received through a microphone, converted into basic text via a speech-to-text plug-in in the Unity Assets Store, accessed through network communication and sent to a dedicated large language model fine-tuned and trained based on an OpenAI interface for translation and return of a series of custom command sequences, thereby achieving the inclusion of multiple expressions of natural language commands with the same semantics of the user. After receiving this command sequence, the user end performs bit-by-bit parsing, and executes operations of generating a character, adjusting body shape, adjusting posture, and deleting a character according to the parsing results.

[0033] For the character generation operation, a virtual character prefab is generated according to the gender and body shape indicated by the user's natural language command, and is placed at the coordinate point specified by the user.

[0034] For the body shape adjustment operation, that is, according to the adjustment part and its change trend indicated by the user's natural language command, the overall or partial body shape of the currently selected virtual character is made fatter or thinner, and the current adjustment part and its change trend are recorded in real time, allowing the user to continue or reverse the changes in the current trend through the left-hand handle buttons.

[0035] For posture adjustment operations, that is, according to the adjustment part and its change trend indicated by the user's natural language command, the specified joint space rotation of the currently selected virtual character is changed, and the current adjustment joint, the root node of the current adjustment joint and the current change trend are recorded at the same time, allowing the user to continue or reverse the changes in the current trend through the left hand handle button or long press the trigger button of the right hand handle to map the handle rotation to the rotation of the joint root node in real time to achieve further adjustment.

[0036] For the character deletion operation, the virtual character currently selected by the user is directly deleted from the scene and the memory occupied by it is released.

[0037] The present invention can be applied to the editing of virtual characters and the construction of virtual scenes. The voice interaction solution designed by the invention without limiting the form of expression can effectively lower the operational threshold for generating and adjusting virtual humanoid characters in virtual scenes based on head-mounted displays, and the button fine-tuning and rotation mapping solutions can also further improve its efficiency, making the overall process more intuitive, clear and easy to use. The organic combination of the above modules can effectively assist in tasks such as digital human editing and basic scene construction.

[0038] The execution environment of the present invention uses a 4.8GHZ CPU and a Core 16-core computer with 64G bytes of memory. It is developed and run based on Unity Engine with NVIDIA GeForce RTX 3080Ti Laptop GPU. The main program is written based on C# language. At the same time, the PyTorch framework based on Python language and Blender software are used to assist the experiment to obtain important data. Under the premise that the computer memory and video memory allow, the basic operation of the present invention can also be based on other execution environments, such as NVIDIA GeForce GTX 4060 and other environments.

[0039] Embodiment 1

[0040] like Figure 1 As shown, the present invention provides a method for generating and editing a humanoid character in a virtual scene, and the steps are as follows:

[0041] Step 1: The user uses a head-mounted display device to enter a virtual scene written based on a game engine.

[0042] The above specifically include:

[0043] The head-mounted display device can be a consumer-grade head-mounted display device, such as: PICO Neo 3, PICO 4, MetaQuest 3 and other all-in-one VR devices or any PCVR head-mounted display device, and the head-mounted display device is used as an interactive medium.

[0044] The game engine may be a mainstream game engine such as Unity Engine, Unreal Engine, etc., and the virtual scene obtained by programming using the game engine is used as a carrier for presentation and interaction.

[0045] Step 2: Use the right handle button to select a coordinate point or a virtual character within the movable area.

[0046] In this step, the coordinate point is the coordinate point of the virtual character to be generated, and the virtual character is the virtual character that the user currently points to and intends to edit and adjust. The ray detection method is used to determine the specific coordinate point of the generated virtual character or obtain the virtual character that the user currently points to, that is, the virtual character that the user intends to edit and adjust.

[0047] Step 3. After the user selects a coordinate point or a virtual character, long press the trigger button of the left handle to enable the microphone to input natural language commands. At the same time, the user's natural language commands are converted into text in real time through the voice-to-text plug-in.

[0048] Step 4: Access the fine-tuned large language model based on the OpenAI interface through network communication, and transmit the natural language command converted into text in step 3 to the large language model. The large language model translates the natural language command into a string of instruction sequences according to custom rules, and then transmits it back to the user end via network communication.

[0049] The customized rules described in step 4 are as follows Figure 4 As shown, specifically, the instruction sequence is designed to be a 1x6 matrix, and each column of this matrix represents "operation type", "generated gender", "body part for adjusting posture", "trend for adjusting body shape", "joint point corresponding to completing this adjustment command", and "movement trend of the corresponding joint point".

[0050] The fine-tuning and training method of the large language model based on the OpenAI interface described in step 4 is as follows: by using ChatGPT to assist in generating and expanding data, that is, first setting the natural language and its corresponding instruction sequence, and then using ChatGPT to understand the semantics and generate as many natural language expressions as possible that can represent this instruction sequence, and then integrating them to obtain a data set. Based on this data set, preprocessing is performed in Python, and the OpenAI interface is connected, and then ChatGPT is used to fine-tune the large language model. The data example is as follows: Figure 5 shown.

[0051] Step 5: Parse the instruction sequence obtained in step 4 bit by bit and execute the corresponding operation type, which includes: generating a character, adjusting body shape, adjusting posture and deleting a character.

[0052] The process of generating a character includes: generating a virtual character prefabricated body according to the gender and body shape indicated by the user's natural language command, and placing it at the coordinate point specified by the user, as shown in Figure 7 (a) and Figure 7 (b). It is worth noting that the virtual humanoid character model in the present invention adopts the SMPL model. Specifically, SMPL (Skinned Multi-Person LinearModel, human mesh model with skin) is a deep learning model, which is a more realistic skin model based on vertices, and can parameterize the posture and posture of the human body, wherein the posture parameter (ShapeParameter) is a 1x10 vector, wherein the ten values ​​are all scalars, and each scalar value can be interpreted as the expansion / contraction amount of the human body in a certain direction (to represent the appearance characteristics of the human body); the posture parameter (Pose Parameter) is a 24x3 vector, wherein each set of values ​​is used to maintain the relative rotation of the joint relative to its parameter, and each rotation is represented by an axis-angle rotation, and is encoded as an arbitrary 3D vector. For the posture parameter β, when it is a vector of all zeros, it represents an average posture, and the change of its specific value can also be understood as the difference between the current posture and the average posture. The setting of the initial body shape of the virtual character and the subsequent adjustment of the overall reminder in the present invention are obtained by adjusting this parameter.

[0053] It is worth adding that the body parameter β cannot show meaningful linear changes for local areas, that is, when the value of a certain dimension is adjusted, the changes reflected in the skin model are often global, illogical, and difficult to summarize. For example, β1 (the value of the second dimension) mainly reflects the difference in body fatness, but when the skin model undergoes a horizontal proportion, that is, a linear change in fatness, the vertical proportions of its various parts will also change accordingly. To this end, the present invention attempts to fit a suitable β parameter based on graphics and gradient descent methods, that is, by scaling the bones of the standard SMPL skin model to simulate changes in body shape, and separately saving the model vertex information before and after the change, to obtain meshes with the same total number of vertices but different topological structures, such as Figure 6 As shown. Then, the vertex information of the mesh model obtained by different combinations of SMPL body shape parameters β is exhaustively enumerated, and the loss is calculated by the loss function MSE with the vertex information of the mesh model obtained by the bone scaling change mentioned above. When the loss is small enough, it is considered that the value sequence of β fitted at this time is the parameter that satisfies this body shape change.

[0054] The process of adjusting the body shape includes: according to the adjustment part and its change trend indicated by the user's natural language command, making the overall body shape or local body shape of the currently selected virtual character fatter or thinner, as shown in Figure 8 (a), and recording the current adjustment part and its change trend in real time, and allowing the user to continue or reverse the change in the current trend through the left-hand handle button, as shown in Figure 8 (b). The error is calculated by using the mean square formula and used as the loss function to fit the overall body shape change parameter β of SMPL to achieve the adjustment of the overall body shape of the virtual character. It is worth noting that since the changes controlled by the body shape parameter β of SMPL are all changes of global nature, the present invention only obtains the β parameter that makes the human body model fatter / thinner as a whole through the above experiment, and it is temporarily impossible to change its specific parts separately. Therefore, only the overall body shape change of the virtual character and the generation of the default body shape are completed using the SMPL β parameter, and the remaining local body shape changes are completed based on bone scaling.

[0055] The process of adjusting the posture includes: according to the adjustment part and its change trend indicated by the user's natural language command, the specified joint space rotation of the currently selected virtual character changes, as shown in Figure 9 (a), and the current adjustment joint, the root node of the current adjustment joint and the current change trend are recorded at the same time, allowing the user to continue or reverse the change in the current trend through the left handle button or long press the trigger button of the right handle to map the handle rotation to the rotation of the joint root node in real time to achieve further adjustment, as shown in Figure 9 (b) and Figure 9 (c). It is worth noting that since the VR scene in the present invention depends on UnityEngine, and SMPL does not provide an interface for its posture parameter θ in Unity, the control of the posture part in this project is completely based on the adjustment of the rotation of each bone node of the skin model in three dimensions, rather than through the posture parameter θ of SMPL.

[0056] In the present invention, the 17 most important joints of the human body are summarized (1 pelvis, 2 vertebrae, 1 neck, 1 head, 2 hip joints, 2 knees, 2 ankle joints, 2 shoulder joints, 2 elbow joints, and 2 wrist joints). Figure 3 As shown, based on these 17 joints, the natural language commands are converted into a command sequence in the manner described above, and the joints to be adjusted and their movement trends are extracted from the command sequence, so that they are rotated on the corresponding reference plane to achieve a preliminary posture adjustment, and the currently adjusted joints and their movement trends are temporarily stored, so that the user can continue to adjust this trend or make a reverse adjustment by pressing a button before issuing the next command, or adjust the rotation angle of this joint and its associated body parts in real time through the rotation angle of the handle to achieve a better adjustment effect. The mapping relationship between joint rotation and handle rotation is as follows:

[0057] (1)

[0058] in, Represents the rotation offset of the player's controller. represents its current rotation, Indicates the rotation of the previous frame.

[0059] (2)

[0060] in, Indicates the rotation of the corresponding joint point in the next frame, which is obtained by the rotation offset of the player's controller and the current rotation of this joint point.

[0061] The process of deleting a character includes: directly deleting the virtual character currently selected by the user from the scene and releasing the memory occupied by it.

[0062] In summary, a method for generating and editing a humanoid character in a virtual scene is proposed, including:

[0063] Step 1: Use a large language model to translate the user's natural language command into a 1×6 instruction sequence. The instruction sequence is a 1×6 matrix, and each column of the matrix represents: "operation type", "generated gender", "body part to adjust posture", "trend of adjusting body shape", "joint point corresponding to completing this adjustment command", and "motion trend of the corresponding joint point".

[0064] Step 2: by parsing the instruction sequence obtained from method 1, the virtual character is generated, deleted, and the body shape and posture are adjusted;

[0065] Among them, posture adjustment: by analyzing the instruction sequence, the body part to be adjusted and its change trend are obtained, so that the rotation amount of the part is discretely changed by a certain amplitude on the corresponding axis;

[0066] Body shape adjustment: by parsing the instruction sequence, the body part to be adjusted and its changing trend are obtained, so as to change the body shape parameters or change the scaling amount of the corresponding part in the corresponding axis.

[0067] In one embodiment, the body shape adjustment includes: calculating the error by using the mean square formula and taking it as the loss function to fit and obtain the overall body shape change parameters of the skinned human mesh model SMPL, so as to adjust the overall body shape of the virtual character.

[0068] Posture adjustment includes: mapping the spatial rotation information of the user's handle to the spatial rotation of the root node of the virtual character's currently adjusted joint, so as to achieve further adjustment of the virtual character's posture.

[0069] Step two includes: parsing the received instruction sequence at the user end, obtaining the information of "current operation type", "generated gender", "body part for adjusting posture", "trend for adjusting body shape", "joint point corresponding to completing this adjustment command" and "movement trend of corresponding joint point", and further calling the corresponding functional module.

[0070] Furthermore, body shape adjustment may include: simulating body shape changes by scaling the bones of the standard SMPL, and saving the model vertex information before and after the change respectively, to obtain a mesh model with the same total number of vertices and different topological structures, then exhaustively enumerating the vertex information of the mesh obtained by different combinations of the SMPL overall body shape change parameter β, and calculating a loss through the loss function mean square error (MSE) by comparing it with the vertex information of the mesh model obtained after the bone scaling change, and performing gradient descent based on the loss to continuously reduce its value. When the loss decreases sufficiently, it is considered that the parameter value sequence fitted at this time is the parameter that satisfies this body shape change.

[0071] Embodiment 2

[0072] like Figure 2 As shown, an embodiment of the present invention provides a system for generating and editing a humanoid character in a virtual scene, including a speech recognition module and an SMPL skinning model, allowing a user to directly use voice and a handle in a virtual environment to generate a virtual humanoid character and adjust the posture and body shape. The main modules include: a large language model access and receiving module, a speech recognition and content copy module, a command parsing module, a posture adjustment module, and a body shape adjustment module.

[0073] Large language model access and receiving module: Access the trained large language model through the OpenAI interface, send the natural language commands input by the user to it, wait for and receive the command sequence returned by it, and give it to the command parsing module for parsing.

[0074] Speech recognition and content copy module: The text input of the user's natural language command is obtained through the speech recognition plug-in, and it is transmitted to the large language model through network communication access. The large language model is responsible for translating the natural language command into the corresponding instruction sequence. It is worth noting that the speech recognition plug-in here uses the paid plug-in Undertone in the Unity Assets Store. This plug-in is based on a deep learning network model and can achieve real-time speech input and convert it into text to obtain the user's natural language command.

[0075] Instruction parsing module: Receives instruction sequences translated from a large language model through network communication, parses each bit, and performs corresponding operations based on the parsing results. It can be divided into two modes: directly performing corresponding operations or indirectly calling corresponding other modules to complete corresponding operations.

[0076] Posture adjustment module: By parsing the instruction sequence, the body part to be adjusted and its change trend can be obtained, so as to achieve a discrete change of the rotation amount of the part in a certain amplitude on the corresponding axis. Furthermore, the user can use the handle button to make the current part continue the current trend or reverse the current trend to make continuous changes, and can also map the rotation of the root node of the current part node through the rotation of the handle to achieve real-time calibration.

[0077] Body shape adjustment module: By analyzing the instruction sequence, the body part to be adjusted and its change trend can be obtained, so as to change the body shape parameters or change the scaling of the corresponding part in the corresponding axis. Furthermore, the user can also use the handle button to make the current part continue the current trend or reverse the current trend for continuous change.

[0078] The present invention can be applied to the tasks of editing virtual characters and constructing virtual scenes. The voice interaction scheme designed by the present invention without limiting the form of expression can effectively reduce the operational threshold for generating and adjusting virtual humanoid characters in virtual scenes based on head-mounted displays, and the key fine-tuning and rotation mapping schemes can further improve its efficiency, making the overall process more intuitive, clear and easy to use. Compared with the existing methods of directly estimating postures through neural networks (such as PyMAF, CycleAdapt, ROMP, VIBE, HyberIK, etc.), the posture editing method based on a large language model and rotation mapping proposed by the present invention has a higher upper limit of restoration accuracy. Specifically, compared with the same Ground Truth (the real situation, which can be used as a reference in this application as a virtual character model with posture information), the scheme of the present invention can achieve lower average joint errors and average vertex errors at a lower cost on the basis of traditional schemes, that is, it is considered to restore the Ground Truth more. The specific effect is compared with Fig.10 The specific values ​​are shown in the following table.

[0079]

[0080] Among them, MPJPE (Mean Per Joint Position Error), that is, the average joint position error, is used to evaluate the accuracy of the posture. The smaller the value, the better the effect. MPVPE (Mean Per Vertex Position Error), that is, the average vertex position error, is used to evaluate the accuracy of mesh vertices (corresponding to the real body shape). The smaller the value, the better the effect.

[0081] In addition, the method described in the present invention also has strong interactivity that traditional pure posture estimation methods do not have. To a certain extent, it has taken a small step towards the process of moving away from the screen editor, and is more suitable for more immersive application scenarios based on head-mounted display devices.

[0082] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the technical solution content of the present invention still belongs to the technology of the present invention.

Claims

1. A method for generating and editing a humanoid character in a virtual scene, characterized in that: include: Step 1: Use a large language model to translate the user's natural language command into a 1×6 instruction sequence. The instruction sequence is a 1×6 matrix, and each column of the matrix represents: "operation type", "generated gender", "body part to adjust posture", "trend of adjusting body shape", "joint point corresponding to completing this adjustment command", and "motion trend of the corresponding joint point". Step 2: by parsing the instruction sequence obtained from step 1, the virtual character is generated, deleted, and the body shape and posture are adjusted; Among them, posture adjustment: by analyzing the instruction sequence, the body part to be adjusted and its change trend are obtained, so that the rotation amount of the part is discretely changed by a certain amplitude on the corresponding axis; Body shape adjustment: by parsing the instruction sequence, the body part to be adjusted and its changing trend are obtained, so as to change the body shape parameters or change the scaling amount of the corresponding part in the corresponding axis.

2. The method for generating and editing a humanoid character in a virtual scene according to claim 1, characterized in that: Step one includes fine-tuning and training of the large language model. The fine-tuning and training process includes: using ChatGPT to assist in generating and expanding data, that is, first setting the natural language and its corresponding instruction sequence, and then using ChatGPT to understand the semantics and generate as many natural language expressions as possible that can represent this instruction sequence, and then integrating them to obtain a data set. Based on this data set, preprocessing is performed in Python and connected to the OpenAI interface, and then ChatGPT is used to fine-tune and train the large language model.

3. The method for generating and editing a humanoid character in a virtual scene according to claim 1, characterized in that: The step 2 includes: parsing the received instruction sequence at the user end, obtaining the information of "current operation type", "generated gender", "body part to adjust posture", "trend of adjusting body shape", "joint point corresponding to completing this adjustment command" and "movement trend of corresponding joint point", and further calling the corresponding function module.

4. The method for generating and editing a humanoid character in a virtual scene according to claim 1, characterized in that: The body shape adjustment includes: simulating the change of body shape by scaling the bones of the standard SMPL, and saving the vertex information of the model before and after the change respectively, so as to obtain a mesh model with the same total number of vertices but different topological structures, and then exhaustively enumerating the vertex information of the mesh obtained by different combinations of the SMPL overall body shape change parameter β, and calculating a loss through the loss function mean square error by combining it with the vertex information of the mesh model obtained by bone scaling, and performing gradient descent based on this loss to continuously reduce its value. When the loss decreases small enough, it is considered that the parameter value sequence fitted at this time is the parameter that satisfies this body shape change.

5. The method for generating and editing a humanoid character in a virtual scene according to claim 1, characterized in that: Posture adjustment includes: extracting the joint points to be adjusted and their movement trends from the instruction sequence, making them rotate on the corresponding reference plane to achieve preliminary posture adjustment, and temporarily storing the joint point number and its movement trend to be adjusted. The rotation mapping calculation method is as follows: （1） in, Indicates the rotation offset of the handle, represents its current rotation, Indicates that the previous frame was rotated; （2） in, Indicates the rotation of the corresponding joint point in the next frame, which is obtained by the rotation offset of the handle and the current rotation of this joint point.

6. A system for generating and editing humanoid characters in a virtual scene, characterized in that: include: Large language model access and receiving module, speech recognition and content copy module, instruction parsing module, posture adjustment module and body shape adjustment module; Large language model access and receiving module: access the trained fine-tuned large language model through the OpenAI interface, send the natural language commands input by the user to it through network communication, wait for and receive the command sequence returned by it, and give it to the command parsing module for parsing; Speech recognition and content copy module: The speech recognition plug-in obtains the user's natural language command input in text form, accesses and transmits it to the large language model through network communication, and the large language model is responsible for translating the natural language command into the corresponding instruction sequence; Instruction parsing module: receives the instruction sequence translated by the large language model through network communication, parses each bit, and performs corresponding operations based on the parsing results, either directly performing the corresponding operation or indirectly calling other corresponding modules to complete the corresponding operation; Posture adjustment module: by parsing the instruction sequence, it obtains the body part to be adjusted and its change trend, so that the rotation amount of the part is discretely changed by a certain amplitude on the corresponding axis; Body shape adjustment module: by parsing the instruction sequence, the body part to be adjusted and its changing trend are obtained, so as to change the body shape parameters or change the scaling amount of the corresponding part in the corresponding axis.

Images