Hand-object interaction 3D result generation method and system based on large language model, terminal and storage medium

Abstract

The invention discloses a hand-object interaction 3D result generation method and system based on a large language model, a terminal and a storage medium, and the method comprises the steps: obtaining hand-object interaction data, and carrying out the preprocessing and text labeling processing, and obtaining a target 3D interaction data set; determining a preset large language model, and performing model training and model fine tuning on the preset large language model according to the target 3D interaction data set to obtain an interaction 3D generation model; obtaining a user description text, inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data; and performing optimization processing on hand parameters in the interaction 3D generation model according to the preliminary interaction 3D data to obtain optimized hand MANO parameters, and obtaining a target hand-object interaction 3D result according to the optimized hand MANO parameters. According to the method and the device, the interaction 3D generation model is constructed, and the interaction 3D generation model is optimized through the user description text, so that the hand-object interaction 3D result can be quickly and accurately output.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, terminal and computer-readable storage medium for generating 3D results of hand-object interaction based on a large language model. Background Art

[0002] A hand-object interaction 3D model refers to a model that can realize natural and intuitive interaction between users and three-dimensional objects in a virtual environment. This model can not only provide a rich visual experience, but also enhance the user's immersion and participation through advanced interaction technology.

[0003] However, in the prior art, after the hand-object interaction 3D model is generated, there is a lack of effective optimization mechanism to adjust the interaction posture between the user's hand and the object, which results in poor hand-object interaction results generated by the hand-object interaction 3D model in actual applications and fails to meet user needs.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for generating 3D hand-object interaction results based on a large language model, aiming to solve the problem that after the hand-object interaction 3D model is generated, there is a lack of effective optimization mechanism to adjust the interaction posture between the user's hand and the object, which leads to poor hand-object interaction results generated by the hand-object interaction 3D model in actual applications and cannot meet user needs.

[0006] To achieve the above object, the present invention provides a method for generating a 3D result of hand-object interaction based on a large language model, and the method for generating a 3D result of hand-object interaction based on a large language model comprises the following steps: Acquire hand-object interaction data, and perform preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set; Determine a preset large language model, and perform model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model; Obtaining user description text, and inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data; The hand parameters in the interactive 3D generation model are optimized according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and the target hand-object interaction 3D result is obtained according to the optimized hand MANO parameters.

[0007] Optionally, the method for generating 3D results of hand-object interaction based on a large language model, wherein the step of acquiring hand-object interaction data and performing preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set specifically includes: Acquire an open source hand dataset, extract hand-object interaction data from the open source hand dataset, and preprocess the hand-object interaction data to obtain a first 3D interaction dataset; Performing text annotation processing on the hand-object interaction data to obtain a second 3D interaction data set; The first 3D interaction dataset and the second 3D interaction dataset are merged and divided to obtain a target 3D interaction dataset.

[0008] Optionally, in the method for generating 3D results of hand-object interaction based on a large language model, the hand-object interaction data includes a 3D model of the object and a first hand MANO parameter; The preprocessing of the hand-object interaction data to obtain a first 3D interaction data set specifically includes: Performing model face reduction processing, data filtering processing and normalization processing on the 3D model of the object to obtain an integer object model; Performing integer processing on the first hand MANO parameter to obtain an integer hand MANO parameter; The integer object model and the integer hand MANO parameters are combined to obtain a first 3D interaction data set.

[0009] Optionally, the method for generating 3D results of hand-object interaction based on a large language model, wherein the step of performing text annotation processing on the hand-object interaction data to obtain a second 3D interaction data set specifically includes: Performing multi-perspective rendering processing on the hand-object interaction data to obtain multi-perspective 2D images; Performing text description processing on the multiple perspective 2D images to obtain multiple image description texts; Extracting image features and text features from a plurality of the image description texts, and calculating the similarity between the image features and the text features, to obtain a target text description with the highest similarity in each of the image description texts; A second 3D interaction dataset is constructed based on the target text description with the highest similarity in each of the image description texts.

[0010] Optionally, in the method for generating 3D results of hand-object interaction based on a large language model, the target 3D interaction dataset includes a training set and an evaluation set; The determining of a preset large language model, and performing model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model specifically includes: Determining a preset large language model, and performing model training processing on the preset large language model according to the training set to obtain an initial interactive 3D generation model; The LoRA algorithm is used to perform model fine-tuning processing on the initial interactive 3D generation model according to the evaluation set to obtain an interactive 3D generation model.

[0011] Optionally, the method for generating 3D results of hand-object interaction based on a large language model, wherein the acquiring of user description text, inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data, specifically comprises: Obtain user description text and set custom hyperparameters, wherein the custom hyperparameters include text randomness and text length; Inputting the user description text and the custom hyperparameters into the interactive 3D generation model, and performing word segmentation, encoding and formatting processing on the user description text through the interactive 3D generation model to obtain serialized data; The context features of the serialized data are extracted through a self-attention mechanism, and the context features are decoded to obtain preliminary interactive 3D data.

[0012] Optionally, the method for generating a hand-object interaction 3D result based on a large language model, wherein the hand parameters in the interaction 3D generation model are optimized according to the preliminary interaction 3D data to obtain optimized hand MANO parameters, and the target hand-object interaction 3D result is obtained according to the optimized hand MANO parameters, specifically comprising: Segmenting the preliminary interactive 3D data to obtain an object OBJ model and a second hand MANO parameter; Calculate the corresponding object BPS distribution according to the object OBJ model, and calculate the surface distance between the hand vertex and the object according to the second hand MANO parameter; Optimizing the hand parameters of the interactive 3D generated model according to the BPS distribution of the object, the surface distance between the hand vertex and the object, and the second hand MANO parameters to obtain optimized hand MANO parameters; The hand MANO parameters are combined with the object OBJ model to obtain a target hand-object interaction 3D result.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides a hand-object interaction 3D result generation system based on a large language model, wherein the hand-object interaction 3D result generation system based on a large language model comprises: A hand-object interaction data processing module is used to obtain the hand-object interaction data, and perform preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set; An interactive 3D generation model construction module is used to determine a preset large language model, and perform model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model; A preliminary interactive 3D data output module, used for acquiring user description text, inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data; The target hand-object interaction 3D result generation module is used to optimize the hand parameters in the interactive 3D generation model according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and obtain the target hand-object interaction 3D result according to the optimized hand MANO parameters.

[0014] In the present invention, hand-object interaction data is obtained, and the hand-object interaction data is preprocessed and text-annotated to obtain a target 3D interaction data set; a preset large language model is determined, and model training and model fine-tuning are performed on the preset large language model according to the target 3D interaction data set to obtain an interactive 3D generation model; user description text is obtained, and the user description text is input into the interactive 3D generation model to output preliminary interactive 3D data; hand parameters in the interactive 3D generation model are optimized according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and target hand-object interaction 3D results are obtained according to the optimized hand MANO parameters. The present invention collects hand-object interaction data, and preprocesses and text-annotates to construct a target 3D interaction data set, and performs model training and model fine-tuning on the large language model according to the target 3D interaction data set to obtain an interactive 3D generation model, and then optimizes the interactive 3D generation model through user description text, so that the hand-object interaction 3D results can be output quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 2 It is a schematic diagram of the overall structure implementation process of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 3It is a schematic diagram of data set generation of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 4 3. It is a data set preprocessing schematic diagram of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 5 It is a schematic diagram of a text annotation process of a preferred embodiment of a method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 6 It is a schematic diagram of model fine-tuning of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 7 It is a schematic diagram of the processing flow of the interactive 3D generation model of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Figure 8 It is a schematic diagram of the processing flow of the interactive 3D optimization model of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Fig. 9 It is a schematic diagram of optimizing the generation of hand MANO parameters of a preferred embodiment of the method for generating 3D results of hand-object interaction based on a large language model of the present invention; Fig.10 3D result generation diagram of a target hand-object interaction according to a preferred embodiment of the method for generating a hand-object interaction 3D result based on a large language model of the present invention; Fig.11 It is a structural diagram of a preferred embodiment of the hand-object interaction 3D result generation system based on a large language model of the present invention; Fig.12 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] The existing 3D generation models have the following major problems in the generation of hand-object interaction 3D: 1. The annotation effect of interactive 3D models is poor: the existing annotation process uses a direct description method for 3D models, which cannot fully and detailedly describe the details of hand-object interaction. There is a lack of annotation processes for interaction. The system can only roughly describe the 3D model as a whole, and cannot accurately describe the specific holding method of a specific object. 2. The large language model is not pre-trained enough for hand-object interaction: the existing 3D generation models (such as MeshXL) are mainly pre-trained for a large number of 3D object models, and lack special training for hand-object interaction scenes. Therefore, when these models generate hand-object interaction 3D models, it is difficult to accurately understand the interaction description input by the user (such as "pick up a water cup with your hand"), and it is also impossible to generate a 3D model that conforms to physical laws and visual realism. 3. The ability of interactive 3D optimization models is limited: after generating hand-object interaction 3D models, the existing technology lacks an effective optimization mechanism to adjust the interaction posture of the user's hand and the object. For example, key factors such as the distance between the hand vertices and the object surface and the influence of the object shape on the hand posture have not been fully considered, resulting in obvious physical irrationality in the generated 3D model in practical applications (such as the hand penetrating the object, unnatural holding posture, etc.).

[0018] To solve the above problems, the present invention provides a method and system for generating 3D hand-object interaction based on large language model fine-tuning technology, and the specific purposes include: 1. Improving the 3D model annotation effect: through an innovative multimodal annotation process, calling BLIP (Bootstrapping Language-Image Pre-training, a visual pre-training framework) and CLIP (Contrastive Language-Image Pre-Training, a deep learning model, a pre-training model that can process text and images at the same time), generating detailed and accurate hand-object interaction 3D model annotation. A detailed description is made for the specific holding method of a specific object (such as picking up, passing, using, etc.), solving the problem of incomplete and inaccurate annotation information in the prior art, and providing high-quality data support for model fine-tuning. 2. Enhancing the interaction understanding ability of the large language model: by fine-tuning the pre-trained large language model, it can better understand the interaction description input by the user, and preliminarily generate a 3D model that conforms to physical laws and visual realism, solving the problem of insufficient pre-training of the existing large language model in the interaction scene. 3. Optimize the interactive posture of the generated model: By introducing an optimization module, the generated 3D model of hand-object interaction is finely adjusted to ensure that the interactive posture of the hand and the object conforms to the laws of physics (avoiding the hand from penetrating the object and making the holding posture natural, etc.), solve the problem of limited optimization ability of the generated model in the existing technology, and improve the practicality and realism of the 3D model. 4. Build an end-to-end 3D generation system: By designing a complete process from text description to 3D model generation, including data collection, preprocessing, annotation, model fine-tuning, generation and optimization modules, an efficient and accurate hand-object interaction 3D generation system is built to solve the problems of scattered processes and low efficiency in the existing technology, and significantly improve the generation efficiency and quality of 3D models.

[0019] The technologies used in the present invention include: 1. Natural Language Processing: Using a pre-trained large language model (such as LLama, Large Language Model Meta AI), the user text description is converted into a 3D hand-object interaction model through fine-tuning technology. 2. Computer Vision: Use multi-view rendering technology to render the 3D model into a 2D image, and run the BLIP model to generate an image description. 3. Multimodal Artificial Intelligence: By fusing text, images and 3D data, combining BLIP and CLIP models, multi-view 3D interaction data annotations are generated, and multimodal technology is used to improve the accuracy and diversity of 3D model annotations. 4. 3D Model Generation Technology: By fine-tuning the LLama model, a 3D object model (OBJ format) and hand posture parameters (MANO model) are directly generated from the text description, and the interaction posture between the hand and the object is adjusted using the optimization module to generate a high-quality hand-object interaction 3D model.

[0020] The method for generating 3D results of hand-object interaction based on a large language model according to a preferred embodiment of the present invention is as follows: Figure 1 As shown, the method for generating 3D results of hand-object interaction based on a large language model includes the following steps: Step S10: Acquire hand-object interaction data, and perform preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set, wherein the hand-object interaction data includes an object 3D model and a first hand MANO parameter.

[0021] like Figure 2 As shown in the figure, the present invention proposes a hand-object 3D interaction generation system based on large language model (LLM) fine-tuning technology. Through the two stages of generation and optimization, the system can realize the generation from user text description to 3D model. The system workflow is as follows: 1. Collect multiple open source hand datasets (i.e. Figure 2 3D data samples in the 3D data sample) in the hand-object interaction data, pre-process the objects and hands, and construct an unlabeled 3D interaction dataset (belonging to Figure 2 2. Use BLIP and CLIP to annotate the object interaction data and construct annotated 3D interaction dataset (part of Figure 23. Use the annotated 3D model dataset to fine-tune the open source large language model LLama as the interactive 3D generation model. 4. Use the interactive 3D generation model to generate a preliminary interactive 3D model (object and hand) based on the user's text description. 5. Extract the distance between the hand vertices and the object surface in the preliminary interactive 3D model, the object BPS distribution (Basis Point Sets, a point cloud encoding technology that aims to efficiently convert three-dimensional point cloud information into fixed-length feature vectors. This method selects fixed base points in space and calculates the vectors from these base points to the nearest points in the point cloud as unique representation features) and the hand MANO parameters as input to the optimization module, and optimizes the generated hand model as the final output.

[0022] This paper fully combines the image understanding and annotation capabilities of the (multimodal) large language model technology, designs a new 3D model annotation method, and innovatively uses the interactive 3D model as the fine-tuning target, allowing the LLama language model to directly generate a hand-object interaction 3D model for the first time.

[0023] Specifically, an open source hand dataset is obtained, and hand-object interaction data in the open source hand dataset is extracted, and the object 3D model is subjected to model face reduction processing, data filtering processing, and normalization processing to obtain an integer object model; the first hand MANO parameters are integerized to obtain integer hand MANO parameters; the integer object model and the integer hand MANO parameters are merged to obtain a first 3D interaction dataset.

[0024] like Figure 3 As shown, the dataset generation process is as follows: 1. Collect hand-object interaction datasets: widely collect hand-object interaction data from open source hand datasets, including various ways of holding different objects by different subjects. 2. Call the data preprocessing process to generate 3D interaction data: use an innovative data preprocessing process to generate a high-quality 3D interaction dataset for subsequent training and data annotation. 3. Call the text annotation process to generate a complete 3D interaction dataset: use an innovative text annotation method to generate a detailed and complete 3D interaction text description, and build a complete 3D interaction dataset for model fine-tuning. 4. Call the dataset post-processing process: merge the preprocessed 3D interaction data with the text annotation, organize the dataset in the LLama3 (a large language model) format, and use it for LLama3 model fine-tuning.

[0025] Introduction to data preprocessing process: This paper collects a large amount of high-quality and diverse 3D hand-object interaction sample data. Preprocessing and text annotation of these data play a key role in improving the model's ability to generate 3D interaction models. The following is the specific process of data collection and data set preprocessing: 1. Data collection: The present invention extracts hand-object 3D interaction data from the Grab dataset (referring to the dataset of whole-body movements for interacting and grabbing 3D objects) and the Oakink dataset (referring to the dataset for image-to-3D tasks) (the dataset itself is composed of object 3D models and hand MANO parameters (hand MANO parameters can also be called hand posture parameters), and each sample contains one data item: hand-object contact area. In the present invention, the object model and hand parameters corresponding to the frame with the largest hand-object contact area in a video sequence in the dataset can be directly selected as training data). These datasets are composed of continuous video frames. In order to ensure the reliability of the collected data, the 3D interaction data of the frame with the largest contact area between the hand and the object in each video is extracted as available data. Considering the diversity of objects and hand movements, the dataset collects a total of up to 157 objects, such as water cups, mobile phones, hammers, mice, staplers, etc. that can interact with the hands, and at the same time, different interaction methods are performed on the objects, such as picking up, passing, using, etc., to avoid the model from overfitting to the characteristic interaction actions of specific objects.

[0026] 2. Dataset preprocessing: Dataset preprocessing includes two parts: object model processing and hand MANO parameter processing (the hand-object interaction data in the present invention includes two parts, one is the object 3D model, and the other is the hand MANO parameter (also called hand posture parameter). The two together are called hand-object interaction data, also called hand-object 3D interaction data). The overall process is as follows: Figure 4 As shown: The object model processing process is as follows: In the present invention, the object mesh is expressed in OBJ plain text format, and the preprocessing process is as follows: a. Model reduction (i.e. Figure 4 "Model simplification": Too many faces of an object will cause too many tokens to be occupied in the fine-tuning of the language model. The Quadric Edge Collapse Decimation method is used to reduce the face of the object model (the Quadric Edge Collapse Decimation method is used for model reduction. Quadric Edge Collapse Decimation is an algorithm for mesh simplification in 3D graphics. It improves rendering efficiency by reducing the number of vertices in the mesh while retaining the shape and details of the model as much as possible. The core idea of ​​the algorithm is to simplify the mesh by gradually merging adjacent vertices. Each time a merge occurs, the algorithm calculates the cost of the edge, selects the edge with the smallest cost for contraction, and updates the error metric of the merged vertices to minimize the impact on the appearance of the model. It is widely used in 3D model optimization and real-time rendering). b. Data filtering (i.e. Figure 4"Remove low-quality objects" in the above example): Both methods of face reduction may cause distortion of the object model. The object model is filtered according to the original shape and representation semantics of the model to filter out the data with model distortion and face reduction failure. c. Data sorting (i.e. Figure 4 In the “integerized to 0-64” section, in order to adapt to the autoregressive characteristics of the large language model, the model needs to gradually predict the next token according to the context of the sequence, and the data needs to be organized in a certain order to maintain logical consistency. The 3D coordinates of all object models are normalized and integerized to the range of 0-64. At the same time, the vertices of the OBJ file are sorted in the order of xyz coordinates from small to large to ensure the order and consistency of the coordinate data, so as to facilitate the subsequent tokenization of the data and use it for fine-tuning training. Among them, the hand model processing process is as follows: In the present invention, the MANO parameters are used to express the hand 3D model, the hand MANO parameters are integerized to the range of 0-512, and merged with the object OBJ file (string concatenation) to construct interactive 3D data.

[0027] Introduction to 3D objects in OBJ format: The present invention uses OBJ (Wavefront OBJ) plain text format to express object meshes. It is a plain text file format used to represent three-dimensional geometric shapes and is commonly used to store object meshes. It describes geometric information such as vertices, faces, and normal vectors of objects, which facilitates the storage and exchange of models. Among them, the introduction of geometric information is as follows: 1. Vertex (v): The coordinates of a point in three-dimensional space are represented by v, x, y, and z. 2. Normal vector (vn): The normal direction of the vertex is represented by vn, x, y, and z, which is used for lighting calculations. 3. Texture coordinates (vt): The coordinates of the texture map are represented by vt, u, and v. 4. Face (f): The polygonal face is represented by f v1 / vt1 / vn1, f v2 / vt2 / vn2..., where the vertex index is described in the order of vertex, texture, and normal.

[0028] OBJ files are simple and readable. In order to retain the most important information of the 3D model and facilitate subsequent model fine-tuning, only the vertex (v) and face (f) data in the OBJ format are retained to construct the 3D interactive dataset.

[0029] Perform multi-perspective rendering processing on the hand-object interaction data to obtain multiple-perspective 2D images; perform text description processing on the multiple perspective 2D images to obtain multiple image description texts; extract image features and text features from the multiple image description texts, and calculate the similarity between the image features and the text features to obtain the target text description with the highest similarity in each of the image description texts; construct a second 3D interaction dataset based on the target text description with the highest similarity in each of the image description texts; merge and divide the first 3D interaction dataset and the second 3D interaction dataset to obtain a target 3D interaction dataset.

[0030] Introduction to text annotation process: This paper improves and innovates the Cap3D annotation method, combines BLIP and clip models, and adjusts the calling method and annotation merging method of the BLIP model, so that it can accurately and detailedly describe the collected 3D interaction data and generate diverse and rich description texts, such as Figure 5 As shown, the steps include: 1. 3D object rendering: observe and render the collected hand-object interaction 3D models from multiple perspectives (use Blender (an open source 3D computer graphics software) to load the 3D model, set the camera's perspective at different positions, and then generate 8 images from different perspectives) into 2D images for subsequent text annotation. 2. Call the BLIP model to generate preliminary text descriptions: Use the QA mode (the QA (Question Answering) mode of the BLIP model is a multimodal question-answering capability based on visual-language pre-training, which can combine image and text information to answer questions raised by users. In this mode, users may input text questions into the model multiple times, and the model answers based on the text input and image) to call the BLIP model multiple times to describe images from different perspectives, first ask the type of object in the image, set the object type as the prompt for the next inquiry, and require the BLIP model to describe the objects in the image and the details of the hand interaction. In order to ensure the correctness of the description generation, the BLIP model is required to generate 5 results at the same time for subsequent text description screening and merging. 3. Call CLIP for text screening and merging: For the five text descriptions generated by the BLIP model, use the CLIP model to extract the image features and the five text description features under this perspective, and use similarity calculation to screen out the one of the five text descriptions that is closest to the image features as the description under this perspective.

[0031] For the text descriptions from eight perspectives generated by a 3D interaction model, a large language model is called to merge the text descriptions from the eight perspectives into one, emphasizing the details of the hand-object interaction and ignoring the background in the description. The text descriptions from multiple perspectives are integrated to make the description more detailed and complete.

[0032] Step S20: determine a preset large language model, and perform model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model. The target 3D interactive data set includes a training set and an evaluation set.

[0033] This paper generates a high-quality 3D hand-object interaction dataset through data collection, preprocessing, text annotation and post-processing, which conforms to the LLama3.1 fine-tuning format. Key frames are extracted from the Grab and Oakink datasets, object models and hand parameters are processed, and detailed interaction descriptions are generated by combining BLIP, CLIP and ChatGPT. The final output is a standardized JSON file containing text descriptions and 3D data, which is directly used for model fine-tuning.

[0034] Specifically, a preset large language model is determined, and model training processing is performed on the preset large language model according to the training set to obtain an initial interactive 3D generation model; and a LoRA algorithm is used to perform model fine-tuning processing on the initial interactive 3D generation model according to the evaluation set to obtain an interactive 3D generation model.

[0035] Introduction to the data set post-processing process: 1. Data set organization: Large model fine-tuning requires organizing the data set in a certain format. For the LLama3.1 model used in this invention, the LLama3 template is used to organize the data. Specifically, the description obtained by text annotation is used as Instruction, the object OBJ format data and hand MANO parameters in the 3D data set are combined as Output, and the Input is set to an empty string. Multiple 3D data are organized into json files in the above format for large model fine-tuning. 2. Divide the data set: Randomly shuffle the organized data set and divide the data set into 80% training set and 20% evaluation set. 3. Data set preservation: After the data processing is completed, the training set and the evaluation set are stored separately for subsequent use.

[0036] like Figure 6 As shown in the figure, the specific process of model fine-tuning is as follows: 1. Fine-tuning training: Take the training set from the data set and perform fine-tuning training for hand-object 3D interaction generation. 2. Evaluation and verification: During the fine-tuning process, regularly select the evaluation set data for model evaluation and check the model fine-tuning results.

[0037] The training process of the model is as follows: use SFT (supervised fine-tuning) to fine-tune the LLama model, use LoRA to update and fine-tune some parameters of the LLama model, and the fine-tuning strategy is autoregressive next Token prediction. The loss is calculated based on the probability distribution of the next Token predicted each time and the actual next Token, and the gradient descent algorithm is used to update some LoRA parameters in the LLama model.

[0038] Algorithm model introduction: 1. Fine-tuning settings: According to the characteristics of the interactive 3D generation task, design a suitable fine-tuning scheme, select hyperparameters such as the number of fine-tuning layers, learning rate, batch-size, and number of training rounds, and determine the training target and loss function. 2. Model fine-tuning: Load the pre-trained LLama model and the prepared hand-object interaction 3D data into the fine-tuning script using the LoRA fine-tuning technology to start the fine-tuning process. LoRA (Low-Rank Adaptation) technology is an efficient model fine-tuning method that fine-tunes the model behavior by introducing small, low-rank matrices in the key layers of the model without making major changes to the entire model structure. The model learns the overall description of the hand-object interaction 3D data in a supervised manner, and what kind of object is held in what posture. Fine-tuning (subtle adjustment or fine-tuning) the original parameters to adapt to the interactive 3D generation task. 3. Evaluation and verification: Regularly evaluate the generation effect of the model during the fine-tuning process, use methods such as cross-validation to evaluate the performance of the model in interactive 3D generation, adjust the fine-tuning strategy based on the evaluation results, and optimize the model. 4. Model saving: After the fine-tuning process is completed, save the fine-tuned model parameters for subsequent use and deployment.

[0039] The core function of model fine-tuning is to fine-tune the LLama model in a targeted manner to improve the generation capability for interactive 3D, which serves as the input for the subsequent interactive 3D optimization model. At the same time, the fine-tuning model adopts a unified data organization form, which facilitates the direct extraction of the generated object mesh and hand parameters from the model output, which is convenient for subsequent optimization.

[0040] Step S30: Obtain user description text, input the user description text into the interactive 3D generation model, and output preliminary interactive 3D data.

[0041] like Figure 7 As shown, the specific processing flow of the interactive 3D generation model is as follows: 1. Receiving user input: The user calls the web Gradio platform developed by the present invention, enters a text description in a specified position, and sets custom hyperparameters (such as temperature, max_length, etc.), and automatically obtains the parameters as the input of the interactive 3D generation model. 2. Interactive 3D model generation: Obtain the text description and hyperparameters input by the user, and the interactive 3D generation model generates preliminary interactive 3D data.

[0042] There are two custom hyperparameters, temperature and max_length, which control the randomness and length of the generated text respectively. temperature is used to adjust the diversity of the generated text. The higher the value, the more random and diverse the generated text is. It may generate a more flexible 3D interaction model based on the user's input. Conversely, it is more fixed and generates a more conservative but accurate 3D interaction model. max_length is used to control the maximum length of the generated text. If the generated interaction model is more complex, it is necessary to adjust the output length to fully output the interaction model.

[0043] Specifically, a user description text is obtained, and custom hyperparameters are set, wherein the custom hyperparameters include text randomness and text length; the user description text and the custom hyperparameters are input into the interactive 3D generation model, and the user description text is segmented, encoded, and formatted by the interactive 3D generation model to obtain serialized data; context features of the serialized data are extracted by a self-attention mechanism, and the context features are decoded to obtain preliminary interactive 3D data.

[0044] Introduction to the algorithm model: LLama (Large Language Model Meta AI) is a high-performance large-scale language model for natural language processing tasks. In the LLama system, a series of preprocessing operations are first performed on the input text data, including word segmentation, encoding, and format standardization, to convert the original text into serialized data in a unified format. Subsequently, the text data will be input into a multi-layer Transformer architecture, and the contextual features of the text will be extracted through the self-attention mechanism to understand the semantics and structural relationships in the sentence.

[0045] Furthermore, the LLama model generates task-related output content through a highly optimized decoding process. Whether it is text generation, translation, or question-answering tasks, LLama can provide high-quality results based on context. Its core lies in its powerful pre-training mechanism and highly adjustable fine-tuning capabilities, which enable it to quickly adapt to different application scenarios.

[0046] The present invention can generate the corresponding 3D object (expressed in OBJ format) and its gesture parameters for hand interaction (based on MANO model) according to the text description input by the user by fine-tuning the LLama-based hand-object interaction 3D data generation model. By combining semantic understanding and 3D data mapping capabilities, the model can accurately generate high-fidelity objects and hand interaction gestures that meet the description, providing reliable data support for application scenarios such as physical simulation, visual analysis, and virtual reality.

[0047] The interactive 3D generation model is the first generation of the hand-object 3D interaction. Its main function is to generate a preliminary object OBJ model and the hand MANO parameters for interaction with it based on the user's text description. It is the key to optimizing the hand parameters based on the object later. Output the object 3D OBJ format model and hand MANO parameters for subsequent hand parameter optimization.

[0048] Step S40: Optimize the hand parameters in the interactive 3D generation model according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and obtain the target hand-object interaction 3D result according to the optimized hand MANO parameters.

[0049] like Figure 8 As shown in the figure, the specific processing flow of the interactive 3D optimization model is as follows: 1. Preliminary interactive 3D data segmentation: Segment the output of the interactive 3D generation model according to the standard form of data set organization and extract the object OBJ model (i.e. Figure 8 obj model in the object) and the hand MANO parameters (i.e. Figure 8The hand MANO parameters in the interactive 3D optimization model are input into the interactive 3D optimization model. 2. 3D model optimization: Calculate the object BPS distribution based on the generated object OBJ model, render the hand model based on the hand MANO parameters and calculate the point-to-point signed distance between the hand vertex and the object (the hand 3D model rendered by the hand MANO parameters contains 778 vertices. For each hand vertex, select a vertex of the object model closest to it and calculate the signed distance between the two (the hand vertex is a positive number outside the object model and a negative number inside the object model)). Combine the generated hand MANO parameters as the input of the interactive 3D optimization model to optimize the hand parameters (the 3D optimization model is a neural network. The input is "object BPS distribution, hand-object vertex signed distance, hand MANO parameters", and the output is the optimized hand MANO parameters. For each training sample, there is a MANO parameter when the data set is collected as the true label. By calculating the error loss between the generated hand MANO parameters and the true MANO parameters, the 3D optimization model parameters are updated by the gradient descent method, so that the model can finally optimize the hand parameters according to the input hand-object information and generate reasonable hand-object interaction results). 3. Generation of interactive 3D results: Input the MANO model according to the MANO parameters of the hand optimized by the interactive optimization model to render the hand model, combine the hand model with the object model to obtain a complete hand-object interactive 3D result, and use the Trimesh library (Trimesh is a pure Python tool library for loading and using triangular Mesh meshes) to load and display it to the specified location of the Web Gradio interface (referring to the Web interface of the machine learning model created using the Gradio Python library) developed by the present invention.

[0050] Specifically, the preliminary interactive 3D data is segmented to obtain an object OBJ model and a second hand MANO parameter; the corresponding object BPS distribution is calculated according to the object OBJ model, and the surface distance between the hand vertices and the object is calculated according to the second hand MANO parameter; the hand parameters of the interactive 3D generated model are optimized according to the object BPS distribution, the surface distance between the hand vertices and the object, and the second hand MANO parameter to obtain optimized hand MANO parameters; the hand MANO parameters are combined with the object OBJ model to obtain a target hand-object interaction 3D result.

[0051] The interactive 3D module is a new deep learning neural network designed by the present invention. It is a neural network model for generating three-dimensional data of hand-object interaction. The model takes hand-object distance features, hand posture rotation matrix and translation vector as input, combines object vertex data, and generates accurate hand parameters (including global direction, hand posture, translation vector and complete posture matrix) through iterative optimization.

[0052] like Fig. 9 As shown, after the fine-tuning training of the present invention, the interactive 3D module can learn more complex hand-object interaction patterns, optimize the preliminary hand-object 3D interaction model generated by a given text description, and generate a reliable hand-object interaction model that conforms to the text description. The specific steps are as follows: 1. MANO parameter processing: Use the MANO rendering model to render the generated MANO parameters into a hand 3D model, and extract its vertex coordinate information for subsequent calculation of the directed distance between the object. 2. Object model processing: Use the BPS encoding method to encode the object model into a 4096-dimensional vector. At the same time, calculate the directed distance between each vertex of the hand model and the object model, and both serve as inputs to the 3D optimization module. 3. Generate optimized hand parameters: The 3D optimization module receives three inputs: the BPS representation of the object, the hand-object vertex distance, and the hand MANO parameters. The fine-tuned neural network can optimize the hand parameters to make its interaction with the object more realistic and reasonable.

[0053] Introduction to MANO rendering model: MANO rendering model is a parametric model used to generate 3D hand meshes. Its input includes 48-dimensional hand posture parameters and 3-dimensional spatial translation parameters. The posture parameters use axis angles to represent joint rotations, and the translation parameters are used to control the spatial position of the hand. The model output is a 3D hand mesh containing 778 vertices, and a dynamic hand 3D model is generated through linear blending deformation.

[0054] Introduction to BPS encoding: BPS encoding (Basis Point Set) is a method of representing 3D objects as fixed-dimensional vectors. This method first defines a set of fixed base points in the 3D space of the object. The number of these base points determines the dimension of the final encoding, for example, 4096 base points correspond to a 4096-dimensional vector. For each base point, BPS calculates its Euclidean distance to the nearest point on the surface of the object, and arranges these distances to form a feature vector. In this way, BPS is able to encode the shape information of 3D objects into vectors of fixed length while maintaining sensitivity to the spatial structure of the object. This encoding method has the characteristic of being independent of the rotation and translation of the object, so it is suitable for a variety of 3D shape analysis tasks.

[0055] It can be understood that the present invention optimizes the generated object OBJ model and hand MANO parameters, and iteratively adjusts the hand parameters to generate more realistic and reasonable hand-object interaction results by taking the object BPS distribution, hand-object vertex directed distance and initial hand MANO parameters as input. The output of the model is the optimized hand MANO parameters, including global direction, hand posture, translation vector and complete posture matrix, which are used for final result visualization.

[0056] like Fig.10 As shown, the present invention uses the MANO rendering model to render the optimized hand MANO parameters into a 3D hand model, and uses the Trimesh library to combine the object model and the hand model to generate the final hand-object interaction 3D model.

[0057] Beneficial effects of the present invention: 1. 3D data annotation process: Combining large language models such as BLIP and CLIP, a new 3D interactive data annotation process is proposed. Through multi-view rendering and multi-model collaboration, high-quality and diverse 3D interactive data annotations are generated, significantly improving data availability.

[0058] 2. Fine-tuning of the large language model: Use 3D data of hand-object interaction to fine-tune the LLama model to directly generate 3D objects and hand gestures from text descriptions, breaking through the limitations of traditional language models and expanding the application of language models in 3D generation tasks.

[0059] 3. Efficient hand parameter optimization module: Design a new hand optimization module to optimize hand posture parameters through object BPS representation and hand-object distance, significantly improve the authenticity and rationality of the generated model, and ensure that the interactive posture conforms to the laws of physics.

[0060] 4. Construction of high-quality 3D datasets: Through innovative preprocessing and annotation processes, a diverse 3D hand-object interaction dataset is constructed, covering a variety of objects and interaction methods, ensuring data reliability and diversity, and providing a solid foundation for model training.

[0061] 5. Flexible user interaction and generation: Develop user interfaces based on the Web Gradio platform, which supports rapid generation and optimization of 3D interactive models through text descriptions and parameter settings to meet personalized needs and improve user experience and system practicality.

[0062] This paper proposes a hand-object interaction 3D generation system based on large language model (LLM) fine-tuning technology. By combining multimodal large models such as BLIP, it realizes the generation and optimization of high-quality hand-object interaction 3D models from user text descriptions. Its core innovations are as follows: 1. This paper innovates the 3D data annotation process and proposes a new 3D data annotation method by combining multimodal models such as BLIP, CLIP and ChatGPT. By performing multi-view rendering on the 3D hand-object interaction model to generate a 2D image, and using an improved BLIP calling method to generate detailed and accurate 3D interaction text annotations, the annotation quality and diversity are significantly improved, providing high-quality data support for model training.

[0063] 2. This invention has achieved a breakthrough in the fine-tuning technology of the large language model. For the first time, the 3D data of hand-object interaction is used to fine-tune the LLama large language model, enabling it to generate 3D object models and hand MANO parameters directly from text descriptions. This innovation expands the application scope of the large language model, enabling it to understand complex interaction semantics and generate high-fidelity 3D models, providing technical support for scenarios such as physical simulation and virtual reality.

[0064] 3. The present invention designs an efficient hand parameter optimization module, which iteratively optimizes the hand posture by calculating the directed distance between the hand vertices and the object surface and combining the object BPS distribution and hand MANO parameters. This process ensures that the generated hand-object interaction posture conforms to the laws of physics and significantly improves the authenticity and rationality of the model.

[0065] 4. This paper has made an important contribution to the construction of high-quality 3D datasets. Through innovative preprocessing and annotation processes, a diverse dataset covering a large number of objects and multiple interaction modes has been constructed. After preprocessing steps such as face reduction, filtering and sorting, the high quality and diversity of the data are ensured, providing a solid foundation for model training and related research.

[0066] The proposal of this invention marks the deep integration of large language models and 3D generation technology, and opens up a new path for the generation and optimization of hand-object interaction 3D models. In the future, with the development of multimodal technology, the technical framework of this invention is expected to be expanded to the fields of virtual reality, robot operation, etc., promoting the innovation and progress of human-computer interaction technology.

[0067] Furthermore, if Fig.11 As shown, based on the above-mentioned method for generating 3D results of hand-object interaction based on a large language model, the present invention also provides a system for generating 3D results of hand-object interaction based on a large language model, wherein the system for generating 3D results of hand-object interaction based on a large language model includes: A hand-object interaction data processing module 51 is used to obtain the hand-object interaction data, and perform preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set; An interactive 3D generation model building module 52 is used to determine a preset large language model, and perform model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model; A preliminary interactive 3D data output module 53 is used to obtain user description text, input the user description text into the interactive 3D generation model, and output preliminary interactive 3D data; The target hand-object interaction 3D result generation module 54 is used to optimize the hand parameters in the interactive 3D generation model according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and obtain the target hand-object interaction 3D result according to the optimized hand MANO parameters.

[0068] Furthermore, if Fig.12 As shown, based on the above-mentioned method and system for generating 3D results of hand-object interaction based on a large language model, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig.12 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0069] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a hand-object interaction 3D result generation program 40 based on a large language model is stored on the memory 20, and the hand-object interaction 3D result generation program 40 based on a large language model can be executed by the processor 10, thereby realizing the hand-object interaction 3D result generation method based on a large language model in the present application.

[0070] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the method for generating 3D results of hand-object interaction based on a large language model.

[0071] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0072] In one embodiment, when the processor 10 executes the hand-object interaction 3D result generation program 40 based on the large language model in the memory 20, the steps of the hand-object interaction 3D result generation method based on the large language model are implemented.

[0073] In summary, the present invention provides a method, system and terminal for generating 3D results of hand-object interaction based on a large language model, the method comprising: obtaining hand-object interaction data, and preprocessing and text annotation processing the hand-object interaction data to obtain a target 3D interaction data set; determining a preset large language model, and performing model training and model fine-tuning on the preset large language model according to the target 3D interaction data set to obtain an interactive 3D generation model; obtaining user description text, and inputting the user description text into the interactive 3D generation model to output preliminary interactive 3D data; optimizing the hand parameters in the interactive 3D generation model according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and obtaining the target hand-object interaction 3D result according to the optimized hand MANO parameters. The present invention collects hand-object interaction data, and performs preprocessing and text annotation processing to construct a target 3D interaction data set, and performs model training and model fine-tuning on the large language model according to the target 3D interaction data set to obtain an interactive 3D generation model, and then optimizes the interactive 3D generation model through the user description text, so as to quickly and accurately output the hand-object interaction 3D result.

[0074] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0075] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0076] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating 3D results of hand-object interaction based on a large language model, characterized in that: The method for generating 3D results of hand-object interaction based on a large language model includes: Acquire hand-object interaction data, and perform preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set; Determine a preset large language model, and perform model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model; Obtaining user description text, and inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data; The hand parameters in the interactive 3D generation model are optimized according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and the target hand-object interaction 3D result is obtained according to the optimized hand MANO parameters.

2. The method for generating 3D results of hand-object interaction based on a large language model according to claim 1, characterized in that: The acquiring of the hand-object interaction data, and performing preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set specifically includes: Acquire an open source hand dataset, extract hand-object interaction data from the open source hand dataset, and preprocess the hand-object interaction data to obtain a first 3D interaction dataset; Performing text annotation processing on the hand-object interaction data to obtain a second 3D interaction data set; The first 3D interaction dataset and the second 3D interaction dataset are merged and divided to obtain a target 3D interaction dataset.

3. The method for generating 3D results of hand-object interaction based on a large language model according to claim 2, characterized in that: The hand-object interaction data includes a 3D model of the object and a first hand MANO parameter; The preprocessing of the hand-object interaction data to obtain a first 3D interaction data set specifically includes: Performing model face reduction processing, data filtering processing and normalization processing on the 3D model of the object to obtain an integer object model; Performing integer processing on the first hand MANO parameter to obtain an integer hand MANO parameter; The integer object model and the integer hand MANO parameters are combined to obtain a first 3D interaction data set.

4. The method for generating 3D results of hand-object interaction based on a large language model according to claim 2, characterized in that: The performing text annotation processing on the hand-object interaction data to obtain a second 3D interaction data set specifically includes: Performing multi-perspective rendering processing on the hand-object interaction data to obtain multi-perspective 2D images; Performing text description processing on the multiple perspective 2D images to obtain multiple image description texts; Extracting image features and text features from a plurality of the image description texts, and calculating the similarity between the image features and the text features, to obtain a target text description with the highest similarity in each of the image description texts; A second 3D interaction dataset is constructed based on the target text description with the highest similarity in each of the image description texts.

5. The method for generating 3D results of hand-object interaction based on a large language model according to claim 1, characterized in that: The target 3D interaction dataset includes a training set and an evaluation set; The determining of a preset large language model, and performing model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model specifically includes: Determining a preset large language model, and performing model training processing on the preset large language model according to the training set to obtain an initial interactive 3D generation model; The LoRA algorithm is used to perform model fine-tuning processing on the initial interactive 3D generation model according to the evaluation set to obtain an interactive 3D generation model.

6. The method for generating 3D results of hand-object interaction based on a large language model according to claim 1, characterized in that: The obtaining of user description text, inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data specifically includes: Obtain user description text and set custom hyperparameters, wherein the custom hyperparameters include text randomness and text length; Inputting the user description text and the custom hyperparameters into the interactive 3D generation model, and performing word segmentation, encoding and formatting processing on the user description text through the interactive 3D generation model to obtain serialized data; The context features of the serialized data are extracted through a self-attention mechanism, and the context features are decoded to obtain preliminary interactive 3D data.

7. The method for generating 3D results of hand-object interaction based on a large language model according to claim 1, characterized in that: The step of optimizing the hand parameters in the interactive 3D generation model according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and obtaining a target hand-object interactive 3D result according to the optimized hand MANO parameters, specifically includes: Segmenting the preliminary interactive 3D data to obtain an object OBJ model and a second hand MANO parameter; Calculate the corresponding object BPS distribution according to the object OBJ model, and calculate the surface distance between the hand vertex and the object according to the second hand MANO parameter; Optimizing the hand parameters of the interactive 3D generated model according to the BPS distribution of the object, the surface distance between the hand vertex and the object, and the second hand MANO parameters to obtain optimized hand MANO parameters; The hand MANO parameters are combined with the object OBJ model to obtain a target hand-object interaction 3D result.

8. A hand-object interaction 3D result generation system based on a large language model, characterized in that: The hand-object interaction 3D result generation system based on a large language model includes: A hand-object interaction data processing module is used to obtain the hand-object interaction data, and perform preprocessing and text annotation processing on the hand-object interaction data to obtain a target 3D interaction data set; An interactive 3D generation model construction module is used to determine a preset large language model, and perform model training and model fine-tuning on the preset large language model according to the target 3D interactive data set to obtain an interactive 3D generation model; A preliminary interactive 3D data output module, used for acquiring user description text, inputting the user description text into the interactive 3D generation model, and outputting preliminary interactive 3D data; The target hand-object interaction 3D result generation module is used to optimize the hand parameters in the interactive 3D generation model according to the preliminary interactive 3D data to obtain optimized hand MANO parameters, and obtain the target hand-object interaction 3D result according to the optimized hand MANO parameters.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a hand-object interaction 3D result generation program based on a large language model stored in the memory and executable on the processor. When the hand-object interaction 3D result generation program based on a large language model is executed by the processor, the steps of the hand-object interaction 3D result generation method based on a large language model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for generating a hand-object interaction 3D result based on a large language model, and when the program for generating a hand-object interaction 3D result based on a large language model is executed by a processor, the steps of the method for generating a hand-object interaction 3D result based on a large language model as described in any one of claims 1 to 7 are implemented.

Images