Tactile sense common sense reasoning-oriented tactile sense-language large model modeling method

Abstract

The invention discloses a tactile common sense reasoning-oriented tactile-language large model modeling method. The method comprises the steps of S1, preparing a first data set and a second data set; s2, constructing a tactile-language large model, wherein the tactile-language large model comprises a tactile encoder, a tactile-language adapter and a large language model LLM; s3, performing two-stage training on the tactile-language large model, and training the tactile-language large model by adopting the first data set in a first training stage; in a second training stage, training the tactile-language large model after the first training stage by adopting a second data set to obtain a trained tactile-language large model; and S4, in a test stage, inputting the second data set into the trained tactile-language large model to obtain text description. According to the tactile-language large model modeling method, the parameter coupling risk is reduced, robust fusion of efficient convergence and cross-modal representation is realized, and the training stability is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a touch-linguistic large model modeling method for touch common sense reasoning. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large language models (LLM) are widely used in transportation, healthcare, finance, autonomous driving, and other fields. In open environments for robot interaction, visual presence is obstructed, and traditional visual perception methods cannot perform object recognition, exploration, and manipulation tasks. In this case, touch signals are particularly important in common sense reasoning of object attributes, providing timely feedback to robots about the surrounding environment, helping robots identify the shape, hardness, and other surface characteristics of objects, enabling robots to accurately understand the physical properties of objects and make reasonable judgments, and enhancing the human-robot interaction experience. Combined with the powerful common sense reasoning capabilities of LLM, collaborative reasoning of touch and LLM can effectively address such challenges and improve the adaptability and decision-making capabilities of robots in complex environments.

[0003] Currently, there are few works that combine haptics with LLMs. Mainly, by mapping haptic features to the semantic space of LLMs, LLMs generate natural language descriptions that highly match haptic perceptions, or based on language instructions, guide the output of text information related to haptic interaction behaviors. This technology relies on the powerful semantic understanding ability of LLMs, realizes the deep integration of haptic signals and language expressions, and enables robots to make accurate physical reasoning and decisions in the absence of visual information or complex interactive environments. Yu et al. proposed Octopi tactile-language large model (Samson Yu, et al. Octopi: Object property reasoning with large tactile-language models [C]. In Robotics: science and systems, 2024.), which integrates haptic signals into LLMs through a three-stage training strategy and dynamic tagging mechanism to handle common sense reasoning tasks for object properties. Yang et al. used contrastive learning to align haptic embeddings with existing visual embeddings in visual-language models (Fengyu Yang, et al. Binding touch to everything: Learning unified multimodal tactile representations [C]. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024.), thereby constructing a tactile-language large model Touch-LLM that can handle tactile question-answering tasks including haptic common sense reasoning.

[0004] However, haptics and language modalities have different characteristics. In tactile-language large models, a haptic encoder is specifically designed to process the haptic modality into haptic features, and the language modality is directly processed into language features via the word embedding layer in the LLM. Then, existing tactile-language large models treat haptic features as language features and concatenate them with original language features for processing by the LLM. That is, these tactile-language large models use a haptic encoder to map haptic signals into a semantic space similar to the language representation space of the LLM, and then force the haptic and language representations to adapt to the same Transformer architecture, without considering the differences between haptic and language features. However, ignoring these differences may hinder the model's understanding of the subtle differences between the two modalities. Therefore, the fusion of haptics and language needs further research and breakthroughs, and more efficient haptic semantic learning mechanisms need to be explored. SUMMARY

[0005] The application aims to provide a haptics-language large model modeling method oriented to haptics common sense reasoning, so as to solve the problem that the difference between haptics features and language features is not considered in the prior art.

[0006] To this end, the application adopts the following technical solutions:

[0007] A haptics-language large model modeling method oriented to haptics common sense reasoning, comprising:

[0008] S1, preparing a first data set and a second data set, the first data set comprising: haptics time series data X touch1 and its sentence-level text description, the second data set comprising: haptics time series data X touch2 and its textual description instruction;

[0009] S2, constructing a haptics-language large model:

[0010] The haptics-language large model comprises: a haptics encoder, a haptics-language adapter and a large language model LLM;

[0011] The input of the haptics encoder is haptics time series data X touch1 or haptics time series data X touch2 , and the output is video-level haptics label sequence Z1' or video-level haptics label sequence Z2'; the input of the haptics-language adapter is the video-level haptics label sequence Z1' or the video-level haptics label sequence Z2', and the output is semantic embedding feature V1 or semantic embedding feature V2;

[0012] In the first training stage, the input of the large language model LLM is the semantic embedding feature V1 and the sentence-level text description in S1, and the output is the text description, and the large language model LLM comprises: a word embedding layer, a splicing module and a Transformer group connected in sequence;

[0013] In the second training stage, the input of the large language model LLM is the semantic embedding feature V2 and the textual description instruction in S1, and the output is a text description. The large language model LLM comprises, in sequence, a word embedding layer, a splicing module and an improved Transformer group. The input of the word embedding layer is the textual description instruction, and the output is a text token sequence T'. The splicing module is used for splicing the text token sequence T' and the semantic embedding feature V2, and the output of the splicing module is used as the input of the improved Transformer group. The improved Transformer group comprises a plurality of alternately connected Transformer blocks and improved Transformer blocks, and the output of the Transformer group is the text description. The improved Transformer block comprises a self-attention layer, a first addition and normalization module, a hybrid expert layer and a second addition and normalization module. The hybrid expert layer comprises a router module and a plurality of feedforward neural networks. The input of each feedforward neural network is the output of the router module.

[0014] S3, training the tactile-language large model, comprising:

[0015] S3-1, in the first training stage, keeping the initial parameters of the tactile encoder and the large language model LLM unchanged, training the tactile-language large model by using the first data set as the target of minimizing the loss function, and obtaining the tactile-language large model after the first training stage;

[0016] S3-2, in the second training stage, keeping the initial parameters of the tactile encoder and the word embedding layer in the large language model LLM unchanged in the tactile-language large model after the first training stage, training the tactile-language large model after the first training stage by using the second data set as the target of minimizing the loss function, and obtaining the trained tactile-language large model;

[0017] S4, in the test stage, inputting the second data set into the trained tactile-language large model to obtain a text description.

[0018] The calculation process of the improved Transformer block comprises:

[0019] Step 1, the self-attention layer adopts a self-attention mechanism to calculate the input data, and the output of the self-attention layer is used as the input of the first addition and normalization module. The input of the first addition and normalization module is added to the input data of the self-attention layer, and the result after addition is normalized. The normalized result is used as the output of the first addition and normalization module, and the output of the first addition and normalization module is used as the input of the hybrid expert layer.

[0020] Step 2, the router module adopts a Top-2 activation strategy, calculates the matching score of the input of the hybrid expert layer with each feedforward neural network, and outputs the weights of the two feedforward neural networks with the highest matching scores;

[0021] Step 3, according to the output of the router module, select the two feedforward neural networks with the highest matching scores from the plurality of feedforward neural networks for prediction, obtain the weighted prediction result, take the prediction result as the output of the hybrid expert layer, add the input of the hybrid expert layer and the output of the hybrid expert layer, normalize the added result, and take the normalized result as the output of the improved Transformer block and the input of the next Transformer block.

[0022] In the first training stage, the input of the word embedding layer of the large language model LLM is the sentence-level text description in S1, and the output is the text token sequence T; the splicing module is used to splice the text token sequence T and the semantic embedding feature V1, and the output of the splicing module is taken as the input of the Transformer group; the Transformer group includes a plurality of sequentially connected Transformer blocks, and the output of the Transformer group is a text description.

[0023] The loss function of the haptic-language large model in the second training stage is:

[0024] L total =L ce +L aux

[0025] In the formula, L ce is a cross-entropy loss, L aux is a load balancing loss, alpha is a scaling factor, F i represents the probability of assigning the expert E i to the token, G i represents the probability of assigning the router to the expert E i , and K represents the total number of feedforward neural networks in the improved Transformer block.

[0026] In the first training stage, the Transformer group includes 32 sequentially connected Transformer blocks.

[0027] The loss function of the haptic-language large model in the first training stage is a cross-entropy loss function.

[0028] The loss function of the haptic-language large model in the second training stage is:

[0029] L total =Lce +L aux

[0030] In the formula, L ce is the cross-entropy loss, L aux is the load balancing loss, alpha is a scaling factor, F i represents the probability of assigning the expert E i labeled probability, G i represents the probability of assigning the router to the expert E i K represents the total number of feedforward neural networks in the improved Transformer block.

[0031] In the large language model LLM of the second training stage, the improved Transformer group includes: 16 Transformer blocks and 16 improved Transformer blocks connected alternately, and the improved Transformer block includes 4 feedforward neural networks.

[0032] In the step S1, the first data set is the Touch100k data set, and the second data set is the PhysiCLeAR data set.

[0033] In the test stage, the structure of the large language model LLM is consistent with that of the large language model LLM in the second training stage.

[0034] The haptic encoder is constructed by using the haptic representation model in TLV-Link, the haptic-language adapter is constructed by using a linear projection layer, and the large language model LLM in the first training stage is constructed by using Vicuna-7B v1.5.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] 1. The present application adopts a two-stage progressive training strategy to decouple training targets in stages, and realizes efficient convergence of the training process and robust fusion of cross-modal representation by optimizing the targets in stages, which significantly improves the training stability and cross-modal alignment capability. The two-stage progressive training strategy of the present application simplifies the training process as much as possible, improves the training efficiency, and at the same time ensures the stability and efficiency of the haptic-language large model in the haptic common sense reasoning task.

[0037] 2. In the first training stage, the present application optimizes the haptic-language adapter by freezing the parameters of the large language model LLM, establishes the consistency of the haptic-language semantic space, and avoids the optimization shock caused by the modal difference in the traditional end-to-end training.

[0038] 3. By introducing a mixed expert layer (MoE) in the haptic- language large model in the second training stage, the haptic- language large model can dynamically allocate expert resources according to the difference between the haptic signal and the language semantics, avoiding the limitation of the traditional model that forces uniform multi-modal features.

[0039] 4. The load balancing loss is introduced in the loss function of the haptic- language large model in the second training stage to optimize the task allocation between experts, preventing specific experts from being over-relied on or idle. Not only does it overcome the difference between the haptic- language modalities and suppress the parameter conflict between the haptic and language modalities, but it also preserves the independence of their representations through a differentiated routing strategy, enabling the haptic- language large model to adaptively handle the variable interaction patterns in open scenarios, thereby enhancing its robustness in complex tasks.

[0040] 5. In the second training stage, the invention introduces a parameter-efficient fine-tuning strategy and a router module, which gradually activates the multi-modal collaborative reasoning capability while preserving the representation ability of the haptic- language adapter after the first training stage. Through load balancing constraints and differentiated division of labor among experts, the depth interaction between haptic temporal dynamics and language semantic logic is further strengthened, ensuring the performance of the model in open scenarios for haptic common sense reasoning. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 Flowchart of the haptic- language large model modeling method of the invention;

[0042] Figure 2 Schematic diagram of a single frame of image in an embodiment of the invention;

[0043] Figure 3 Schematic diagram of 5 frames of image in an embodiment of the invention;

[0044] Figure 4 Structure schematic diagram of the haptic- language large model in the first training stage in the invention;

[0045] Figure 5 Structure schematic diagram of the haptic- language large model in the second training stage in the invention;

[0046] Figure 6 Structure schematic diagram of the improved Transformer block of the haptic- language large model in the second training stage in the invention; Figure 7 Input of the haptic- language large model in the test stage in the invention. DETAILED DESCRIPTION

[0047] EMBODIMENT

[0048] As Figure 1As shown, the haptics-commonsense-reasoning-oriented haptics-language large model modeling method comprises the following steps:

[0049] S1, preparing a data set, comprising:

[0050] (1) preparing a Touch100k data set. The Touch100k data set is used as input data of the haptics-language large model in the first training stage, and the Touch100k data set comprises dynamic haptic time series data X touch1 and a corresponding sentence-level text description. The dynamic haptic time series data X touch1 ∈R N×H×W×C , wherein N is the total number of frames, H and W are the resolution of a single frame image, and C is the channel number of a single frame image; the sentence-level text description is a specific written description of the haptic signal.

[0051] Figure 2 As shown, the haptics time series data X touch1 in the Touch100k data set in an embodiment of the present application comprises a schematic diagram of a single frame image, and the sentence-level text description corresponding to the frame image is "The object is a rough-cut piece of limestone. The point of contact is the surface of the stone. Its material composition is calcite. The texture is slightly rough."

[0052] (2) preparing a PhysiCLeAR data set. The PhysiCLeAR data set is used as input data of the haptics-language large model in the second training stage and the test stage, and the PhysiCLeAR data set comprises dynamic haptic time series data X touch2 and a corresponding written description instruction;

[0053] Figure 3 As shown, the haptics time series data X touch2 in the PhysiCLeAR data set in an embodiment of the present application comprises a schematic diagram of 5 frame images, and the written description instruction corresponding to the 5 frame images is "Describe the physical properties of".

[0054] S2, constructing a haptics-language large model:

[0055] As shown in Figure 4 and Figure 5As shown, the touch-language large model is composed of a touch encoder, a touch-language adapter, and a large language model LLM.

[0056] In an embodiment of the present application, the touch encoder adopts the touch representation model construction in TLV-Link (Ning Cheng, et al. Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation [J]. arXiv preprint arXiv: 2406.03813, 2024); the input of the touch encoder is touch time series data X touch1 or touch time series data X touch2 , and the output is video-level touch label sequence Z1' or video-level touch label sequence Z2', Z1', Z2' ∈ R P×C . The touch encoder independently encodes each frame of image in touch time series data X touch1 or touch time series data X touch2 , to obtain frame-level touch label sequence Z1 or frame-level touch label sequence Z2, Z1, Z2 ∈ R N×P×C , where P is the number of labels per frame, P = H × W / 14 2 = 256, and each label corresponds to an image block with a size of 14 × 14; the frame-level touch label sequence Z1 or frame-level touch label sequence Z2 is subjected to average pooling to obtain video-level touch label sequence Z1' or video-level touch label sequence Z2', and the video-level touch label sequence Z1' and the video-level touch label sequence Z2' are both implicit spatio-temporal features.

[0057] The touch representation model in TLV-Link is obtained through cross-modal joint pre-training, and the touch representation model has a highly consistent semantic embedding space with the language mode; the touch encoder implicitly captures the spatio-temporal patterns of multiple frames of images, avoiding explicit and complex time series modeling while ensuring high computational efficiency.

[0058] The touch-language adapter is constructed by using a linear projection layer, the input of the touch-language adapter is the video-level touch label sequence Z1' or the video-level touch label sequence Z2', and the output is semantic embedding feature V1 (aligned with the language mode) or semantic embedding feature V2, V1, V2 ∈ R P×D, D is the hidden layer dimension of the large language model LLM; the haptic-language adapter is configured to map the video-level haptic token sequence Z1' or the video-level haptic token sequence Z2' to a semantic space of the large language model LLM to obtain the semantic embedding feature V1 or the semantic embedding feature V2.

[0059] In an embodiment of the present application, in the first training stage, the large language model LLM is constructed by using Vicuna-7B v1.5 (Wei-Lin Chiang, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%*chatgpt quality [Z]. https: / / vicuna.lmsys.org , 2023), the input of the large language model LLM is the sentence-level text description in the semantic embedding feature V1 and S1, and the output is a text description. The large language model LLM comprises a word embedding layer, a concatenation module and a Transformer group connected in sequence. Wherein:

[0060] The input of the word embedding layer is the sentence-level text description in S1, and the output is a text token sequence T = [t1, t2, …, t M ]∈R M×D , M is the length of the text token sequence, and D is the hidden layer dimension of the large language model LLM; the concatenation module is configured to concatenate the text token sequence T and the semantic embedding feature V1, and the output of the concatenation module is taken as the input of the Transformer group; the Transformer group comprises a plurality of Transformer blocks (decoders) connected in sequence, and the output of the Transformer group is a text description.

[0061] In this embodiment, the Transformer group comprises 32 Transformer blocks connected in sequence.

[0062] In the second training stage, the input of the large language model LLM is the semantic embedding feature V2 and the text description instruction in S1, and the output is a text description. The large language model LLM comprises a word embedding layer, a concatenation module and an improved Transformer group connected in sequence, wherein the input of the word embedding layer is a text description instruction, and the output is a text token sequence T'; the concatenation module is configured to concatenate the text token sequence T' and the semantic embedding feature V2, and the output of the concatenation module is taken as the input of the improved Transformer group; the improved Transformer group comprises a plurality of Transformer blocks (decoders) and improved Transformer blocks connected alternately, and the output of the Transformer group is a text description. As Figure 6As shown, the improved Transformer block includes: a self-attention layer, a first addition and normalization module, a hybrid expert layer (MoE), and a second addition and normalization module, wherein the hybrid expert layer includes: a router module and a plurality of feedforward neural networks, and the inputs of the feedforward neural networks are all the outputs of the router module.

[0063] The calculation process of the improved Transformer block includes:

[0064] Step 1: The self-attention layer calculates the input data using the self-attention mechanism, and takes the output of the self-attention layer as the input of the first addition and normalization module; the input of the first addition and normalization module is added to the input data of the self-attention layer, and the result is normalized, and the normalized result is taken as the output of the first addition and normalization module, and the output of the first addition and normalization module is taken as the input of the hybrid expert layer.

[0065] Step 2: The router module uses a Top-2 activation strategy to calculate the matching score of the input of the hybrid expert layer and each feedforward neural network, and outputs the weights of the two feedforward neural networks with the highest matching score.

[0066] Step 3: According to the output of the router module, select the two feedforward neural networks with the highest matching score from the plurality of feedforward neural networks for prediction, and obtain the weighted prediction result, take the prediction result as the output of the hybrid expert layer, add the output of the hybrid expert layer to the input of the hybrid expert layer, and normalize the result, and take the normalized result as the output of the improved Transformer block, and as the input of the next Transformer block.

[0067] In this embodiment, the improved Transformer group includes: 16 Transformer blocks (decoders) and 16 improved Transformer blocks connected alternately, and the improved Transformer block includes 4 feedforward neural networks.

[0068] The improved Transformer block dynamically activates the most suitable expert (feedforward neural network) for the haptic modality and the language modality through the router module of the hybrid expert layer, optimizes the information fusion and reasoning process, efficiently processes diversified data, and thus improves the model performance. The hybrid expert layer not only improves the collaboration between the haptic modality and the language modality, but also realizes more fine-grained modality differentiation, management and interpretation, as well as decoupling optimization of haptic detail analysis and language logical reasoning.

[0069] In the test phase, the structure of the large language model LLM is consistent with that of the large language model LLM in the second training phase.

[0070] S3, training the haptic-linguistic large model with the data set prepared in S1, the training process adopts a two-stage progressive optimization framework, as follows:

[0071] S3-1, in the first training stage, the initial parameters of the haptic encoder and the large language model LLM in the haptic-linguistic large model remain unchanged in the first training stage, the haptic-linguistic large model is trained using the Touch100k data set, the loss function is calculated and the haptic-linguistic large model is optimized to minimize the loss function, so that the parameters of the haptic-linguistic adapter in the haptic-linguistic large model are continuously optimized, driving the implicit alignment of haptic features and language space, and the semantic embedding features V output by the haptic-linguistic adapter can be effectively adapted to the large language model LLM, wherein the loss function is a cross-entropy loss function.

[0072] The first training stage ensures the integrity of the original knowledge representation of the language model while establishing a benchmark semantic interface for haptic-linguistic cross-modal interaction, providing a stable collaborative foundation for subsequent dynamic routing and expert (feedforward neural network) allocation.

[0073] S3-2, in the second training stage, the initial parameters of the haptic encoder and the word embedding layer in the large language model LLM in the haptic-linguistic large model after the first training stage remain unchanged in the second training stage, the haptic-linguistic large model after the first training stage is trained on the PhysiCLeAR data set using low-rank adaptation (LoRA), the loss function is calculated and the haptic-linguistic large model after the first training stage is optimized to minimize the loss function, and the parts of the large language model LLM except the word embedding layer are fine-tuned, while the haptic-linguistic adapter is fine-tuned with all parameters, enhancing the differential processing capability of the haptic-linguistic large model after the first training stage for multi-modal input, and finally obtaining the trained haptic-linguistic large model.

[0074] wherein the loss function is:

[0075] L total =L ce +L aux

[0076] In the formula, L ce is the cross-entropy loss, L aux is the load balancing loss, α is a scaling factor, F i represents the probability of assigning a label to the expert E i (the i-th feedforward neural network), and G i represents the probability of assigning a label to the expert E iThe probability of the router of the (i-th feedforward neural network) K represents the total number of feedforward neural networks in the improved Transformer block.

[0077] The second training stage enhances the multi-modal inference ability through dynamic architecture, while enhancing the multi-modal understanding and generation ability of the haptic-language large model after the first training stage, realizes dynamic expert allocation for tokens, and improves the processing ability of the model for complex haptic-language input. The second training stage realizes the dynamic cooperation of haptic and language modalities through architectural innovation, and the core goal is to enhance the semantic understanding ability of the model for multi-modal tokens through a differentiated expert allocation mechanism. The loss function of the haptic-language large model in the second training stage introduces a load balancing loss as an auxiliary optimization target.

[0078] The two-stage progressive training strategy realizes the dynamic cooperation of the haptic and language modalities of the haptic-language large model, and the two-stage training strategy makes the haptic-language large model gradually transition from preliminary haptic semantic adaptation to complex reasoning optimization in an open scenario, effectively balancing training efficiency and performance while improving modality cooperation robustness.

[0079] In the present embodiment, the input of the trained haptic-language large model is as shown in Figure 3 The expected result of the text description output by the corresponding haptic-language large model is "The surface feels yielding, exhibits a notable roughness and there are many big bumps due to grains present. Overall, it presents a soft and rough surface with big bumps."

[0080] S4, in the test stage, the haptic time series data X in the PhysiCLeAR dataset touch2 and its corresponding text description instruction are input into the trained haptic-language large model to obtain a text description.

[0081] In the present embodiment, the input of the trained haptic-language large model is as shown in Figure 7 ;

[0082] The expected output is: First object: The surface feels moderately hard, lacks noticeable roughness and there are no bumps or irregularities present. Overall, it presents a moderately hard and smooth surface with no bumps. Second object: The surface feels yielding, has a subtle roughness and there are small irregularities and bumps. Overall, it presents a moderately hard and slightly rough surface with small bumps. Conclusion: Yes, the first object is smoother.

[0083] The text description output by the trained tactile- language large model is: First object: The surface feels solid and sturdy, lacks noticeable roughness and there are no prominent bumps or ridges. Overall, it presents a hard and smooth surface with no bumps. Second object: The surface feels soft, exhibits a slight roughness and there are minor textural variations due to the weave. Overall, it presents a moderately hard and slightly rough surface with small bumps. Conclusion: Yes, the first object is smoother.

[0084] The input as shown in Figure 7 is input into the Octopi-7B model and the Octopi-13B model to obtain the output results.

[0085] The output results of the Octopi-7B model are as follows: First object: The surface feels moderately hard, lacks noticeable roughness and there are no bumps or irregularities present. Overall, it presents a moderately hard and smooth surface with no bumps. Second object: The surface feels soft and yielding, lacks noticeable roughness and there are some small textural variations. Overall, it presents a soft and smooth surface with small bumps. Conclusion: Both objects are similar in terms of roughness.

[0086] The output results of the Octopi-13B model are as follows: the first object: the surface feels solid and sturdy, lacks noticeable roughness and there are no textural variations or fine ridges. Overall, it presents a hard and smooth surface with no bumps. The second object: the surface feels relatively firm, exhibits a slight roughness and there are small bumps and ridges present. Overall, it presents a moderately hard and slightly rough surface with small bumps. Conclusion: No, the second object is smoother.

[0087] According to the text description output by the trained tactile- language large model and the output results of the Octopi-7B model and the Octopi-13B model, only the text description output by the tactile- language large model of the present application is consistent with the expected output conclusion, that is, "Yes, the first object is smoother (Yes, the first object is smoother)." The three evaluation indexes CIDEr, B@4 and METEOR are calculated on the PhysiCLeAR dataset, as shown in Table 1:

[0088] Table 1

[0089] Model name CIDEr B@4 METHOR Octopi-7B 138.60 64.16 77.63 Octopi-13B 141.20 64.33 77.79 The invention 195.03 68.03 82.58

[0090] As can be known from Table 1, the trained tactile-language large model of the present application is superior to the Octopi-7B model and the Octopi-13B model in three indexes, and a great improvement is achieved, compared with the suboptimal Octopi-13B model, the CIDEr, B@4 and METEOR three evaluation indexes are improved by 53.83, 3.70 and 4.79 respectively, so that the trained tactile-language large model of the present application is superior to the Octopi-7B model and the Octopi-13B model.

[0091] Wherein, the evaluation index CIDEr refers to: Ramakrishna Vedantam, et al. Cider: Consensus-based image description evaluation [C]. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. The evaluation index B@4 refers to: Kishore Papineni, et al. Bleu: a method for automatic evaluation of machine translation [C]. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002. The evaluation index METEOR refers to: Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments [C]. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and / or summarization, 2005.

Claims

1. A haptic-linguistic large model modeling method oriented to haptic common sense reasoning, characterized in that, The method comprises the following steps: S1, prepare a first dataset and a second dataset, the first dataset comprising: haptic temporal data X touch1 and its sentence-level textual description, the second dataset comprising: haptic temporal data X touch2 and its textual description instructions; S2, constructing a haptic-language large model: The haptic-language large model comprises a haptic encoder, a haptic-language adapter and a large language model LLM; The input of the haptic encoder is haptic temporal data X touch1 or haptic temporal data X touch2 and the output is video-level haptic label sequence Z1' or video-level haptic label sequence Z2'; the input of the haptic-language adapter is the video-level haptic label sequence Z1' or video-level haptic label sequence Z2', and the output is semantic embedding feature V1 or semantic embedding feature V2; In the first training stage, the input of the large language model LLM is the semantic embedding feature V1 and the sentence-level text description in S1, and the output is a text description. The large language model LLM comprises a word embedding layer, a splicing module and a Transformer group connected in sequence; In the second training stage, the input of the large language model LLM is the semantic embedding feature V2 and the text description instruction in S1, and the output is a text description. The large language model LLM comprises a word embedding layer, a splicing module and an improved Transformer group connected in sequence. The input of the word embedding layer is the text description instruction, and the output is a text token sequence T'. The splicing module is used for splicing the text token sequence T' and the semantic embedding feature V2, and the output of the splicing module is used as the input of the improved Transformer group. The improved Transformer group comprises a plurality of alternately connected Transformer blocks and improved Transformer blocks, and the output of the Transformer group is a text description. The improved Transformer block comprises a self-attention layer, a first addition and normalization module, a hybrid expert layer and a second addition and normalization module. The hybrid expert layer comprises a router module and a plurality of feedforward neural networks. The input of each feedforward neural network is the output of the router module. S3, training the haptic-language large model, comprising: S3-1, in the first training stage, keeping the initial parameters of the haptic encoder and the large language model LLM unchanged, training the haptic-language large model by using the first data set as the target of minimizing the loss function, and obtaining the haptic-language large model after the first training stage; S3-2, in the second training stage, keeping the initial parameters of the haptic encoder and the word embedding layer in the large language model LLM in the haptic-language large model after the first training stage unchanged, training the haptic-language large model after the first training stage by using the second data set as the target of minimizing the loss function, and obtaining the trained haptic-language large model; S4, in the test stage, inputting the second data set into the trained haptic-language large model to obtain a text description.

2. The tactile-linguistic large model modeling method of claim 1, wherein, The calculation process of the improved Transformer block comprises: Step 1, the self-attention layer adopts a self-attention mechanism to calculate the input data, and the output of the self-attention layer is used as the input of the first addition and normalization module. The input of the first addition and normalization module is added to the input data of the self-attention layer, and the result is normalized. The normalized result is used as the output of the first addition and normalization module, and the output of the first addition and normalization module is used as the input of the hybrid expert layer. Step 2, the router module adopts a Top-2 activation strategy to calculate the matching score of the input of the hybrid expert layer with each feedforward neural network, and outputs the weights of the two feedforward neural networks with the highest matching scores; Step 3, according to the output of the router module, the two feedforward neural networks with the highest matching scores are selected from the plurality of feedforward neural networks for prediction, and a weighted prediction result is obtained; the prediction result is taken as the output of the hybrid expert layer, the output of the hybrid expert layer is added to the input of the hybrid expert layer, the added result is normalized, and the normalized result is taken as the output of the improved Transformer block and the input of the next Transformer block.

3. The tactile-linguistic large model modeling method of claim 1, wherein, In the first training stage of the large language model LLM, the input of the word embedding layer is the sentence-level text description in S1, and the output is the text token sequence T; the concatenation module is used to concatenate the text token sequence T and the semantic embedding feature V1, and the output of the concatenation module is taken as the input of the Transformer group; the Transformer group includes a plurality of sequentially connected Transformer blocks, and the output of the Transformer group is the text description.

4. The tactile-linguistic large model modeling method of claim 3, characterized in that: In the first training stage of the large language model LLM, the Transformer group includes 32 sequentially connected Transformer blocks.

5. The tactile-linguistic large model modeling method of claim 1, wherein, The loss function of the first training stage of the tactile-linguistic large model is a cross-entropy loss function.

6. The tactile-linguistic large model modeling method of claim 1, wherein, The loss function of the second training stage of the tactile-linguistic large model is: L total = L ce + L aux where L ce is the cross-entropy loss, L aux is the load balancing loss, α is a scaling factor, F i denotes the probability of being assigned to an expert E i labeled G i denotes the probability of being assigned to a router of an expert E i K denotes the total number of feed-forward neural networks in the improved Transformer block.

7. The tactile-linguistic large model modeling method of claim 1, wherein, In the second training stage of the large language model LLM, the improved Transformer group includes 16 Transformer blocks and 16 improved Transformer blocks connected alternately, and the improved Transformer block includes four feedforward neural networks.

8. The tactile-linguistic large model modeling method of claim 1, wherein, In step S1, the first data set is the Touch100k data set, and the second data set is the PhysiCLeAR data set.

9. The tactile-linguistic large model modeling method of claim 1, wherein, In the test stage, the structure of the large language model LLM is consistent with that of the large language model LLM in the second training stage.

10. The tactile-linguistic large model modeling method according to any one of claims 1-9, characterized in that: The tactile encoder is constructed by using the tactile representation model in TLV-Link; the tactile-linguistic adapter is constructed by using a linear projection layer; and the large language model LLM in the first training stage is constructed by using Vicuna-7B v1.5.

Images