Model deployment method and device for visual-tactile sensor, equipment and medium

Abstract

The application discloses a model deployment method and device for a visual-tactile sensor, equipment and medium, relates to the technical field of machine learning, and comprises the following steps: training a first target model by using a source domain data set, obtaining a second target model, and determining model parameters of the second target model; training a third target model by using the model parameters and a target domain data set, obtaining a fourth target model; determining the fourth target model as a teacher model, and constructing a corresponding student model based on the computing power of a tactile perception device; training the student model by using the teacher model, obtaining a trained student model, and deploying the trained student model to the tactile perception device, so that the trained student model infers and generates a target force value corresponding to tactile data based on the tactile data perceived by the tactile perception device, and the tactile perception device performs a preset force feedback control operation based on the target force value.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method, apparatus, device, and medium for deploying models for visual and tactile sensors. Background Technology

[0002] Vision-based force-tactile sensors rely on cameras to capture the deformation of elastic bodies to decouple contact forces. With their advantages of high spatial resolution, multi-dimensional force detection capabilities, and low cost, they have become a research hotspot in the field of force sensing. However, these sensors only demonstrate good performance in laboratory environments. In the process of moving from prototype development to industrial mass production and practical deployment, they are constrained by core technological bottlenecks, making it difficult to meet the standardization and real-time requirements of industrial applications.

[0003] Existing technologies have significant shortcomings in mass production calibration. Microscopic differences between individual sensors cause the general model to fail during transfer, making it difficult to balance calibration efficiency and cost. In mass production, existing solutions either directly transfer the general model trained on a small number of sensors to all sensors, or independently complete the entire process of data acquisition and model training for each sensor. However, visual force and tactile sensors have complex coupling between optics and mechanics. Even with standardized manufacturing, microscopic differences such as marker distribution, assembly tolerances, and adhesive layer thickness will still occur. Deep learning models are highly sensitive to these features, and directly transferring the model will lead to a significant decrease in force prediction accuracy and linearity, failing to meet industrial measurement requirements. Calibration for each sensor requires the acquisition of massive amounts of image-force ground truth data, which is time-consuming, labor-intensive, and significantly increases manufacturing costs, becoming a core obstacle to mass production. Meanwhile, edge deployment of sensor models faces challenges in terms of computing power and real-time performance. To achieve high-precision force decoupling, existing technologies often employ complex architectures such as deep convolutional neural networks and visual Transformers, resulting in a large number of model parameters, high computational complexity, and high inference latency, making it difficult to meet the ultra-high frequency response requirements of scenarios such as robot force feedback control. Furthermore, the trained models cannot be directly adapted to resource-constrained edge devices such as embedded chips and robotic arm controllers, lacking efficient model compression and adaptation methods, resulting in poor hardware resource adaptability.

[0004] Therefore, how to solve the calibration data dependence caused by individual sensor differences while ensuring high sensor detection accuracy is an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a model deployment method, apparatus, device, and medium for visual-tactile sensors, which can solve the calibration data dependency phenomenon caused by individual sensor differences while ensuring high sensor detection accuracy. The specific solution is as follows: Firstly, this application provides a model deployment method for visual-tactile sensors, including: Based on historical data collected by a preset visual-tactile sensor, a source domain dataset is constructed to determine the target visual-tactile sensor to be calibrated. During the calibration operation for the target visual-tactile sensor, a target domain dataset is constructed based on the calibration data collected by the target visual-tactile sensor. The first target model containing a convolutional neural network is trained using the source domain dataset to obtain a second target model, and the model parameters of the second target model are determined. The third target model is trained using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor; The fourth target model is determined as the teacher model, and a student model corresponding to the tactile sensing device is constructed based on the computing power of the tactile sensing device. The student model is trained using the teacher model, composite loss function, and knowledge distillation technique to obtain a trained student model. The trained student model is then deployed to the tactile sensing device so that the trained student model can infer and generate a target force value corresponding to the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value.

[0006] Optionally, the construction of the source domain dataset based on historical data collected through a preset visual-tactile sensor includes: Data pairs containing optical flow image data and force values ​​are collected from preset visual and tactile sensors in the target production line. These data pairs are identified as historical data, and a source domain dataset is constructed based on the historical data.

[0007] Optionally, constructing the source domain dataset based on the historical data includes: The optical flow image data in the historical data is determined, and the global mean and standard deviation of the optical flow image data are determined. The optical flow image data is then normalized using the Z-score normalization formula, the global mean, and the standard deviation to obtain normalized data. Determine the force values ​​in the historical data, and divide the force values ​​using a stratified sampling strategy that uses a preset force value range and the sampling probability is inversely proportional to the number of samples in the preset force value range, to obtain the divided force values; Linear interpolation is performed on the historical data to obtain virtual data; The source domain dataset is constructed based on the normalized data, the partitioned force values, and the virtual data.

[0008] Optionally, training a first target model containing a convolutional neural network using the source domain dataset to obtain a second target model includes: A first target model is determined, which includes a convolutional neural network and a task head for predicting normal and shear forces. The first target model is then trained using the source domain dataset to obtain a second target model.

[0009] Optionally, training the third target model using the model parameters and the target domain dataset to obtain a fourth target model adapted to the characteristics of the target visual-tactile sensor includes: The model parameters are determined as the initial weights of the third target model to complete the initialization operation of the third target model and obtain the initialized model. Freeze the first target parameters of the convolutional layer used for feature extraction in the initialized model, and unfreeze the second target parameters of the fully connected layer and the output layer in the initialized model to complete the adjustment operation of the initialized model and obtain the adjusted model; The fully connected layer and the output layer in the initialized model are determined as the unfrozen layer, and the unfrozen layer is trained using the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor.

[0010] Optionally, the construction of the student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device includes: Tensor transformation is used to arrange the spatial dimension pixel blocks of optical flow image data into the channel dimension of the model to complete the first model construction operation; An SE channel attention mechanism is embedded in the convolutional blocks of the model to complete the second model construction operation; Embed the Swish activation function into the model to complete the third model construction operation; Based on the computing power of the tactile sensing device, the first model construction operation, the second model construction operation, and the third model construction operation, a student model corresponding to the tactile sensing device is constructed.

[0011] Optionally, training the student model using the teacher model, the composite loss function, and knowledge distillation techniques to obtain the trained student model includes: Determine a first predicted value of the force value by the student model, and determine a first mean square error between the first predicted value and the true value corresponding to the force value, so as to determine a hard label loss term based on the first mean square error; Determine a second predicted value of the force value by the teacher model, and determine a second mean square error between the first predicted value and the second predicted value, so as to determine a soft label loss term based on the second mean square error; The channel mean of the feature maps extracted by the student model and the teacher model is determined, and the channel mean is used as a spatial attention map to determine the attention transfer loss term based on the spatial attention map. Determine the distance matrix between the sample data received by the student model and the teacher model respectively, and determine the knowledge distillation loss term based on the distance matrix; A composite loss function is constructed based on the hard label loss term, the soft label loss term, the attention transfer loss term, and the knowledge distillation loss term; The student model is trained using the teacher model, the composite loss function, and the knowledge distillation technique to obtain the trained student model.

[0012] Secondly, this application provides a model deployment device for visual-tactile sensors, comprising: The dataset construction module is used to construct a source domain dataset based on historical data collected by a preset visual-tactile sensor, determine the target visual-tactile sensor to be calibrated, and construct a target domain dataset based on the calibration data collected by the target visual-tactile sensor during the calibration operation triggered for the target visual-tactile sensor. The parameter determination module is used to train a first target model containing a convolutional neural network using the source domain dataset to obtain a second target model, and to determine the model parameters of the second target model. The model training module is used to train the third target model using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor. The model building module is used to determine the fourth target model as the teacher model and to build a student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device. The model deployment module is used to train the student model using the teacher model, the composite loss function, and the knowledge distillation technique to obtain the trained student model, and to deploy the trained student model to the tactile sensing device. This allows the trained student model to infer and generate a target force value corresponding to the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned model deployment method for visual-touch sensors.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned model deployment method for visual-tactile sensors.

[0015] In this application, a source domain dataset is constructed based on historical data collected by a preset visual-tactile sensor to determine the target visual-tactile sensor to be calibrated. During the calibration operation triggered for the target visual-tactile sensor, a target domain dataset is constructed based on the calibration data collected by the target visual-tactile sensor. A first target model containing a convolutional neural network is trained using the source domain dataset to obtain a second target model, and the model parameters of the second target model are determined. A third target model is trained using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor. The fourth target model is determined as a teacher model, and a student model corresponding to the tactile sensing device is constructed based on the computing power of the tactile sensing device. The student model is trained using the teacher model, a composite loss function, and knowledge distillation technology to obtain a trained student model. The trained student model is then deployed to the tactile sensing device so that the trained student model can infer and generate a target force value corresponding to the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value. As can be seen from the above, in this application, a source domain dataset is constructed using historical data collected by a preset visual-tactile sensor, and the target visual-tactile sensor to be calibrated is determined. When the calibration process of the target visual-tactile sensor is initiated, a target domain dataset is constructed based on the calibration data collected by the sensor. The source domain dataset is used to train a first target model containing a convolutional neural network to obtain a second target model and determine its model parameters. Based on the above model parameters and the target domain dataset, a third target model is trained to obtain a fourth target model that matches the characteristics of the target visual-tactile sensor. The fourth target model is used as a teacher model, and a corresponding student model is constructed according to the computing power of the tactile sensing device. The student model is trained using the teacher model, a composite loss function, and knowledge distillation techniques. The trained student model is then deployed to the tactile sensing device, enabling it to infer the corresponding target force value based on the collected tactile data and execute a preset force feedback control operation based on the target force value. In other words, this application is a data-driven algorithm processing flow designed to bridge the gap between mass manufacturing of sensors and real-time industrial applications. This solution does not rely on a specific sensor physical structure. Instead, it transforms uncalibrated raw sensors into intelligent sensing units with high-precision, low-latency inference capabilities through a two-stage processing at the software algorithm level. In this way, this application can resolve the calibration data dependency caused by individual sensor differences while ensuring high sensor detection accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart of a model deployment method for visual-tactile sensors disclosed in this application; Figure 2 This is a flowchart of a specific model deployment method for visual-tactile sensors disclosed in this application; Figure 3 This is a schematic diagram of a transfer learning model architecture disclosed in this application; Figure 4 This is a schematic diagram of a student model for knowledge distillation disclosed in this application; Figure 5 This is a schematic diagram of a model deployment device for visual-tactile sensors disclosed in this application; Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Currently, existing technologies have significant shortcomings in mass production calibration. Due to the microscopic differences between individual sensors, the transfer of general models fails, making it difficult to balance calibration efficiency and cost. In mass production, existing solutions either directly transfer the general model trained on a small number of sensors to all sensors, or independently complete the entire process of data acquisition and model training for each sensor. However, visual force and tactile sensors have complex coupling between optics and mechanics. Even with standardized manufacturing, microscopic differences such as marker distribution, assembly tolerances, and adhesive layer thickness will still occur. Deep learning models are highly sensitive to these features, and directly transferring the model will lead to a significant decrease in force prediction accuracy and linearity, failing to meet industrial measurement requirements. On the other hand, calibrating each sensor individually requires collecting massive amounts of image-force ground truth data, which is time-consuming, labor-intensive, and significantly increases manufacturing costs, becoming a core obstacle to mass production. Meanwhile, edge deployment of sensor models faces challenges in terms of computing power and real-time performance. Existing technologies, in order to achieve high-precision force decoupling, often employ complex architectures such as deep convolutional neural networks and visual Transformers. These models have a large number of parameters, high computational complexity, and high inference latency, making it difficult to meet the ultra-high frequency response requirements of scenarios such as robot force feedback control. Furthermore, the trained models cannot be directly adapted to resource-constrained edge devices such as embedded chips and robotic arm controllers, lacking efficient model compression and adaptation methods, resulting in poor hardware resource adaptability. To address these issues, this application provides a model deployment method, apparatus, device, and medium for visual-tactile sensors, which can solve the calibration data dependency phenomenon caused by individual sensor differences while ensuring high sensor detection accuracy.

[0020] See Figure 1 As shown, this embodiment of the invention discloses a model deployment method for visual-tactile sensors, including: Step S11: Construct a source domain dataset based on historical data collected by a preset visual-tactile sensor, determine the target visual-tactile sensor to be calibrated, and construct a target domain dataset based on calibration data collected by the target visual-tactile sensor during the calibration operation triggered for the target visual-tactile sensor.

[0021] In this embodiment, the source domain dataset is constructed based on historical data collected by a preset visual-tactile sensor. When collecting historical data, a preset visual-tactile sensor in a stable operating state within the target production line is typically selected to ensure that the collected data accurately reflects the sensor's response characteristics under typical operating conditions. Data pairs containing optical flow image data and force values ​​are collected from the preset visual-tactile sensor in the target production line; these data pairs are identified as historical data, and the source domain dataset is constructed based on this historical data.

[0022] To improve the quality of the source domain dataset and meet the needs of subsequent model training, a series of preprocessing operations are required on the original historical data. For the optical flow image data, considering the potential differences in overall brightness or contrast between data collected from different sensors or batches, the optical flow image data within the historical data is identified, along with its global mean and standard deviation. The optical flow image data is then normalized using the Z-score standardization formula, the global mean, and the standard deviation to obtain normalized data. This process helps enhance the model's robustness to the inherent noise of different sensors.

[0023] Secondly, the force values ​​in historical data may be unevenly distributed, and too many or too few samples in certain force value intervals can affect the model's generalization ability. Therefore, the force values ​​in the historical data are determined, and the force values ​​are divided using a stratified sampling strategy where the sampling probability is inversely proportional to the number of samples in the preset force value interval. This results in divided force values, ensuring that the model can learn sufficiently across all force value ranges.

[0024] Furthermore, to expand the dataset size and simulate possible data changes based on limited historical data, data augmentation operations were performed on the historical data. Specifically, linear interpolation was performed on the historical data to obtain virtual data. Finally, a source domain dataset was constructed based on the normalized data, the partitioned force values, and the virtual data.

[0025] Simultaneously or subsequently after the source domain dataset is constructed, it is necessary to identify the target visual-tactile sensor to be calibrated. The target visual-tactile sensor typically refers to a newly installed sensor that needs to be put into use but has not yet been precisely calibrated, or a sensor that needs recalibration due to environmental changes, device aging, or other reasons. Once the target sensor is identified, the calibration process for it is triggered. During this calibration process, the target visual-tactile sensor is controlled to operate under specific or similar operating conditions to a preset sensor, and its raw calibration data is collected. This calibration data also includes optical flow image sequences and corresponding force readings, used to construct a target domain dataset reflecting the specific characteristics of this sensor, providing a target domain data foundation for subsequent model transfer.

[0026] Step S12: Train the first target model containing the convolutional neural network using the source domain dataset to obtain the second target model, and determine the model parameters of the second target model.

[0027] In this embodiment, the specific structure of the first target model is first determined, namely, a first target model containing a convolutional neural network and a task head for predicting normal and shear forces. Then, the first target model is trained using the source domain dataset to obtain a second target model. Finally, the model parameters of the second target model are extracted and saved; these model parameters constitute the knowledge base for subsequent transfer learning.

[0028] Step S13: Train the third target model using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor.

[0029] In this embodiment, to effectively transfer the model knowledge learned from the source domain to the target sensor, the model is first initialized. The model parameters are determined as the initial weights of the third target model to complete the initialization operation of the third target model, resulting in the initialized model. Essentially, this operation uses the general visual-tactile mapping relationship learned on the source domain sensor as the starting point for model learning.

[0030] After obtaining the initialized model, a parameter adjustment strategy needs to be set. The first target parameters of the convolutional layers used for feature extraction in the initialized model are frozen, and the second target parameters of the fully connected layers and output layer in the initialized model are unfrozen to complete the adjustment operation and obtain the adjusted model. This strategy aims to retain the general feature representation capabilities learned by the source model, while focusing on adapting the latter part of the model to the specific response characteristics of the target sensor.

[0031] Next, the adjusted model is trained using the target domain dataset. The training process focuses on the fully connected layers and the output layer in the model, which are collectively referred to as the thawing layers. The fully connected layers and the output layer in the initialized model are identified as thawing layers, and the thawing layers are trained using the target domain dataset to obtain a fourth target model adapted to the characteristics of the target visual-touch sensor.

[0032] Step S14: Determine the fourth target model as the teacher model, and construct a student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device.

[0033] In this embodiment, the fourth target model, which has been trained and is well-adapted to the characteristics of the target sensor, is established as the teacher model. This model has high prediction accuracy, but its computational complexity is usually also high. Considering that the tactile sensing devices that actually deploy the model often have strict limitations in computing power, storage, and power consumption, directly deploying the teacher model may not meet the real-time requirements. Therefore, it is necessary to construct a lightweight student model that is specifically adapted to the computing power conditions of the target device.

[0034] Specifically, tensor transformation operations are used to arrange the spatial dimension pixel blocks of optical flow image data into the channel dimension of the model to complete the first model construction operation; an SE channel attention mechanism is embedded in the convolutional blocks of the model to complete the second model construction operation; a Swish activation function is embedded in the model to complete the third model construction operation; and a student model corresponding to the tactile sensing device is constructed based on the computing power of the tactile sensing device, the first model construction operation, the second model construction operation, and the third model construction operation.

[0035] Step S15: Train the student model using the teacher model, composite loss function, and knowledge distillation technique to obtain a trained student model. Deploy the trained student model to the tactile sensing device so that the trained student model can infer and generate a target force value corresponding to the tactile data based on the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value.

[0036] In this embodiment, a composite loss function comprising four loss terms is first constructed. Specifically, a first predicted value of the force value by the student model is determined, and a first mean squared error between the first predicted value and the true value corresponding to the force value is determined, to determine a hard label loss term based on the first mean squared error; a second predicted value of the force value by the teacher model is determined, and a second mean squared error between the first predicted value and the second predicted value is determined, to determine a soft label loss term based on the second mean squared error; the channel mean of the feature maps extracted by the student model and the teacher model is determined, and the channel mean is used as a spatial attention map, to determine an attention transfer loss term based on the spatial attention map; and a distance matrix between the sample data received by the student model and the teacher model is determined, to determine a knowledge distillation loss term based on the distance matrix. Finally, a composite loss function is constructed based on the hard label loss term, the soft label loss term, the attention transfer loss term, and the knowledge distillation loss term.

[0037] Furthermore, the student model is trained using the teacher model, the composite loss function, and knowledge distillation technology to obtain a trained student model. After training, the trained student model is deployed to the target tactile sensing device. The deployment process may include converting the model into a format suitable for the device's inference engine and integrating it. Subsequently, the trained student model can run on the device, performing inference based on tactile data collected in real time by the tactile sensing device to generate corresponding target force values ​​in real time. The core controller of the tactile sensing device receives this target force value, using it as a key feedback input to execute preset force feedback control operations, such as adjusting the gripping force of the robotic arm, achieving compliant assembly, or providing on-site tactile interaction.

[0038] As can be seen from the above, in this application, a source domain dataset is constructed using historical data collected by a preset visual-tactile sensor, and the target visual-tactile sensor to be calibrated is determined. When the calibration process of the target visual-tactile sensor is initiated, a target domain dataset is constructed based on the calibration data collected by the sensor. The source domain dataset is used to train a first target model containing a convolutional neural network to obtain a second target model and determine its model parameters. Based on the above model parameters and the target domain dataset, a third target model is trained to obtain a fourth target model that matches the characteristics of the target visual-tactile sensor. The fourth target model is used as a teacher model, and a corresponding student model is constructed according to the computing power of the tactile sensing device. The student model is trained using the teacher model, a composite loss function, and knowledge distillation techniques. The trained student model is then deployed to the tactile sensing device, enabling it to infer the corresponding target force value based on the collected tactile data and execute a preset force feedback control operation based on the target force value. In other words, this application is a data-driven algorithm processing flow designed to bridge the gap between mass manufacturing of sensors and real-time industrial applications. This solution does not rely on a specific sensor physical structure. Instead, it transforms uncalibrated raw sensors into intelligent sensing units with high-precision, low-latency inference capabilities through a two-stage processing at the software algorithm level. In this way, this application can resolve the calibration data dependency caused by individual sensor differences while ensuring high sensor detection accuracy.

[0039] The core technical solutions of the embodiments of this application will be described in detail below.

[0040] The core technical solution of this application can be mainly divided into two stages. In the first stage, the main focus is on efficient data calibration based on a "unified model" and transfer learning (TL). This invention establishes a rapid adaptation mechanism from "general features of the group" to "specific features of the individual". First, a "unified model" with strong generalization ability is trained as a base using historical data (i.e., source domain data) from multiple general sensors on the production line. When a new sensor (i.e., the target domain) needs to be calibrated, it is no longer necessary to collect the full dataset and train from scratch. This invention uses the aforementioned unified model as initialization parameters, collecting only a very small number of calibration samples from the new sensor, such as 50% or less of the full data. By freezing the general feature layer and fine-tuning the task-specific heads, a high-precision model adapted to the specific sensor is quickly generated.

[0041] In the second stage, the main focus is on hardware-adaptive lightweight deployment based on knowledge distillation (KD). This invention establishes a compression mechanism from a "high-precision heavy model" to a "lightweight fast model" to adapt to different terminal computing devices. The fine-tuned model generated in the first stage is used as the high-precision "Teacher Model." Based on the computing power constraints and operator support of target deployment devices such as embedded GPUs (Graphics Processing Units), DSPs (Digital Signal Processors), and FPGAs (Field Programmable Gate Arrays), a simplified "Student Model" is constructed. Through a multi-objective loss function including hard-label loss, soft-label loss, and attention map alignment, the student model is forced to mimic the output distribution and intermediate feature responses of the teacher model. The final output is a lightweight model that retains the high precision of the teacher model while possessing extremely high inference speeds, such as sub-millisecond levels, which can be directly used for industrial real-time control.

[0042] Reference Figure 2 As shown, the core steps of this application mainly include data preparation and preprocessing, construction of a teacher model based on transfer learning (TL), and compression and deployment of a student model based on knowledge distillation (KD). The core steps in this application are explained in detail below.

[0043] Specifically, before training the model, data preparation and preprocessing first involve creating a dataset containing source domain data and target domain data.

[0044] For source domain data acquisition, collect "optical flow image-force true value" data pairs from multiple, such as 10, general sensors on the production line under standardized loading to construct a large-scale source domain dataset; for target domain data acquisition, for new sensors to be calibrated, only collect a small amount of data, such as 10%-50% of the full data, as target domain calibration data.

[0045] Furthermore, data augmentation and normalization can be performed on the collected data. Global normalization calculates the global mean and standard deviation of the source domain data, using Z-score standardization to project the input data from all sensors into a unified feature space. Stratified sampling addresses the scarcity of high-force samples by employing a stratified sampling strategy, where the sampling probability is proportional to the reciprocal of the number of samples in that force range, ensuring the model's ability to learn extreme force values. Mixup regularization performs linear interpolation on the training samples. In this formula, For generated virtual samples; These are two different samples randomly selected from the training set. The mixed weighting coefficients are used to generate virtual samples to smooth the decision boundary and prevent overfitting during fine-tuning with small samples.

[0046] Secondly, refer to Figure 3 As shown, the step of building a high-precision "teacher model" based on transfer learning (TL) aims to solve the problem of direct transfer failure caused by individual differences in sensors, and this step mainly includes unified model pre-training and transfer fine-tuning.

[0047] For unified model pre-training, specifically, a multi-task convolutional neural network is constructed, such as based on the VGG (Visual Geometry Group) architecture, which includes a shared feature extraction backbone and task heads for predicting normal force (Pressure) and shear force (Shear), respectively. The model is then trained in a fully supervised manner using a source domain dataset, enabling it to learn common visual features shared by all sensors, such as optical flow texture and deformed edges, thus obtaining a pre-trained "unified model".

[0048] For transfer fine-tuning, specifically, initialization is performed, that is, the parameters of the unified model are used as the initial weights of the sensor model to be calibrated. Furthermore, a freezing strategy and a fast adaptation strategy can be implemented on the sensor model to be calibrated. The freezing strategy freezes all convolutional layer parameters of the feature extraction backbone, maintaining the general feature extraction capability unchanged, and only unfreezes the fully connected layers and output header parameters. Fast adaptation uses a small amount of data from the target domain, such as 50% of the data volume, to fine-tune the unfrozen layers. Due to the significant reduction in the number of trainable parameters, this process can converge quickly, generating a high-precision "teacher model" adapted to the specific sensor.

[0049] Finally, refer to Figure 4 As shown, the step of building a lightweight “student model” based on knowledge distillation (KD) aims to compress the teacher model into a real-time model suitable for deployment on edge devices. This step may include student model building and composite loss function distillation training.

[0050] Specifically, for student model construction, a lightweight architecture is designed for deployment hardware such as embedded GPUs. The student model will include downsampling, an attention module, and activation functions. Downsampling (i.e., Space-to-Depth, S2D) replaces traditional strided convolutional or pooling layers as the first layer with an S2D module. This rearranges spatial dimension pixel blocks to channel dimension, reducing resolution while fully preserving high-frequency spatial information of micro-deformations, avoiding information loss. The attention module (i.e., SEBlock) embeds a "squeeze-and-excitation" module into the convolutional blocks, achieving adaptive recalibration of channel features with extremely low computational cost. Activation function optimization replaces ReLU with the Swish activation function to improve the smoothness and stability of the regression task.

[0051] For distillation training of the composite loss function, the teacher model is used to guide the student model training, minimizing the composite loss function composed of the following four parts (i.e., ... ): Hard label loss (i.e. This refers to the mean square error (MSE) between the student model's predicted values ​​and the ground truth values, ensuring basic accuracy. Soft label loss (i.e., ...) The mean squared error (MSE) is the difference between the student model's predictions and the teacher model's predictions. The teacher model's output contains a smoother distribution than the true values, which helps with noise resistance. Attention shift loss (i.e., ...) This involves calculating the channel mean of the feature maps of the teacher and student as a spatial attention map (i.e., an attention map), forcing them to align. This ensures that the student model "sees" the same stress deformation region as the teacher model. Relational knowledge distillation loss (i.e., This means that, based on the distance matrix between samples, the student model is forced to maintain the same relative relationship between samples as the teacher model (i.e., Relational Structure), so as to better learn the order of power values.

[0052] Model deployment involves exporting the trained student model and deploying it in the edge computing unit of an industrial robot or tactile sensing terminal. Due to structural simplification and S2D optimization, the student model's inference latency can be reduced by approximately 50%, such as from 0.9ms to 0.4ms, enabling real-time force feedback control at >1000Hz.

[0053] Additionally, it's worth noting that during transfer learning, the backbone network can be kept frozen during the transfer learning phase, while trainable lightweight adapter modules (i.e., adapters) are inserted between convolutional layers, or low-rank adaptation (LoRA) techniques can be used. This approach also achieves the effect of reducing the number of parameters and achieving fast convergence. Alternatively, the backbone network can not be completely frozen; instead, a very low learning rate can be applied to the backbone network, while a high learning rate can be applied to the task heads for joint training.

[0054] In this embodiment, in addition to using a custom Space-to-Depth+SE architecture, you can also use industry-standard lightweight networks as student models, such as MobileNet, ShuffleNet, EfficientNet-Lite, or even lightweight Vision Transformers, such as MobileViT.

[0055] Furthermore, this embodiment illustrates an offline distillation method where the teacher is trained before the student is taught. Alternatively, the teacher and student models can be trained simultaneously, mutually guiding each other—this is online distillation. Self-distillation can also be used, where a separate teacher model is not trained; instead, the model's high-level or deep features guide the shallow features, achieving compression and acceleration.

[0056] Accordingly, see Figure 5 As shown, this application embodiment provides a model deployment device for visual-tactile sensors, including: The dataset construction module 11 is used to construct a source domain dataset based on historical data collected by a preset visual-tactile sensor, determine the target visual-tactile sensor to be calibrated, and construct a target domain dataset based on calibration data collected by the target visual-tactile sensor during the calibration operation triggered for the target visual-tactile sensor. The parameter determination module 12 is used to train a first target model containing a convolutional neural network using the source domain dataset to obtain a second target model, and to determine the model parameters of the second target model. The model training module 13 is used to train the third target model using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-touch sensor. Model building module 14 is used to determine the fourth target model as the teacher model and to build a student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device. The model deployment module 15 is used to train the student model using the teacher model, the composite loss function, and the knowledge distillation technique to obtain the trained student model, and to deploy the trained student model to the tactile sensing device, so that the trained student model can infer and generate a target force value corresponding to the tactile data based on the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value.

[0057] In some specific embodiments, the dataset construction module 11 specifically includes: The dataset construction unit is used to collect data pairs containing optical flow image data and force values ​​from preset visual and tactile sensors in the target production line, identify the data pairs as historical data, and construct a source domain dataset based on the historical data.

[0058] In some specific embodiments, the dataset construction unit specifically includes: The data processing subunit is used to determine the optical flow image data in the historical data, determine the global mean and standard deviation of the optical flow image data, and normalize the optical flow image data using the Z-score normalization formula, the global mean and the standard deviation to obtain normalized data. The force value segmentation subunit is used to determine the force values ​​in the historical data. It uses a hierarchical sampling strategy that divides the force values ​​using a preset force value range and a sampling probability that is inversely proportional to the number of samples in the preset force value range to obtain the segmented force values. The data interpolation subunit is used to perform linear interpolation on the historical data to obtain virtual data; The dataset construction subunit is used to construct the source domain dataset based on the normalized data, the partitioned force values, and the virtual data.

[0059] In some specific embodiments, the parameter determination module 12 specifically includes: The first model training unit is used to determine a first target model containing a convolutional neural network and a task head for predicting normal and shear forces, and to train the first target model using the source domain dataset to obtain a second target model.

[0060] In some specific embodiments, the model training module 13 specifically includes: The model initialization unit is used to determine the model parameters as the initial weights of the third target model, so as to complete the initialization operation of the third target model and obtain the initialized model. The model adjustment unit is used to freeze the first target parameters of the convolutional layer used for feature extraction in the initialized model, and unfreeze the second target parameters of the fully connected layer and the output layer in the initialized model, so as to complete the adjustment operation of the initialized model and obtain the adjusted model. The unfrozen layer training unit is used to determine the fully connected layer and the output layer in the initialized model as unfrozen layers, and to train the unfrozen layers using the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-touch sensor.

[0061] In some specific embodiments, the model building module 14 specifically includes: The first model building unit is used to arrange the spatial dimension pixel blocks of optical flow image data into the channel dimension of the model using tensor transformation operations to complete the first model building operation. The second model building unit is used to embed the SE channel attention mechanism into the convolutional blocks of the model to complete the second model building operation; The third model building unit is used to embed the Swish activation function into the model to complete the third model building operation; The student model construction unit is used to construct a student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device, the first model construction operation, the second model construction operation, and the third model construction operation.

[0062] In some specific embodiments, the model deployment module 15 specifically includes: The first loss term determination unit is used to determine the first predicted value of the force value by the student model, and to determine the first mean square error between the first predicted value and the true value corresponding to the force value, so as to determine the hard label loss term based on the first mean square error. The second loss term determination unit is used to determine the second predicted value of the force value by the teacher model, and to determine the second mean square error between the first predicted value and the second predicted value, so as to determine the soft label loss term based on the second mean square error; The third loss term determination unit is used to determine the channel mean of the feature maps extracted by the student model and the teacher model respectively, and to determine the channel mean as a spatial attention map, so as to determine the attention transfer loss term based on the spatial attention map; The fourth loss term determination unit is used to determine the distance matrix between the sample data received by the student model and the teacher model respectively, so as to determine the knowledge distillation loss term based on the distance matrix; The function construction unit is used to construct a composite loss function based on the hard label loss term, the soft label loss term, the attention transfer loss term, and the knowledge distillation loss term; The second model training unit is used to train the student model using the teacher model, the composite loss function, and the knowledge distillation technique to obtain the trained student model.

[0063] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the model deployment method for visual-touch sensors disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0064] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0065] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0066] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the model deployment method for visual and tactile sensors executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0067] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed model deployment method for visual-tactile sensors. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0068] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0069] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0070] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0071] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for deploying a model for visual-tactile sensors, characterized in that, include: Based on historical data collected by a preset visual-tactile sensor, a source domain dataset is constructed to determine the target visual-tactile sensor to be calibrated. During the calibration operation for the target visual-tactile sensor, a target domain dataset is constructed based on the calibration data collected by the target visual-tactile sensor. The first target model containing a convolutional neural network is trained using the source domain dataset to obtain a second target model, and the model parameters of the second target model are determined. The third target model is trained using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor; The fourth target model is determined as the teacher model, and a student model corresponding to the tactile sensing device is constructed based on the computing power of the tactile sensing device. The student model is trained using the teacher model, composite loss function, and knowledge distillation technique to obtain a trained student model. The trained student model is then deployed to the tactile sensing device so that the trained student model can infer and generate a target force value corresponding to the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value.

2. The model deployment method for visual-tactile sensors according to claim 1, characterized in that, The source domain dataset constructed based on historical data collected through preset visual-tactile sensors includes: Data pairs containing optical flow image data and force values ​​are collected from preset visual and tactile sensors in the target production line. These data pairs are identified as historical data, and a source domain dataset is constructed based on the historical data.

3. The model deployment method for visual-tactile sensors according to claim 2, characterized in that, The construction of the source domain dataset based on the historical data includes: The optical flow image data in the historical data is determined, and the global mean and standard deviation of the optical flow image data are determined. The optical flow image data is then normalized using the Z-score normalization formula, the global mean, and the standard deviation to obtain normalized data. Determine the force values ​​in the historical data, and divide the force values ​​using a stratified sampling strategy that uses a preset force value range and the sampling probability is inversely proportional to the number of samples in the preset force value range, to obtain the divided force values; Linear interpolation is performed on the historical data to obtain virtual data; The source domain dataset is constructed based on the normalized data, the partitioned force values, and the virtual data.

4. The model deployment method for visual-tactile sensors according to claim 1, characterized in that, The step of training a first target model containing a convolutional neural network using the source domain dataset to obtain a second target model includes: A first target model is determined, which includes a convolutional neural network and a task head for predicting normal and shear forces. The first target model is then trained using the source domain dataset to obtain a second target model.

5. The model deployment method for visual-tactile sensors according to claim 1, characterized in that, The step of training the third target model using the model parameters and the target domain dataset to obtain a fourth target model adapted to the characteristics of the target visual-tactile sensor includes: The model parameters are determined as the initial weights of the third target model to complete the initialization operation of the third target model and obtain the initialized model. Freeze the first target parameters of the convolutional layer used for feature extraction in the initialized model, and unfreeze the second target parameters of the fully connected layer and the output layer in the initialized model to complete the adjustment operation of the initialized model and obtain the adjusted model; The fully connected layer and the output layer in the initialized model are determined as the unfrozen layer, and the unfrozen layer is trained using the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor.

6. The model deployment method for visual-tactile sensors according to claim 1, characterized in that, The construction of a student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device includes: Tensor transformation is used to arrange the spatial dimension pixel blocks of optical flow image data into the channel dimension of the model to complete the first model construction operation; An SE channel attention mechanism is embedded in the convolutional blocks of the model to complete the second model construction operation; Embed the Swish activation function into the model to complete the third model construction operation; Based on the computing power of the tactile sensing device, the first model construction operation, the second model construction operation, and the third model construction operation, a student model corresponding to the tactile sensing device is constructed.

7. The model deployment method for visual-tactile sensors according to any one of claims 1 to 6, characterized in that, The process of training the student model using the teacher model, composite loss function, and knowledge distillation technique to obtain the trained student model includes: Determine a first predicted value of the force value by the student model, and determine a first mean square error between the first predicted value and the true value corresponding to the force value, so as to determine a hard label loss term based on the first mean square error; Determine a second predicted value of the force value by the teacher model, and determine a second mean square error between the first predicted value and the second predicted value, so as to determine a soft label loss term based on the second mean square error; The channel mean of the feature maps extracted by the student model and the teacher model is determined, and the channel mean is used as a spatial attention map to determine the attention transfer loss term based on the spatial attention map. Determine the distance matrix between the sample data received by the student model and the teacher model respectively, and determine the knowledge distillation loss term based on the distance matrix; A composite loss function is constructed based on the hard label loss term, the soft label loss term, the attention transfer loss term, and the knowledge distillation loss term; The student model is trained using the teacher model, the composite loss function, and the knowledge distillation technique to obtain the trained student model.

8. A model deployment device for visual-tactile sensors, characterized in that, include: The dataset construction module is used to construct a source domain dataset based on historical data collected by a preset visual-tactile sensor, determine the target visual-tactile sensor to be calibrated, and construct a target domain dataset based on the calibration data collected by the target visual-tactile sensor during the calibration operation triggered for the target visual-tactile sensor. The parameter determination module is used to train a first target model containing a convolutional neural network using the source domain dataset to obtain a second target model, and to determine the model parameters of the second target model. The model training module is used to train the third target model using the model parameters and the target domain dataset to obtain a fourth target model that is adapted to the characteristics of the target visual-tactile sensor. The model building module is used to determine the fourth target model as the teacher model and to build a student model corresponding to the tactile sensing device based on the computing power of the tactile sensing device. The model deployment module is used to train the student model using the teacher model, the composite loss function, and the knowledge distillation technique to obtain the trained student model, and to deploy the trained student model to the tactile sensing device. This allows the trained student model to infer and generate a target force value corresponding to the tactile data perceived by the tactile sensing device, so that the tactile sensing device can perform a preset force feedback control operation based on the target force value.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the model deployment method for visual-tactile sensors as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the model deployment method for visual-tactile sensors as described in any one of claims 1 to 7.

Images