A scalable cross-modal haptic signal generation method for emergency rescue scenarios

Abstract

This invention discloses a scalable cross-modal tactile signal generation method for emergency rescue scenarios. First, audio and video signals are encoded on the device side, and contrastive learning is used to reduce inter-modal differences. Then, the edge terminal senses network bandwidth information and feeds it back to the device side in a timely manner, guiding the device to complete scalable semantic encoding. Finally, after receiving the corresponding cross-modal fused semantic information or different levels of fused semantic representation, the edge terminal uses a designed adaptive tactile signal generation strategy to generate tactile signals of the appropriate granularity. This invention effectively solves the problem of tactile signal loss caused by the difficulty in directly acquiring tactile signals and dynamically fluctuating transmission bandwidth in emergency rescue, ultimately affecting the operator experience when performing remote control. It achieves relevant semantic learning and cross-modal generation of multimodal signals, ensuring real-time and reliable acquisition of tactile signals under conditions of dynamically fluctuating network bandwidth, and ultimately improving the operator's rescue efficiency.

Claims

1. A scalable cross-modal tactile signal generation method for emergency rescue scenarios, characterized in that: The cross-modal tactile signal generation method includes the following steps: Step 1: Use a residual neural network as a feature encoder to extract low-dimensional features from visual and audio data, and use a contrastive learning method to align the extracted visual and audio features. Step 2: Design a cross-modal fusion method. This method first normalizes multimodal features, calculates intermediate representations of different modalities, then uses multi-head attention blocks to extract cross-modal interaction information, and finally merges the features by summing element by element to obtain fused features. Step 3: Utilize the fusion features extracted in the previous step to achieve scalable semantic encoding, thereby obtaining hierarchical semantic representation. At the same time, the network situation awareness module enables a comprehensive understanding of the current network environment and selects the optimal transmission strategy based on network bandwidth. Step 4: After receiving the semantic representation transmitted from the device, the edge device performs semantic decoding and cross-modal tactile signal generation. Depending on the semantic granularity, the edge device adaptively adopts the corresponding generation strategy, namely coarse-grained cross-modal retrieval, fine-grained cross-modal retrieval, and cross-modal generation. Step 5: Train the entire model using an optimization algorithm to obtain the optimal model parameters for the subsequent testing phase. Step 6: Input the paired audio and visual signals into the optimal network model, which extracts the features of the audio and visual signals and performs fusion. Then, it executes the optimal tactile signal recovery strategy based on the current network state.

2. The scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1-1, input the visual image v i and the spectrogram a i of the audio signal into ResNet-18, and extract a low-dimensional feature vector f V and f A ; Steps 1-2: To mitigate modal differences, contrastive learning is used to align features from different modalities, and a similarity function is used to evaluate the similarity between features extracted from different modalities. The similarity function is defined as follows: in and This represents a linear projection that maps features to a low-dimensional representation; V represents video features, and A represents audio features. The value of S ranges from 0 to 1, with higher values ​​indicating greater similarity between features; For each visual and audio feature, calculate the similarity between visual-to-audio and audio-to-visual, using the following formula: where B is batch size, λ represents temperature parameter, V b , A b respectively represent the video feature and the audio feature at the b-th sample. Finally, the contrastive loss is defined as cross-entropy, expressed as: in and This indicates that the probability of a true similarity pair is 1, while the probability of a negative similarity pair is 0.

3. The scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 1, characterized in that: Step 2 includes the following steps: Step 2-1: Feature fusion is performed using two Transformer blocks. Each block consists of a Multi-Head Attention (MHA) block, a Normalization Layer (NL) layer, and a Multi-Layer Perceptron (MLP). The MHA is used to extract the semantic association between visual and audio features. The specific steps are as follows: Step 211, first, normalize the input multi-modal features to obtain f V-norm and f A-norm ; Step 212: Next, use linear projection to calculate the intermediate representations of different modalities: query Q, key K, and value V; Step 213, then, cross-modal interaction information f is obtained through MHA V-iter and f A-iter ; the process is described as follows: Q a K v V v Q v K a V a It is an intermediate representation of visual and audio features, where d represents the dimension; Step 214: Combine the outputs of the two Transformer blocks by summing element by element to obtain f. V-A The entire process can be described as follows:

4. The scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3-1: Use the concise representation of the fusion features extracted in the previous step to achieve scalable semantic encoding, that is, use feature maps of different depths to represent different semantic granularities. Step 311: The fusion feature with complete semantics is represented as f V-A The shallow and deep feature maps extracted from the fused features through the neural network are represented as f, respectively. V-A-s and f V-A-d ; Feature f V-A It will be used for direct tactile generation, while f V-A-s and f V-A-d They will be used for fine-grained and coarse-grained retrieval respectively, mainly for indirect tactile generation; Step 312: After determining the category, search for tactile signals of the same category in the existing tactile database; the information required for classification is a subset of the information required for complete signal reconstruction; similarly, for retrieval tasks of different granularities, the information required for coarse-grained tasks is a subset of the information required for fine-grained tasks; the bandwidth required to transmit this information is represented as M1, M2, and M3, where M1 > M2 > M3; in addition, the bandwidth is closely related to the feature dimension, the frame rate of acquiring visual signals, and the channel coding strategy; Step 3-2: At the same time, in the above semantic encoding process, it is necessary to deal with different network conditions and select to use a bandwidth prediction network that combines a heuristic-based controller and a deep reinforcement learning controller. Step 321: Initially, when the input data is limited, it uses a heuristic-based controller; Step 322: Once enough input data has been collected, the bandwidth prediction network will use deep reinforcement learning to adapt to different network conditions; the device dynamically adjusts the scalable coding network based on the predicted bandwidth information.

5. A scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4-1: After receiving the semantic representation transmitted by the device, the edge device performs semantic decoding and haptic generation. Depending on the granularity of the semantics, the edge device adaptively executes the corresponding generation strategy, namely coarse-grained cross-modal retrieval, fine-grained cross-modal retrieval, and cross-modal generation. Step 411: First, perform semantic decoding to obtain semantic information; Step 422: Execute the corresponding signal recovery strategy according to the granularity of the received semantic information.

6. A scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 5, characterized in that: The retrieval model progresses from coarse-grained to fine-grained: under limited bandwidth conditions, deep feature maps containing less semantic information are used for coarse-grained retrieval, while shallow feature maps containing more semantic information are used for fine-grained retrieval; after obtaining category information, tactile signals of the same category are retrieved from the existing tactile database and presented to the operator; the retrieval model includes fully connected layers, and the structure of the final layer is determined by the number of categories in the database; the output of the final layer is used to calculate scores for different categories, which are then normalized using the softmax function; The ultimate goal is to minimize the cross-entropy between the predicted distribution and the actual labels. This goal is defined as: in, It represents the one-hot encoding of the true labels in fine-grained and coarse-grained retrieval models. and It is the predicted multi-class probability.

7. A scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 5, characterized in that: Cross-modal generation: With sufficient bandwidth, the device transmits complete fused features, enabling cross-modal generation at the edge. Generative Adversarial Networks (GANs) are used to generate tactile signals. Two discriminator networks, D1 and D2, are combined to enhance the consistency between the generated tactile signal h′ and the real tactile signal h. Specifically, D1 aims to distinguish two pairs of signals: h paired with itself, and h paired with the generated signal h′. This process ensures the structural similarity of the signals, effectively aligning h and h′ from a global perspective. Discriminator D2 functions similarly to a standard GAN. The adversarial learning loss function of D1 and D2 is expressed as: Where, θ g θ d1 θ d2 These are the parameters of G, D1, and D2, where p(*) represents the signal distribution, and E... h′～p(h′),h～p(h) [] indicates that h′ and h are sampled from distributions p(h′) and p(h), respectively, and the expected value is calculated for the expression within the brackets; E h～p(h) [] indicates sampling h from distribution p(h) and calculating the expected value of the expression within the brackets; E h′～p(h′) [] indicates that h′ is sampled from distribution p(h′), and the expected value is calculated for the expression within the brackets; In addition, the L2 loss between the generated result and the actual tactile signal is calculated in the following way: L G = ||hh′||2; Overall, the total loss of cross-modal generation is defined as: L gen =L D1 +L D2 +L G ; With the above objective function, the generation and discriminant models are iteratively trained to generate tactile signals closer to the real tactile signals; finally, the overall objective function of the cross-modal tactile generation scheme is written as: L total =L contra +L fine +L coarse +L gen 。 8. A scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 1, characterized in that: Step 5 includes the following steps: Step 5-1: Use visual signals, audio signals, and tactile signals to train and optimize the encoding network, fusion, and generation network. After freezing the parameters of the visual encoding network, audio encoding network, and cross-modal encoding network, then train the fine-grained retrieval network, and then freeze the parameters of the fine-grained retrieval network, and then train the fine-grained retrieval network to obtain the trained network parameters. The specific process is as follows: Step 511: Initialize the parameters of the visual encoding network, audio encoding network, and cross-modal encoding network, set the number of iterations: epoch1, set the learning rates: μ1, μ2, μ3, and start training the encoding, fusion, and generation network; Step 512: Update θ using the Adam optimizer. v The corresponding loss function L contra +L gen Update θ a The corresponding loss function L contra +L gen Update θ cmf The corresponding loss function L contra +L gen Update θ g The corresponding loss function L contra +L gen Update θ d1 The corresponding loss function L D1 Update θ d2 The corresponding loss function L D2 The specific format is as follows: Where θ v θ a θ cmf θ g θ d1 θ d2 These are the visual encoder, audio encoder, cross-modal feature fusion module, generator network, discriminator d1, and discriminator d2, respectively. ▽ represents the partial derivative with respect to each loss function. Step 513: If epoch < epoch1, then repeat Step 512. After epoch1 rounds of iteration, obtain the network that converges to the optimum, and at the same time update and freeze the parameters of the visual encoding network, audio encoding network, and cross-modal encoding network; Step 514: Set the number of iterations: epoch2, and start training the fine-grained retrieval network; Step 515: Update θ using the Adam optimizer. ssc and θ c-fine The corresponding loss function L c-coarse +L c-fine The specific formula is as follows: i ssc =θ ssc -μ1▽ θssc (L c-coarse +L c-fine ), Where θ ssc θ c-fine These are all fine-grained retrieval network parameters, and ▽ represents the partial derivative with respect to each loss function; Step 516: If epoch < epoch2, then repeat Step 515. After epoch2 rounds of iteration, obtain the network that converges to the optimum, and at the same time update and freeze the parameters of the fine-grained retrieval network; Step 517: Set the number of iterations: epoch3, and start training the coarse-grained retrieval network; Step 518: Update θ using the Adam optimizer. c-coarse The corresponding loss function L c-coarse +L c-fine The specific formula is as follows: i c-coarse =θ c-coarse -μ1▽ θc-coarse (L c-coarse ), Where θ c-coarse represents the parameters of the coarse-grained retrieval network, and ▽ represents the partial derivatives with respect to each loss function; Step 519: If epoch < epoch3, then repeat Step 518. After epoch3 rounds of iteration, obtain the network that converges to the optimum, and obtain the optimal model parameters for use in the subsequent test phase.

9. A scalable cross-modal tactile signal generation method for emergency rescue scenarios according to claim 1, characterized in that: Step 6 includes the following steps: Step 6-1: Input the paired audio and visual signals into the optimal network model, which will extract the features of the audio and visual signals, perform fusion, and then execute the optimal tactile signal recovery strategy according to the current network state. The specific steps are as follows: Step 611: Collect visual and audio samples; Step 612: Perform visual encoding and tactile encoding on the samples; Step 613: Perform feature extraction and cross-modal feature fusion to generate features; Step 614: Based on network state perception, determine the current network bandwidth M; Step 615: The device performs different processing based on the received bandwidth information. If M > M1, the transmission fusion feature f... V-A At the edge computing end, a generative network is used to generate tactile signals across modalities; if M2 < M < M1, shallow feature maps f are transmitted. V-A-s At the edge computing end, a fine-grained cross-modal retrieval network is used to perform fine-grained cross-modal retrieval, searching for tactile signals based on the existing database; if M3 < M < M2, shallow feature maps f are transmitted. V-A-d At the edge computing end, a coarse-grained cross-modal retrieval network is used to perform coarse-grained cross-modal retrieval, searching for tactile signals based on existing databases; Step 616: The edge side finally transmits the generated tactile signal to the user.

Images