Future frame prediction method for cross-modal communication

By establishing a multi-scale prediction network and a converged network in cross-modal communication, using haptic data to assist in predicting video frames, the fuzzy problem of visual data prediction in cross-modal communication is solved, and higher prediction accuracy and performance are achieved. Mining the correlation of multimodal data, the generated predicted frame quality is higher.

CN118900337APending Publication Date: 2024-11-05NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410574260.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-10
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

In cross-modal communication, traditional methods are difficult to effectively predict and recover lost visual data, especially when correlations between multimodal signals are not fully utilized, resulting in blurred and inaccurate prediction results.

Method used

A future frame prediction method for cross-modal communication is designed. By acquiring tactile data and video data, a multi-scale prediction network and a multi-scale fusion network are established, using generators and discriminators, combining absolute errors and average squared errors between images. , and other loss functions such as confrontation generation loss, train cross-modal video frame prediction models to generate high-quality future video frames.

Benefits of technology

By leveraging the correlation between tactile signals and visual signals, the fuzzy problem of traditional single-modal prediction is overcome, and higher prediction accuracy and performance indicators are achieved. In-depth exploration of the intrinsic connections between multimodal data and breaking different modal data The barrier between the generated predicted frames are closer to the real frame at the pixel level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118900337A_ABST
    Figure CN118900337A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of deep learning, and discloses a future frame prediction method for cross-modal communication, and the method comprises the steps: obtaining tactile data and video data of a receiving end of a cross-modal communication system, cutting a video frame image of the video data, and packaging the cut video frame image and a tactile signal of the corresponding tactile data as a training set; establishing a cross-modal video frame prediction model under the assistance of the tactile data; training the cross-modal video frame prediction model under the assistance of the tactile data, and calculating to obtain a plurality of loss parameters for updating the cross-modal video frame prediction model; after the parameters of the model converge, an optimal cross-modal video frame prediction model and parameters are obtained, tactile data corresponding to historical video frames and future frames received by a receiving end of the cross-modal communication system are input into the trained optimal cross-modal video frame prediction model, and the future video frames are output; according to the method, the internal relation among the multi-modal data is deeply mined, and the fuzzy problem caused by traditional single-modal video frame prediction is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a future frame prediction method for cross-modal communication. Background Art

[0002] With the continuous rise of digitalization, intelligence and personalization needs, multimodal services that can be heard, seen and touched have emerged, bringing users unprecedented sensory experience. Such services can not only realistically reproduce multiple sensory information such as hearing, touch and vision at the same time, but also play a pivotal role in new service paradigms such as telemedicine, virtual reality enhancement and distance education. Considering that people's cognition of the external environment is a diversified and multi-level integrated process, the various sensory signals obtained, such as audio, video, touch, etc., are actually descriptions of the same thing in different dimensions. There must be a strong correlation between these descriptions. In order to deeply explore the connection between different modes, cross-modal communication technology has emerged by using modal linkage;

[0003] When conducting cross-modal communication, it is equally important to consider tactile information and visual information, whether in terms of material recognition or mutual contact force estimation; researchers have begun using sensors to obtain tactile information and visual information at the source, and try to predict the visual data lost in the transmission process at the remote end; therefore, the rational use of the correlation between visual information and tactile information can help the task of predicting and estimating missing visual information. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems and to design a future frame prediction method for cross-modal communication.

[0005] The present invention provides a future frame prediction method for cross-modal communication, and the future frame prediction method for cross-modal communication comprises the following steps:

[0006] Acquire tactile data and video data from a receiving end of the cross-modal communication system, and crop video frame images of the video data and package the corresponding tactile signals of the tactile data as a training set;

[0007] Establishing a cross-modal video frame prediction model assisted by the tactile data, wherein the cross-modal video frame prediction model includes a multi-scale prediction network and a multi-scale fusion network, the multi-scale prediction network includes a generator and a discriminator, and the multi-scale fusion network includes convolution, upsampling and channel addition operations;

[0008] The cross-modal video frame prediction model assisted by the tactile data is trained, the absolute error loss and the mean square error loss between images are calculated according to the generated multi-scale images, the gradient loss between images is calculated according to the final generated image and the real image, the adversarial generation loss of the generated image is calculated according to the generative adversarial network and combined with the mean square error, and the calculated multiple losses are used to update the parameters of the cross-modal video frame prediction model;

[0009] When the parameters of the model converge, the optimal cross-modal video frame prediction model and parameters are obtained, and the tactile data corresponding to the historical video frames and future frames received by the receiving end of the cross-modal communication system are input into the trained optimal cross-modal video frame prediction model, and the output is the future video frame.

[0010] Optionally, in a first implementation of the present invention, the multi-scale prediction network process of the cross-modal video frame prediction model assisted by tactile data includes:

[0011] Cropped video data and tactile data {h 1 ,h 2 ,h 3 ,…,h k} to obtain a training set, and input the training set into the multi-scale network {N 1 ,N 2 ,N 3 ,N 4} for training;

[0012] Generator G k The input is the cropped image and tactile data {h j ,h j+1 ,h j+2 ,,h j+i-1}, by extracting the features of the historical frame, generating the pixel values ​​of each position of the predicted frame, and obtaining a predicted frame of size k Where T is the prediction step length, and the generator network includes a convolution layer and an activation function layer;

[0013] The convolution layer convolves the input features with the filter bank to generate output features, where the equation of the convolution layer c can be expressed as:

[0014]

[0015] Among them, n c is the number of convolution kernels in layer c, is the convolution kernel corresponding to the p channel of the c-th layer and the q channel of the c-1-th layer, is the offset of the p channel of the cth layer, z c is the output of the cth convolutional layer without activation function, φc is the output of the cth layer after the activation function. Except for the last convolutional layer whose activation function is Tanh, the activation functions of the remaining convolutional layers are all ReLU, that is,

[0016]

[0017] Discriminator D k The input is the generator G k Generated prediction frame and the corresponding real frame The discriminator uses multi-level convolution operations to deeply process the input samples and gradually convert the input samples into feature vectors;

[0018] After feature extraction, the discriminator inputs the obtained feature vector into the fully connected layer, which is expressed as follows:

[0019]

[0020] Among them, W c represents the weight of the c-th layer of the fully connected network, ρ c Represents the connectivity between nodes in the cth layer, which conforms to the Bernoulli distribution;

[0021] The last fully connected layer uses the Sigmoid activation function to limit the prediction results to the range of [0, 1].

[0022] Optionally, in a second implementation of the present invention, after the feature extraction is completed, the discriminator inputs the obtained feature vector into a fully connected layer, including:

[0023] When processing the input feature map, the discriminator uses convolution operations to extract and transform features, and uses a pooling layer to reduce the size of the feature map and extract key information. Before inputting the obtained feature vector into the fully connected layer, the multi-dimensional feature data processed by pooling is compressed into a one-dimensional vector form;

[0024] Through the activation function, the feature vector is mapped to a probability value between 0 and 1. If the probability value exceeds 0.5, the sample is judged to be from the real data set; if it is lower than 0.5, it is regarded as a sample generated by the generator.

[0025] Optionally, in a third implementation of the present invention, when training the discriminator, a real frame sequence is used. As the training target of positive samples, use the frame sequence generated by the generator As the training target of negative samples, the loss function of the discriminator can be expressed as:

[0026]

[0027] The training loss function of the generator can be defined as:

[0028]

[0029] Among them, L bce is the binary cross entropy loss, defined as

[0030]

[0031] Among them, v i ∈[0,1],

[0032] Optionally, in a fourth implementation of the present invention, a distance loss function L is used p As an additional loss function, we use To train the generator, for (x,y) and The pixel at the position whose L p The loss is expressed as:

[0033]

[0034] Optionally, in a fifth implementation of the present invention, an image gradient loss GDL is added. For the real image v gt and predicted image The GDL definition is as follows:

[0035]

[0036] Here, γ is an integer greater than or equal to 1.

[0037] Optionally, in a sixth implementation of the present invention, the multi-scale fusion network process of the cross-modal video frame prediction model assisted by tactile data includes:

[0038] The image with the smallest scale is taken as the top layer of the pyramid. The larger scale images are integrated into the pyramid in sequence by gradually upsampling and horizontal connection. At the bottom layer of the pyramid, an image integrating features of all scales is obtained.

[0039] Assume that the images of four scales are in is the minimum scale image, For the maximum scale image, the pyramid fusion process is recursively defined as:

[0040]

[0041] Among them, F n represents the fused feature map of the nth layer of the pyramid, Upsamping(·) represents the upsampling operation, Indicates the additive fusion in the channel dimension, Conv 1×1 Represents a 1×1 convolution operation, which is used to adjust the number of channels, F 4 It is the output image that integrates four scale features.

[0042] Optionally, in a seventh implementation of the present invention, the image with the smallest scale is used as the top layer of the pyramid, and the images with larger scales are successively integrated into the pyramid by stepwise upsampling and lateral connection, and an image integrating features of all scales is obtained at the bottom layer of the pyramid, including:

[0043] In each fusion step, the small-scale image is first upsampled so that the small-scale image has the same spatial resolution as the larger-scale image at the next level;

[0044] The upsampled image is added to the larger scale image in the channel dimension by using 1×1 convolution for lateral connection.

[0045] Optionally, in an eighth implementation of the present invention, training the cross-modal video frame prediction model assisted by the tactile data includes:

[0046] Get the training set containing n frames of video as V = {v 1 ,v 2 ,v 3 ,…,v t ,…,v n}, where v t Represents the video frame at time t, each video frame in V has a corresponding tactile information h;

[0047] Each i frame in the video V is taken as a set of input I group = {v j ,v j+1 ,v j+2 ,…,v j+i-1}, where v j ,v j+1 ,v j+2 ,…,v j+i-t As the input history frame, v j+i-t-1 ,…,v j+i-1 As a reference frame for comparison with the predicted frame;

[0048] Each i frame in the video V is taken as a set of input I group = {v j ,v j+1 ,v j+2 ,…,v j+i-1}, where v j ,v j+1 ,v j+2 ,…,vj+i-t As the input history frame, v j+i-t-1 ,…,v j+i-1 As a reference frame for comparison with the predicted frame;

[0049] Assume that the pixel difference between two adjacent frames at the (x, y) position is Then the pixel difference of adjacent cropped images of size c×c is:

[0050]

[0051] Set Threshold I threshold ,when All adjacent frames in Should As the input of the prediction network, otherwise it will be discarded;

[0052] Initialize the parameters of the cross-modal video frame prediction model assisted by tactile data, and iteratively train the cross-modal video frame prediction model assisted by tactile data using the gradient descent algorithm and the Adam optimizer until the maximum number of iterations is reached.

[0053] Optionally, in a ninth implementation of the present invention, the parameters of the cross-modal video frame prediction model assisted by tactile data are initialized, and the cross-modal video frame prediction model assisted by tactile data is iteratively trained using a gradient descent algorithm and an Adam optimizer until a maximum number of iterations is reached, including:

[0054] Set the learning rate of the generator and discriminator, set the loss function coefficient, set the network parameter saving frequency and test frequency;

[0055] The set network parameters are updated based on the gradient descent algorithm. When the number of iterations reaches the specified maximum number, the indicator results and predicted images of the last test are saved.

[0056] The advantages of the future frame prediction method for cross-modal communication of the present invention are:

[0057] (1) The present invention can overcome the blurring problem caused by traditional single-modality video frame prediction by leveraging the correlation between tactile signals and visual signals, that is, the prediction frame takes a compromise blurring result in order to maintain high performance indicators;

[0058] (2) The present invention makes full use of the correlation between different modalities, deeply explores the intrinsic connection between multimodal data, and breaks down the barriers between different modal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0060] Figure 1 It is a complete network structure schematic diagram of the present invention.

[0061] Figure 2 It is a multi-scale prediction network model diagram in the cross-modal video frame prediction model assisted by tactile data of the present invention;

[0062] Figure 3 It is a multi-scale fusion network model diagram in the cross-modal video frame prediction model assisted by tactile data of the present invention;

[0063] Figure 4 This is a diagram of the prediction results of the next two frames of the comparative test of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific implementation examples.

[0065] The present invention provides a future frame prediction method for cross-modal communication, and its complete network structure diagram is as follows: Figure 1 As shown, the method comprises the following steps:

[0066] Step 101: Acquire tactile data and video data from a receiving end of a cross-modal communication system, crop video frame images of the video data and package the tactile signals of the corresponding tactile data as a training set;

[0067] The dataset is resized and the number of multi-scale networks of the prediction network is k. Then the divided video frame groups I group Cut into k different sizes, namely It is packaged with the corresponding tactile data and sent to the multi-scale network for training and prediction. Figure 2 Shown on the left.

[0068] The present invention uses 4 generative adversarial networks, so the images are cropped into four sizes, namely 4×4, 8×8, 16×16, and 32×32, which can not only significantly improve computational efficiency and reduce resource consumption, but also enable the model to capture more abstract key features, thereby enhancing adaptability to scale changes and improving generalization capabilities. These small-sized images are selected for training to achieve a balance in multiple aspects such as computational efficiency, feature abstraction, scale robustness, and model generalization, so as to achieve more efficient learning and prediction of image-related tasks.

[0069] The cropped image must contain the largest part of the change. Assume that the pixel difference between two adjacent frames at the (x, y) position is It is considered that the pixel difference of adjacent cropped images of size c×c is:

[0070]

[0071] In order to filter out image blocks with rich information, the present invention sets a threshold I threshold , only when All adjacent frames in Should Only then will it be used as the input of the prediction network, otherwise it will be discarded. This screening mechanism ensures that the data input into the multi-scale network has sufficient information, thereby helping the network to extract more useful features and improve the accuracy of training and prediction.

[0072] Step 102: establishing a cross-modal video frame prediction model assisted by tactile data, wherein the cross-modal video frame prediction model includes a multi-scale prediction network and a multi-scale fusion network, the multi-scale prediction network includes a generator and a discriminator, and the multi-scale fusion network includes convolution, upsampling and channel addition operations;

[0073] By using the historical visual signals and the tactile signals corresponding to the future frames, a multi-scale generative adversarial network is used to obtain the multi-scale future video frames as the input of the multi-scale fusion network. Figure 2 Shown on the right.

[0074] The first multi-scale generative network is composed of a convolutional network with 3×3 convolution kernels, and the corresponding filter sizes are 128, 256, 128, and 3. Then the first size video frame generated by the generative network is input into the first multi-scale discriminative network. The first discriminator network consists of a convolutional network with 64 3×3 convolution kernels, a pooling layer, a flattening layer, and a fully connected layer. The number of neurons in the fully connected layer is 512, 256, and 1.

[0075] The second multi-scale generative network is composed of a convolutional network with 3×3 and 5×5 convolutional kernels, with the number of convolutional kernels being 128, 256, 128, and 3; the second multi-scale discriminative network is composed of three layers of 3×3 convolutional networks, pooling layers, flattening layers, and fully connected layers. The number of convolutional kernels in the convolutional network is 64, 128, and 128, respectively, and the number of neurons in the fully connected layer is 1024, 512, and 1. The third multi-scale generative network is also composed of a convolutional network with 3×3 and 5×5 convolutional kernels, with the number of convolutional kernels being 128, 256, 512, 256, 128, and 3; the third multi-scale discriminative network is composed of three layers of 5×5 convolutional networks, pooling layers, flattening layers, and fully connected layers. The number of convolutional kernels in the convolutional network is 128, 256, and 256, respectively, and the number of neurons in the fully connected layer is 1024, 512, and 1. The fourth multi-scale generative network is also composed of a convolutional network with 5×5 and 7×7 convolutional kernels, and the number of convolutional kernels is 128, 256, 512, 256, 128, and 3; the fourth multi-scale discriminative network is composed of a 5×5 and 7×7 convolutional network, a pooling layer, a flattening layer, and a fully connected layer. The number of convolutional kernels in the convolutional network is 128, 256, 512, and 256, respectively, and the number of neurons in the fully connected layer is 1024, 512, and 1.

[0076] In the four generating networks, the activation function used in the convolutional neural network except the last convolutional layer is Tanh, and the activation function used in the remaining layers is ReLU.

[0077] The discriminator is trained and optimized by discriminative adversarial loss. The specific definition of discriminative loss is as follows:

[0078]

[0079] Among them, L bce is the binary cross entropy loss, defined as

[0080]

[0081] Among them, v i ∈[0,1],

[0082] Similarly, the generator is trained and optimized by generating adversarial loss, and the specific definition of generating adversarial loss is as follows:

[0083]

[0084] In order to ensure that the predicted video frame does not produce edge blur at the pixel level, the present invention uses the distance loss function L p As an additional loss function, for (x, y) and The pixel at the position whose L p Loss is specifically defined as:

[0085]

[0086] Therefore, we can use To train the generator. adv and α p is the loss ratio coefficient.

[0087] In order to enable the network to pay more attention to the edge and texture information of the image during the training process and improve the consistency between the predicted image and the real image at the gradient level, the present invention also adds the image gradient loss (Gradient Difference Loss, GDL). gt and predicted image Its GDL definition is as follows:

[0088]

[0089] Where γ is an integer greater than or equal to 1. By calculating the gradient difference between adjacent pixels of the predicted image and the real image, the quality of the generated image is improved.

[0090] Therefore, the total loss function of the generator is:

[0091]

[0092] Among them, α GDL is the scaling factor of the gradient loss.

[0093] Step 103: training the cross-modal video frame prediction model assisted by tactile data, calculating the absolute error loss and mean square error loss between images based on the generated multi-scale images, calculating the gradient loss between images based on the final generated image and the real image, calculating the adversarial generation loss of the generated image based on the generative adversarial network and combined with the mean square error, and using the calculated multiple losses to update the parameters of the cross-modal video frame prediction model;

[0094] Design a multi-scale prediction frame fusion network; the multi-scale fusion network structure diagram is as follows Figure 3 as shown:

[0095] After obtaining the predicted frames of four different scales from the prediction network, the smallest prediction frame is first upsampled to make its features consistent with the size of the second smallest prediction frame; then the second smallest prediction frame is extracted into image features through a 1×1 convolution operation, and the two features are added channel by channel to obtain the first fused feature. The first fused feature is upsampled and added to the features of the second largest prediction frame after the convolution operation in the channel dimension to obtain the second fused feature, and the third fused feature is obtained by the same operation.

[0096] The specific process is as follows:

[0097] The four scales of images are in is the minimum scale image, is the maximum scale image. The pyramid fusion process can be recursively defined as:

[0098]

[0099] Among them, F n represents the fused feature map of the nth layer of the pyramid, Upsamping(·) represents the upsampling operation, Indicates the additive fusion in the channel dimension, Conv 1×1 represents a 1×1 convolution operation, which is used to adjust the number of channels. Finally, F 4 That is, the image features of the four scale features are integrated, and finally the final predicted frame with the same size as the original video frame is obtained through deconvolution.

[0100] The multi-scale prediction network is trained using an optimization method to obtain the optimal model parameters for the subsequent testing phase. The multi-scale generation network and the multi-scale discrimination network are trained using continuous multi-scale video frames and corresponding tactile data. The specific process is as follows:

[0101] Set the clipping threshold I threshold =100;

[0102] Set the batch size batch = 4, the distance loss p = 1, the historical input frames are 4, and the predicted future frames are 2;

[0103] Set the learning rate l of the multi-scale generation network G =0.04, the learning rate l of the multi-scale discrimination network D =0.02;

[0104] Set γ = 1 and the loss coefficient α adv =0.05,α p =1,α GDL =1;

[0105] Start iteration and set the number of iterations to 15000;

[0106] The gradient descent algorithm and Adam optimizer are used for training updates.

[0107] Step 104: After the parameters of the model converge, the optimal cross-modal video frame prediction model and parameters are obtained, and the tactile data corresponding to the historical video frames and future frames received by the receiving end of the cross-modal communication system are input into the trained optimal cross-modal video frame prediction model, and the output is the future video frame.

[0108] The paired historical input frames and future frame tactile signals of the test set are input into the optimal multi-scale prediction network, which is used to generate prediction frames of 4 sizes. The generated prediction frames are fused using the fusion network to generate the final prediction frame. A trained multi-scale prediction network is used; a set of historical frames and future frame tactile signals are sent into the multi-scale prediction network to complete multi-scale prediction and multi-scale fusion, and finally the corresponding future video frame is obtained.

[0109] The following experimental results show that compared with the prediction network without adding tactile data, the present invention uses the correlation and complementarity of multimodal signals to realize the prediction of visual signals and achieves better generation effect.

[0110] The present invention uses a visual-contact force dataset to test the performance of the proposed method. Figure 4 As shown in Figure 1, the dataset contains contact videos of four objects (paper cup, sponge, stapler and hose). The video content is touched from four different angles (0°, 10°, 20°, 30°) and three lighting conditions (350, 550 and 750 lux). The total data volume is 17×4×4×3 video clips, with a total of 816 video clips, each of which contains the complete process of four contacts. Each clip contains about 500 continuous video frames. There are a total of 440,967 images, each of which is 128×128 (RGB) pixels. Each video frame corresponds to a contact force value measured by the sensor. Therefore, there are 816 text files recording the contact force values. The video frame rate of the dataset is 120fps, that is, 120 video frames are taken per second. The tactile data acquisition device consists of an RC servo motor and a probe, which avoids the possibility that the probe may be excluded as noise due to the probe being too small, or the deformation of the contact position may be too large due to the probe being too large to affect the observation. For elastically deformable objects made of a single material, such as sponges and paper cups, applying force at different locations on the surface can produce an obviously deformable object without causing difficulty in feature extraction due to the different locations of the contact points. However, for inelastically deformable objects made of multiple materials, such as staplers and hoses, applying force at different locations will produce deformations that vary due to the different materials, which may affect feature extraction and reduce performance. Therefore, this dataset applies force at the center of the object and measures the force value only in the z direction.

[0111] In this embodiment, the performance indicators used to evaluate the future frame prediction method proposed by the present invention are divided into two categories: peak signal-to-noise ratio and structural similarity.

[0112] Peak signal-to-noise ratio: Peak signal-to-noise ratio (PSNR) is a visual signal evaluation indicator. It is based on the error between pixels and calculates the ratio of the peak signal energy to the average energy of the noise. It is the most common and widely used objective evaluation indicator of visual signals. The higher the PSNR value, the smaller the distortion.

[0113] Structural similarity: Structural similarity (SSIM) measures the similarity of visual signals from three aspects: brightness, contrast, and structure. The value range of SSIM is [0,1]. The larger the value, the smaller the distortion of the visual signal. Structural similarity is more consistent with the human eye's perception judgment than PSNR in evaluating the quality of visual signals.

[0114] Table 1 Comparison of prediction network prediction performance for 2 frames

[0115]

[0116]

[0117] From Table 1 and Figure 4 It can be clearly seen that with the assistance of tactile data, the proposed method shows better performance for both the first frame prediction and the second frame prediction. This means that by incorporating tactile information, the prediction model can more accurately capture the dynamic changes of video frames in the environment of the generative adversarial network, thereby generating higher quality prediction frames. This result not only verifies the effectiveness of tactile data in enhancing the accuracy of video frame prediction, but also demonstrates the superiority and adaptability of the proposed method in complex network structures. By comparing the performance indicators of different prediction frames, it can be further confirmed that the proposed method has stable performance in continuous frame prediction tasks and can provide reliable support for practical application scenarios. In the prediction of the sponge and hose contact video frames, the proposed method predicts the movement trend of the probe more accurately than without tactile assistance.

[0118] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A future frame prediction method for cross-modal communication, characterized in that: The future frame prediction method for cross-modal communication comprises the following steps: Acquire tactile data and video data from a receiving end of the cross-modal communication system, and crop video frame images of the video data and package the corresponding tactile signals of the tactile data as a training set; Establishing a cross-modal video frame prediction model assisted by the tactile data, wherein the cross-modal video frame prediction model includes a multi-scale prediction network and a multi-scale fusion network, the multi-scale prediction network includes a generator and a discriminator, and the multi-scale fusion network includes convolution, upsampling and channel addition operations; The cross-modal video frame prediction model assisted by the tactile data is trained, the absolute error loss and the mean square error loss between images are calculated according to the generated multi-scale images, the gradient loss between images is calculated according to the final generated image and the real image, the adversarial generation loss of the generated image is calculated according to the generative adversarial network and combined with the mean square error, and the calculated multiple losses are used to update the parameters of the cross-modal video frame prediction model; When the parameters of the model converge, the optimal cross-modal video frame prediction model and parameters are obtained, and the tactile data corresponding to the historical video frames and future frames received by the receiving end of the cross-modal communication system are input into the trained optimal cross-modal video frame prediction model, and the output is the future video frame.

2. A future frame prediction method for cross-modal communication according to claim 1, characterized in that: The multi-scale prediction network process of the cross-modal video frame prediction model assisted by tactile data includes: Cropped video data and tactile data {h1,h2,h3,…,h k } to obtain a training set, and input the training set into a multi-scale network {N1, N2, N3, N4} for training; Generator G k The input is the cropped image and tactile data {h j ,h j+1 ,h j+2 ,,h j+i-1 }, by extracting the features of the historical frame, generating the pixel values ​​of each position of the predicted frame, and obtaining a predicted frame of size k Where T is the prediction step length, and the generator network includes a convolution layer and an activation function layer; The convolution layer convolves the input features with the filter bank to generate output features, where the equation of the convolution layer c can be expressed as: Among them, n c is the number of convolution kernels in layer c, is the convolution kernel corresponding to the p channel of the c-th layer and the q channel of the c-1-th layer, is the offset of the p channel of the cth layer, z c is the output of the cth convolutional layer without activation function, φ c is the output of the cth layer after the activation function. Except for the last convolutional layer whose activation function is Tanh, the activation functions of the remaining convolutional layers are all ReLU, that is, Discriminator D k The input is the generator G k Generated prediction frame and the corresponding real frame The discriminator uses multi-level convolution operations to deeply process the input samples and gradually convert the input samples into feature vectors; After feature extraction, the discriminator inputs the obtained feature vector into the fully connected layer, which is expressed as follows: Among them, W c represents the weight of the c-th layer of the fully connected network, ρ c Represents the connectivity between nodes in the cth layer, which conforms to the Bernoulli distribution; The last fully connected layer uses the Sigmoid activation function to limit the prediction results to the range of [0, 1].

3. A future frame prediction method for cross-modal communication according to claim 2, characterized in that: After the feature extraction is completed, the discriminator inputs the obtained feature vector into the fully connected layer, including: When processing the input feature map, the discriminator uses convolution operations to extract and transform features, and uses a pooling layer to reduce the size of the feature map and extract key information. Before inputting the obtained feature vector into the fully connected layer, the multi-dimensional feature data processed by pooling is compressed into a one-dimensional vector form; Through the activation function, the feature vector is mapped to a probability value between 0 and 1. If the probability value exceeds 0.5, the sample is judged to be from the real data set; if it is lower than 0.5, it is regarded as a sample generated by the generator.

4. A future frame prediction method for cross-modal communication according to claim 2, characterized in that: When training the discriminator, use the real frame sequence As the training target of positive samples, use the frame sequence generated by the generator As the training target of negative samples, the loss function of the discriminator can be expressed as: The training loss function of the generator can be defined as: Among them, L bce is the binary cross entropy loss, defined as Among them, v i ∈[0,1], 5. A future frame prediction method for cross-modal communication according to claim 2, characterized in that: Use the distance loss function L p As an additional loss function, we use To train the generator, for (x,y) and The pixel at the position whose L p The loss is expressed as:

6. A future frame prediction method for cross-modal communication according to claim 2, characterized in that: Add image gradient loss GDL, for the real image v gt and predicted image The GDL definition is as follows: Here, γ is an integer greater than or equal to 1.

7. A future frame prediction method for cross-modal communication according to claim 1, characterized in that: The multi-scale fusion network process of the cross-modal video frame prediction model assisted by tactile data includes: The image with the smallest scale is taken as the top layer of the pyramid. The larger scale images are integrated into the pyramid in sequence by gradually upsampling and horizontal connection. At the bottom layer of the pyramid, an image integrating features of all scales is obtained. Assume that the images of four scales are in is the minimum scale image, For the maximum scale image, the pyramid fusion process is recursively defined as: Among them, F n represents the fused feature map of the nth layer of the pyramid, Upsamping(·) represents the upsampling operation, Indicates the additive fusion in the channel dimension, Conv 1×1 It represents a 1×1 convolution operation, which is used to adjust the number of channels. F4 is the output image that integrates four scale features.

8. A future frame prediction method for cross-modal communication according to claim 7, characterized in that: The image with the smallest scale is used as the top layer of the pyramid. The images with larger scales are integrated into the pyramid in sequence by stepwise upsampling and lateral connection. At the bottom layer of the pyramid, an image integrating features of all scales is obtained, including: In each fusion step, the small-scale image is first upsampled so that the small-scale image has the same spatial resolution as the larger-scale image at the next level; The upsampled image is added to the larger scale image in the channel dimension by using 1×1 convolution for lateral connection.

9. A future frame prediction method for cross-modal communication according to claim 1, characterized in that: Training the cross-modal video frame prediction model assisted by the tactile data includes: Get the training set containing n frames of video V = {v1, v2, v3, ..., v t ,…,v n }, where v t Represents the video frame at time t, each video frame in V has a corresponding tactile information h; Each i frame in the video V is taken as a set of input I group = {v j ,v j+1 ,v j+2 ,…,v j+i-1 }, where v j ,v j+1 ,v j+2 ,…,v j+i-t As the input history frame, v j+i-t-1 ,…,v j+i-1 As a reference frame for comparison with the predicted frame; Each i frame in the video V is taken as a set of input I group = {v j ,v j+1 ,v j+2 ,…,v j+i-1 }, where v j ,v j+1 ,v j+2 ,…,v j+i-t As the input history frame, v j+i-t-1 ,…,v j+i-1 As a reference frame for comparison with the predicted frame; Assume that the pixel difference between two adjacent frames at the (x, y) position is Then the pixel difference of adjacent cropped images of size c×c is: Set Threshold I threshold ,when All adjacent frames in Should As the input of the prediction network, otherwise it will be discarded; Initialize the parameters of the cross-modal video frame prediction model assisted by tactile data, and iteratively train the cross-modal video frame prediction model assisted by tactile data using the gradient descent algorithm and the Adam optimizer until the maximum number of iterations is reached.

10. A future frame prediction method for cross-modal communication according to claim 9, characterized in that: Initializing the parameters of the cross-modal video frame prediction model assisted by tactile data, and iteratively training the cross-modal video frame prediction model assisted by tactile data using a gradient descent algorithm and an Adam optimizer until a maximum number of iterations is reached, includes: Set the learning rate of the generator and discriminator, set the loss function coefficient, set the network parameter saving frequency and test frequency; The set network parameters are updated based on the gradient descent algorithm. When the number of iterations reaches the specified maximum number, the indicator results and predicted images of the last test are saved.