Visual Motion Prediction Acceleration Method and System Directly Based on Pre-trained Model

Abstract

The present invention relates to the field of artificial intelligence, and discloses a method and system for accelerating visual motion prediction directly based on a pre-trained model, including: denoising an input signal based on UNet, where UNet includes an encoder, an intermediate block, and a decoder; calculating the feature difference of the output features of the encoder between adjacent denoising steps; judging whether the current denoising step is a non-critical step or a critical step according to the feature difference; using the encoder features and intermediate block features of the most recent critical step as the input of the decoder block in subsequent non-critical steps to obtain noise prediction features. A new method without retraining - the fast strategy - is proposed, which can be regarded as a powerful and accelerated alternative to the diffusion strategy for learning visual motion robot control. The comparison results with existing acceleration methods show that the fast strategy has the highest success rate in visual motion inference speed, demonstrating its effectiveness and superiority.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method and system for accelerating visual motion prediction directly based on a pre-trained model. Background Art

[0002] Diffusion models have become the mainstream paradigm of generative AI and achieved significant breakthroughs in various applications, from low-level computer vision to high-level vision-language tasks. In addition, some recent studies have also demonstrated excellent performance of diffusion models in imitation learning for robot control. However, the diffusion strategy often results in slow inference speed due to numerous sampling steps, which limits the application of diffusion models in various generative applications and resource-constrained robot systems.

[0003] To improve the inference speed of diffusion models, various acceleration methods have been proposed in the field of text-to-image generation in recent years. These methods can be roughly divided into two categories: reducing the number of sampling steps and knowledge distillation. The first category of methods reduces the number of sampling steps to achieve acceleration by minimizing the first-order discretization error of ordinary differential equations. For example, implicit denoising diffusion models and design space methods for diffusion-based generative models. Although these methods can reduce the sampling steps to a certain extent, the subsequent problem of decreased prediction performance caused by factors such as modifying the Markov property will lead to poor predicted noise distribution and decreased prediction performance. The second category of methods gradually distills a faster-inference student model from a slower-inference teacher model, and then achieves fast inference based on the trained student model. However, this type of method requires further superimposing the training cost of the student model on the training cost of the pre-trained teacher model, which takes more time and is difficult to generalize to other tasks.

[0004] In the field of robotics, there are currently only a few methods such as the consistency strategy to accelerate the diffusion strategy. However, these methods often cannot balance inference speed and accuracy simultaneously. Summary of the Invention

[0005] The main objective of the present invention is to solve the technical problem in the prior art that inference speed and accuracy cannot be balanced simultaneously. A method for accelerating visual motion prediction directly based on a pre-trained model includes the following steps:

[0006] Based on UNet, denoise the input signal, where the UNet includes an encoder, an intermediate block, and a decoder;

[0007] Calculate the feature difference of the output features of the encoder between adjacent denoising steps;

[0008] Based on the feature difference, determine whether the current denoising step is a non-critical step or a critical step;

[0009] Use the encoder features and intermediate block features of the most recent key step as the input to the decoder block of subsequent non - key steps to obtain noise prediction features;

[0010] Complete visual motion prediction based on the noise prediction features.

[0011] Calculating the feature difference of the output features of the computing encoder between adjacent denoising steps includes:

[0012] Calculate the mean square error of the output features of each module of the one - dimensional convolutional UNet between adjacent denoising steps;

[0013] Normalize the calculated mean square error according to the batch size, dimension, and field of view length of the features to obtain the feature difference. The formula is as follows:

[0014]

[0015] where b is the batch size, d is the dimension, h is the field of view length, i represents the input of the current module, is the feature output by the k - th module at the n - th denoising step, is the feature output by the k - th module at the (n - 1)-th denoising step, represents the feature difference of the k - th module between the n - th denoising step and the (n - 1)-th denoising step.

[0016] Judging whether the current denoising step is a non - key step or a key step according to the feature difference is specifically as follows:

[0017] The first denoising step is a key step, and the judgment method for subsequent denoising steps is as follows:

[0018] Map the decoder output features to the frequency domain through Fourier transform and calculate the spectral energy E. The formula is:

[0019]

[0020] where F(·) represents the component amplitude, and (u, v) represents the position in the spectrogram;

[0021] Construct a scoring function :

[0022]

[0023] where s1, s2 are skip - connection features, b is the output feature of the intermediate module; n represents the n - th denoising step interval, M represents the mean square error of the output features corresponding to the denoising step interval, T represents different tasks, and w is the weight;

[0024] Set the number of key steps;

[0025] Select the step corresponding to the step interval with the highest score as the key step according to the set number of key steps, and the others are non-key steps.

[0026] Dynamically adjust the noise prediction feature f of the non-key step through the scaling factor c d2 and output the corrected frequency domain feature f′ d2 , and the formula is:

[0027] .

[0028] The scaling factor c is calculated based on the ratio of the spectral energy change between the output features of the non-key step and the corresponding key step, and the formula is:

[0029]

[0030] where E nonkey is the spectral energy of the output feature of the non-key step, and E key is the spectral energy of the output feature of the key step.

[0031] The scaling factor c is obtained through the following steps:

[0032] Pre-construct correction parameter dictionaries, denoted as dict{DDIM} and dict{EDM}, which contain the scaling factors for each task.

[0033] The second aspect of the present invention provides a visual motion prediction acceleration system that can be directly based on a pre-trained model, including:

[0034] A denoising processing unit for denoising the input signal based on UNet, and the UNet includes an encoder, an intermediate block, and a decoder;

[0035] A feature difference calculation unit for calculating the feature difference of the output features of the encoder between adjacent denoising steps;

[0036] A judgment unit for judging whether the current denoising step is a non-key step or a key step according to the feature difference, specifically:

[0037] The first denoising step is a key step, and the judgment method for subsequent denoising steps is as follows:

[0038] Map the decoder output feature to the frequency domain through Fourier transform and calculate the spectral energy E, and the formula is:

[0039]

[0040] where F(·) represents the component amplitude, and (u,v) represents the position in the spectrogram;

[0041] Construct a scoring function :

[0042]

[0043] Among them, s1 and s2 are skip connection features, b is the output feature of the intermediate module; n represents the nth denoising step interval, M represents the mean square error of the output feature of the corresponding denoising step interval, T represents different tasks, and w is the weight;

[0044] Set the number of key steps;

[0045] According to the set number of key steps, select the step corresponding to the step interval with the highest score as the key step, and the others as non-key steps;

[0046] The noise prediction feature unit is used to use the encoder feature and the intermediate block feature of the most recent key step as the input of the decoder block of the subsequent non-key steps to obtain the noise prediction feature.

[0047] The third aspect of the present invention provides an electronic device, including: a memory and at least one processor, instructions are stored in the memory, and the memory and the at least one processor are interconnected by a line; the at least one processor calls the instructions in the memory to enable the electronic device to execute the above-mentioned visual motion prediction acceleration method that can be directly based on a pre-trained model.

[0048] The fourth aspect of the present invention provides a computer-readable storage medium, instructions are stored in the computer-readable storage medium, and when it runs on a computer, it enables the computer to execute the above-mentioned visual motion prediction acceleration method that can be directly based on a pre-trained model.

[0049] The present invention has the following beneficial effects:

[0050] The present invention proposes a novel fast inference robot control strategy. Without training a student model, by repeatedly using encoder features, inference acceleration is achieved and performance similar to the original model is maintained.

[0051] The present invention designs a dynamic key step selection strategy based on Fourier spectrum energy, which can adaptively select key steps according to different robot tasks, rather than using a fixed uniform sampling method, to more accurately retain task performance. This mechanism can directly screen key steps according to the characteristics of the noise prediction network itself for different tasks, and does not require additional training of a neural network to select better key steps.

[0052] The present invention designs a Fourier spectrum energy noise correction strategy. By calculating the change ratio of the energy of non-critical steps relative to the energy of critical steps, the noise of non-critical steps is adjusted, thereby effectively avoiding performance degradation and ensuring the accuracy of the generated trajectory. At the same time, a method of constructing a correction parameter dictionary is proposed to avoid the time loss caused by repeatedly calculating correction parameters during the inference process. Description of the Drawings

[0053] Figure 1 It is a comparison result diagram of the diffusion strategy (DP), the consistency strategy (CP), and the fast strategy (FP) of the present invention in the simulation experiment.

[0054] Figure 2 It is a characteristic difference diagram between denoising steps of each module.

[0055] Figure 3 It is a spectrogram after Fourier transform of the second decoder module under the PushT task. Detailed Implementation Manner

[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of the present invention are used to distinguish similar objects and do not have to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that illustrated or described herein. In addition, the terms "comprising" or "having" and any variation thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0057] The object of the present invention is to solve the requirement in the field of robotics that the diffusion strategy inference is too slow to achieve real-time and accurate control of the robot, and provides a visual motion prediction acceleration strategy that can be directly based on the pre-trained model.

[0058] Steps for dividing the visual motion prediction acceleration strategy:

[0059] Conduct feature analysis on the diffusion strategy based on the UNet (U-shaped network) architecture: During the denoising process of a specific task, the encoder features change little and are highly similar, while the decoder features fluctuate greatly in different denoising steps. Multiplex the key encoder features for non-critical steps within a certain time range, thereby avoiding repeatedly calculating these non-critical encoder features with little change to reduce the computational overhead;

[0060] Key step selection: For the requirements of different robot control tasks, the noise prediction network parameters of the pre-trained diffusion policy vary. However, in essence, the frequency domain analysis method in the image field can be adopted. Therefore, the present invention designs a key step selection strategy based on Fourier spectrum energy. The influence degrees of the encoder (input to the decoder through skip connections) and the intermediate module in the UNet on the final noise are quantified according to the spectrum energy as weights, and are adaptively combined with the mean square error of the corresponding module feature changes obtained in the feature analysis to screen more critical denoising steps.

[0061] Predicted noise correction: To compensate for the possible performance degradation and prediction trajectory deviation caused by reusing the encoder features of the key step in the non-critical step, the present invention further designs a noise correction strategy, that is, to correct the noise of the non-critical step according to the change ratio of the non-critical step energy relative to the key step energy, so that the noise distribution obtained by predicting after reusing the features of the previous key step in the non-critical step is close to the noise distribution when the features of the key step are not reused in the non-critical step itself. At the same time, it is found that for the same task under different initial settings, the energy change is relatively consistent, so a correction parameter dictionary is directly constructed to avoid repeated calculation of the correction parameters during the inference process.

[0062] Note: DP (Diffusion Policy), CP (Consistency Policy), DDPM (Denoising Diffusion Probabilistic Model), EDM (Design Space Method for Diffusion-Based Generative Models), DDIM (Implicit Denoising Diffusion Model).

[0063] For ease of understanding, the specific process of the embodiment of the present invention is described below. The first embodiment of the visual motion prediction acceleration method based on the pre-trained model in the embodiment of the present invention includes:

[0064] Based on UNet, denoise the input signal, and the UNet includes an encoder, an intermediate block, and a decoder;

[0065] Calculate the feature difference of the output features of the encoder between adjacent denoising steps;

[0066] According to the feature difference, determine whether the current denoising step is a non-critical step or a key step, specifically:

[0067] The first denoising step is a key step, and the judgment method for the subsequent denoising steps is as follows:

[0068] Map the decoder output features to the frequency domain through Fourier transform, and calculate the spectrum energy E. The formula is:

[0069]

[0070] Where F(·) represents the component amplitude, and (u, v) represents the position in the spectrogram;

[0071] Construct a scoring function :

[0072]

[0073] Among them, s1 and s2 are skip connection features, b is the output feature of the intermediate module; n represents the nth denoising step interval, M represents the mean square error of the output features of the corresponding denoising step interval, T represents different tasks, and w is the weight;

[0074] Set the number of key steps;

[0075] According to the set number of key steps, select the step corresponding to the step interval with the highest score as the key step, and the others as non-key steps.

[0076] Use the encoder features and intermediate block features of the most recent key step as the input to the decoder block of subsequent non-key steps to obtain noise prediction features;

[0077] Complete visual motion prediction based on the noise prediction features.

[0078] Preferably, the calculation of the feature difference of the output features of the encoder between adjacent denoising steps includes:

[0079] Calculate the mean square error of the output features of each module of the one-dimensional convolutional UNet between adjacent denoising steps;

[0080] Normalize the calculated mean square error according to the batch size, dimension, and field of view length of the features to obtain the feature difference. The formula is as follows:

[0081]

[0082] Among them, b is the batch size, d is the dimension, h is the field of view length, and i represents the input of the current module, is the feature output by the kth module at the nth denoising step, is the feature output by the kth module at the (n - 1)th denoising step, represents the feature difference of the kth module between the nth denoising step and the (n - 1)th denoising step.

[0083] Dynamically adjust the noise prediction feature f of the non-key step through the scaling factor c d2 to output the corrected frequency domain feature f' d2 , and the formula is:

[0084] .

[0085] The scaling factor c is calculated based on the ratio of the spectral energy change between the output features of the non-key step and the corresponding key step. The formula is:

[0086] 。

[0087] Among them, E nonkey is the spectral energy of the output feature of the non-critical step, and E key is the spectral energy of the output feature of the critical step.

[0088] The scaling factor c is obtained through the following steps:

[0089] Pre-construct correction parameter dictionaries, denoted as dict{DDIM} and dict{EDM}, which contain the scaling factors for each task.

[0090] The above describes the visual motion prediction acceleration method directly based on pre-trained models in the embodiments of the present invention. Next, the visual motion prediction acceleration device directly based on pre-trained models in the embodiments of the present invention will be described:

[0091] A denoising processing unit for denoising the input signal based on UNet, where the UNet includes an encoder, an intermediate block, and a decoder;

[0092] A feature difference calculation unit for calculating the feature difference of the output features of the encoder between adjacent denoising steps;

[0093] A judgment unit for judging whether the current denoising step is a non-critical step or a critical step according to the feature difference, specifically:

[0094] The first denoising step is a critical step, and the judgment method for subsequent denoising steps is as follows:

[0095] Map the decoder output feature to the frequency domain through Fourier transform, and calculate the spectral energy E. The formula is:

[0096]

[0097] where F(·) represents the component amplitude, and (u, v) represents the position in the spectrogram;

[0098] Construct a scoring function :

[0099]

[0100] where s1, s2 are skip connection features, b is the output feature of the intermediate module; n represents the nth denoising step interval, M represents the mean square error of the output feature of the corresponding denoising step interval, T represents different tasks, and w is the weight;

[0101] Set the number of critical steps;

[0102] According to the set number of key steps, select the step corresponding to the step interval with the highest score as the key step, and the others as non-key steps;

[0103] The noise prediction feature unit is used to take the encoder feature and the intermediate block feature of the most recent key step as the input of the decoder block of subsequent non-key steps to obtain the noise prediction feature.

[0104] The present invention is a visual motion prediction acceleration strategy that can be directly based on a pre-trained model, including the following steps:

[0105] 1. A typical denoising network uses a UNet-based architecture, which consists of an encoder E, an intermediate module M, and a decoder D. The noise prediction network based on a convolutional neural network still retains the core elements of UNet, and the features at different levels extracted by the encoder E are still input into the decoder D through skip connections. In this experiment, the encoder or decoder features in the UNet structure of the diffusion strategy are reused in subsequent denoising steps for noise prediction. The calculation redundancy is reduced by ignoring the calculations of the encoder and the intermediate module of the non-key steps in the noise prediction process to achieve acceleration.

[0106] The present invention first calculates the mean square error of the output features of each module of the one-dimensional convolutional UNet between adjacent denoising steps, and then normalizes the calculated feature differences according to the batch size, dimension, and field of view length of the features to obtain the final feature differences.

[0107]

[0108] b is the batch size, d is the dimension (number of channels), h is the field of view length, i represents the input of the current module, represents the feature difference between the k-th module at the n-th denoising step and the (n - 1)-th denoising step. The feature differences between the denoising steps of each module under DDPM sampling are as shown in (a) of Figure 2 The present invention finds that when using DDPM sampling steps, the feature differences between the encoder and decoder modules of the one-dimensional UNet in the diffusion strategy change less between the later denoising steps. However, for the DDPM sampling strategy, the number of 100 denoising steps will cause the span between key steps to be too large when setting a small number of key steps, resulting in excessive error accumulation, which indirectly leads to setting too many key steps under DDPM sampling, thus requiring a large amount of time to find the best key step setting among numerous key step combinations. As shown in Figure 2As shown in (b) of , the characteristic differences of the encoder module change more smoothly and subtly compared to those of the decoder module. Although DDIM sampling changes the noise characteristics during the denoising process, the characteristic differences between denoising steps are small. At the same time, due to the small number of denoising steps, excessive errors will not accumulate. Similarly, the analysis of EDM sampling also shows similar results, that is, with a small number of sampling steps, the encoder feature changes very slightly between denoising steps. Therefore, the method of reusing encoder features in the present invention can also be applied to DDIM sampling and EDM sampling.

[0109] 2. According to the previous feature analysis results, reusing the features of key steps in non-critical steps is an ideal method. The core idea of the FP strategy (FastPolicy) of the present invention is to set the key denoising steps T and reuse the encoder features in these steps. Since during the denoising process, the encoder features passed through skip connections and the outputs of intermediate blocks change very little, the present invention directly uses the encoder features and intermediate block features of the current key step as the inputs of the corresponding decoder blocks in subsequent non-critical steps. In this way, the computational redundancy of the encoder and intermediate blocks in non-critical steps is eliminated, thus accelerating the inference process.

[0110] 3. Previous studies usually only focused on a single task (such as image generation) or tried to find better key steps by designing neural networks, but these methods often have poor generalization performance or require additional time to train the model. The present invention proposes a dynamic key step selection mechanism based on Fourier spectrum energy for adaptively selecting key steps. Each task of the diffusion strategy requires training a corresponding model. Therefore, the key steps designed for a certain task may not generalize well to other types of tasks. For example, the key steps of the PushT task may not be the best key steps for the Square subtask under Robomimic. The goal of the present invention is to dynamically select key steps based on the inherent characteristics of the noise prediction network for each task. For the one-dimensional UNet in the diffusion strategy, the finally predicted noise is determined by three variables: the first skip connection, the second skip connection, and the output of the intermediate module, denoted as s1, s2, and b respectively. The denoising steps with larger characteristic differences should be prioritized as key steps to reduce error accumulation. However, the final noise is affected by multiple variables, and their impacts on noise prediction are different. UNet was originally designed for image processing, and noise characteristic analysis usually uses Fourier transform to study the amplitude in the frequency domain. Although the diffusion strategy predicts an action sequence, UNet is still used as the noise prediction network. Therefore, the analysis methods used for UNet in the image field are also applicable here. The present invention combines Fourier transform to analyze the denoising process in the frequency domain. The final 1D convolution is only used for scaling. Therefore, the present invention performs frequency domain analysis on the output of the second decoder block (denoted as d2), and the schematic diagram of the transformed spectrum is as Figure 3As shown. The energy calculation formula is: , where F(·) represents the component amplitude, and (u, v) represents the position in the spectrogram, similar to pixel coordinates. The present invention can quantify the contributions of s1, s2, and b to the energy of d2 respectively to evaluate their impacts. For this purpose, the present invention constructs the following scoring function:

[0111]

[0112] where n represents the nth denoising step interval, M corresponds to the mean square error of this step interval, and T represents different tasks (such as PushT). w is the weight, which is calculated according to the contribution ratios of s1, s2, and b to the energy respectively. Then, according to the set number of key steps (set to 3 in this experiment), the step corresponding to the step interval with the highest score is selected as the key step. It should be noted that the first denoising step must be used as the key step.

[0113] For 1D UNet, since the final noise prediction depends on the skip connections s1, s2, and the output of the intermediate module, the scoring function of each denoising step should be composed of these three elements; the impacts of these elements on the final result are visualized through spectral energy; the proportion of spectral energy contribution is normalized and used as the weight. After obtaining the weight, the score is calculated by combining the feature differences corresponding to each module between denoising steps analyzed before, and the one with the highest score is selected as the key step.

[0114] 4. Reusing features may lead to deviations in the predicted noise, thereby affecting the accuracy of the final trajectory prediction. To solve this problem, the present invention calculates a scaling factor c based on the spectral energy change ratio between the d2 output features of the non-key step and the corresponding key step, and the formula is . It should be noted that during the denoising process, the key step always precedes the non-key step. Then, the noise correction formula is , where f d2 and f′ d2 represent the frequency-domain noise features before and after correction respectively. To avoid repeated calculation of the scaling factor during the inference process, the present invention pre-constructs correction parameter dictionaries, denoted as dict{DDIM} and dict{EDM}, which contain the scaling factors for each task. In particular, for the second-order sampler with two noise predictions, since the second noise prediction is based on the result of the first noise prediction, the present invention only compensates the first noise prediction to minimize the additional inference time.

[0115] It should be noted that the method of the present invention aims to reduce the inference time while maintaining a certain task completion rate, and does not require retraining the student model. However, the sampling of the important baseline methods (diffusion strategy and consistency strategy) of the present invention is completely different, namely DDIM and EDM respectively. Therefore, in order to fairly compare the performance, the present invention adopts the same sampling strategy as the baseline method in the strategy of the present invention, which to a certain extent proves the generalization ability of the present invention. Except for the UNet architecture, the present invention maintains the same module configuration (such as image encoder and normalization method) as the diffusion strategy and consistency strategy in all experiments and methods. In addition, the present invention also uses the observed data within two frames (including wrist camera images, third-person perspective camera images, and end effector poses) as input and outputs a continuous string of end effector poses.

[0116] Experimental results: All other metrics were calculated using the same settings on an NVIDIA GPU L40 graphics card.

[0117] Figure 1 shows the results of the diffusion strategy (DP), consistency strategy (CP), and the fast strategy (FP) of the present invention in the simulation experiment, indicating that the method of the present invention significantly speeds up the inference while maintaining the success rate.

[0118] See Figure 2 , which shows the characteristic differences between denoising steps of each module under sampling of the PushT task in the denoising diffusion probabilistic model (DDPM), implicit denoising diffusion model (DDIM), and design space method of diffusion-based generative model (EDM) (the number of discrete steps is bins = 80 and bins = 8 respectively). The zoomed-in figure highlights the regions with less significant feature changes.

[0119] Table 1 presents a comprehensive comparison of the strategy of the present invention and the DDIM-based diffusion model in terms of performance. It can be seen that whether it is a single-arm task or a dual-arm task, the method of the present invention can be comparable to the baseline method (diffusion strategy) in terms of accuracy, and even slightly improve the accuracy in some cases. More importantly, the inference time of the present invention is approximately half of that of the baseline method.

[0120] Explanation of Table 1: Shows the inference time (milliseconds) and success rate (%) of the latest checkpoint and the best checkpoint during the training process. All models were trained for 1000 epochs. w / o NC: without noise correction; w / NC: with noise correction.

[0121] Table 1

[0122]

[0123] Table 2 shows the performance comparison of the strategy of the present invention with the diffusion strategy and the consistency strategy under EDM. It should be noted that all pre-trained diffusion models were only trained for 400 rounds, except that the student model of the consistency strategy was trained for 500 rounds, in order to make a fair comparison under the configuration of the consistency strategy. All experimental results are based on data collected by professionals. The present invention observes that the accuracy performance of the present invention is even better than that of the diffusion strategy, which may be attributed to the better noise distribution in non-critical steps. In fact, the present invention also has an advantage in inference speed compared with the diffusion strategy. Compared with the consistency strategy, the inference speed of the method of the present invention is relatively slow, because the student model of the consistency strategy realizes further inference by distilling the knowledge of the teacher model into itself. However, in more complex tasks, the performance of the consistency strategy is not as good as that of the method of the present invention, which may be related to its training instability. More importantly, compared with the consistency strategy, the advantage of the present invention is that there is no need to retrain the student model, which further improves the deployment ability of the method of the present invention on resource-constrained real robot devices.

[0124] Table 2 Note: Comparison of the fast strategy, consistency strategy and diffusion strategy based on EDM sampling in terms of simulation experiment performance. The present invention shows the inference time (milliseconds) and average success rate (%) of the best checkpoint during training. All models were trained for 400 epochs except the student model of the consistency strategy (CP) which was trained for 500 epochs. All tasks were evaluated on PH data. In the setting of the consistency strategy, when using a second-order solver and setting the critical step to 3, the number of inference function calls of the fast strategy of the present invention is 7.

[0125] Table 2

[0126]

[0127] An embodiment of the present invention also provides an electronic device, which may vary greatly due to configuration or performance differences, and may include one or more processors (central processing units, CPUs) (for example, one or more processors) and a memory, and one or more storage media for storing applications or data (for example, one or more mass storage devices). Among them, the memory and the storage medium may be transient storage or persistent storage. The program stored in the storage medium may include one or more modules, and each module may include a series of instruction operations on the electronic device. Further, the processor may be configured to communicate with the storage medium and execute a series of instruction operations in the storage medium on the electronic device.

[0128] The electronic device may further include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the structure of the electronic device in this embodiment does not constitute a limitation on the electronic device, and it may include more or fewer components, or combine certain components, or have different component arrangements.

[0129] An electronic device structure provided by an embodiment of the present invention may vary greatly due to different configurations or performances. It may include one or more processors (central processing units, CPUs) (for example, one or more processors) and a memory, and one or more storage media for storing applications or data (such as one or more mass storage devices). Among them, the memory and the storage medium may be transient storage or persistent storage. The program stored in the storage medium may include one or more modules, and each module may include a series of instruction operations on the electronic device. Further, the processor may be configured to communicate with the storage medium and execute a series of instruction operations in the storage medium on the electronic device.

[0130] The electronic device may further include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the electronic device structure does not constitute a limitation on the electronic device, and it may include more or fewer components than the foregoing, or combine certain components, or have different component arrangements.

[0131] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions run on the computer, the computer is caused to execute the steps of the foregoing method.

[0132] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described system or device and unit can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0133] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0134] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A visual motion prediction acceleration method directly based on a pre-trained model, characterized in that, It includes the following steps: Based on a one-dimensional UNet, denoise the input signal. The UNet includes an encoder, an intermediate block, and a decoder; Calculate the feature difference of the output features of the encoder between adjacent denoising steps; According to the feature difference, determine whether the current denoising step is a non-critical step or a critical step. Specifically: The first denoising step is a critical step. The determination method for subsequent denoising steps is as follows: Map the decoder output features to the frequency domain through Fourier transform and calculate the spectral energy E. The formula is: where F(·) represents the component amplitude, and (u, v) represents the position in the spectrogram; Construct a scoring function : where s1, s2 are skip connection features, b is the output feature of the intermediate module; n represents the nth denoising step interval, M represents the mean square error of the output features corresponding to the denoising step interval, T represents different tasks, and w is the weight; Set the number of critical steps; According to the set number of critical steps, select the step corresponding to the step interval with the highest score as the critical step, and the others as non-critical steps; Use the encoder features and intermediate block features of the most recent critical step as the input of the decoder block for subsequent non-critical steps to obtain noise prediction features.

2. The visual motion prediction acceleration method directly based on a pre-trained model according to claim 1, wherein The calculation of the feature difference of the output features of the encoder between adjacent denoising steps includes: Calculate the mean square error of the output features of each module of the one-dimensional convolutional UNet between adjacent denoising steps; Normalize the calculated mean square error according to the batch size, dimension, and field of view length of the features to obtain the feature difference. The formula is as follows: Among them, b is the batch size, d is the dimension, h is the field of view length, i represents the input of the current module, is the feature output by the k-th module at the n-th denoising step, is the feature output by the k-th module at the (n-1)-th denoising step, represents the feature difference between the k-th module at the n-th denoising step and the (n-1)-th denoising step.

3. The visual motion prediction acceleration method directly based on a pre-trained model according to claim 1, characterized in that Dynamically adjust the noise prediction feature f of the non-critical step by a scaling factor c d2 to output the corrected frequency domain feature f′ d2 , and the formula is: 。 4. The visual motion prediction acceleration method directly based on a pre-trained model according to claim 3, wherein The scaling factor c is calculated based on the spectral energy change ratio between the output features of the non-critical step and the corresponding critical step. The formula is: Among them, E nonkey is the spectral energy of the output feature of the non-critical step, and E key is the spectral energy of the output feature of the critical step.

5. The visual motion prediction acceleration method directly based on a pre-trained model according to claim 4, wherein The scaling factor c is obtained through the following steps: Pre-construct a correction parameter dictionary, denoted as dict{DDIM} and dict{EDM}, which contains the scaling factor for each task.

6. A visual motion prediction acceleration system that can be directly based on a pre-trained model, characterized in that, The system includes: A denoising processing unit for denoising the input signal based on the UNet. The UNet includes an encoder, an intermediate block, and a decoder; A feature difference calculation unit for calculating the feature difference of the output features of the encoder between adjacent denoising steps; A judgment unit for judging whether the current denoising step is a non-critical step or a critical step according to the feature difference. Specifically: The first denoising step is a critical step. The determination method for subsequent denoising steps is as follows: Map the decoder output features to the frequency domain through Fourier transform and calculate the spectral energy E. The formula is: where F(·) represents the component amplitude, and (u, v) represents the position in the spectrogram; Construct a scoring function : where s1, s2 are skip connection features, b is the output feature of the intermediate module; n represents the nth denoising step interval, M represents the mean square error of the output features corresponding to the denoising step interval, T represents different tasks, and w is the weight; Set the number of critical steps; According to the set number of critical steps, select the step corresponding to the step interval with the highest score as the critical step, and the others as non-critical steps; A noise prediction feature unit for using the encoder features and intermediate block features of the most recent critical step as the input of the decoder block for subsequent non-critical steps to obtain noise prediction features.

7. An electronic device, the electronic device includes a memory and at least one processor, and instructions are stored in the memory; The at least one processor invokes the instructions in the memory to cause the electronic device to execute each step of the visual motion prediction acceleration method that can be directly based on a pre-trained model as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed by the processor, each step of the visual motion prediction acceleration method that can be directly based on a pre-trained model as described in any one of claims 1-5 is implemented.

Images