Motion estimation model training method, motion estimation method and related equipment

Abstract

The present application provides a motion estimation model training method, a motion estimation method and related equipment. The motion estimation model training method obtains a training data set of a mobile game scene, including a first sample image of adjacent frames and its corresponding sample motion information; based on downsampling the first sample image, a second sample image and a third sample image are obtained, and the motion estimation model is trained with the sample motion information. The motion estimation method is applied to an electronic device that has deployed a game application and a motion estimation model. First, images of adjacent frames are obtained from the game application, and then downsampled to obtain the first image and the second image, and these two frames of images are input into the motion estimation model to obtain the target motion information. Using the downsampled image as the input of the model in both the model training stage and the inference stage can speed up the inference speed, reduce power consumption, and realize efficient and real-time motion estimation in the mobile game scene, thereby improving the rendering effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a motion estimation model training method, a motion estimation method, and related equipment. Background Art

[0002] As mobile applications (such as gaming apps) continue to evolve, the rendering processes required by these apps, especially those that display video, are becoming increasingly complex and diverse. For example, these processes include frame interpolation (to increase frame rate), super-resolution (to increase resolution), and anti-aliasing (to eliminate jagged edges) to enhance the user's visual experience when using the app, such as when playing games.

[0003] Motion estimation can be used to obtain the motion information between two adjacent frames. In the rendering process, motion information can be used as prior information in the time domain for processing such as interpolation, super-resolution, and anti-aliasing to achieve better rendering effects.

[0004] Among current motion estimation technologies, non-artificial intelligence (AI) methods have high computational overhead, are not robust, and have limited applicability. AI methods have complex model structures and are difficult to meet the real-time and low-power requirements of gaming scenarios on mobile devices.

[0005] Therefore, how to perform motion estimation of game scenes on mobile devices becomes a key issue. Summary of the Invention

[0006] This application provides a motion estimation model training method, a motion estimation method and related equipment, which can realize efficient and real-time motion estimation in mobile game scenes, improve rendering effects, and thus enhance user experience.

[0007] In a first aspect, a motion estimation model training method is provided, comprising: obtaining a training data set, the training data set comprising M adjacent frame image groups and M sample motion information, each adjacent frame image group comprising two adjacent frames of first sample images, the M sample motion information respectively being motion information of the M adjacent frame image groups, M being a positive integer, the training data set being obtained from a first application on a mobile terminal, the first sample images being images constituting a to-be-displayed screen of the first application; downsampling the two first sample images included in the mth adjacent frame image group in the M adjacent frame image groups respectively to obtain corresponding second sample images and third sample images, 1≤m≤M and m being an integer, the motion information of the mth adjacent frame image group being the mth sample motion information in the M sample motion information; inputting the second sample image and the third sample image into a motion estimation model to obtain motion information; obtaining a loss function value based on the difference between the motion information and the mth sample motion information; and adjusting the parameters to be trained in the motion estimation model based on the loss function value.

[0008] Understandably, downsampling can reduce image size. Using downsampled images to train motion estimation models and subsequently using downsampled images for inference can speed up model inference and reduce power consumption, allowing the model to be deployed on mobile devices and meet the frame rate and power consumption requirements of mobile application scenarios (such as gaming), enabling efficient and real-time motion estimation and improving rendering effects.

[0009] It can also be understood that the trained motion estimation model can obtain motion information based on the input adjacent frame images. Compared to the motion estimation method based on three-dimensional (3D) motion fields, which requires obtaining 3D information and rotation matrices, on the one hand, since motion information can be obtained without obtaining 3D information, it can be applied to scenes where 3D information cannot be obtained, such as transparent objects and shadows. On the other hand, since fewer resources are obtained, the computational overhead is lower. Compared to the motion estimation method based on optical flow, since motion estimation does not rely on search areas and matching, the effect of motion estimation is not affected by the size of the search area or the computational complexity of matching.

[0010] In a possible embodiment, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

[0011] In a possible embodiment, a loss function value is obtained based on the difference between the motion information and the motion information of the mth sample, including: obtaining a loss function value based on the difference between the motion information and the motion information of the mth sample, and an attention weight, the attention weight being the weight of pixels at corresponding positions in the second sample image and the third sample image, and the attention weight being used to indicate the user's attention to the pixels at the corresponding position.

[0012] That is, the parameters of the motion estimation model are adjusted using a loss function determined according to the attention weights.

[0013] As you can understand, by setting the attention weights, the trained motion estimation model can have different training accuracies for different pixels in the input image. Furthermore, by adjusting the attention weights, the training accuracy of the motion estimation model for different pixels can be adjusted. This allows for targeted allocation of computing power during model inference, resulting in a better user experience for the final displayed image, even with the same computing power.

[0014] In a possible embodiment, among the pixels at corresponding positions, pixels located within a target attention area are determined as target pixels, and the attention weight of the target pixel is greater than the attention weights of other pixels, wherein the target attention area includes the central areas of the second sample image and the third sample image.

[0015] Optionally, the target attention area can be further subdivided into different attention gradients. That is, the target attention area is further subdivided into different areas according to the level of attention, and each area corresponds to a different attention weight.

[0016] Optionally, the target focus area may be determined based on a source of the training dataset, ie, a type of game application.

[0017] In one possible embodiment, obtaining a training data set includes: obtaining M adjacent frame image groups and image-related data from a first application; determining a target method for obtaining motion information based on the type of the image-related data; and determining M sample motion information based on the image-related data through the target method; wherein, when the type of the image-related data includes three-dimensional information of the first sample image, the target method includes three-dimensional reprojection; or, when the type of the image-related data does not include three-dimensional information of the first sample image, the target method includes a method for motion estimation based on an optical flow method or a neural network on a personal computer (PC).

[0018] The above solution, by constructing a motion estimation dataset for mobile gaming scenarios, can obtain motion information that is closer to real-world usage scenarios (i.e., gaming scenarios) as a dataset, thereby improving the effectiveness of motion estimation. This classification method can obtain motion information from more gaming scenarios as accurately as possible. To obtain more accurate motion information, as many resources as possible are obtained, providing greater computational overhead, thereby making the training dataset more complete, and also improving the training effect and efficiency of the motion estimation model.

[0019] In a possible embodiment, when the type of image-related data includes three-dimensional information of a first sample image and the target method includes three-dimensional reprojection, the three-dimensional information of the first sample image includes a scene depth value and a coordinate transformation matrix; based on the image-related data, M sample motion information is determined in a target manner, including: based on the scene depth values ​​and coordinate transformation matrices of the two frames of first sample images in each adjacent frame image group, obtaining the sample motion information corresponding to each adjacent frame image group based on three-dimensional reprojection.

[0020] In a possible embodiment, when the type of image-related data includes three-dimensional information of the first sample image and the target method includes three-dimensional reprojection, the three-dimensional information of the first sample image includes a scene depth value and a coordinate transformation matrix; based on the image-related data, M motion information is determined in a target manner, including: based on the scene depth values ​​and coordinate transformation matrices of the two frames of the first sample images in each adjacent frame image group, the motion information corresponding to each adjacent frame image group is obtained based on the three-dimensional reprojection.

[0021] In a possible embodiment, the second sample image and the third sample image are input into a motion estimation model to obtain motion information, including: processing the second sample image based on the first input layer to obtain first sample image data, and processing the third sample image based on the second input layer to obtain second sample image data; performing feature extraction on the first sample image data based on the first hidden layer to obtain a first feature map, and performing feature extraction on the second sample image data based on the second hidden layer to obtain a second feature map; processing the first feature map and the second feature map based on an association module to obtain motion information, wherein the association module is used to calculate the correlation between the first feature map and the second feature map.

[0022] In a possible embodiment, the first feature map and the second feature map are processed based on the association module to obtain motion information, including: sampling the first feature map based on the association module to obtain a first sampling feature; sampling the second feature map based on the association module to obtain a second sampling feature; calculating the correlation degree based on the first sampling feature and the second sampling feature based on the association module to obtain a correlation degree feature map of the first feature map and the second feature map; and decoding the correlation degree feature map to obtain motion information.

[0023] In a possible embodiment, sampling the first feature map based on the association module to obtain a first sampling feature includes: determining the first target feature based on the association module; sampling with the first target feature as the center block and a sampling range of K×K blocks to obtain the first sampling feature; sampling the second feature map based on the association module to obtain a second sampling feature, including: determining the second target feature based on the association module; sampling with the second target feature as the center block and a sampling range of K×K blocks to obtain the second sampling feature; wherein K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

[0024] The above scheme has been experimentally proven to use a narrower sampling range for sampling on the mobile terminal, which has almost no impact on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0025] Understandably, a wider sampling area can capture more information, allowing the correlation module to extract sufficient information and abstract motion representations based on the correlations between features, thereby obtaining motion information. However, this also leads to a more complex network model and increased computational effort, placing considerable pressure on real-time mobile gaming scenarios. This makes real-time inference difficult to achieve on mobile devices and results in excessive power consumption.

[0026] In a possible embodiment, the number of first hidden layers and second hidden layers is L, and the L first hidden layers correspond to the L second hidden layers in one-to-one correspondence, and the number of association modules is P; the first feature map and the second feature map are processed based on the association modules to obtain motion information, including: based on the P association modules, respectively processing the P first sub-feature maps output by part or all of the first hidden layers in the L first hidden layers and the P second sub-feature maps output by the corresponding second hidden layers to obtain motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

[0027] In a possible embodiment, the first association module among the P association modules includes a first association layer and a first decoder, the i-th association module among the P association modules includes an i-th association layer and an i-th decoder, and the P-th association module includes a P-th association layer and a P-th decoder. Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the P second sub-feature maps output by the corresponding second hidden layer are calculated to obtain motion information, including: the first association layer calculates the association degree of the first first sub-feature map in the P first sub-feature maps and the first second sub-feature map in the P second sub-feature maps to obtain a first association feature map, and inputs the first association feature map into the first decoder; the first decoder calculates the first association degree. The feature map is decoded to obtain the first intermediate feature, and the first intermediate feature is input into the second decoder; the i-th correlation layer calculates the correlation between the i-th first sub-feature map and the i-th second sub-feature map to obtain the i-th correlation feature map, and the i-th correlation feature map is input into the i-th decoder; the i-th decoder decodes the i-th correlation feature map to obtain the i-th intermediate feature, and inputs the i-th intermediate feature into the i+1-th decoder, 2≤i≤P-1 and i is an integer; the P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain the P-th correlation feature map, and inputs the P-th correlation feature map into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain motion information.

[0028] In a possible embodiment, when P=3, based on P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layers, and the P second sub-feature maps output by the corresponding second hidden layers are processed respectively to obtain motion information, including: inputting the cth first sub-feature map output by the cth first hidden layer and the cth second sub-feature map output by the cth second hidden layer into the first association module to obtain the first intermediate feature; inputting the bth first sub-feature map output by the bth first hidden layer and the bth second sub-feature map output by the bth second hidden layer, and the first intermediate feature into the second association module to obtain the second intermediate feature; inputting the ath first sub-feature map output by the ath first hidden layer and the ath second sub-feature map output by the ath second hidden layer, and the second intermediate feature into the third association module to obtain motion information, 1≤a<b≤L-1, c=L, and a, b, and c are all integers.

[0029] Experiments have shown that the above scheme can achieve a better compromise between the accuracy and power consumption of the motion estimation model by calculating the correlation based on the feature maps output by the shallower, middle and deepest hidden layers.

[0030] In a possible embodiment, L<10, the L-layer first hidden layer includes L-1 layers of first convolutional layers and 1 layer of first pooling layer, and the L-layer second hidden layer includes L-1 layers of second convolutional layers and 1 layer of second pooling layer; wherein, when L=6, the 6-layer first hidden layer includes 5 layers of first convolutional layers and 1 layer of first pooling layer, and the 6-layer second hidden layer includes 5 layers of second convolutional layers and 1 layer of second pooling layer, the a-th layer first hidden layer is the 2-th layer first convolutional layer, the b-th layer first hidden layer is the 5-th layer first convolutional layer, the c-th layer first hidden layer is the first pooling layer, the a-th layer second hidden layer is the 2-th layer second convolutional layer, the b-th layer second hidden layer is the 5-th layer second convolutional layer, and the c-th layer second hidden layer is the second pooling layer.

[0031] In this application, the network structure of the motion estimation model is shallower than that of the PC-side neural network, so that it can be deployed and operate normally on the mobile side. Experiments have shown that on the mobile side, reducing the number of hidden layers and making the network structure shallower can reduce computational overhead while maintaining the accuracy of the motion estimation model.

[0032] It is understandable that the number of hidden layers in the current PC-based neural network structure is generally greater than 10. By adopting a deeper network structure, it is possible to extract effective high-dimensional features, thereby obtaining sufficient information to calculate the relationship between the two images and improve the accuracy of the motion estimation model.

[0033] In a second aspect, a motion estimation method based on a motion estimation model is provided, which is applied to an electronic device, wherein the electronic device is deployed with a first application, and the motion estimation model is deployed on the electronic device, and the motion estimation model is trained by a motion estimation model training method such as any one of the first aspect and its difficult embodiments, and the method includes: obtaining two adjacent frames of images from the first application, and downsampling the two adjacent frames of images respectively to obtain a first image and a second image, wherein the two adjacent frames of images are images of a to-be-displayed screen constituting the first application; inputting the first image and the second image into the motion estimation model to obtain target motion information of the two adjacent frames of images, including: processing the first image based on the first input layer to obtain first image data, and processing the second image based on the second input layer to obtain second image data; extracting features from the first image data based on the first hidden layer to obtain a first feature map, and extracting features from the second image data based on the second hidden layer to obtain a second feature map; processing the first feature map and the second feature map based on the association module to obtain target motion information, wherein the association module is used to calculate the correlation between the first feature map and the second feature map.

[0034] The above solution first downsamples the images obtained from the gaming application before inputting them into the motion estimation model. This can speed up inference and reduce power consumption so that the model can be deployed on mobile devices. It can also meet the frame rate and power consumption requirements of mobile gaming scenarios, achieve efficient and real-time motion estimation, and improve rendering effects.

[0035] Furthermore, by setting attention weights during training, the trained motion estimation model can achieve different training accuracies for different pixels in the input image. Furthermore, by adjusting the attention weights, the training accuracy of the motion estimation model for different pixels can be adjusted. This allows for targeted allocation of computing power during model inference, resulting in a better user experience for the final displayed image, even with the same computing power.

[0036] In a possible embodiment, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

[0037] In a possible embodiment, the first feature map and the second feature map are processed based on the association module to obtain target motion information, including: sampling the first feature map based on the association module to obtain a first sampling feature; sampling the second feature map based on the association module to obtain a second sampling feature; calculating the first sampling feature and the second sampling feature to obtain a correlation feature map of the first feature map and the second feature map; and decoding the correlation feature map to obtain target motion information.

[0038] In a possible embodiment, sampling the first feature map based on the association module to obtain a first sampling feature includes: determining a first target feature; sampling with the first target feature as the center block and a sampling range of K×K blocks to obtain the first sampling feature; sampling the second feature map based on the association module to obtain a second sampling feature includes: determining a second target feature; sampling with the second target feature as the center block and a sampling range of K×K blocks to obtain the second sampling feature; wherein K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

[0039] The above scheme has been experimentally proven to use a narrower sampling range for sampling on the mobile terminal, which has almost no impact on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0040] Understandably, a wider sampling area can capture more information, allowing the correlation module to extract sufficient information and abstract motion representations based on the correlations between features, thereby obtaining motion information. However, this also leads to a more complex network model and increased computational effort, placing considerable pressure on real-time mobile gaming scenarios. This makes real-time inference difficult to achieve on mobile devices and results in excessive power consumption.

[0041] In a possible embodiment, the number of first hidden layers and second hidden layers is L, and the L first hidden layers correspond to the L second hidden layers in one-to-one correspondence. The number of association modules is P. The first feature map and the second feature map are processed based on the association modules to obtain target motion information, including: based on the P association modules, respectively processing the P first sub-feature maps output by part or all of the first hidden layers in the L first hidden layers and the P second sub-feature maps output by the corresponding second hidden layers to obtain target motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

[0042] In a possible embodiment, the first association module among the P association modules includes the first association layer and the first decoder, the i-th association module among the P association modules includes the i-th association layer and the i-th decoder, and the P-th association module includes the P-th association layer and the P-th decoder. Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the P second sub-feature maps output by the corresponding second hidden layer are calculated to obtain target motion information, including: the first association layer calculates the association degree of the first first sub-feature map in the P first sub-feature maps and the first second sub-feature map in the P second sub-feature maps to obtain the first association feature map, and inputs the first association feature map into the first decoder; the first decoder calculates the first association degree. The feature map is decoded to obtain the first intermediate feature, and the first intermediate feature is input into the second decoder; the i-th correlation layer calculates the correlation between the i-th first sub-feature map and the i-th second sub-feature map to obtain the i-th correlation feature map, and the i-th correlation feature map is input into the i-th decoder; the i-th decoder decodes the i-th correlation feature map to obtain the i-th intermediate feature, and the i-th intermediate feature is input into the i+1-th decoder, 2≤i≤P-1 and i is an integer; the P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain the P-th correlation feature map, and the P-th correlation feature map is input into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain the target motion information.

[0043] In a possible embodiment, when P=3, based on P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the P second sub-feature maps output by the corresponding second hidden layer are processed respectively to obtain target motion information, including: inputting the c-th first sub-feature map output by the c-th first hidden layer and the c-th second sub-feature map output by the c-th second hidden layer into the first association module to obtain the first intermediate feature; inputting the b-th first sub-feature map output by the b-th first hidden layer and the b-th second sub-feature map output by the b-th second hidden layer, and the first intermediate feature into the second association module to obtain the second intermediate feature; inputting the a-th first sub-feature map output by the a-th first hidden layer and the a-th second sub-feature map output by the a-th second hidden layer, and the second intermediate feature into the third association module to obtain target motion information, 1≤a<b≤L-1, c=L, and a, b, and c are all integers.

[0044] Experiments have shown that the above scheme can achieve a better compromise between the accuracy and power consumption of the motion estimation model by calculating the correlation based on the feature maps output by the shallower, middle and deepest hidden layers.

[0045] In a possible embodiment, L<10, the L-layer first hidden layer includes L-1 layers of first convolutional layers and 1 layer of first pooling layer, and the L-layer second hidden layer includes L-1 layers of second convolutional layers and 1 layer of second pooling layer; wherein, when L=6, the 6-layer first hidden layer includes 5 layers of first convolutional layers and 1 layer of first pooling layer, and the 6-layer second hidden layer includes 5 layers of second convolutional layers and 1 layer of second pooling layer, the a-th layer first hidden layer is the 2-th layer first convolutional layer, the b-th layer first hidden layer is the 5-th layer first convolutional layer, the c-th layer first hidden layer is the first pooling layer, the a-th layer second hidden layer is the 2-th layer second convolutional layer, the b-th layer second hidden layer is the 5-th layer second convolutional layer, and the c-th layer second hidden layer is the second pooling layer.

[0046] It is understandable that the number of hidden layers in the current PC-based neural network structure is generally greater than 10. By adopting a deeper network structure, it is possible to extract effective high-dimensional features, thereby obtaining sufficient information to calculate the relationship between the two images and improve the accuracy of the motion estimation model.

[0047] In this application, the network structure of the motion estimation model is shallower than that of the PC-side neural network, so that it can be deployed and operate normally on the mobile side. Experiments have shown that on the mobile side, reducing the number of hidden layers and making the network structure shallower can reduce computational overhead while maintaining the accuracy of the motion estimation model.

[0048] In a possible embodiment, the first image is a non-predicted image that has been displayed, the second image is a non-predicted image that has not been displayed, and the first image is the previous non-predicted image of the second image in the display timing. The method also includes: performing pixel shifting and padding on the second image and the first image according to target motion information to obtain one or more predicted images; and displaying the second image after displaying the one or more predicted images.

[0049] The above solution applies the motion estimation model in the interpolation scenario, which improves the accuracy of obtaining target motion information and reduces the computational overhead of obtaining target motion information, thereby improving the accuracy of the interpolation process and reducing the computational overhead of the interpolation process.

[0050] In a third aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code comprises computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0052] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0053] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0054] It is understandable that the electronic device provided in the third aspect, the chip system provided in the fourth aspect, the computer storage medium provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to perform the methods provided in this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram illustrating possible problems with motion estimation using a non-artificial intelligence approach according to an embodiment of the present application;

[0056] Figure 2 FIG2 shows a hardware structure diagram of an electronic device 100 provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram illustrating the interaction of data flows between a central processing unit (CPU), a graphics processing unit (GPU), and a neural-network processing unit (NPU) provided in an embodiment of the present application;

[0058] Figure 4 A flowchart of a motion estimation method provided in an embodiment of the present application;

[0059] Figure 5 A schematic flow chart of a motion estimation model training method 200 provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of an example of several possible target focus areas including the central area of ​​the second sample image provided in an embodiment of the present application;

[0061] Figure 7 This is a schematic diagram of an example of the sampling area of ​​the current neural network on the PC;

[0062] Figure 8 A schematic diagram of an example of a sampling area of ​​a motion estimation model provided in an embodiment of the present application;

[0063] Figure 9 A schematic diagram of an example of a training flow chart for a motion estimation model provided in an embodiment of the present application;

[0064] Figure 10 A schematic diagram of an example of a motion estimation method based on a motion estimation model provided in an embodiment of the present application;

[0065] Figure 11 A schematic diagram of an example of an inference flow chart of a motion estimation model provided in an embodiment of the present application;

[0066] Figure 12 This is a software structure block diagram of the electronic device 100 according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0068] As mobile games continue to evolve, the rendering processes of mobile game engines are becoming increasingly complex and diverse. For example, the rendering process includes processing such as frame interpolation (to increase frame rate), super-resolution (to increase resolution), and anti-aliasing (to eliminate jagged edges) to enhance the user's visual experience while playing games.

[0069] In gaming scenarios, motion estimation can be used to obtain motion information between two adjacent frames. This motion information can be used as prior information in the temporal domain for processing such as interpolation, super-resolution, and anti-aliasing, resulting in better rendering results.

[0070] Motion estimation is the process of extracting the key feature points or contours of an object by processing continuous physical image frames, and inferring the motion state of the object based on the position changes of these feature points.

[0071] Current motion estimation technologies can include: two-dimensional (2D) algorithms such as optical flow, motion estimation based on three-dimensional (3D) motion fields, and motion estimation based on deep learning on the PC side.

[0072] First, motion estimation based on a 3D motion field requires obtaining 3D information and a rotation matrix, then using 3D reprojection to match adjacent frames to calculate inter-frame motion information. This method can achieve relatively good results, but it requires a lot of resources and is expensive, and is not suitable for scenes lacking 3D information, such as transparent objects and shadows.

[0073] Figure 1 A schematic diagram of possible problems in performing motion estimation using a non-artificial intelligence method according to an embodiment of the present application. Figure 1 (a) and (b) correspond to the motion estimation method based on 3D motion field. Figure 1 (a) is the image #1 displayed on the display screen of the electronic device in the game scene. Figure 1 (b) is a visualization image of the motion information of image #1 (actually a color image, the corresponding grayscale image is used in this application), and the ellipse marks the Figure 1 Pixels at the same position in (a) and (b). In the visualization of motion information, the depth of the pixel color represents the magnitude of the pixel's motion speed; the color of the pixel represents the direction of motion, with red and green corresponding to the two directions respectively.

[0074] like Figure 1 As shown in (a) in the figure, image #1 is a frame image of a person running, and the shadow of the person on the ground can be seen in the ellipse.

[0075] like Figure 1As shown in (b), the character's body is different from the background color, indicating that the character is moving. Figure 1 In (b), the ellipse does not reflect the motion information of the shadow. Therefore, this motion estimation method will fail to estimate the motion information of the shadow.

[0076] Second, 2D algorithms like optical flow use 2D image information to calculate motion based on pixel-to-pixel matching. Because they only require 2D image information, these methods are suitable for most scenarios. However, their effectiveness depends on the size of the search area and the computational complexity of the matching, so achieving good results often requires considerable computational overhead.

[0077] For example, optical flow is a method that analyzes the brightness changes of pixels in an image to infer the direction and speed of an object's movement. Optical flow is a two-dimensional vector field that represents the displacement of a point from one frame to the next.

[0078] Figure 1 (c) and (d) correspond to the motion estimation method based on the optical flow method. Figure 1 (c) in the figure is image #2 displayed on the display screen of the electronic device in the game scene. Figure 1 (d) is a visualization image of the motion information of image #2 (actually a color image, the corresponding grayscale image is used in this application), and the ellipse marks the Figure 1 Pixels at the same position in (c) and (d).

[0079] like Figure 1 As shown in (c) in the figure, image #2 is a frame of the character running, and the oval is the background (grass). The character is running, while the background is not moving.

[0080] like Figure 1 As shown in (d), within the ellipse, pixels of varying colors and shades appear on the grassy background. Understandably, since the grassy background, aside from the person, is not moving, it should appear pure in the motion visualization, with no visible motion. However, because the optical flow method relies solely on 2D information, errors can be significant, resulting in significant motion artifacts against the static background.

[0081] Third, PC-based motion estimation, based on deep learning, is a data-driven approach that can also utilize only 2D images, achieving good motion estimation results with minimal computational effort. However, current PC-based neural network models are relatively complex, making them difficult to run directly on mobile devices, preventing real-time inference. Even if they can operate normally, the data transmission overhead and resulting power consumption are substantial.

[0082] In summary, among current motion estimation technologies, non-artificial intelligence (AI) methods have high computational overhead, are not robust, and have limited applicability (for example, motion information in complex scenes such as translucency, shadows, and special effects is difficult to obtain); while AI methods have complex model structures and cannot meet the real-time and low-power requirements of gaming scenarios on mobile devices.

[0083] Therefore, how to perform motion estimation of game scenes on mobile devices becomes a key issue.

[0084] In view of this, the present application provides a motion estimation model training method to obtain a training data set of a mobile game scene, which includes a first sample image of adjacent frames and its corresponding sample motion information, and trains a motion estimation model based on downsampling the first sample image to obtain a second sample image and a third sample image, as well as the sample motion information.

[0085] Understandably, downsampling can reduce image size. Using downsampled images to train motion estimation models and subsequently using downsampled images for inference can speed up model inference and reduce power consumption, enabling model deployment on mobile devices and meeting the frame rate and power consumption requirements of mobile gaming scenarios. This allows for efficient and real-time motion estimation and improves rendering quality.

[0086] In some embodiments, parameters of the motion estimation model are adjusted using a loss function determined according to attention weights, where the attention weights are used to indicate the user's attention to pixels in the second sample image and the third sample image.

[0087] As you can understand, by setting the attention weights, the trained motion estimation model can have different training accuracies for different pixels in the input image. Furthermore, by adjusting the attention weights, the training accuracy of the motion estimation model for different pixels can be adjusted. This allows for targeted allocation of computing power during model inference, resulting in a better user experience for the final displayed image, even with the same computing power.

[0088] In some embodiments, the sample motion information in the training dataset is obtained based on game scene classification. In one case, when the game application provides 3D information about the first sample image, the sample motion information can be obtained based on a 3D motion field. In another case, when the game application does not provide 3D information about the first sample image, the sample motion information can be obtained based on optical flow or a neural network on the PC.

[0089] Based on the analysis of current motion estimation algorithms above, it's clear that this classification approach can accurately capture more motion information from game scenes. To obtain more precise motion information, we maximize the resources and computational overhead required to improve the training dataset, resulting in better training results and greater efficiency for the motion estimation model.

[0090] It can also be understood that the trained motion estimation model can obtain motion information based on the input adjacent frame images. Compared to the motion estimation method based on 3D motion field, which requires obtaining 3D information and rotation matrix, on the one hand, since motion information can be obtained without obtaining 3D information, it can be applied to scenes where 3D information cannot be obtained, such as transparent objects and shadows; on the other hand, since fewer resources are obtained, the computational overhead is lower. Compared to the motion estimation method based on optical flow, since motion estimation does not rely on search area and matching, the effect of motion estimation is not affected by the size of the search area and the computational complexity of matching.

[0091] This application also provides a motion estimation method based on a motion estimation model, which is applied to an electronic device that deploys a gaming application and a motion estimation model. The electronic device first obtains images of adjacent frames from the gaming application, then downsamples them to obtain a first image and a second image. The first and second images are then input into the motion estimation model to obtain target motion information.

[0092] Therefore, the beneficial effects of the motion estimation model during inference can be found in the corresponding description of the training process above.

[0093] In some embodiments, the motion estimation model's network structure is shallower than the PC-based neural network structure to facilitate deployment and proper operation on mobile devices. Experiments have shown that a shallower network structure can reduce computational overhead while maintaining the accuracy of the motion estimation model.

[0094] In some embodiments, the motion estimation model includes a correlation module that first samples the first image and the second image and calculates a correlation feature map between the first image and the second image. The sampling range of the correlation module is narrower than that of current neural network sampling on PCs. Experiments have shown that using a narrower sampling range on mobile devices has little impact on the accuracy of the motion estimation model, while significantly reducing its complexity.

[0095] In some embodiments, the motion estimation model removes the feature remapping (warping) operation compared to the PC-side neural network. Experiments have shown that removing this operation has little impact on the accuracy of the motion estimation model, but significantly reduces its complexity.

[0096] With respect to the above three embodiments, it can be seen that the motion estimation model has a streamlined structure while ensuring accuracy, thereby enabling real-time reasoning in mobile gaming scenarios and efficiently obtaining motion information.

[0097] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are first introduced.

[0098] 1. Motion information, or it can be called motion vector or motion vector information (this application does not limit the name): the instantaneous velocity vector of the pixel motion of a spatial moving object on the observation imaging plane; or, the distance and direction (including x / y directions) that a spatial moving object moves per unit time of pixels on the observation imaging plane; or, the motion relationship between each pixel on adjacent image frames, including information such as position, motion speed, and motion direction; the motion vector between two frames of images is the displacement of a dynamic object obtained by motion estimation of the two frames of images, and both frames of images include the dynamic object.

[0099] 2. Downsampling (also known as downsampling or image reduction): In image processing, this refers to reducing the resolution of an image, usually by discarding some pixels. This can reduce file size for easier storage and transmission while also maintaining image quality to a certain extent.

[0100] 3. Frame interpolation technology: also known as frame rate conversion technology, is a technology that improves the smoothness of the video by adding one or more frames to every two frames of the original video, shortening the display time between each frame, and correcting the illusion formed by the persistence of vision of the human eye.

[0101] 4. Super-resolution: Super-resolution reconstruction involves increasing the resolution of an original image through hardware or software. This process involves obtaining a high-resolution image or video from a series of low-resolution images or videos. Super-resolution rendering is used to increase the resolution of image data. This resolution increase can be from low resolution to high resolution or ultra-high resolution, or from high resolution to ultra-high resolution.

[0102] 5. Anti-aliasing (AA): also known as anti-aliasing or anti-aliasing, is a technology that eliminates jagged edges of image objects in the output image.

[0103] 6. High-dimensional features: This generally refers to situations in machine learning and data analysis where a dataset contains a large number of features. These features may be derived from raw data through transformations or processing. For example, in deep learning, input data is often converted into high-dimensional feature vectors. The presence of high-dimensional features allows the model to capture more information and details, thereby improving its predictive performance.

[0104] However, high-dimensional features also bring some problems, such as increased computational complexity, possible redundant features, and increased data sparsity.

[0105] 7. A hardware buffer is a physical memory area used for data transfer between different components or peripherals. It allows the sender to store data in the buffer until the receiver is ready and read it at the appropriate rate. By using a hardware buffer, speed mismatches between the sender and receiver can be resolved, thereby improving system reliability and performance.

[0106] 8. In computer hardware, shared memory refers to a block of random access memory (RAM) that can be accessed by multiple CPUs in a multiprocessor computer system. In computer software, shared memory refers to a method of interprocess communication, or a way for simultaneously running programs to exchange data. One process creates an area in RAM that other processes can access.

[0107] 9. Open Graphics Library for Embedded Systems (OpenGL ES) is an application programming interface (API) for embedded systems. In the Android system architecture, developers can use OpenGL ES to create and manipulate two-dimensional and three-dimensional graphics.

[0108] Please refer to Figure 2 , Figure 2 FIG1 shows a hardware structure diagram of an electronic device 100 provided in an embodiment of the present application. Figure 2 The electronic device 100 may include a processor 110 , and optionally a memory 120 and a display screen 130 .

[0109] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0110] The processor 110 may include one or more processing units. For example, the processor 110 may include a CPU 111 , a GPU 112 , and an NPU 113 , and may also include an application processor.

[0111] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0112] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 130. The MIPI interface includes, for example, a display serial interface (DSI). In some embodiments, the processor 110 and the display 130 communicate via the DSI interface to implement the display function of the electronic device 100.

[0113] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0114] In the motion estimation method provided in this application, CPU 111 applies for shared memory so that data can be transmitted between GPU 112 and NPU 113. Specifically, the data flow interaction between CPU 111, GPU 112 and NPU 113 can be seen in Figure 3 .

[0115] Electronic device 100 implements display functionality through a GPU 112, a display screen 130, and an application processor. GPU 112 is a microprocessor for image processing that connects display screen 130 and the application processor. GPU 112 is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs 112, which execute program instructions to generate or modify display information.

[0116] In the motion estimation method provided in this application, the GPU 112 provides an image for motion estimation, including obtaining an image from a game application and rendering it at a normal resolution, and downsampling the rendered image at a normal resolution to obtain a downsampled image. Specifically, the data flow interaction between the CPU 111, the GPU 112, and the NPU 113 can be seen in Figure 3 .

[0117] The display screen 130 is used to display images, videos, and the like. The display screen 130 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device 100 may include one or N display screens 130, where N is an integer greater than one.

[0118] NPU 113 is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. NPU 113 enables intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0119] In the motion estimation method provided in this application, the NPU 113 calls the motion estimation model to obtain motion information. Specifically, the data flow interaction between the CPU 111, GPU 112 and NPU 113 can be seen in Figure 3 .

[0120] The internal memory, i.e., the memory 120, can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0121] Figure 3 Schematic diagram of the data flow interaction between the CPU, GPU and NPU provided in the embodiment of the present application.

[0122] It can be understood that in this application, the electronic device running the motion estimation model provided in this application requires data flow interaction between the CPU, GPU and NPU.

[0123] Perform steps A to E in advance.

[0124] In step A, the CPU creates a hardware buffer and requests shared memory from the hardware buffer. Alternatively, the CPU creates hardware buffer shared memory. The CPU can request shared memory from the hardware buffer in random access memory (RAM). One way to request shared memory is for the CPU to request storage space for a GPU shader storage buffer object (SSBO) from the hardware buffer. After the fragment shader or compute shader in the GPU receives the downsampled image frame, it writes the downsampled image frame to the GPU SSBO. Motion estimation results output by the NPU can also be written to shared memory.

[0125] Exemplarily, after step A, the CPU notifies the GPU of the pointer address of the shared memory so that the shared memory can be used as storage memory for the downsampled image frame. The CPU also notifies the NPU of the pointer address of the shared memory so that the shared memory can be used as storage memory for the motion estimation result.

[0126] Hardware Buffer is a type of memory. When the CPU, GPU, and NPU access the Hardware Buffer, there are certain requirements for the format of the Hardware Buffer. For example, when applying for shared memory as storage space accessible to the Fragment Shader or Compute Shader in the GPU, you can specify the Hardware Buffer format as AHARDWAREBUFFER_FORMAT_BLOB.

[0127] Another way to request shared memory is for the CPU to request two blocks of shared memory from the hardware buffer. One block is used to store the downsampled image frames obtained by the GPU, which serves as input data for the NPU. The other block is used to store NPU output data, such as motion estimation results, which serves as input data for the GPU during UI rendering. The two blocks of shared memory requested by the CPU must have the same format, for example, AHARDWAREBUFFER_FORMAT_BLOB.

[0128] Step B, create an EGLImage.

[0129] EGLImage represents a shared resource type created by EGL client APIs (such as OpenGLQ and OpenVG). Its original purpose was to share 2D image data, but it did not explicitly define the format or purpose of the shared data. Therefore, in theory, applications and related client APIs can create any type of shared data for any purpose.

[0130] Exemplarily, step B is performed by the GPU. After the GPU creates the EGLImage, the GPU can share data with the CPU and the NPU through data transmission between the EGLImage and the shared memory.

[0131] Step C, GPU binds EGLImage and input texture / output texture (inputTexture / outputTexture).

[0132] Among them, inputTexture corresponds to the downsampled image frame, and outputTexture corresponds to the result of motion estimation.

[0133] Thus, the GPU can share inputTexture / outputTexture based on EGLImage.

[0134] In step D, the GPU and inputTexture / outputTexture are bound based on the glBlitFramebuffer function.

[0135] The glBlitFramebuffer function is a function provided by OpenGL that copies a certain area of ​​a frame buffer (Framebuffer) to another Framebuffer.

[0136] Step E: Bind the GPU to the inputTexture / outputTexture using the glBindTexture function.

[0137] glBindTexture is the function used to bind a texture in OpenGL.

[0138] Then, steps 1-N and 2-(N-1) are executed in a loop. Step 1-1 is to input the first frame of image into the motion estimation model, and there is no corresponding step 2-(N-1). After executing steps 1-1 and 1-2, step 2-1 is to output the motion information of the second frame of image and the first frame of image.

[0139] Step 1-N: Input the Nth frame image to the motion estimation model. Figure 3 The current frame image processed by the motion estimation model is the Nth frame, and the previous frame image is the N-1th frame.

[0140] The CPU instructs the NPU to obtain the Nth frame image from the shared memory.

[0141] Subsequently, the motion estimation model performs motion estimation on the Nth frame image and the N-1th frame image.

[0142] Step 2-(N-1): output motion information of the Nth frame image and the N-1th frame image.

[0143] Specifically, steps 1-N and 2-(N-1) can be found in Figure 4 Description related to steps (7) and (8) in .

[0144] Figure 4 This is a flowchart of a motion estimation method provided in an embodiment of the present application.

[0145] like Figure 4 As shown in the figure, the motion estimation method mainly involves four parts: offline training, OPENGLES thread, shared memory and AI computing thread.

[0146] Among them, the OPENGLES thread, shared memory and AI computing thread are implemented on a mobile electronic device that executes the motion estimation method (for the convenience of explanation, referred to as the first electronic device below), and offline training is implemented on another electronic device (for the convenience of explanation, referred to as the second electronic device below).

[0147] The general process is to first use the motion estimation model training method provided in this application on the second electronic device to train the motion estimation model provided in this application, and then deploy the motion estimation model on the first electronic device. For the first electronic device, the OPENGLES thread (executed on the GPU) first transfers the image to be motion estimated to the shared memory, and the AI ​​computing thread (executed on the NPU) performs motion estimation on the image to be motion estimated using the motion estimation model, and transfers the motion estimation result to the shared memory. The OPENGLES thread obtains the motion estimation result from the shared memory, performs subsequent processing (such as rendering) on ​​the result, and then transfers it to the display screen for display.

[0148] For the offline training part: the second electronic device performs model training based on the training data set obtained from the game APP of the mobile terminal (which may include the first electronic device) to obtain a motion estimation model, and then deploys the model file of the motion estimation model to the first electronic device. Specifically, the training data set of the motion estimation model and the model training process will be combined below. Figure 5 Provide a detailed introduction.

[0149] For the First Electronic Device:

[0150] In the OPENGLES thread part, (1) the game application generates an image rendering event, the GPU obtains the game frame (the image frame of the game scene) and triggers the initialization of the AI ​​model (i.e., the motion estimation model).

[0151] For example, the game frames may be a video stream.

[0152] (2) Initialize the AI ​​model (i.e., motion estimation model), which includes runtime checking, model loading, model compilation, and memory configuration.

[0153] The initialization of the motion estimation model includes runtime checking, model loading, model compilation, and memory configuration. The runtime checking is to determine whether to run the motion estimation model in the NPU. Model loading, model compilation, and memory configuration are to ensure that the motion estimation model can run normally.

[0154] In some implementations, the runtime check includes checking whether an NPU is configured in the electronic device and checking the resolution of the application; if an NPU is configured, running a motion estimation model in the NPU; if an NPU is not configured in the electronic device or the resolution of the application is low, prohibiting the running of the motion estimation model, and the CPU sends rendering instructions to the GPU, and the GPU responds to the rendering instructions for rendering.

[0155] Model loading is to convert the motion estimation model into a model file that can be recognized by the NPU. The model file is loaded into the memory during the initialization phase. Model compilation is to verify that the model file can run successfully. Memory configuration is to allocate memory for the motion estimation model. The allocated memory is used to store the input data and output data of the motion estimation model. In this embodiment, the memory allocated for the motion estimation model can be CPU memory or shared memory.

[0156] (3) Render the game frame at normal resolution to obtain an image at normal resolution.

[0157] It can be understood that, assuming there are X frames in the game, one frame is rendered each time, and the current normal resolution image is the Nth frame. 1≤N≤X, where N and X are both integers.

[0158] (4) Down-sample the image at normal resolution to obtain the current frame (frame N).

[0159] Understandably, normal-resolution image frames are too large and computationally expensive to implement real-time inference. Downsampling and reducing the image size can speed up inference and reduce power consumption during subsequent AI model inference, thus meeting the game's frame rate and power requirements.

[0160] If N is equal to 1, (5) is not executed. If N>1, (5) is executed.

[0161] (5) Wait for the AI ​​computing thread to enter the waiting state.

[0162] It is understood that when N>1, the AI ​​operation thread needs to process the motion estimation of the N-1 frame image before processing the motion estimation of the N-1 frame image. If the AI ​​operation thread is still processing the N-1 frame image, it has not entered the waiting state and step (6) is not executed. If the AI ​​operation thread has processed the N-1 frame image, it enters the waiting state and can continue to execute step (6). This is to avoid transferring the N-1 frame image to the shared memory before the motion estimation of the N-1 frame image is completed.

[0163] For example, the GPU may determine whether the NPU enters the waiting state through signaling interaction with the NPU.

[0164] (6) Input the current frame image.

[0165] The GPU writes the current frame image to the shared memory based on the pointer address of the shared memory. The CPU notifies the NPU to read the current frame image and the previous frame image from the shared memory.

[0166] like Figure 4 As shown, the shared memory stores the input current frame (Frame N) image and the input previous frame (Frame N-1) image.

[0167] (7) Input Frame N and Frame N-1 into the AI ​​operation thread. The AI ​​operation thread includes a motion estimation model.

[0168] The NPU reads Frame N and Frame N-1 based on the pointer address of the shared memory.

[0169] It is understood that after the AI ​​operation thread completes the motion estimation of Frame N-2 / N-1, it triggers the AI ​​motion thread to enter the waiting state. After inputting Frame N and Frame N-1 to the AI ​​operation thread in step (7), it needs to receive a wake-up instruction before it can perform motion estimation.

[0170] (8) Based on the pointer address of the shared memory, the NPU writes the motion estimation results of Frame N-2 / N-1 into the shared memory.

[0171] It should be noted that the Frame N-2 / N-1 motion estimation is understood to be the motion estimation of the N-1th and N-2th frames. The motion estimation result of Frame N-2 / N-1 can also be referred to as the motion estimation result of Frame N-1. The meaning of Frame N-1 / N is similar and is not further explained below.

[0172] The CPU notifies the GPU to read the motion estimation results of Frame N-2 / N-1 in the shared memory.

[0173] (9) The GPU reads the motion estimation result of the previous frame based on the pointer address of the shared memory.

[0174] This application does not limit the order in which steps (7) and (8) are performed, nor does it limit the order in which steps (7) and (9) are performed.

[0175] (10) The GPU determines to start the AI ​​computing thread.

[0176] (11) The GPU asynchronously starts the AI ​​computing thread, which instructs the NPU to wake up the motion estimation of the current frame.

[0177] (12) The AI ​​computing thread wakes up Frame N-1 / N motion estimation.

[0178] (13) The motion estimation model performs Frame N-1 / N motion estimation.

[0179] (14) Post-processing and Frame N-1 user interface (UI) rendering.

[0180] Post-processing is performed based on the motion estimation results of Frame N-2 / N-1, and UI rendering is performed for Frame N-1.

[0181] Exemplarily, if the motion estimation result is applied to an interpolation scenario, post-processing includes pixel shifting and filling for the motion estimation result; if the motion estimation result is applied to a super-resolution scenario, post-processing includes determining the correlation between the N-1th frame and the N-2th frame image based on the motion estimation result of the previous frame, so as to facilitate subsequent super-resolution processing; if the motion estimation result is applied to an anti-aliasing scenario, post-processing includes determining the correlation between the N-1th frame and the N-2th frame image based on the motion estimation result of the previous frame, so as to facilitate subsequent anti-aliasing processing.

[0182] The order of the above (14) and (13) is not limited and they can also be executed simultaneously.

[0183] (15) Send Frame N-1 after post-processing and UI rendering to the screen for display.

[0184] It is understandable that the motion estimation result of Frame N-1 / N is also processed with reference to the above process, and Frame N is also sent to the screen for display with reference to the above process, which will not be described in detail.

[0185] It can be understood that after the GPU obtains the motion estimation result of the previous frame, it asynchronously starts the AI ​​computing thread to perform motion estimation for the current frame. On the one hand, it can reduce the computational pressure of the motion estimation model and improve the inference time. On the other hand, it can execute step (13) at the same time as executing steps (14) and (15), reducing the NPU calculation time, the delay caused by the communication between the NPU and the GPU, and the display, thereby improving the efficiency of the entire process.

[0186] The above Figure 4 The process of applying the motion estimation method provided by this application to interpolation, super-resolution or anti-aliasing scenarios is introduced. The following describes the embodiments of this application from the perspectives of model training and model inference.

[0187] Figure 5 This is a schematic flow chart of a motion estimation model training method 200 provided in an embodiment of the present application, corresponding to the model training side.

[0188] S201, obtain a training data set.

[0189] The training data set includes M adjacent frame image groups and M sample motion information, each adjacent frame image group includes two adjacent frames of first sample images, and the M sample motion information is motion information of the M adjacent frame image groups, where M is a positive integer.

[0190] Exemplarily, the M groups of adjacent frame images may be one or more groups of consecutive frames, or one or more video streams.

[0191] For example, M adjacent frame image groups are a group of continuous frames, and the continuous frames include Y frame images. Then, every two adjacent frame images can form an adjacent frame image group, that is, M=Y-1.

[0192] Taking the example of M adjacent frame image groups being 3 video streams (the image frames between the video streams are discontinuous), the first video stream includes frame A images, the second video stream includes frame B images, and the third video stream includes frame C images, then every two adjacent frame images among the A frame images, B frame images, and C frame images can form an adjacent frame image group, that is, M=A+B+C-3.

[0193] The first sample image is obtained from a first application on a mobile device, and is an image that constitutes a screen to be displayed by the first application. Alternatively, the first sample image is a portion of all images that constitute the screen to be displayed by the first application. Exemplarily, the screen to be displayed here is a dynamic screen, a video to be displayed, or a video to be displayed.

[0194] For example, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario. This application mainly uses the game scenario as an example for illustration, but does not limit its scope of protection.

[0195] For example, a game application generally refers to various mobile game applications, and a motion estimation model trained based on the first sample image has good generalization performance. For another example, a game application refers to a certain type of game, such as a real-time strategy game, a card game, or a business simulation game. A motion estimation model trained based on the first sample image has better motion estimation performance for game frames of the same type of game. For another example, a game application specifically refers to a specific game application, and a motion estimation model trained based on the first sample image has extremely high adaptability to that game application.

[0196] In one possible implementation of S201, taking the first application as a game application as an example, the sample motion information in the training data set is obtained by classifying the game scenes. The game scenes are roughly divided into two categories, and different methods are used to obtain motion information for different game scenes.

[0197] Understandably, data annotation for motion estimation datasets is very complex, time-consuming, and labor-intensive because it requires knowing the motion relationship between each pixel. Currently used datasets are generally divided into two categories: large synthetic datasets and small real-world datasets. These datasets are more targeted at autonomous driving or animation rather than mobile gaming scenarios. By constructing a motion estimation dataset for mobile gaming scenarios, we can obtain motion information that is closer to real-world usage scenarios (i.e., gaming scenarios) as a dataset, thereby improving the effect of motion estimation.

[0198] Step 1-a: Obtain M adjacent frame image groups and image-related data from a first application.

[0199] Exemplarily, the type of image-related data includes 2D information or 3D information.

[0200] Step 1-b: Determine the target method for obtaining motion information according to the type of image-related data.

[0201] The first type of image-related data includes three-dimensional information of a first sample image, and the target method includes three-dimensional reprojection.

[0202] Alternatively, the second category is that the type of image-related data does not include three-dimensional information of the first sample image, and the target method includes a method for motion estimation based on an optical flow method or a neural network on a PC.

[0203] Step 1-c: Determine M sample motion information in a targeted manner based on the image-related data.

[0204] Corresponding to the first category, image-related data includes scene depth and coordinate transformation matrix. The motion information of two adjacent frames is calculated as follows:

[0205]

[0206] Among them, v 3D is the motion information of two adjacent frames, VP n-1 and VP n are the coordinate transformation matrices of the nth frame image and the n-1th frame image, d pre and d cur They are the scene depth values ​​of the n-th frame image and the n-1-th frame image respectively.

[0207] Corresponding to the second category, when drawing translucent objects, shadows, and skill effects, since 3D information such as scene depth and coordinate transformation matrix cannot be obtained from the game application, it is not possible to use the first category of calculations to match pixels between frames using 3D information. Image-related data includes 2D pixel information, and motion information between adjacent frames is calculated based on 2D pixels. In this case, motion information is obtained through optical flow calculation algorithms or PC-side neural network estimation.

[0208] Based on the analysis of current motion estimation algorithms above, this classification method can accurately capture more motion information from game scenes. To obtain more precise motion information, we maximize the use of resources and increase computational overhead, thereby improving the training dataset and resulting in better training results and higher efficiency for the motion estimation model.

[0209] S202 : Down-sampling the two first sample images included in the mth adjacent frame image group among the M adjacent frame image groups to obtain corresponding second sample images and third sample images.

[0210] Wherein, 1≤m≤M and m is an integer, and the motion information of the m-th adjacent frame image group is the m-th sample motion information among the M sample motion information.

[0211] As you can understand, downsampling can reduce image size. Using the downsampled images to train the motion estimation model and subsequently using the downsampled images for inference can speed up inference and reduce power consumption, thus meeting the game's frame rate and power requirements.

[0212] It is understood that each time the first application displays a frame of image, the electronic device must draw, render, and synthesize the image for display before the image can be displayed on the display screen. When the application generates a rendering event, the electronic device's CPU can obtain the image resource to be rendered and perform the drawing, rendering, and synthesizing operations for the current real image.

[0213] Exemplarily, the first sample image is a non-predicted image (that is, a real image), or the second sample image and the third sample image are non-predicted images (that is, real images).

[0214] S203, inputting the second sample image and the third sample image into the motion estimation model; S204, the motion estimation model outputs motion information.

[0215] S205 : Obtain a loss function value according to the difference between the motion information and the motion information of the m th sample.

[0216] Exemplarily, the loss function value is obtained based on the difference between the motion information and the motion information of the mth sample, and the attention weight, where the attention weight is the weight of the pixels at corresponding positions in the second sample image and the third sample image, and the attention weight is used to indicate the user's attention to the pixels at the corresponding positions.

[0217] The pixels at corresponding positions in the second sample image and the third sample image may be understood as the pixels at the same position in the two frames of the second sample image.

[0218] It can be understood that, taking the second sample image as an example, the user's attention to the pixels here is actually the user's attention to the pixels displayed on the display screen, and the pixels displayed on the display screen (the first sample image before the second sample image is down-sampled, and the corresponding pixels of the image after being displayed) and the pixels of the second sample image frame correspond one to one, so it can be understood as the user's attention to the pixels of the second sample image frame.

[0219] It can also be understood that the level of attention can also be understood as the level of sensitivity, or the level of obviousness perceived by the user.

[0220] For example, the loss function is as follows:

[0221]

[0222] Where l is the number of feature layers or the lth layer in the L-layer hidden layer in the motion estimation model, α l is the weight of the l-th layer feature or the weight of the l-th hidden layer, F l (x) is the motion estimation value of pixel x corresponding to the l-th layer feature, is the reference value corresponding to the l-th layer feature, that is, the sample motion information in the training data set, ω x is the attention weight of pixel x, where pixel x is the pixel in the image input to the motion estimation model.

[0223] It is understandable that since the screen size of the mobile terminal is smaller than that of the PC terminal, and the area that the human eye focuses on during the game (for the sake of convenience, referred to as the target focus area below) is actually smaller (such as the main character, skills, etc.), the rest of the area (such as the large background, UI icons, etc.) is not paid much attention to. Therefore, in this case, if the motion estimation of each pixel is trained with the same high precision requirement, the motion estimation model obtained by training will require greater computational overhead and power consumption during operation; if the motion estimation of each pixel is trained with the same lower precision requirement in order to save computational overhead, it will easily lead to low accuracy in the area where the user focuses on, thereby reducing the user experience.

[0224] In this application, the parameters of the motion estimation model are adjusted based on the pixel attention weight, which can provide a better user experience at the same power consumption.

[0225] As you can understand, by setting the attention weights, the trained motion estimation model can have different training accuracies for different pixels in the input image. Furthermore, by adjusting the attention weights, the training accuracy of the motion estimation model for different pixels can be adjusted. This allows for targeted allocation of computing power during model inference, resulting in a better user experience for the final displayed image, even with the same computing power.

[0226] In one possible implementation, the pixel located within the target attention area among the pixels at the corresponding position is determined as the target pixel, and the attention weight of the target pixel is greater than the attention weight of other pixels (the pixels at the corresponding position except the target pixel).

[0227] Exemplarily, the attention weight of the target attention area is 0.5, and the attention weight of other areas outside the target attention area is 0.3.

[0228] Optionally, the target attention area is further subdivided into different areas according to the level of attention, each corresponding to a different attention weight. For example, the target attention area includes a first attention area and a second attention area, and the attention weight of the first attention area is greater than the attention weight of the second attention area.

[0229] Exemplarily, the attention weights of other areas outside the target attention area are 0.3, and the attention weight of the target attention area is greater than 0.3, wherein the attention weight of the first attention area is 0.5, and the attention weight of the second attention area is 0.4.

[0230] The target focus area includes the central area (or central point, or a circle with the central point as the center and the first threshold as the radius) of the second sample image and the third sample image. The following description will take the second sample image as an example.

[0231] For example, the training data set of the game scene is obtained from various game applications on mobile terminals. During the training process, the target focus area can be determined as the central area.

[0232] For another example, the training data set of the game scene is obtained from a certain type of mobile game, and the target focus area can be adjusted accordingly according to the characteristics of this type of game.

[0233] Figure 6 The embodiment of the present application provides several possible target regions of interest including a schematic diagram of an example of a central region of a second sample image.

[0234] It is understandable that for the sake of convenience, Figure 6 In the examples, the target focus area is shown by taking the image after the second sample image is sent for display as an example. As mentioned above, the pixels of the image after the second sample image is sent for display correspond to the pixels of the second sample image frame one by one. Figure 6 By way of example, the target focus area of ​​the second sample image can also be reflected.

[0235] like Figure 6 As shown in (a), taking a business simulation game as an example (such as a city building game), in the image after the second sample image #1 is displayed, the user generally focuses on the area in the center of the display interface to view the situation of the business simulation, such as the target focus area #1.

[0236] like Figure 6 As shown in (b), taking a single-player game as an example (such as a racing game), in the image after the second sample image #2 is displayed, the user generally focuses on the area in the center of the display interface and the area slightly above the center to view the currently operated device and the situation in front, such as the target focus area #2.

[0237] like Figure 6 As shown in (c), taking a role-playing game as an example (for example, a player plays a game character and explores a virtual world), in the image after the second sample image #3 is displayed, the user generally pays attention to the character played and its surroundings. Since the character played is generally close to the central area, for example, the target focus area #3 includes the character played and the central area.

[0238] For another example, the training data set of a game scene is obtained from a specific game application on a mobile terminal, and the target focus area can be adjusted accordingly according to the characteristics of the game.

[0239] For example, semantic recognition can be incorporated into the motion estimation model to identify the visual features of the game on the display interface and determine the attention weight of the screen based on these features. For example, in a specific role-playing game, the current character can be identified and the areas of high user attention can be determined based on the character's behavioral characteristics (such as walking, jumping, and other methods, as well as the skills and related special effects possessed). The associated attention weights can also be determined.

[0240] S206: Adjust parameters that need to be trained in the motion estimation model according to the loss function value.

[0241] Understandably, downsampling can reduce image size. Using downsampled images to train motion estimation models and subsequently inferring them with the downsampled images can speed up inference and reduce power consumption, ensuring that the game's frame rate and power requirements are met. This enables efficient and real-time motion estimation in mobile gaming scenarios, improving rendering quality and enhancing the user experience.

[0242] Part of the process in the motion estimation model training process, namely process #1, includes S203 and S204.

[0243] As a possible implementation of process #1, process #2 includes the following steps.

[0244] S203a: Process the second sample image based on the first input layer to obtain first sample image data.

[0245] S203b: Process the third sample image based on the second input layer to obtain second sample image data.

[0246] It can be understood that S203a and S203b are possible specific examples of S203.

[0247] S205a, performing feature extraction on the second sample image data based on the first hidden layer; S206a, the first hidden layer outputs a first feature map.

[0248] For example, the first hidden layer includes one or more first convolutional layers and one or more first pooling layers.

[0249] S205b, extracting features from the third sample image data based on the second hidden layer; S206b, the second hidden layer outputs a second feature map.

[0250] For example, the second hidden layer includes one or more second convolutional layers and one or more second pooling layers.

[0251] S207, processing the first feature map and the second feature map based on the association module; S204a, the association module outputs motion information.

[0252] It can be understood that S204a is a possible specific example of S204.

[0253] In one possible implementation, the association module includes an association layer and a decoder.

[0254] S207a: The correlation layer calculates the correlation between the first feature map and the second feature map, and outputs a correlation feature map.

[0255] S207b, the decoder is used to decode the correlation feature map; S204b, the decoder outputs motion information.

[0256] It can be understood that S204b is a possible specific example of S204, and S207a and S207b are a possible specific example of S207.

[0257] The motion estimation model provided in this application is described in detail below from the perspective of model structure.

[0258] Model structure: The motion estimation model's network structure is shallower than that of current PC-based neural networks. For example, compared to current PC-based neural networks, the motion estimation model has fewer hidden layers and a more streamlined decoder structure (e.g., the decoder includes fewer convolutional layers and a more streamlined activation function).

[0259] Exemplarily, the number of the first hidden layer and the second hidden layer is L, where L<10. For example, L can be 5, 6, or 7. Preferably, when L=5, experiments have shown that the accuracy and performance of the motion estimation model achieve the best compromise.

[0260] It is understandable that the number of hidden layers in the current PC-based neural network structure is generally greater than 10. By adopting a deeper network structure, it is possible to extract effective high-dimensional features, thereby obtaining sufficient information to calculate the relationship between the two images and improve the accuracy of the motion estimation model.

[0261] In this application, the network structure of the motion estimation model is shallower than that of the PC-side neural network, so that it can be deployed and operate normally on the mobile side. Experiments have shown that on the mobile side, reducing the number of hidden layers and making the network structure shallower can reduce computational overhead while maintaining the accuracy of the motion estimation model.

[0262] Model structure 2: Compared with the current PC-side neural network, the network structure of the motion estimation model has a narrower sampling area of ​​the association module (specifically, the association layer).

[0263] As a possible implementation of S207, the association module samples the first feature map and the second feature map to obtain corresponding sampling features, and performs calculation based on the sampling features to obtain a correlation feature map.

[0264] For example, in step 2-a, the first feature map is sampled based on the association module to obtain the first sampling feature; in step 2-b, the second feature map is sampled based on the association module to obtain the second sampling feature; in step 2-c, the first sampling feature and the second sampling feature are calculated to obtain the correlation feature map of the first feature map and the second feature map.

[0265] As a possible implementation of step 2-a, in step 3-a, a first target feature is determined; and in step 3-b, a first sampled feature is obtained by sampling K×K blocks with the first target feature as the center block, where K is less than 7 and K is a positive integer.

[0266] Similarly, as a possible implementation of step 2-b, step 4-a determines the second target feature. Step 4-b samples K×K blocks with the second target feature as the center block to obtain the second sampled feature. Here, K < 7 and K is a positive integer. The position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

[0267] It can be understood that each block of the first feature map and the second feature map is sampled, that is, each block in the first feature map can be the first target feature, and each block in the second feature map can be the second target feature.

[0268] Figure 7 This is a schematic diagram of an example of the sampling area of ​​the current PC-side neural network.

[0269] Figure 7 In (a), (b) and (c), 9×9 blocks (including white blocks and dark gray blocks) are used as examples to represent the features to be sampled, blocks including black dots are used as examples to represent target features, white blocks are used as examples to represent features within the sampling range, and dark gray blocks are used as examples to represent features outside the sampling range.

[0270] like Figure 7 As shown in (a), the target feature is used as the central block, the sampling range is 9×9, and the number of sampling blocks is 81.

[0271] like Figure 7 As shown in (b), the target feature is used as the central block, the sampling range is 7×7, and the number of sampling blocks is 49.

[0272] like Figure 7 As shown in (c), the target feature is used as the central block, the sampling range is 9×9, and the number of sampling blocks is 53.

[0273] Understandably, a wider sampling area can capture more information, allowing the correlation module to extract sufficient information and abstract motion representations based on the correlations between features, thereby obtaining motion information. However, this also leads to a more complex network model and increased computational effort, placing considerable pressure on real-time mobile gaming scenarios. This makes real-time inference difficult to achieve on mobile devices and results in excessive power consumption.

[0274] In addition, in some cases, compared to Figure 7The sampling method shown in (b) is as follows: Figure 7 The sampling method shown in (c) in Figure 1 has a higher accuracy of motion estimation on the PC side. However, experiments have shown that these two sampling methods have almost no effect on the accuracy of the motion model on the mobile side. Figure 7 The sampling method shown in (c) is more complex, with increased computational effort and greater power consumption. Therefore, a regularly shaped sampling area is preferred on the mobile terminal.

[0275] Figure 8 A schematic diagram of an example of a sampling area of ​​a motion estimation model provided in an embodiment of the present application.

[0276] Figure 8 In the figure, a 9×9 block (including a white block and a dark gray block) is taken as an example to represent the features to be sampled, a block including black dots is taken as an example to represent the target features, a white block is taken as an example to represent the features within the sampling range, and a dark gray block is taken as an example to represent the features outside the sampling range.

[0277] For example, if the feature map to be sampled is a 12×12 block, each block in the feature map to be sampled is used as the target feature for sampling, that is, each block is used as the center point to sample the features within the sampling range. Figure 8 The diagrams given are not diagrams of the feature maps.

[0278] like Figure 8 As shown in the figure, the target feature is taken as the central block, the sampling range is 5×5, and the number of sampling blocks is 25 for sampling.

[0279] As shown above, in this application, K<7 and K is a positive integer. Experiments have shown that using a narrower sampling range for sampling on the mobile terminal has almost no effect on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0280] It should be noted that K=5 is used here for illustration only. K can also have other values, such as 4 or 6 or other integers, without limitation. K=5 is the preferred value verified by experiments in this application. When K=5, the accuracy and performance of the motion estimation model are optimally compromised.

[0281] It should be noted that the sampling area involved in this application can also be called a sampling range. The smaller or narrower the sampling area or sampling range is, the fewer the number of sampled features is.

[0282] Model structure three: Compared with the current PC-side neural network, the network structure of the motion estimation model removes the feature remapping (warp, also known as distortion) operation.

[0283] In current PC-based neural networks, the warp operation is used for feature mapping. This operation maps the features of one image to those of another, further mining the correlation between the two images. Experiments have shown that removing this operation has little impact on the accuracy of the motion estimation model, but significantly reduces its complexity.

[0284] In summary, it can be seen from one or more of the above model structures one to three that the accuracy of the motion estimation model is guaranteed while reducing the complexity of the model, making the motion estimation model effective and concise, so that real-time inference can be performed in mobile gaming scenarios to estimate motion information.

[0285] The motion estimation model provided in this application is described in detail below from the perspective of the computational logic between network layers within the motion estimation model.

[0286] The network layer includes hidden layers and association modules. The hidden layers include convolutional layers and pooling layers; the association modules include association layers and decoders.

[0287] First, let's introduce the calculation logic between the hidden layer and the associated module. Figure 10 Reasoning process #2 in the middle for better understanding.

[0288] The number of the first hidden layer and the second hidden layer is L, and the L first hidden layers correspond to the L second hidden layers. The number of association modules is P.

[0289] In one possible implementation of S207 and S204a, step 5-a processes, based on P correlation modules, P first sub-feature maps output by some or all of the first hidden layers in the L first hidden layers, and P second sub-feature maps output by the corresponding second hidden layers, to obtain motion information. The P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

[0290] It can be understood that when P=L, the first sub-feature map calculated by each first hidden layer in the L-layer first hidden layer and the second sub-feature calculated by the corresponding second hidden layer in the L-layer second hidden layer must be processed based on an association module.

[0291] That is to say, the calculation result of the first hidden layer of the lth layer in the L-layer first hidden layer should be output not only to the first hidden layer of the l+1th layer, but also to the lth associated module, 1≤l≤L and l is an integer.

[0292] Alternatively, when P is less than L, there are P first hidden layers in the L first hidden layers, and the first sub-feature map calculated by each first hidden layer in the P first hidden layers, and the second sub-feature calculated by the corresponding second hidden layer in the L second hidden layers are both processed based on an association module.

[0293] That is to say, the calculation result of the first hidden layer of the pth layer in the first hidden layer of the P layer should not only be output to the first hidden layer of the next layer in the first hidden layer of the L layer, but also to the pth associated module, 1≤p≤P and p is an integer.

[0294] As a possible implementation of S207a, the correlation layer may calculate the correlation by referring to the following formula:

[0295]

[0296] Where l represents the first hidden layer of the first hidden layer in L layers, or the second hidden layer of the second hidden layer in L layers; f1 l The feature of the first hidden layer of the second sample image in the lth layer, or the first sampling feature obtained by sampling the first hidden layer of the second sample image in the lth layer based on the K×K sampling range (including K×K blocks #1); N represents the feature of the third sample image in the second hidden layer of the lth layer, or the feature calculated for the third sample image in the second hidden layer of the lth layer, and the second sampling feature obtained by sampling based on the above K×K sampling range (including K×K blocks #2); l is the characteristic length of the second hidden layer or the first hidden layer of the lth layer.

[0297] Understandable, It means that the K×K blocks #1 included in the first sampling feature are multiplied by the K×K blocks #2 included in the second sampling feature, and the feature map obtained by the multiplication includes K×K blocks #3.

[0298] Preferably, the feature maps output by the shallower, middle, and deepest hidden layers in the first hidden layer or the second hidden layer of the L layers can be processed based on the correlation module. Experiments have shown that by calculating the correlation based on the feature maps output by the shallower, middle, and deepest hidden layers, a better compromise between the accuracy and power consumption of the motion estimation model can be achieved.

[0299] Exemplarily, when P=3, in one implementation of step 5-a, in step 6-a, the cth first sub-feature map output by the first hidden layer of the cth layer and the cth second sub-feature map output by the second hidden layer of the cth layer are input into the first association module to obtain the first intermediate feature; in step 6-b, the bth first sub-feature map output by the first hidden layer of the bth layer and the bth second sub-feature map output by the second hidden layer of the bth layer, and the first intermediate feature are input into the second association module to obtain the second intermediate feature; in step 6-c, the ath first sub-feature map output by the first hidden layer of the ath layer and the ath second sub-feature map output by the second hidden layer of the ath layer, and the second intermediate feature are input into the third association module to obtain motion information, 1≤a<b≤L-1, c=L, 2≤P≤L and P and L are all integers, and a, b and c are all integers.

[0300] Preferably, when L=6 and the first hidden layer of L includes 5 convolutional layers and 1 pooling layer, the first hidden layer of layer a is the second convolutional layer, the first hidden layer of layer b is the fifth convolutional layer, and the first hidden layer of layer c is the pooling layer. Figure 9 and related descriptions.

[0301] The calculation logic between multiple related modules is given below.

[0302] In one implementation of step 5-a, case 1, for the first association module among the P association modules, including the first association layer and the first decoder. In the first association module, the input of the first association layer is the first first sub-feature map among the P first sub-feature maps and the first second sub-feature map among the P second sub-feature maps, and the output is the first association feature map; the input of the first decoder is the first association feature map, and the output is the first intermediate feature.

[0303] Case 2: For the i-th association module among the P association modules, 2≤i≤P-1 and i is an integer, the i-th association module includes the i-th association layer and the i-th decoder. The i-th association layer calculates the association between the i-th first sub-feature map in the P first sub-feature maps and the i-th second sub-feature map in the P second sub-feature maps to obtain the i-th association feature map, and inputs the i-th association feature map into the i-th decoder; the input of the i-th decoder includes the i-th association feature map and the i-1-th intermediate feature output by the i-1-th decoder. The i-th decoder decodes the i-th association feature map and the i-1-th intermediate feature to obtain the i-th intermediate feature, and inputs it into the i+1-th decoder.

[0304] Case 3: For the Pth association module, the Pth association module includes a Pth association layer and a Pth decoder. The Pth association layer calculates the association between the Pth first sub-feature map and the Pth second sub-feature map to obtain the Pth association feature map, and inputs the Pth association feature map into the Pth decoder;

[0305] The Pth decoder decodes the Pth correlation feature map and the P-1th intermediate feature to obtain motion information.

[0306] For details, please refer to Figure 9 A schematic diagram taking three associated modules as an example.

[0307] Figure 9 The motion estimation model training flowchart 300 provided in the embodiment of the present application corresponds to the model training side.

[0308] like Figure 9 As shown, the motion estimation module includes 6 first hidden layers, 6 second hidden layers, and 3 association modules.

[0309] Taking L=6 as an example, the L-layer first hidden layer includes 5 first convolutional layers and one first pooling layer, and the L-layer second hidden layer includes 5 second convolutional layers and 1 second pooling layer.

[0310] The calculation logic between the hidden layer and the association module is shown in the example of P = 3, a = 2, b = 5, and c = 6. The second first sub-feature map output by the first convolutional layer of the second layer and the second second sub-feature map output by the second convolutional layer of the second layer are input into the association layer #3 of the third association module. The fifth first sub-feature map output by the first convolutional layer of the fifth layer and the fifth second sub-feature map output by the second convolutional layer of the fifth layer are input into the association layer #2 of the second association module. The sixth first sub-feature map output by the first pooling layer and the sixth second sub-feature map output by the second pooling layer are input into the association layer #1 of the first association module.

[0311] It should be noted that L=6 is only an example, and L can also have other values, such as L=7 or 8. When L=6, the values ​​of a, b, and c are only examples, and a, b, and c can also have other values, such as a=1, b=4, c=6, or a=2, b=4, c=6, etc.

[0312] Among them, the calculation logic between multiple association modules takes P=3 as an example. The input of the decoder #1 of the first association module (it should be noted that the decoder #x involved in this application is also referred to as the x-th decoder, x≥1 and x is an integer. Similarly, the association layer #x, the association feature map #x, etc. are expressed in the same way, and are explained uniformly here without further elaboration) is the association feature map #1 output by the association layer #1, and the output is the first intermediate feature, and the first intermediate feature is input to the decoder #2 of the second association module; the input of the decoder #2 of the second association module is the association feature map #2 output by the association layer #2 and the first intermediate feature, and the output is the second intermediate feature, and the second intermediate feature is input to the decoder #3 of the third association module; the input of the decoder #3 of the third association module is the association feature map #3 output by the association layer #3 and the second intermediate feature, and the output is motion information.

[0313] S301a: Input the second sample image into the first input layer, and process the second sample image based on the first input layer to obtain first sample image data.

[0314] S301b: Input the third sample image into the second input layer, and process the third sample image based on the second input layer to obtain second sample image data.

[0315] S302a, input the first sample image data into the first convolution layer of the first layer, the calculation result of the first convolution layer of the first layer is input into the first convolution layer of the second layer, the calculation result of the first convolution layer of the second layer (the second first sub-feature map) is input into the first convolution layer of the third layer and the third association module, the calculation result of the first convolution layer of the third layer is input into the first convolution layer of the fourth layer, the calculation result of the first convolution layer of the fourth layer is input into the first convolution layer of the fifth layer, and the calculation result of the first convolution layer of the fifth layer (the fifth first sub-feature map) is input into the first pooling layer.

[0316] S302b, input the second sample image data into the second convolutional layer of the first layer, the calculation result of the second convolutional layer of the first layer is input into the second convolutional layer of the second layer, the calculation result of the second convolutional layer of the second layer (the second second sub-feature map) is input into the second convolutional layer of the third layer and the third association module, the calculation result of the second convolutional layer of the third layer is input into the second convolutional layer of the fourth layer, the calculation result of the second convolutional layer of the fourth layer is input into the second convolutional layer of the fifth layer, and the calculation result of the second convolutional layer of the fifth layer (the fifth second sub-feature map) is input into the second pooling layer.

[0317] S303a, the calculation result of the first pooling layer (the sixth first sub-feature map) is output to the first association module.

[0318] S303b, the calculation result of the second pooling layer (the 6th second sub-feature map) is output to the first association module.

[0319] S304: The correlation layer #1 calculates the correlation between the sixth first sub-feature graph and the sixth second sub-feature graph to obtain a correlation feature graph #1, and inputs the result to the decoder #1.

[0320] S305: Decoder #1 decodes the correlation feature map #1 to obtain the first intermediate feature, and inputs it to decoder #2.

[0321] S306a, the calculation result of the first convolutional layer of the fifth layer (the fifth first sub-feature map) is also input into the second association module.

[0322] S306b, the calculation result of the 5th second convolutional layer (the 5th second sub-feature map) is also input into the 2nd association module.

[0323] S307: The correlation layer #2 calculates the correlation between the fifth first sub-feature map and the fifth second sub-feature map to obtain a correlation feature map #2, and inputs the result to the decoder #2.

[0324] S308: Decoder #2 decodes the correlation feature map #2 to obtain the second intermediate feature and inputs it to decoder #2.

[0325] S309a, the calculation result of the first convolutional layer of the second layer (the second first sub-feature map) is also input into the third association module.

[0326] S309b, the calculation result of the second convolutional layer of the second layer (the second second sub-feature map) is also input into the third association module.

[0327] S310, the correlation layer #3 calculates the correlation between the second first sub-feature map and the second second sub-feature map to obtain a correlation feature map #3, and inputs it to the decoder #3.

[0328] S311, decoder #3 decodes the correlation feature map #3 and outputs motion information.

[0329] For S312 and S313, please refer to the description related to S205 and S206.

[0330] The beneficial effects of the inference flowchart 300 can be found in the corresponding description of the method 200 and will not be elaborated here.

[0331] Figure 10 This is a schematic diagram of a motion estimation method 400 based on a motion estimation model provided in an embodiment of the present application, corresponding to the model inference side.

[0332] It is understandable that the electronic device is deployed with a game application (which may be the same as or different from the game application mentioned in method 200, and is not limited in this application), and the electronic device is also deployed with a motion estimation model trained by method 200.

[0333] like Figure 10 As shown, the method 400 includes the following steps.

[0334] S401: Acquire two adjacent frames of images from a first application.

[0335] The two adjacent frames of images are images constituting a to-be-displayed screen of the first application program.

[0336] Exemplarily, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

[0337] S402 , down-sampling two adjacent frames of images to obtain a first image and a second image.

[0338] It is understood that each time the first application displays a frame of image, the electronic device must draw, render, and synthesize the image for display before the image can be displayed on the display screen. When the application generates a rendering event, the electronic device's CPU can obtain the image resource to be rendered and perform the drawing, rendering, and synthesizing operations for the current real image.

[0339] For example, the first image is a non-predicted image that has been displayed (i.e., a real image), the second image is a non-predicted image that has not been displayed, and the first image is the previous non-predicted image in the display sequence of the second image. Alternatively, image #1 before the first image is down-sampled is a non-predicted image that has been displayed, and image #2 before the second image is down-sampled is a non-predicted image that has not been displayed, and image #1 is the previous non-predicted image in the display sequence of image #2.

[0340] S403: Input the first image and the second image into a motion estimation model.

[0341] S404: The motion estimation model outputs target motion information of two adjacent frames of images.

[0342] The reasoning process #1 of the motion estimation module includes S403 and S404. As a possible implementation of the reasoning process #1, the reasoning process #2 includes the following steps.

[0343] S403a, processing the first image based on the first input layer; S405a, the first input layer outputs first image data.

[0344] S403b, processing the second image based on the second input layer; S405a, the second input layer outputs second image data.

[0345] It can be understood that S403a and S403b are possible specific examples of S403.

[0346] S405a also includes inputting the first image data into the first hidden layer, and performing feature extraction on the first image data based on the first hidden layer; S406a, the first hidden layer outputs a first feature map.

[0347] For example, the first hidden layer includes one or more first convolutional layers and first pooling layers.

[0348] S405b further includes inputting the second image data into a second hidden layer, and performing feature extraction on the second image data based on the second hidden layer; S406b, the second hidden layer outputs a second feature map.

[0349] For example, the second hidden layer includes one or more layers of a second convolutional layer and a second pooling layer.

[0350] S407, processing the first feature map and the second feature map based on the association module; S404a, the association module outputs target motion information.

[0351] It can be understood that S404a is a possible specific example of S404.

[0352] In one possible implementation, the association module includes an association layer and a decoder.

[0353] S407a: The correlation layer calculates the correlation between the first feature map and the second feature map, and outputs a correlation feature map.

[0354] S407b, the decoder is used to decode the correlation feature map; S404b, the decoder outputs the target motion information.

[0355] It can be understood that S404b is a possible specific example of S404, and S407a and S407b are a possible specific example of S407.

[0356] The above solution first downsamples the images obtained from the gaming application before inputting them into the motion estimation model. This can speed up inference and reduce power consumption so that the model can be deployed on mobile devices. It can also meet the frame rate and power consumption requirements of mobile gaming scenarios, achieve efficient and real-time motion estimation, and improve rendering effects.

[0357] During training, by setting attention weights, the trained motion estimation model can achieve different training accuracies for different pixels in the input image. Furthermore, by adjusting the attention weights, the training accuracy of the motion estimation model for different pixels can be adjusted. This allows for targeted allocation of computing power during model inference, resulting in a better user experience for the displayed image while using the same computing power.

[0358] S401 to S407b in method 400 may refer to the corresponding descriptions of S201 to S207b in method 200 on the model training side, with the following differences:

[0359] (1) The parameters in the motion estimation model on the model inference side are the parameters that have been updated after the loss function value is transferred backward, or the trained parameters. The parameters in the motion estimation model on the model training side are the parameters to be updated during the training process.

[0360] (2) The first feature map in method 200 is the first feature map of the second sample image, while the first feature map in method 400 is the first feature map of the first image. The first feature map and the first feature map in method 400 are different. The feature maps and other data obtained during the intermediate calculation process of the motion estimation model are similar and are not described in detail below.

[0361] (3) Compared with method 200, in method 400, the first sample image, the second sample image, the first sample image data, the second sample image data and the motion information are replaced by the first image, the second image, the first image data, the second image data and the target motion information, respectively.

[0362] (4) Model inference side, refer to the description of the model training side Figure 9 Related content, replaced by see Figure 11 Related content (with Figure 9 For example, refer to the description of the model training side. Figure 9 The schematic diagram of 3 related modules is replaced by, see Figure 11 The diagram in FIG. 3 is a diagram of three associated modules as an example. Other replacements are similar and will not be described in detail here.

[0363] Similarly, the structure of the motion estimation model can also refer to the corresponding description of the model training side above.

[0364] Similarly, the computational logic between network layers within the motion estimation model can also refer to the corresponding description of the model training side above.

[0365] Similarly, the calculation logic between multiple related modules can also refer to the corresponding description of the model training side above.

[0366] Figure 11The inference flowchart 500 of the motion estimation model provided in the embodiment of the present application corresponds to the model inference side.

[0367] like Figure 11 As shown, the motion estimation module includes 6 first hidden layers, 6 second hidden layers, and 3 association modules.

[0368] Taking L=6 as an example, the L-layer first hidden layer includes 5 first convolutional layers and one first pooling layer, and the L-layer second hidden layer includes 5 second convolutional layers and 1 second pooling layer.

[0369] The calculation logic between the hidden layer and the association module is shown in the example of P = 3, a = 2, b = 5, and c = 6. The second first sub-feature map output by the first convolutional layer of the second layer and the second second sub-feature map output by the second convolutional layer of the second layer are input into the association layer #3 of the third association module. The fifth first sub-feature map output by the first convolutional layer of the fifth layer and the fifth second sub-feature map output by the second convolutional layer of the fifth layer are input into the association layer #2 of the second association module. The sixth first sub-feature map output by the first pooling layer and the sixth second sub-feature map output by the second pooling layer are input into the association layer #1 of the first association module.

[0370] It should be noted that L=6 is only an example, and L can also have other values, such as L=7 or 8. When L=6, the values ​​of a, b, and c are only examples, and a, b, and c can also have other values, such as a=1, b=4, c=6, or a=2, b=4, c=6, etc.

[0371] The calculation logic between multiple association modules takes P=3 as an example. The input of decoder #1 of the first association module is the association feature map #1 output by the association layer #1, and the output is the first intermediate feature. The first intermediate feature is input to decoder #2 of the second association module. The input of decoder #2 of the second association module is the association feature map #2 output by the association layer #2 and the first intermediate feature. The output is the second intermediate feature. The second intermediate feature is input to decoder #3 of the third association module. The input of decoder #3 of the third association module is the association feature map #3 output by the association layer #3 and the second intermediate feature. The output is target motion information.

[0372] S501a, inputting the first image into the first input layer, and processing the first image based on the first input layer; S502a, the first input layer outputs first image data.

[0373] S501b, input the second image into the second input layer, and process the second image based on the second input layer; S502b, the first input layer outputs the second image data.

[0374] S502a also includes inputting the first image data into the first convolution layer of the first layer, inputting the calculation result of the first convolution layer of the first layer into the first convolution layer of the second layer, inputting the calculation result of the first convolution layer of the second layer (the second first sub-feature map) into the first convolution layer of the third layer and the third association module, inputting the calculation result of the first convolution layer of the third layer into the first convolution layer of the fourth layer, inputting the calculation result of the first convolution layer of the fourth layer into the first convolution layer of the fifth layer, and inputting the calculation result of the first convolution layer of the fifth layer (the fifth first sub-feature map) into the first pooling layer.

[0375] S502b also includes inputting the second image data into the first second convolutional layer, inputting the calculation result of the first second convolutional layer into the second second convolutional layer, inputting the calculation result of the second second convolutional layer (the second second sub-feature map) into the third second convolutional layer and the third association module, inputting the calculation result of the third second convolutional layer into the fourth second convolutional layer, inputting the calculation result of the fourth second convolutional layer into the fifth second convolutional layer, and inputting the calculation result of the fifth second convolutional layer (the fifth second sub-feature map) into the second pooling layer.

[0376] S503a, the calculation result of the first pooling layer (the sixth first sub-feature map) is output to the first association module.

[0377] S503b, the calculation result of the second pooling layer (the 6th second sub-feature map) is output to the first association module.

[0378] S504: The correlation layer #1 calculates the correlation between the sixth first sub-feature graph and the sixth second sub-feature graph to obtain a correlation feature graph #1, and inputs the result to the decoder #1.

[0379] S505: Decoder #1 decodes the correlation feature map #1 to obtain the first intermediate feature, and inputs it to decoder #2.

[0380] S506a, the calculation result of the first convolutional layer of the fifth layer (the fifth first sub-feature map) is also input into the second association module.

[0381] S506b, the calculation result of the 5th second convolutional layer (the 5th second sub-feature map) is also input into the 2nd association module.

[0382] S507: The correlation layer #2 calculates the correlation between the fifth first sub-feature map and the fifth second sub-feature map to obtain a correlation feature map #2, and inputs the result to the decoder #2.

[0383] S508: Decoder #2 decodes the correlation feature map #2 to obtain the second intermediate feature and inputs it to decoder #2.

[0384] S509a, the calculation result of the first convolutional layer of the second layer (the second first sub-feature map) is also input into the third association module.

[0385] S509b, the calculation result of the second convolutional layer of the second layer (the second second sub-feature map) is also input into the third association module.

[0386] S510, the correlation layer #3 calculates the correlation between the second first sub-feature map and the second second sub-feature map to obtain a correlation feature map #3, and inputs it to the decoder #3.

[0387] S511, decoder #3 decodes the correlation feature map #3 and outputs target motion information.

[0388] The beneficial effects of the inference flowchart 500 can be found in the corresponding description of the method 400 and will not be elaborated here.

[0389] Figure 12 This is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer (not shown in the figure), the system layer, and the kernel layer.

[0390] The application layer can include a series of application packages.

[0391] like Figure 12 As shown, the application package may include a first application, which may be a game, live broadcast, video, etc. The application package may also include other applications, such as camera, gallery, calendar, phone, map, WLAN, Bluetooth, music, short message, etc. (not shown in the figure).

[0392] Among them, games, videos, or live broadcast applications require certain rendering effects when playing pictures. Based on this requirement, the motion estimation model provided in the embodiment of the present application can realize efficient and real-time motion estimation in mobile game scenes, thereby improving the rendering effect.

[0393] The system layer may include multiple functional modules, such as an image processing library, an image resource processing module, a motion estimation module, and a post-processing and UI rendering module. It may also include an image engine, a media library, etc. (not shown in the figure).

[0394] The image resource processing module is used to obtain image resources (real image frames) and transmit the image resources to the motion estimation module.

[0395] The image processing library is used to draw, render, composite, and display real-world images. It includes both a 2D graphics library and a 3D graphics library. The 3D graphics library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is a drawing engine for 2D graphics.

[0396] The motion estimation module is used to perform motion estimation on the image resources and transmit the motion estimation results to the post-processing and UI rendering module. The motion estimation module can call the motion estimation model provided in this application to perform motion estimation.

[0397] The functions of post-processing and UI rendering modules can be found in Figure 4 Related instructions for step (14) in .

[0398] The kernel layer is the layer between hardware and software. The kernel layer includes at least the display driver. It may also include camera drivers, Wi-Fi drivers, communication drivers, audio drivers, sensor drivers, Bluetooth drivers, etc. (not shown).

[0399] The following combination Figure 12 , taking the first application as a game as an example, Figure 12 The rendering process is explained in the following sections.

[0400] The game application generates an image rendering event and sends the image resource data to the image processing library. The three-dimensional graphics library in the image processing library performs image drawing, rendering and other operations based on the received image resource data. The image resource processing module intercepts the image resource data (current real image frame) sent by the game application to the image processing library, and transmits the intercepted image resource data to the prediction frame processing module. The motion estimation module obtains the previous real image frame of the current real image frame based on the image resource data (current real image frame), performs motion estimation based on the current real image frame and the previous real image frame, and obtains motion information. The post-processing and UI rendering unit performs post-processing and rendering processing on the current real image frame based on the motion information (for example, it can perform corresponding processing based on interpolation, super-resolution or anti-aliasing scenes) to obtain the image frame to be displayed, and sends the image frame to be displayed to the display driver for display.

[0401] The present invention provides a chip system comprising one or more processors configured to retrieve and execute instructions stored in a memory, thereby executing the method of the present invention. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0402] Among them, the chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0403] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0404] The computer program product can be stored in a memory and finally converted into an executable target file that can be executed by a processor through preprocessing, compilation, assembly and linking.

[0405] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0406] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0407] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0408] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0409] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0410] It should be understood that the term "plurality" used herein refers to two or more. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0411] The terms (or numbers) "first", "second", ... etc. that appear in the embodiments of the present application are only used for descriptive purposes, that is, they are only used to distinguish different objects, such as different "coordinates", etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", ... etc. may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one of the following (item)" or similar expressions refers to any combination of these items, including any combination of a single (item) or plural (items).

[0412] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. A motion estimation model training method, characterized in that: The method comprises: Obtaining a training data set, the training data set including M adjacent frame image groups and M sample motion information, each adjacent frame image group including two adjacent frames of first sample images, the M sample motion information respectively being motion information of the M adjacent frame image groups, M being a positive integer, the training data set being obtained from a first application on a mobile terminal, the first sample images being images constituting a to-be-displayed screen of the first application; downsampling two first sample images included in an m-th adjacent frame image group among the M adjacent frame image groups to obtain corresponding second sample images and third sample images, where 1≤m≤M and m is an integer, and the motion information of the m-th adjacent frame image group is the m-th sample motion information among the M sample motion information; Inputting the second sample image and the third sample image into the motion estimation model to obtain motion information; Obtaining a loss function value according to a difference between the motion information and the motion information of the mth sample; Parameters that need to be trained in the motion estimation model are adjusted according to the loss function value.

2. The method according to claim 1, wherein The first application is a game application, and the training data set is obtained from the game application in a game scenario; Alternatively, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; Alternatively, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

3. The method according to claim 1 or 2, wherein: The obtaining a loss function value according to the difference between the motion information and the motion information of the m-th sample includes: The loss function value is obtained based on the difference between the motion information and the mth sample motion information, and the attention weight, where the attention weight is the weight of the pixels at corresponding positions in the second sample image and the third sample image, and the attention weight is used to indicate the user's attention to the pixels at the corresponding positions.

4. The method according to claim 3, wherein Among the pixels at the corresponding positions, pixels located within a target attention area are determined as target pixels, and an attention weight of the target pixel is greater than attention weights of other pixels, wherein the target attention area includes central areas of the second sample image and the third sample image.

5. The method according to claim 1, wherein The obtaining of the training data set includes: Acquire the M adjacent frame image groups and image-related data from the first application; determining a target method for acquiring motion information according to the type of the image-related data; Determining the M sample motion information according to the image-related data in the target manner; In which, when the type of image-related data includes three-dimensional information of the first sample image, the target method includes three-dimensional reprojection; or, when the type of image-related data does not include three-dimensional information of the first sample image, the target method includes a method for motion estimation based on optical flow method or neural network on a personal computer.

6. The method according to claim 5, wherein In a case where the type of the image-related data includes three-dimensional information of the first sample image and the target mode includes three-dimensional reprojection, the three-dimensional information of the first sample image includes a scene depth value and a coordinate transformation matrix; The determining, based on the image-related data and in the target manner, the M pieces of motion information includes: The motion information corresponding to each of the adjacent frame image groups is obtained based on the three-dimensional reprojection according to the scene depth values ​​and coordinate transformation matrix of the two first sample images in each of the adjacent frame image groups.

7. The method according to claim 1, wherein Inputting the second sample image and the third sample image into the motion estimation model to obtain motion information includes: Processing the second sample image based on the first input layer to obtain first sample image data, and processing the third sample image based on the second input layer to obtain second sample image data; Performing feature extraction on the first sample image data based on the first hidden layer to obtain a first feature map, and performing feature extraction on the second sample image data based on the second hidden layer to obtain a second feature map; The first feature map and the second feature map are processed based on an association module to obtain the motion information, wherein the association module is used to calculate the association degree between the first feature map and the second feature map.

8. The method according to claim 7, wherein The processing of the first feature map and the second feature map based on the association module to obtain the motion information includes: Sampling the first feature map based on the association module to obtain a first sampling feature; Sampling the second feature map based on the association module to obtain a second sampling feature; Based on the association module, calculating the association degree according to the first sampling feature and the second sampling feature to obtain a correlation degree feature graph of the first feature graph and the second feature graph; The correlation feature map is decoded to obtain the motion information.

9. The method according to claim 8, wherein The sampling of the first feature map based on the association module to obtain a first sampling feature includes: determining a first target feature based on the association module; sampling K×K blocks with the first target feature as a center block to obtain the first sampling feature; The sampling of the second feature map based on the association module to obtain a second sampling feature includes: determining a second target feature based on the association module; sampling K×K blocks with the second target feature as the center block to obtain the second sampling feature; Wherein, K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

10. The method according to claim 7, wherein: The number of layers of the first hidden layer and the second hidden layer is L, and the L first hidden layers and the L second hidden layers correspond one to one, and the number of the association modules is P; The processing of the first feature map and the second feature map based on the association module to obtain the motion information includes: Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layers, and the P second sub-feature maps output by the corresponding second hidden layers are processed respectively to obtain the motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

11. The method according to claim 10, wherein The first association module among the P association modules includes a first association layer and a first decoder, the i-th association module among the P association modules includes an i-th association layer and an i-th decoder, and the P-th association module includes a P-th association layer and a P-th decoder. The calculation based on the P association modules is performed on the P first sub-feature maps output by part or all of the first hidden layers in the L layer, and the P second sub-feature maps output by the corresponding second hidden layers to obtain the motion information, including: The first correlation layer calculates the correlation between the first of the P first sub-feature maps and the first of the P second sub-feature maps to obtain a first correlation feature map, and inputs the first correlation feature map into the first decoder; the first decoder decodes the first correlation feature map to obtain a first intermediate feature, and inputs the first intermediate feature into the second decoder; The i-th correlation layer calculates the correlation between the i-th first sub-feature graph and the i-th second sub-feature graph to obtain the i-th correlation feature graph, and inputs the i-th correlation feature graph into the i-th decoder; the i-th decoder decodes the i-th correlation feature graph to obtain the i-th intermediate feature, and inputs the i-th intermediate feature into the i+1-th decoder, where 2≤i≤P-1 and i is an integer; The P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain a P-th correlation feature map, and inputs the P-th correlation feature map into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain the motion information.

12. The method according to claim 10, wherein When P=3, the P association modules are used to process the P first sub-feature maps output by part or all of the first hidden layers in the L layers, and the P second sub-feature maps output by the corresponding second hidden layers, to obtain the motion information, including: Inputting the cth first sub-feature map output by the cth first hidden layer and the cth second sub-feature map output by the cth second hidden layer into a first association module to obtain a first intermediate feature; Inputting the bth first sub-feature map output by the bth first hidden layer, the bth second sub-feature map output by the bth second hidden layer, and the first intermediate feature into a second association module to obtain a second intermediate feature; The ath first sub-feature map output by the ath first hidden layer, the ath second sub-feature map output by the ath second hidden layer, and the second intermediate feature are input into the third association module to obtain the motion information, 1≤a<b≤L-1, c=L, and a, b and c are all integers.

13. The method according to claim 10, wherein L<10, L layers of the first hidden layer include L-1 layers of the first convolution layer and 1 layer of the first pooling layer, and L layers of the second hidden layer include L-1 layers of the second convolution layer and 1 layer of the second pooling layer; Among them, when L=6, the 6-layer first hidden layer includes 5 layers of the first convolutional layer and 1 layer of the first pooling layer, and the 6-layer second hidden layer includes 5 layers of the second convolutional layer and 1 layer of the second pooling layer, the a-th layer first hidden layer is the 2-th layer first convolutional layer, the b-th layer first hidden layer is the 5-th layer first convolutional layer, the c-th layer first hidden layer is the first pooling layer, the a-th layer second hidden layer is the 2-th layer second convolutional layer, the b-th layer second hidden layer is the 5-th layer second convolutional layer, and the c-th layer second hidden layer is the second pooling layer.

14. A motion estimation method based on a motion estimation model, applied to electronic equipment, characterized in that: The electronic device is deployed with a first application, the motion estimation model is deployed on the electronic device, and the motion estimation model is trained by the motion estimation model training method according to any one of claims 1 to 13, the method comprising: Acquire two adjacent frames of images from the first application, and perform downsampling processing on the two adjacent frames of images to obtain a first image and a second image, wherein the two adjacent frames of images are images constituting a to-be-displayed screen of the first application; Inputting the first image and the second image into the motion estimation model to obtain target motion information of the two adjacent frames of images includes: Processing the first image based on a first input layer to obtain first image data, and processing the second image based on a second input layer to obtain second image data; Performing feature extraction on the first image data based on a first hidden layer to obtain a first feature map, and performing feature extraction on the second image data based on a second hidden layer to obtain a second feature map; The first feature map and the second feature map are processed based on an association module to obtain the target motion information, wherein the association module is used to calculate the association degree between the first feature map and the second feature map.

15. The method according to claim 14, wherein The first application is a game application, and the training data set is obtained from the game application in a game scenario; Alternatively, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; Alternatively, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

16. The method according to claim 14 or 15, characterized in that The processing of the first feature map and the second feature map based on the association module to obtain the target motion information includes: Sampling the first feature map based on the association module to obtain a first sampling feature; Sampling the second feature map based on the association module to obtain a second sampling feature; Calculating the first sampling feature and the second sampling feature to obtain a correlation feature graph of the first feature graph and the second feature graph; The correlation feature map is decoded to obtain the target motion information.

17. The method according to claim 16, wherein The sampling of the first feature map based on the association module to obtain a first sampling feature includes: determining a first target feature; taking the first target feature as a center block and sampling K×K blocks to obtain the first sampling feature; The sampling of the second feature map based on the association module to obtain a second sampling feature includes: determining a second target feature; taking the second target feature as a center block and sampling K×K blocks to obtain the second sampling feature; Wherein, K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

18. The method according to claim 14, wherein The number of layers of the first hidden layer and the second hidden layer is L, and the L first hidden layers and the L second hidden layers correspond one to one, and the number of the association modules is P; The processing of the first feature map and the second feature map based on the association module to obtain the target motion information includes: Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layers, and the P second sub-feature maps output by the corresponding second hidden layers are processed respectively to obtain the target motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

19. The method according to claim 18, wherein The first association module among the P association modules includes a first association layer and a first decoder, the i-th association module among the P association modules includes an i-th association layer and an i-th decoder, and the P-th association module includes a P-th association layer and a P-th decoder. Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layer and the P second sub-feature maps output by the corresponding second hidden layers are calculated to obtain the target motion information, including: The first correlation layer calculates the correlation between the first of the P first sub-feature maps and the first of the P second sub-feature maps to obtain a first correlation feature map, and inputs the first correlation feature map into the first decoder; the first decoder decodes the first correlation feature map to obtain a first intermediate feature, and inputs the first intermediate feature into the second decoder; The i-th correlation layer calculates the correlation between the i-th first sub-feature graph and the i-th second sub-feature graph to obtain the i-th correlation feature graph, and inputs the i-th correlation feature graph into the i-th decoder; the i-th decoder decodes the i-th correlation feature graph to obtain the i-th intermediate feature, and inputs the i-th intermediate feature into the i+1-th decoder, where 2≤i≤P-1 and i is an integer; The P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain a P-th correlation feature map, and inputs the P-th correlation feature map into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain the target motion information.

20. The method according to claim 19, wherein When P=3, the P association modules are used to process the P first sub-feature maps output by part or all of the first hidden layers in the L layers, and the P second sub-feature maps output by the corresponding second hidden layers, to obtain the target motion information, including: Inputting the cth first sub-feature map output by the cth first hidden layer and the cth second sub-feature map output by the cth second hidden layer into a first association module to obtain a first intermediate feature; Inputting the bth first sub-feature map output by the bth first hidden layer, the bth second sub-feature map output by the bth second hidden layer, and the first intermediate feature into a second association module to obtain a second intermediate feature; The ath first sub-feature map output by the ath first hidden layer, the ath second sub-feature map output by the ath second hidden layer, and the second intermediate feature are input into the third association module to obtain the target motion information, 1≤a<b≤L-1, c=L, and a, b and c are all integers.

21. The method according to claim 20, wherein L<10, L layers of the first hidden layer include L-1 layers of the first convolution layer and 1 layer of the first pooling layer, and L layers of the second hidden layer include L-1 layers of the second convolution layer and 1 layer of the second pooling layer; Among them, when L=6, the 6-layer first hidden layer includes 5 layers of the first convolutional layer and 1 layer of the first pooling layer, and the 6-layer second hidden layer includes 5 layers of the second convolutional layer and 1 layer of the second pooling layer, the a-th layer first hidden layer is the 2-th layer first convolutional layer, the b-th layer first hidden layer is the 5-th layer first convolutional layer, the c-th layer first hidden layer is the first pooling layer, the a-th layer second hidden layer is the 2-th layer second convolutional layer, the b-th layer second hidden layer is the 5-th layer second convolutional layer, and the c-th layer second hidden layer is the second pooling layer.

22. The method of claim 14, wherein: The first image is a non-predicted image that has been displayed, the second image is a non-predicted image that has not been displayed, and the first image is a non-predicted image that is a previous non-predicted image of the second image in a display timing sequence. The method further includes: Performing pixel shifting and padding on the second image and the first image according to the target motion information to obtain one or more predicted images; After displaying the one or more predicted images, the second image is displayed.

23. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 13, or, to execute the method as described in any one of claims 14 to 22.

24. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 13, or to execute the method as described in any one of claims 14 to 22.

25. A computer-readable storage medium, characterized in that The computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 13, or executes the method according to any one of claims 14 to 22.

Images