Motion estimation model training method, motion estimation method and related equipment

Abstract

The invention provides a motion estimation model training method, a motion estimation method and related equipment. The motion estimation model training method comprises the steps that a training data set of a mobile terminal game scene is acquired, and the training data set comprises first sample images of adjacent frames and sample motion information corresponding to the first sample images; and performing down-sampling on the first sample image to obtain a second sample image and a third sample image, and training a motion estimation model according to the sample motion information. The motion estimation method is applied to the electronic equipment in which a game application and a motion estimation model are deployed. The method comprises the following steps: acquiring images of adjacent frames from a game application, performing down-sampling processing to obtain a first image and a second image, and inputting the two frames of images into a motion estimation model to obtain target motion information. In the model training stage and the reasoning stage, the image subjected to down-sampling processing is used as the input of the model, the reasoning speed can be increased, power consumption can be reduced, efficient and real-time motion estimation is achieved in a mobile terminal game scene, and the rendering effect is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a motion estimation model training method, a motion estimation method and related equipment. Background Art

[0002] As mobile applications (such as game applications) continue to be updated and iterated, mobile applications, especially those that need to display videos, require increasingly complex and diverse rendering processes. For example, the rendering process includes frame insertion (used to increase frame rate), super-resolution (used to increase resolution), anti-aliasing (used to eliminate jagged edges of images), and other processing to improve the visual experience of users when using applications (such as playing games).

[0003] Through motion estimation, the motion information between two adjacent frames can be obtained. In the rendering process, the motion information can be used as a priori information in the time domain for processing such as interpolation, super-resolution, and anti-aliasing to obtain better rendering effects.

[0004] Among the current motion estimation technologies, non-artificial intelligence (AI) methods have high computational overhead, are not robust, and have limited applicable scenarios; while AI methods have complex model structures and are difficult to meet the real-time and low-power requirements of gaming scenarios on mobile devices.

[0005] Therefore, how to perform motion estimation for game scenes on mobile devices becomes a key issue. Summary of the invention

[0006] The present application provides a motion estimation model training method, a motion estimation method and related equipment, which can realize efficient and real-time motion estimation in mobile game scenes, improve rendering effects, and thus improve user experience.

[0007] In a first aspect, a motion estimation model training method is provided, comprising: obtaining a training data set, the training data set comprising M adjacent frame image groups and M sample motion information, each adjacent frame image group comprising two adjacent frames of first sample images, the M sample motion information respectively being motion information of the M adjacent frame image groups, M being a positive integer, the training data set being obtained from a first application on a mobile terminal, the first sample images being images constituting a to-be-displayed screen of the first application; down-sampling the two first sample images included in the mth adjacent frame image group of the M adjacent frame image groups respectively to obtain corresponding second sample images and third sample images, 1≤m≤M and m being an integer, the motion information of the mth adjacent frame image group being the mth sample motion information of the M sample motion information; inputting the second sample image and the third sample image into a motion estimation model to obtain motion information; obtaining a loss function value based on the difference between the motion information and the mth sample motion information; and adjusting the parameters to be trained in the motion estimation model based on the loss function value.

[0008] It is understandable that downsampling can reduce the image size. Using the downsampled images to train the motion estimation model, and accordingly, using the downsampled images for subsequent motion estimation model reasoning, can speed up the model reasoning and reduce power consumption, so that the model can be deployed on the mobile terminal and can meet the frame rate and power consumption requirements of mobile application scenarios (such as game scenarios), achieve efficient and real-time motion estimation, and improve rendering effects.

[0009] It can also be understood that the motion estimation model obtained by training can obtain motion information based on the input adjacent frame images. Compared with the motion estimation method based on the three-dimensional (3D) motion field, which needs to obtain 3D information and rotation matrix, on the one hand, since it does not need to obtain 3D information to obtain motion information, it can be applied to scenes such as transparent objects and shadows where 3D information cannot be obtained; on the other hand, since fewer resources are obtained, the computational overhead is smaller. Compared with the motion estimation method based on the optical flow method, since it does not rely on the search area and matching for motion estimation, the effect of motion estimation is not affected by the size of the search area and the computational complexity of matching.

[0010] In a possible embodiment, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

[0011] In a possible embodiment, a loss function value is obtained according to the difference between the motion information and the motion information of the mth sample, including: obtaining the loss function value according to the difference between the motion information and the motion information of the mth sample, and an attention weight, wherein the attention weight is the weight of pixels at corresponding positions in the second sample image and the third sample image, and the attention weight is used to indicate the user's attention to the pixels at the corresponding positions.

[0012] That is, the parameters of the motion estimation model are adjusted using a loss function determined according to the attention weights.

[0013] It can be understood that by setting the attention weight, the trained motion estimation model can have different training accuracy for different pixels in the input image. Moreover, by adjusting the attention weight, the training accuracy of the motion estimation model for different pixels can be adjusted. In this way, the computing power can be allocated in a targeted manner during model inference, so that when the same computing power is used, the final image displayed can provide a better user experience.

[0014] In a possible embodiment, among the pixels at corresponding positions, pixels located within a target focus area are determined as target pixels, and the attention weight of the target pixel is greater than the attention weights of other pixels, wherein the target focus area includes the central areas of the second sample image and the third sample image.

[0015] Optionally, the target attention area can be further subdivided into attention gradients. That is, the target attention area is further subdivided into different areas according to the level of attention, and each area corresponds to a different attention weight.

[0016] Optionally, the target focus area may be determined according to a source of the training dataset, that is, a type of game application.

[0017] In a possible embodiment, obtaining a training data set includes: obtaining M adjacent frame image groups and image-related data from a first application; determining a target method for obtaining motion information based on the type of the image-related data; and determining M sample motion information based on the image-related data by a target method; wherein, in a case where the type of the image-related data includes three-dimensional information of the first sample image, the target method includes three-dimensional reprojection; or, in a case where the type of the image-related data does not include three-dimensional information of the first sample image, the target method includes a method for motion estimation based on an optical flow method or a neural network on a personal computer (PC).

[0018] The above scheme, by constructing a motion estimation data set for mobile game scenes, can obtain motion information that is closer to the actual usage scene (that is, the game scene) as a data set, thereby improving the effect of motion estimation. This classification method can obtain more motion information of game scenes as accurately as possible. In order to obtain more accurate motion information, as many resources as possible are obtained, and greater computing overhead is provided, so that the training data set is more complete, which will also make the training effect of the motion estimation model better and the training efficiency higher.

[0019] In a possible embodiment, when the type of image-related data includes three-dimensional information of a first sample image and the target method includes three-dimensional reprojection, the three-dimensional information of the first sample image includes a scene depth value and a coordinate transformation matrix; according to the image-related data, M sample motion information is determined in a target manner, including: according to the scene depth values ​​and coordinate transformation matrices of two frames of first sample images in each adjacent frame image group, based on three-dimensional reprojection, obtaining the sample motion information corresponding to each adjacent frame image group.

[0020] In a possible embodiment, when the type of image-related data includes three-dimensional information of a first sample image and the target method includes three-dimensional reprojection, the three-dimensional information of the first sample image includes a scene depth value and a coordinate transformation matrix; according to the image-related data, M motion information is determined in a target manner, including: according to the scene depth values ​​and coordinate transformation matrices of two frames of first sample images in each adjacent frame image group, based on three-dimensional reprojection, obtaining the motion information corresponding to each adjacent frame image group.

[0021] In a possible embodiment, the second sample image and the third sample image are input into a motion estimation model to obtain motion information, including: processing the second sample image based on the first input layer to obtain first sample image data, and processing the third sample image based on the second input layer to obtain second sample image data; extracting features of the first sample image data based on the first hidden layer to obtain a first feature map, and extracting features of the second sample image data based on the second hidden layer to obtain a second feature map; processing the first feature map and the second feature map based on an association module to obtain motion information, wherein the association module is used to calculate the degree of association between the first feature map and the second feature map.

[0022] In a possible embodiment, the first feature map and the second feature map are processed based on an association module to obtain motion information, including: sampling the first feature map based on the association module to obtain a first sampling feature; sampling the second feature map based on the association module to obtain a second sampling feature; based on the association module, calculating the correlation degree according to the first sampling feature and the second sampling feature to obtain a correlation degree feature map of the first feature map and the second feature map; decoding the correlation degree feature map to obtain motion information.

[0023] In a possible embodiment, sampling the first feature map based on the association module to obtain the first sampling feature includes: determining the first target feature based on the association module; taking the first target feature as the center block and sampling a sampling range of K×K blocks to obtain the first sampling feature; sampling the second feature map based on the association module to obtain the second sampling feature includes: determining the second target feature based on the association module; taking the second target feature as the center block and sampling a sampling range of K×K blocks to obtain the second sampling feature; wherein K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

[0024] The above scheme has been experimentally proven to have almost no effect on the accuracy of the motion estimation model by using a narrower sampling range for sampling on the mobile terminal, but it greatly reduces the complexity of the motion estimation model.

[0025] It is understandable that a wider sampling area can obtain more sufficient information, so that the association module can obtain sufficient information, abstract the motion representation based on the association between features, and thus obtain motion information. However, it will also make the network model more complex and increase the amount of calculation, which will bring considerable pressure to the real-time mobile game scenes, making it difficult to achieve real-time reasoning on the mobile terminal and the power consumption is too high.

[0026] In a possible embodiment, the number of layers of the first hidden layer and the second hidden layer are both L, and the L-layer first hidden layer corresponds to the L-layer second hidden layer in one-to-one correspondence, and the number of association modules is P; the first feature map and the second feature map are processed based on the association module to obtain motion information, including: based on P association modules, P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer, and P second sub-feature maps output by the corresponding second hidden layer are processed respectively to obtain motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

[0027] In a possible embodiment, the first association module among the P association modules includes the first association degree layer and the first decoder, the i-th association module among the P association modules includes the i-th association degree layer and the i-th decoder, and the P-th association module includes the P-th association degree layer and the P-th decoder. Based on the P association modules, P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the corresponding P second sub-feature maps output by the second hidden layer are calculated to obtain motion information, including: the first association degree layer calculates the association degree of the first first sub-feature map in the P first sub-feature maps and the first second sub-feature map in the P second sub-feature maps to obtain the first association degree feature map, and inputs the first association degree feature map into the first decoder; the first decoder calculates the first association degree feature map. The feature map is decoded to obtain the first intermediate feature, and the first intermediate feature is input into the second decoder; the i-th correlation layer calculates the correlation between the i-th first sub-feature map and the i-th second sub-feature map to obtain the i-th correlation feature map, and the i-th correlation feature map is input into the i-th decoder; the i-th decoder decodes the i-th correlation feature map to obtain the i-th intermediate feature, and inputs the i-th intermediate feature into the i+1-th decoder, 2≤i≤P-1 and i is an integer; the P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain the P-th correlation feature map, and the P-th correlation feature map is input into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain motion information.

[0028] In a possible embodiment, when P=3, based on P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layers and the P second sub-feature maps output by the corresponding second hidden layers are processed respectively to obtain motion information, including: inputting the cth first sub-feature map output by the cth first hidden layer and the cth second sub-feature map output by the cth second hidden layer into the first association module to obtain the first intermediate feature; inputting the bth first sub-feature map output by the bth first hidden layer and the bth second sub-feature map output by the bth second hidden layer, and the first intermediate feature into the second association module to obtain the second intermediate feature; inputting the ath first sub-feature map output by the ath first hidden layer and the ath second sub-feature map output by the ath second hidden layer, and the second intermediate feature into the third association module to obtain motion information, 1≤a<b≤L-1, c=L, and a, b and c are all integers.

[0029] The above scheme has been experimentally proved that by calculating the correlation based on the feature maps output by the shallower, middle and deepest hidden layers, a better compromise can be achieved between the accuracy and power consumption of the motion estimation model.

[0030] In a possible embodiment, L<10, the L-layer first hidden layer includes L-1 layers of first convolutional layers and 1 layer of first pooling layer, and the L-layer second hidden layer includes L-1 layers of second convolutional layers and 1 layer of second pooling layer; wherein, when L=6, the 6-layer first hidden layer includes 5 layers of first convolutional layers and 1 layer of first pooling layer, and the 6-layer second hidden layer includes 5 layers of second convolutional layers and 1 layer of second pooling layer, the a-th first hidden layer is the 2-th first convolutional layer, the b-th first hidden layer is the 5-th first convolutional layer, the c-th first hidden layer is the first pooling layer, the a-th second hidden layer is the 2-th second convolutional layer, the b-th second hidden layer is the 5-th second convolutional layer, and the c-th second hidden layer is the second pooling layer.

[0031] In this application, the network structure of the motion estimation model is shallower than the network structure of the PC-side neural network, so that it can be deployed on the mobile side and run normally. Experiments have shown that on the mobile side, reducing the number of hidden layers and making the network structure shallower can reduce the computational overhead while ensuring the accuracy of the motion estimation model.

[0032] It is understandable that the number of hidden layers in the current PC-side neural network structure is generally greater than 10. By adopting a deeper network structure, effective high-dimensional features can be extracted, so that sufficient information can be obtained to calculate the relationship between the two images, thereby improving the accuracy of the motion estimation model.

[0033] In a second aspect, a motion estimation method based on a motion estimation model is provided, which is applied to an electronic device, wherein the electronic device is deployed with a first application program, and the motion estimation model is deployed on the electronic device, and the motion estimation model is trained by a motion estimation model training method such as any one of the first aspect and its difficult embodiments, and the method comprises: acquiring two adjacent frames of images from the first application program, and downsampling the two adjacent frames of images respectively to obtain a first image and a second image, wherein the two adjacent frames of images are images constituting a to-be-displayed screen of the first application program; inputting the first image and the second image into the motion estimation model to obtain target motion information of the two adjacent frames of images, comprising: processing the first image based on a first input layer to obtain first image data, and processing the second image based on a second input layer to obtain second image data; extracting features from the first image data based on a first hidden layer to obtain a first feature map, and extracting features from the second image data based on a second hidden layer to obtain a second feature map; processing the first feature map and the second feature map based on an association module to obtain target motion information, wherein the association module is used to calculate the degree of association between the first feature map and the second feature map.

[0034] The above solution first downsamples the image obtained from the game application and then inputs it into the motion estimation model, which can speed up the inference speed and reduce power consumption so that the model can be deployed on the mobile terminal and can meet the frame rate and power consumption requirements of the mobile game scene, realize efficient and real-time motion estimation, and improve rendering effects.

[0035] In addition, by setting the attention weights during the training process, the trained motion estimation model can have different training accuracies for different pixels in the input image. Moreover, by adjusting the attention weights, the training accuracy of the motion estimation model for different pixels can be adjusted. This allows for targeted allocation of computing power during model inference, so that the final displayed image user experience is better when the same computing power is used.

[0036] In a possible embodiment, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

[0037] In a possible embodiment, the first feature map and the second feature map are processed based on an association module to obtain target motion information, including: sampling the first feature map based on the association module to obtain a first sampling feature; sampling the second feature map based on the association module to obtain a second sampling feature; calculating the first sampling feature and the second sampling feature to obtain a correlation feature map between the first feature map and the second feature map; decoding the correlation feature map to obtain target motion information.

[0038] In a possible embodiment, sampling the first feature map based on the association module to obtain the first sampling feature includes: determining the first target feature; taking the first target feature as the center block and sampling a sampling range of K×K blocks to obtain the first sampling feature; sampling the second feature map based on the association module to obtain the second sampling feature includes: determining the second target feature; taking the second target feature as the center block and sampling a sampling range of K×K blocks to obtain the second sampling feature; wherein K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

[0039] The above scheme has been experimentally proven to have almost no effect on the accuracy of the motion estimation model by using a narrower sampling range for sampling on the mobile terminal, but it greatly reduces the complexity of the motion estimation model.

[0040] It is understandable that a wider sampling area can obtain more sufficient information, so that the association module can obtain sufficient information, abstract the motion representation based on the association between features, and thus obtain motion information. However, it will also make the network model more complex and increase the amount of calculation, which will bring considerable pressure to the real-time mobile game scenes, making it difficult to achieve real-time reasoning on the mobile terminal and the power consumption is too high.

[0041] In a possible embodiment, the number of layers of the first hidden layer and the second hidden layer are both L, and the L-layer first hidden layer corresponds to the L-layer second hidden layer one-to-one, the number of association modules is P, and the first feature map and the second feature map are processed based on the association modules to obtain target motion information, including: based on P association modules, respectively processing P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the corresponding P second sub-feature maps output by the second hidden layer to obtain target motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

[0042] In a possible embodiment, the first association module among the P association modules includes the first association degree layer and the first decoder, the i-th association module among the P association modules includes the i-th association degree layer and the i-th decoder, and the P-th association module includes the P-th association degree layer and the P-th decoder. Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the P second sub-feature maps output by the corresponding second hidden layer are calculated to obtain the target motion information, including: the first association degree layer calculates the association degree of the first first sub-feature map in the P first sub-feature maps and the first second sub-feature map in the P second sub-feature maps to obtain the first association degree feature map, and inputs the first association degree feature map into the first decoder; the first decoder calculates the first association degree feature map. The i-th feature map is decoded to obtain the first intermediate feature, and the first intermediate feature is input into the second decoder; the i-th correlation layer calculates the correlation between the i-th first sub-feature map and the i-th second sub-feature map to obtain the i-th correlation feature map, and the i-th correlation feature map is input into the i-th decoder; the i-th decoder decodes the i-th correlation feature map to obtain the i-th intermediate feature, and the i-th intermediate feature is input into the i+1-th decoder, 2≤i≤P-1 and i is an integer; the P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain the P-th correlation feature map, and the P-th correlation feature map is input into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain the target motion information.

[0043] In a possible embodiment, when P=3, based on P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L-layer first hidden layer and the P second sub-feature maps output by the corresponding second hidden layer are processed respectively to obtain target motion information, including: inputting the c-th first sub-feature map output by the c-th first hidden layer and the c-th second sub-feature map output by the c-th second hidden layer into the first association module to obtain the first intermediate feature; inputting the b-th first sub-feature map output by the b-th first hidden layer and the b-th second sub-feature map output by the b-th second hidden layer, and the first intermediate feature into the second association module to obtain the second intermediate feature; inputting the a-th first sub-feature map output by the a-th first hidden layer and the a-th second sub-feature map output by the a-th second hidden layer, and the second intermediate feature into the third association module to obtain target motion information, 1≤a<b≤L-1, c=L, and a, b and c are all integers.

[0044] The above scheme has been experimentally proved that by calculating the correlation based on the feature maps output by the shallower, middle and deepest hidden layers, a better compromise can be achieved between the accuracy and power consumption of the motion estimation model.

[0045] In a possible embodiment, L<10, the L-layer first hidden layer includes L-1 layers of first convolutional layers and 1 layer of first pooling layer, and the L-layer second hidden layer includes L-1 layers of second convolutional layers and 1 layer of second pooling layer; wherein, when L=6, the 6-layer first hidden layer includes 5 layers of first convolutional layers and 1 layer of first pooling layer, and the 6-layer second hidden layer includes 5 layers of second convolutional layers and 1 layer of second pooling layer, the a-th first hidden layer is the 2-th first convolutional layer, the b-th first hidden layer is the 5-th first convolutional layer, the c-th first hidden layer is the first pooling layer, the a-th second hidden layer is the 2-th second convolutional layer, the b-th second hidden layer is the 5-th second convolutional layer, and the c-th second hidden layer is the second pooling layer.

[0046] It is understandable that the number of hidden layers in the current PC-side neural network structure is generally greater than 10. By adopting a deeper network structure, effective high-dimensional features can be extracted, so that sufficient information can be obtained to calculate the relationship between the two images, thereby improving the accuracy of the motion estimation model.

[0047] In this application, the network structure of the motion estimation model is shallower than the network structure of the PC-side neural network, so that it can be deployed on the mobile side and run normally. Experiments have shown that on the mobile side, reducing the number of hidden layers and making the network structure shallower can reduce the computational overhead while ensuring the accuracy of the motion estimation model.

[0048] In a possible embodiment, the first image is a non-predicted image that has been displayed, the second image is a non-predicted image that has not been displayed, and the first image is the previous non-predicted image of the second image in the display timing. The method also includes: performing pixel shifting and filling on the second image and the first image according to target motion information to obtain one or more predicted images; after displaying the one or more predicted images, displaying the second image.

[0049] The above scheme applies the motion estimation model in the interpolation scenario, improves the accuracy of obtaining target motion information, reduces the computational overhead of obtaining target motion information, improves the accuracy of the interpolation process, and reduces the computational overhead of the interpolation process.

[0050] In a third aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0052] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible implementation manner of the first aspect.

[0053] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0054] It can be understood that the electronic device provided in the third aspect, the chip system provided in the fourth aspect, the computer storage medium provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of possible problems in performing motion estimation using a non-artificial intelligence method according to an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application is shown;

[0057] Figure 3 A schematic diagram of the interaction of data flows between a central processing unit (CPU), a graphics processing unit (GPU) and a neural-network processing unit (NPU) provided in an embodiment of the present application;

[0058] Figure 4 A flowchart of a motion estimation method provided in an embodiment of the present application;

[0059] Figure 5 A schematic flow chart of a motion estimation model training method 200 provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of several possible target focus areas provided in the embodiment of the present application including an example of a central area of ​​a second sample image;

[0061] Figure 7 This is a schematic diagram of an example of the sampling area of ​​the current neural network on the PC;

[0062] Figure 8 A schematic diagram of an example of a sampling area of ​​a motion estimation model provided in an embodiment of the present application;

[0063] Fig. 9 A schematic diagram of an example of a training flow chart of a motion estimation model provided in an embodiment of the present application;

[0064] Fig.10 A schematic diagram of an example of a motion estimation method based on a motion estimation model provided in an embodiment of the present application;

[0065] Fig.11 A schematic diagram of an example of a reasoning flow chart of a motion estimation model provided in an embodiment of the present application;

[0066] Fig.12 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0068] As mobile games continue to be updated and iterated, the rendering process of the game engine of mobile devices has become more complex and diverse. For example, the rendering process includes frame insertion (used to increase the frame rate), super-resolution (used to increase the resolution), anti-aliasing (used to eliminate jagged edges of the image) and other processing to improve the user's visual experience when playing games.

[0069] In game scenes, motion estimation can be used to obtain motion information between two adjacent frames. The motion information can be used as prior information in the time domain and used in processing processes such as interpolation, super-resolution, and anti-aliasing to obtain better rendering effects.

[0070] Motion estimation is the process of extracting the key feature points or contours of an object by processing continuous physical image frames, and inferring the motion state of the object based on the position changes of these feature points.

[0071] Current motion estimation technologies can include: two-dimensional (2D) algorithms such as optical flow, motion estimation based on three-dimensional (3D) motion fields, and motion estimation based on deep learning on the PC side.

[0072] First, motion estimation based on 3D motion field requires obtaining 3D information and rotation matrix, and using 3D reprojection to match adjacent frames to calculate the motion information between frames. This method can achieve relatively good results, but it requires a lot of resources and is expensive, and is not suitable for scenes that lack 3D information such as transparent objects and shadows.

[0073] Figure 1 A schematic diagram of possible problems in performing motion estimation using a non-artificial intelligence method provided in an embodiment of the present application. Figure 1 (a) and (b) in FIG. 1 correspond to the method of motion estimation based on the 3D motion field. Figure 1 (a) in the figure is image #1 displayed on the display screen of the electronic device in the game scene. Figure 1 (b) is a visualization image of the motion information of image #1 (actually a color image, the corresponding grayscale image is used in this application), and the ellipse marks the Figure 1 The pixels at the same position in (a) and (b). In the visualization of motion information, the depth of the pixel color represents the value of the pixel's motion speed; the color of the pixel represents the direction of motion, with red and green corresponding to two directions respectively.

[0074] like Figure 1 As shown in (a) in the figure, image #1 is a frame image of a person running, and the shadow of the person on the ground can be seen in the ellipse.

[0075] like Figure 1As shown in (b) of the figure, the figure's body is different from the background color, indicating that the figure is moving. Since the figure's shadow theoretically moves with the figure, Figure 1 In (b), the ellipse does not reflect the motion information of the shadow. Therefore, this motion estimation method will fail when estimating the motion information of the shadow.

[0076] Second, 2D algorithms such as optical flow use 2D image information to calculate motion information based on the matching between pixels. Because only 2D image information is required, this type of method is suitable for most scenarios, but the effect depends on the size of the search area and the computational complexity of the matching. Therefore, in order to achieve good results, it usually requires a considerable computational overhead.

[0077] For example, the optical flow method infers the direction and speed of an object by analyzing the brightness changes of pixels in an image. The optical flow is a two-dimensional vector field that represents the displacement of a point from the first frame to the second frame.

[0078] Figure 1 (c) and (d) in the figure correspond to the motion estimation method based on the optical flow method. Figure 1 (c) in the figure is image #2 displayed on the display screen of the electronic device in the game scene. Figure 1 (d) in FIG. 1 is a visualization image of the motion information of image #2 (actually a color image, the corresponding grayscale image is used in this application), and the ellipse marks the Figure 1 Pixels at the same position in (c) and (d).

[0079] like Figure 1 As shown in (c) in the figure, image #2 is a frame of the character running, and the oval is the background (grass). The character is running, while the background is not moving.

[0080] like Figure 1 As shown in (d), there are pixels of different colors and shades on the grass background within the ellipse. It is understandable that the large grass background, except for the person, does not move. In the motion information visualization diagram, it should be a pure background, and there should be no visualization of motion information. However, since the optical flow method only relies on 2D information, the error will be relatively large, and there will be a lot of motion information on the non-moving background.

[0081] Third, PC-based motion estimation based on deep learning is a data-driven method that can also use only 2D images and achieve better motion estimation results with less computation. However, the current neural network model on the PC is relatively complex, and it is very difficult to run it directly on the mobile terminal, and real-time reasoning cannot be achieved; and even if it can run normally, the data transmission overhead generated during the operation will be very large, and the power consumption will also be very high.

[0082] In summary, among the current motion estimation technologies, non-artificial intelligence (AI) methods have high computational overhead, are not robust, and have limited applicable scenarios (for example, motion information of complex scenes such as translucency, shadows, and special effects is difficult to obtain); the AI ​​method has a complex model structure and is difficult to meet the real-time and low-power requirements of game scenes on mobile terminals.

[0083] Therefore, how to perform motion estimation for game scenes on mobile devices becomes a key issue.

[0084] In view of this, the present application provides a motion estimation model training method to obtain a training data set for a mobile game scene, wherein the training data set includes a first sample image of adjacent frames and its corresponding sample motion information, and trains a motion estimation model based on downsampling the first sample image to obtain a second sample image and a third sample image, as well as the sample motion information.

[0085] It is understandable that downsampling can reduce the image size. Using the downsampled images to train the motion estimation model, and accordingly, using the downsampled images for subsequent motion estimation model reasoning, can speed up the model reasoning and reduce power consumption, so that the model can be deployed on the mobile terminal and can meet the frame rate and power consumption requirements of the mobile terminal game scene, realize efficient and real-time motion estimation, and improve the rendering effect.

[0086] In some embodiments, parameters of the motion estimation model are adjusted using a loss function determined according to attention weights, where the attention weights are used to indicate the user's attention to pixels in the second sample image and the third sample image.

[0087] It can be understood that by setting the attention weight, the trained motion estimation model can have different training accuracy for different pixels in the input image. Moreover, by adjusting the attention weight, the training accuracy of the motion estimation model for different pixels can be adjusted. In this way, the computing power can be allocated in a targeted manner during model inference, so that when the same computing power is used, the final image displayed can provide a better user experience.

[0088] In some embodiments, the sample motion information in the training data set is obtained based on the game scene classification. One type is that when the game application provides the three-dimensional information of the first sample image, the sample motion information can be obtained based on the 3D motion field. Another type is that when the game application does not provide the three-dimensional information of the first sample image, the sample motion information can be obtained based on the optical flow method or the neural network on the PC.

[0089] It can be understood that based on the above analysis of the current motion estimation algorithm, this classification method can obtain more motion information of the game scene as accurately as possible. In order to obtain more accurate motion information, more resources are obtained as much as possible, and a larger computing overhead is provided, so that the training data set is more complete, which will also make the training effect of the motion estimation model better and the training efficiency higher.

[0090] It can also be understood that the motion estimation model obtained by training can obtain motion information based on the input adjacent frame images. Compared with the motion estimation method based on the 3D motion field, which needs to obtain 3D information and rotation matrix, on the one hand, since it does not need to obtain 3D information to obtain motion information, it can be applied to scenes such as transparent objects and shadows where 3D information cannot be obtained; on the other hand, since fewer resources are obtained, the computational overhead is smaller. Compared with the motion estimation method based on the optical flow method, since it does not rely on the search area and matching for motion estimation, the effect of motion estimation is not affected by the size of the search area and the computational complexity of matching.

[0091] The present application also provides a motion estimation method based on a motion estimation model, which is applied to an electronic device that deploys a game application and a motion estimation model. The electronic device first obtains images of adjacent frames from the game application, and then performs downsampling processing to obtain a first image and a second image. The first image and the second image are input into the motion estimation model to obtain target motion information.

[0092] Therefore, the beneficial effects of the motion estimation model inference can be found in the corresponding description of the training process above.

[0093] In some embodiments, the network structure of the motion estimation model is shallower than the network structure of the PC-side neural network so that it can be deployed on the mobile terminal and run normally. Experiments have shown that the shallower network structure can reduce the computational overhead while ensuring the accuracy of the motion estimation model.

[0094] In some embodiments, the motion estimation model includes an association module, which first samples the first image and the second image and calculates a correlation feature map of the first image and the second image. The sampling range of the association module is narrower than the sampling range of the current PC-side neural network. Experiments have shown that using a narrower sampling range for sampling on the mobile side has almost no effect on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0095] In some embodiments, the motion estimation model removes the feature remapping (warp, also known as distortion) operation compared to the neural network on the PC side. Experiments have shown that removing this operation has almost no effect on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0096] With respect to the above three embodiments, it can be seen that the motion estimation model has a streamlined structure while ensuring accuracy, so that real-time reasoning can be performed in mobile game scenes to efficiently obtain motion information.

[0097] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are first introduced.

[0098] 1. Motion information, or it can be called motion vector or motion vector information (this application does not limit the name): the instantaneous velocity vector of the pixel motion of a spatial moving object on the observation imaging plane; or, the distance and direction (including x / y directions) that a spatial moving object moves per pixel unit time on the observation imaging plane; or, the motion relationship between each pixel on adjacent image frames, including information such as position, motion speed, and motion direction; the motion vector between two frames of images is the displacement of a dynamic object obtained by motion estimation of the two frames of images, and both frames of images include the dynamic object.

[0099] 2. Downsampling (also called downsampling, image reduction): In the field of image processing, it refers to reducing the resolution of an image, usually by discarding some pixels. This can reduce the file size, facilitate storage and transmission, and also maintain the image quality to a certain extent.

[0100] 3. Interpolation technology: also known as frame rate conversion technology, is a technology that improves the smoothness of the video by adding one or more frames to every two frames of the original video, shortening the display time between each frame, and correcting the illusion caused by the persistence of human vision.

[0101] 4. Super-resolution: Improving the resolution of the original image through hardware or software methods, and obtaining a high-resolution image or video through a series of low-resolution images or videos is called super-resolution reconstruction. That is, super-resolution rendering is used to improve the resolution of image data. The resolution improvement can be from low resolution to high resolution or ultra-high resolution, or from high resolution to ultra-high resolution.

[0102] 5. Anti-aliasing (AA): also known as anti-aliasing or anti-aliasing, is a technology that eliminates jagged edges on the output image.

[0103] 6. High-dimensional features: usually refers to the situation in which the data set contains a large number of features in machine learning and data analysis. These features may be obtained by transforming or processing the original data. For example, in deep learning, the input data is usually converted into a high-dimensional feature vector. The existence of high-dimensional features enables the model to capture more information and details, thereby improving the model's predictive performance.

[0104] However, high-dimensional features also bring some problems, such as increased computational complexity, possible redundant features, and increased data sparsity.

[0105] 7. Hardware Buffer is a physical memory area used for data transmission between different components or peripherals. It allows the sender to store data in the buffer before the receiver is ready and read it at an appropriate rate. By using the hardware buffer, the speed mismatch between the sender and the receiver can be solved, thereby improving the reliability and performance of the system.

[0106] 8. In computer hardware, shared memory refers to a block of random access memory (RAM) that can be accessed by multiple different CPUs in a multiprocessor computer system. In computer software, shared memory refers to: a method of interprocess communication, that is, a way for simultaneously running programs to exchange data. A process creates an area in RAM that other processes can access.

[0107] 9. Open Graphics Library for Embedded Systems (OpenGL ES) is an application programming interface (API) for embedded systems. In the Android system architecture, developers can use OpenGL ES to create and manipulate two-dimensional and three-dimensional graphics.

[0108] Please refer to Figure 2 , Figure 2 FIG. 1 is a schematic diagram showing a hardware structure of an electronic device 100 provided in an embodiment of the present application. Figure 2 The electronic device 100 may include a processor 110 , and may optionally include a memory 120 and a display screen 130 .

[0109] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0110] The processor 110 may include one or more processing units. For example, the processor 110 may include a CPU 111 , a GPU 112 , and an NPU 113 , etc., and may also include an application processor.

[0111] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0112] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 130. The MIPI interface includes a display serial interface (DSI), etc. In some embodiments, the processor 110 and the display screen 130 communicate via the DSI interface to implement the display function of the electronic device 100.

[0113] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0114] In the motion estimation method provided in the present application, CPU 111 applies for shared memory so that data can be transmitted between GPU 112 and NPU 113. Specifically, the data flow interaction between CPU 111, GPU 112 and NPU 113 can be seen in Figure 3 .

[0115] The electronic device 100 implements the display function through the GPU 112, the display screen 130, and the application processor. The GPU 112 is a microprocessor for image processing, which is connected to the display screen 130 and the application processor. The GPU 112 is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs 112, which execute program instructions to generate or change display information.

[0116] In the motion estimation method provided in the present application, the GPU 112 provides an image for motion estimation, including obtaining an image from a game application and rendering it at a normal resolution, and downsampling the rendered image at a normal resolution to obtain an image after downsampling. Specifically, the data flow interaction between the CPU 111, the GPU 112, and the NPU 113 can be seen in Figure 3 .

[0117] The display screen 130 is used to display images, videos, etc. The display screen 130 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 130, where N is an integer greater than 1.

[0118] NPU 113 is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU 113 can realize applications such as intelligent cognition of electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0119] In the motion estimation method provided in the present application, the NPU 113 calls the motion estimation model to obtain motion information. Specifically, the data flow interaction between the CPU 111, the GPU 112 and the NPU 113 can be seen in Figure 3 .

[0120] The internal memory, i.e., the memory 120, can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data (such as audio data, a phone book, etc.) created by the electronic device 100 during use, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0121] Figure 3 A schematic diagram of the interaction of data flows between the CPU, GPU, and NPU provided in an embodiment of the present application.

[0122] It can be understood that in the present application, the electronic device running the motion estimation model provided in the present application requires data flow interaction between the CPU, GPU and NPU.

[0123] Perform steps A to E in advance.

[0124] In step A, the CPU creates a hardware buffer and applies for shared memory from the Hardware Buffer, or in other words, the CPU creates hardware buffer shared memory. The CPU can apply for shared memory from the Hardware Buffer of the random access memory (RAM). One way to apply for shared memory is that the CPU applies for storage space of the GPU shader storage buffer object (SSBO) from the Hardware Buffer, and after the fragment shader in the GPU or the compute shader in the GPU obtains the downsampled image frame, the downsampled image frame is written to the GPU SSBO. The motion estimation results output by the NPU can also be written to the shared memory.

[0125] Exemplarily, after step A, the CPU notifies the GPU of the pointer address of the shared memory so as to use the shared memory as the storage memory of the downsampled image frame. The CPU notifies the NPU of the pointer address of the shared memory so as to use the shared memory as the storage memory of the motion estimation result.

[0126] Hardware Buffer is a kind of memory. When CPU, GPU and NPU access Hardware Buffer, there are certain requirements for the format of Hardware Buffer. For example, when applying for shared memory as storage space accessible to Fragment Shader in GPU or Compute Shader in GPU, you can specify the format of Hardware Buffer as AHARDWAREBUFFER_FORMAT_BLOB.

[0127] Another way to apply for shared memory is that the CPU can apply for two blocks of shared memory from the Hardware Buffer. One block of shared memory is used to store the downsampled image frame obtained by the GPU as the input data of the NPU; the other block of shared memory is used to store the output data of the NPU, such as the result of motion estimation, as the input data of the GPU during UI rendering. The formats of the two blocks of shared memory applied by the CPU are the same, such as the format of the shared memory can be AHARDWAREBUFFER_FORMAT_BLOB.

[0128] Step B, create an EGLImage.

[0129] Among them, EGLImage represents a type of shared resource created by EGL client API (such as OpenGLQ, OpenVG). Its original intention is to share 2D image data, but it does not clearly define the format of shared data and the purpose of sharing, so in theory, applications and related client APIs can create any type of shared data for any purpose.

[0130] Exemplarily, step B is performed by the GPU. After the GPU creates the EGLImage, the GPU can share data with the CPU and the NPU through data transmission between the EGLImage and the shared memory.

[0131] Step C, GPU binds EGLImage and input texture / output texture (inputTexture / outputTexture).

[0132] Among them, inputTexture corresponds to the downsampled image frame, and outputTexture corresponds to the result of motion estimation.

[0133] Thus, the GPU can share inputTexture / outputTexture based on EGLImage.

[0134] Step D: Binding between GPU and inputTexture / outputTexture is performed based on glBlitFramebuffer function.

[0135] The glBlitFramebuffer function is a function provided by OpenGL that copies a certain area of ​​a framebuffer to another Framebuffer.

[0136] Step E: Binding between GPU and inputTexture / outputTexture is performed based on glBindTexture function.

[0137] glBindTexture is the function used to bind a texture in OpenGL.

[0138] Then, step 1-N and step 2-(N-1) are executed in a loop. Step 1-1 is to input the first frame image into the motion estimation model, and there is no corresponding step 2-(N-1) at this time; after executing step 1-1 and step 1-2, step 2-1 is to output the motion information of the second frame image and the first frame image.

[0139] Step 1-N, input the Nth frame image to the motion estimation model. Figure 3 The current frame image processed by the motion estimation model is the Nth frame, and the previous frame image is the N-1th frame.

[0140] The CPU instructs the NPU to obtain the Nth frame image from the shared memory.

[0141] Then, the motion estimation model performs motion estimation on the Nth frame image and the N-1th frame image.

[0142] Step 2-(N-1), output motion information of the Nth frame image and the N-1th frame image.

[0143] Specifically, steps 1-N and 2-(N-1) can be found in Figure 4 Description related to steps (7) and (8) in .

[0144] Figure 4 A flowchart of a motion estimation method provided in an embodiment of the present application.

[0145] like Figure 4 As shown, the motion estimation method mainly involves four parts: offline training, OPENGLES thread, shared memory and AI computing thread.

[0146] Among them, the OPENGLES thread, shared memory and AI computing thread are implemented on a mobile electronic device that executes the motion estimation method (for the convenience of explanation, referred to as the first electronic device below), and offline training is implemented on another electronic device (for the convenience of explanation, referred to as the second electronic device below).

[0147] The general process is to first use the motion estimation model training method provided in this application on the second electronic device to train the motion estimation model provided in this application, and then deploy the motion estimation model on the first electronic device. For the first electronic device, the OPENGLES thread (executed on the GPU) first transfers the image to be motion estimated to the shared memory, and the AI ​​operation thread (executed on the NPU) uses the motion estimation model to perform motion estimation on the image to be motion estimated, and transfers the motion estimation result to the shared memory. The OPENGLES thread obtains the motion estimation result from the shared memory, and after subsequent processing (such as rendering) on ​​the result, it is transmitted to the display screen for display.

[0148] For the offline training part: the second electronic device performs model training based on the training data set obtained from the game APP of the mobile terminal (which may include the first electronic device) to obtain a motion estimation model, and then deploys the model file of the motion estimation model to the first electronic device. Specifically, the training data set of the motion estimation model and the model training process will be described below in conjunction with Figure 5 Provide a detailed introduction.

[0149] For First Electronic Devices:

[0150] In the OPENGLES thread part, (1) the game application generates an image rendering event, the GPU obtains the game frame (the image frame of the game scene) and triggers the initialization of the AI ​​model (i.e., the motion estimation model).

[0151] For example, the game frames may be a video stream.

[0152] (2) Initialize the AI ​​model (i.e., motion estimation model), which includes runtime checking, model loading, model compilation, and memory configuration.

[0153] The initialization of the motion estimation model includes runtime checking, model loading, model compilation, and memory configuration. The runtime checking is to determine whether to run the motion estimation model in the NPU. Model loading, model compilation, and memory configuration are to ensure that the motion estimation model can run normally.

[0154] In some implementations, the runtime check includes checking whether an NPU is configured in the electronic device and checking the resolution of the application; if an NPU is configured, running a motion estimation model in the NPU; if an NPU is not configured in the electronic device or the resolution of the application is low, prohibiting the running of the motion estimation model, and the CPU sends rendering instructions to the GPU, and the GPU responds to the rendering instructions for rendering.

[0155] Model loading is to convert the motion estimation model into a model file recognizable by the NPU. The model file is loaded into the memory during the initialization phase. Model compilation is to verify that the model file can run successfully. Memory configuration is to allocate memory for the motion estimation model. The allocated memory is used to store the input data and output data of the motion estimation model. In this embodiment, the memory allocated for the motion estimation model can be CPU memory or shared memory.

[0156] (3) Render the game frame at normal resolution to obtain an image at normal resolution.

[0157] It can be understood that, assuming that there are X frames in the game, one frame is rendered each time, and the current normal resolution image is the Nth frame. 1≤N≤X, N and X are both integers.

[0158] (4) Down-sample the image with normal resolution to obtain the current frame (frame N).

[0159] Understandably, the image frame of normal resolution is too large and computationally expensive to achieve real-time reasoning. By downsampling and reducing the image size, the reasoning speed can be accelerated and the power consumption can be reduced during subsequent AI model reasoning, so as to meet the frame rate and power consumption requirements of the game.

[0160] If N is equal to 1, (5) is not executed. If N>1, (5) is executed.

[0161] (5) Wait for the AI ​​computing thread to enter the waiting state.

[0162] It can be understood that when N>1, the AI ​​operation thread needs to process the motion estimation of the N-1 frame image before processing the motion estimation of the N-1 frame image. If the AI ​​operation thread is still processing the N-1 frame image, it has not entered the waiting state and will not execute step (6); if the AI ​​operation thread has processed the N-1 frame image, it will enter the waiting state and can continue to execute step (6). This is to avoid transferring the N-1 frame image to the shared memory before the motion estimation of the N-1 frame image is completed.

[0163] Exemplarily, the GPU may determine whether the NPU enters the waiting state through signaling interaction with the NPU.

[0164] (6) Input the current frame image.

[0165] The GPU writes the current frame image into the shared memory based on the pointer address of the shared memory. The CPU notifies the NPU to read the current frame image and the previous frame image in the shared memory.

[0166] like Figure 4 As shown, the shared memory stores the input current frame (Frame N) image and the input previous frame (Frame N-1) image.

[0167] (7) Input Frame N and Frame N-1 into the AI ​​operation thread. The AI ​​operation thread includes a motion estimation model.

[0168] The NPU reads Frame N and Frame N-1 based on the pointer address of the shared memory.

[0169] It is understandable that after the AI ​​operation thread finishes the motion estimation of Frame N-2 / N-1, it will trigger the AI ​​motion thread to enter the waiting state. After inputting Frame N and Frame N-1 to the AI ​​operation thread in step (7), it needs to receive a wake-up instruction before performing motion estimation.

[0170] (8) Based on the pointer address of the shared memory, the NPU writes the motion estimation result of Frame N-2 / N-1 into the shared memory.

[0171] It should be noted that the Frame N-2 / N-1 motion estimation is understood as performing motion estimation on the N-1th frame and the N-2th frame image, and the motion estimation result of the Frame N-2 / N-1 can be called the motion estimation result of the N-1th frame. The meaning of Frame N-1 / N is similar to this and will not be repeated below.

[0172] The CPU notifies the GPU to read the motion estimation results of Frame N-2 / N-1 in the shared memory.

[0173] (9) The GPU reads the motion estimation result of the previous frame based on the pointer address of the shared memory.

[0174] The present application does not limit the order in which steps (7) and (8) are performed, nor does it limit the order in which steps (7) and (9) are performed.

[0175] (10) The GPU determines to start the AI ​​computing thread.

[0176] (11) The GPU asynchronously starts the AI ​​computing thread, that is, instructs the NPU to wake up the motion estimation of the current frame.

[0177] (12) The AI ​​computing thread wakes up Frame N-1 / N motion estimation.

[0178] (13) The motion estimation model performs Frame N-1 / N motion estimation.

[0179] (14) Post-processing and Frame N-1 user interface (UI) rendering.

[0180] Post-processing is performed based on the motion estimation results of Frame N-2 / N-1, and UI rendering is performed for Frame N-1.

[0181] Exemplarily, if the motion estimation result is applied to an interpolation scene, post-processing includes pixel movement and filling for the motion estimation result; if the motion estimation result is applied to a super-resolution scene, post-processing includes determining the correlation between the N-1th frame and the N-2th frame image according to the motion estimation result of the previous frame, so as to facilitate subsequent super-resolution processing; if the motion estimation result is applied to an anti-aliasing scene, post-processing includes determining the correlation between the N-1th frame and the N-2th frame image according to the motion estimation result of the previous frame, so as to facilitate subsequent anti-aliasing processing.

[0182] The order of the above (14) and (13) is not limited and they can also be executed simultaneously.

[0183] (15) Send Frame N-1 after post-processing and UI rendering to the screen for display.

[0184] It can be understood that the motion estimation result of Frame N-1 / N is also processed with reference to the above process, and Frame N is also sent to the screen for display with reference to the above process, which will not be described in detail.

[0185] It can be understood that after the GPU obtains the motion estimation result of the previous frame, it asynchronously starts the AI ​​computing thread to perform the motion estimation of the current frame. On the one hand, it can reduce the computational pressure of the motion estimation model and improve the inference time efficiency; on the other hand, it can execute step (13) while executing steps (14) and (15), reducing the NPU calculation time, the delay caused by the NPU and GPU communication, and the display transmission process, thereby improving the efficiency of the entire process.

[0186] The above Figure 4 The process of applying the motion estimation method provided by the present application to interpolation, super-resolution or anti-aliasing scenes is introduced. The following introduces the embodiments of the present application from the model training side and the model inference side respectively.

[0187] Figure 5 A schematic flowchart of a motion estimation model training method 200 provided in an embodiment of the present application corresponds to the model training side.

[0188] S201, obtaining a training data set.

[0189] The training data set includes M adjacent frame image groups and M sample motion information, each adjacent frame image group includes two adjacent first sample images, the M sample motion information are motion information of the M adjacent frame image groups, and M is a positive integer.

[0190] Exemplarily, the M adjacent frame image groups may be one or more groups of continuous frames, or one or more video streams.

[0191] For example, M adjacent frame image groups are a group of continuous frames, and the continuous frames include Y frame images. Then, every two adjacent frame images can form an adjacent frame image group, that is, M=Y-1.

[0192] Taking the example that M adjacent frame image groups are 3 video streams (the image frames between the video streams are discontinuous), the first video stream includes A frame images, the second video stream includes B frame images, and the third video stream includes C frame images. Then, every two adjacent frame images among the A frame images, B frame images, and C frame images can form an adjacent frame image group, that is, M=A+B+C-3.

[0193] The first sample image is obtained from the first application on the mobile terminal, and the first sample image is an image constituting the screen to be displayed of the first application. Alternatively, the first sample image is a partial image of all images constituting the screen to be displayed of the first application. Exemplarily, the screen to be displayed here is a dynamic screen, or a screen of a video to be displayed, or a video to be displayed.

[0194] For example, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario. This application mainly uses the game scenario as an example for illustration, but does not limit its protection scope.

[0195] For example, a game application generally refers to various game applications on mobile terminals, and the motion estimation model trained based on the first sample image has good generalization performance. For another example, a game application refers to a certain type of game, such as real-time strategy games, card games, and business simulation games, and the motion estimation model trained based on the first sample image has better motion estimation effect for game frames of the same type of games. For another example, a game application specifically refers to a specific game application, and the motion estimation model trained based on the first sample image has extremely high adaptability to the game application.

[0196] In a possible implementation of S201, taking the first application as a game application as an example, the sample motion information in the training data set is obtained according to the classification of game scenes. The game scenes are roughly divided into two categories, and different methods are used to obtain motion information for different types of game scenes.

[0197] It is understandable that the data set for motion estimation needs to know the motion relationship between each pixel, so the data annotation is very complicated, time-consuming and laborious. The currently used data sets are generally divided into two categories: large synthetic data sets and small real-scene data sets. These data sets are more targeted at autonomous driving or animation rather than mobile game scenes. By constructing a motion estimation data set for mobile game scenes, motion information that is closer to the actual usage scenario (that is, the game scene) can be obtained as a data set, thereby improving the effect of motion estimation.

[0198] Step 1-a, obtaining M adjacent frame image groups and image-related data from a first application.

[0199] Exemplarily, the type of image-related data includes 2D information or 3D information.

[0200] Step 1-b, determining a target method for obtaining motion information according to the type of image-related data.

[0201] The first type of image-related data includes three-dimensional information of a first sample image, and the target method includes three-dimensional reprojection.

[0202] Alternatively, the second category is that the type of image-related data does not include three-dimensional information of the first sample image, and the target method includes a method for motion estimation based on an optical flow method or a neural network on a PC.

[0203] Step 1-c, determining M sample motion information in a targeted manner based on the image-related data.

[0204] Corresponding to the first category, image-related data includes scene depth and coordinate transformation matrix. The motion information of two adjacent frames is calculated as follows:

[0205]

[0206] Among them, v 3D is the motion information of two adjacent frames, VP n-1 and VP n are the coordinate transformation matrices of the nth frame image and the n-1th frame image, respectively, d pre and d cur They are the scene depth values ​​of the n-th frame image and the n-1-th frame image respectively.

[0207] Corresponding to the second category, when drawing translucent objects, shadows, and skill effects, since it is impossible to obtain 3D information such as scene depth and coordinate transformation matrix from the game application, it is not possible to use the 3D information to match pixels between frames through the calculation method of the first category. Image-related data includes 2D pixel information, and the motion information of two adjacent frames is calculated based on 2D pixels. At this time, motion information is obtained through optical flow calculation algorithm or PC-side neural network estimation.

[0208] Based on the above analysis of the current motion estimation algorithm, this classification method can obtain more motion information of the game scene as accurately as possible. In order to obtain more accurate motion information, more resources are obtained as much as possible, and greater computing overhead is provided, so that the training data set is more complete, which will also make the training effect of the motion estimation model better and the training efficiency higher.

[0209] S202 : Down-sampling the two first sample images included in the mth adjacent frame image group among the M adjacent frame image groups to obtain corresponding second sample images and third sample images.

[0210] Wherein, 1≤m≤M and m is an integer, and the motion information of the mth adjacent frame image group is the mth sample motion information among the M sample motion information.

[0211] It is understandable that downsampling can reduce the image size. Using the downsampled image to train the motion estimation model, and accordingly, the subsequent motion estimation model reasoning also uses the downsampled image for reasoning, which can speed up the reasoning speed and reduce power consumption, so as to meet the frame rate and power consumption requirements of the game.

[0212] It can be understood that each time the first application displays a frame of an image, the electronic device needs to draw, render, synthesize and display the frame of image before the image can be displayed on the display screen. When the application generates a rendering event, the CPU of the electronic device can obtain the image resource to be rendered to perform the drawing, rendering and synthesis and display operations of the current real image.

[0213] Exemplarily, the first sample image is a non-prediction image (that is, a real image), or the second sample image and the third sample image are non-prediction images (that is, real images).

[0214] S203, inputting the second sample image and the third sample image into the motion estimation model; S204, outputting motion information by the motion estimation model.

[0215] S205 , obtaining a loss function value according to the difference between the motion information and the motion information of the m th sample.

[0216] Exemplarily, the loss function value is obtained according to the difference between the motion information and the m-th sample motion information, and the attention weight, wherein the attention weight is the weight of the pixels at corresponding positions in the second sample image and the third sample image, and the attention weight is used to indicate the user's attention to the pixels at the corresponding positions.

[0217] The pixels at corresponding positions in the second sample image and the third sample image may be understood as the pixels at the same position in two frames of the second sample image.

[0218] It can be understood that, taking the second sample image as an example, the user's attention to the pixels here is actually the user's attention to the pixels displayed on the display screen, and the pixels displayed on the display screen (the first sample image before the second sample image is down-sampled, and the corresponding pixels of the image after being displayed) and the pixels of the second sample image frame correspond one to one, so it can be understood as the user's attention to the pixels of the second sample image frame.

[0219] It can also be understood that the level of attention can also be understood as the level of sensitivity, or the level of obviousness perceived by the user.

[0220] Exemplarily, the loss function is as follows:

[0221]

[0222] Where l is the number of feature layers or the lth layer in the L-layer hidden layer in the motion estimation model, α l is the weight of the l-th layer feature or the weight of the l-th hidden layer, F l (x) is the motion estimation value of pixel x corresponding to the feature of the lth layer, is the reference value corresponding to the l-th layer feature, that is, the sample motion information in the training data set, ω x is the attention weight of pixel x, where pixel x is the pixel in the image input to the motion estimation model.

[0223] It is understandable that since the screen size of the mobile terminal is smaller than that of the PC terminal, and the area that the human eye focuses on during the game (for the sake of convenience, referred to as the target focus area below) is actually smaller (such as the main character, skills, etc.), the rest of the area (such as large-scale background, UI icons, etc.) is not paid much attention to. Therefore, in this case, if the motion estimation of each pixel is trained with the same high precision requirement, the motion estimation model obtained by the training will require greater computational overhead and power consumption during operation; if the motion estimation of each pixel is trained with the same lower precision requirement in order to save computational overhead, it is easy to cause the area where the user focuses on to have low precision, thereby reducing the user experience.

[0224] In the present application, the parameters of the motion estimation model are adjusted based on the attention weight of the pixel, so that a better user experience can be obtained at the same power consumption.

[0225] It can be understood that by setting the attention weight, the trained motion estimation model can have different training accuracy for different pixels in the input image. Moreover, by adjusting the attention weight, the training accuracy of the motion estimation model for different pixels can be adjusted. In this way, the computing power can be allocated in a targeted manner during model inference, so that when the same computing power is used, the final image displayed can provide a better user experience.

[0226] In a possible implementation, pixels in the target focus area among the pixels at the corresponding positions are determined as target pixels, and the attention weight of the target pixel is greater than the attention weights of other pixels (pixels in the corresponding positions except the target pixel).

[0227] Exemplarily, the attention weight of the target attention area is 0.5, and the attention weight of other areas outside the target attention area is 0.3.

[0228] Optionally, the target attention area is further subdivided into different areas according to the level of attention, and each area corresponds to a different attention weight. For example, the target attention area includes a first attention area and a second attention area, and the attention weight of the first attention area is greater than that of the second attention area.

[0229] Exemplarily, the attention weights of other areas outside the target attention area are 0.3, and the attention weight of the target attention area is greater than 0.3, wherein the attention weight of the first attention area is 0.5, and the attention weight of the second attention area is 0.4.

[0230] The target focus area includes the central area (or central point, or a circle with the central point as the center and the first threshold as the radius) of the second sample image and the third sample image. The second sample image is used as an example for description.

[0231] For example, the training data set of the game scene is obtained from various game applications on the mobile terminal. During the training process, the target focus area can be determined as the central area.

[0232] For another example, the training data set of the game scene is obtained from a certain type of game on the mobile terminal, and the target focus area can be adjusted accordingly according to the characteristics of the game.

[0233] Figure 6 A schematic diagram of an example in which several possible target focus areas provided for an embodiment of the present application include a central area of ​​a second sample image.

[0234] Understandably, for the sake of convenience, Figure 6 In the examples, the target focus area is shown by taking the image after the second sample image is displayed as an example. As mentioned above, the pixels of the image after the second sample image is displayed correspond to the pixels of the second sample image frame one by one. Figure 6 By way of example, the target focus area of ​​the second sample image can also be reflected.

[0235] like Figure 6 As shown in (a), taking a business simulation game as an example (such as a city building game), in the image after the second sample image #1 is displayed, the user generally focuses on the area in the center of the display interface to view the situation of the business simulation, such as the target focus area #1.

[0236] like Figure 6 As shown in (b), taking a single-player game as an example (such as a racing game), in the image after the second sample image #2 is displayed, the user generally focuses on the area in the center of the display interface and the area slightly above the center to view the currently operated device and the situation in front, such as the target focus area #2.

[0237] like Figure 6 As shown in (c) in the figure, taking a role-playing game as an example (for example, a player plays a game character and explores a virtual world, etc.), in the image after the second sample image #3 is displayed, the user generally pays attention to the character played and its surroundings. Since the character played is generally close to the central area, for example, the target focus area #3 includes the character played and the central area.

[0238] For another example, the training data set of a game scene is obtained from a specific game application on a mobile terminal, and the target focus area can be adjusted accordingly according to the characteristics of the game.

[0239] For example, semantic recognition can be added to the motion estimation model to identify the characteristics of the game screen on the display interface, and determine the attention weight of the screen based on the characteristics of the screen. For example, in a specific role-playing game, the current character is identified, and the area with high user attention is determined based on the character's behavioral characteristics (such as walking, jumping, etc., such as the skills and related special effects, etc.), and the relevant attention weight is determined.

[0240] S206: Adjust parameters that need to be trained in the motion estimation model according to the loss function value.

[0241] It is understandable that downsampling can reduce the image size. Using the downsampled image to train the motion estimation model, and accordingly, the subsequent motion estimation model inference also uses the downsampled image for inference, which can speed up the inference speed and reduce power consumption, so as to meet the frame rate and power consumption requirements of the game, and realize efficient and real-time motion estimation in mobile game scenes, improve rendering effects, and enhance user experience.

[0242] Part of the process in the motion estimation model training process, namely process #1 includes S203 and S204.

[0243] As a possible implementation of process #1, process #2 includes the following steps.

[0244] S203a, processing the second sample image based on the first input layer to obtain first sample image data.

[0245] S203b: Process the third sample image based on the second input layer to obtain second sample image data.

[0246] It can be understood that S203a and S203b are possible specific examples of S203.

[0247] S205a, extracting features from the second sample image data based on the first hidden layer; S206a, the first hidden layer outputs a first feature map.

[0248] For example, the first hidden layer includes one or more first convolutional layers and one or more first pooling layers.

[0249] S205b, extracting features of the third sample image data based on the second hidden layer; S206b, the second hidden layer outputs a second feature map.

[0250] For example, the second hidden layer includes one or more second convolutional layers and one or more second pooling layers.

[0251] S207, processing the first feature map and the second feature map based on the association module; S204a, the association module outputs motion information.

[0252] It can be understood that S204a is a possible specific example of S204.

[0253] In a possible implementation, the association module includes an association layer and a decoder.

[0254] S207a, the correlation layer calculates the correlation between the first feature map and the second feature map, and outputs a correlation feature map.

[0255] S207b, the decoder is used to decode the correlation feature map; S204b, the decoder outputs motion information.

[0256] It can be understood that S204b is a possible specific example of S204, and S207a and S207b are possible specific examples of S207.

[0257] The motion estimation model provided by this application is described in detail from the perspective of model structure.

[0258] Model structure 1: The network structure of the motion estimation model is shallower than the network structure of the current PC-side neural network. For example, compared with the current PC-side neural network, the motion estimation model has fewer hidden layers and a more streamlined decoder structure (for example, the decoder includes fewer convolutional layers and a more streamlined activation function).

[0259] Exemplarily, the number of the first hidden layer and the number of the second hidden layer are both L, and L<10. For example, L can be 5, 6, or 7. Preferably, when L=5, experiments have shown that the accuracy and performance of the motion estimation model achieve the best compromise.

[0260] It is understandable that the number of hidden layers in the current PC-side neural network structure is generally greater than 10. By adopting a deeper network structure, effective high-dimensional features can be extracted, so that sufficient information can be obtained to calculate the relationship between the two images, thereby improving the accuracy of the motion estimation model.

[0261] In this application, the network structure of the motion estimation model is shallower than the network structure of the PC-side neural network, so that it can be deployed on the mobile side and run normally. Experiments have shown that on the mobile side, reducing the number of hidden layers and making the network structure shallower can reduce the computational overhead while ensuring the accuracy of the motion estimation model.

[0262] Model structure 2: The network structure of the motion estimation model has a narrower sampling area of ​​the association module (specifically, the association layer) compared to the current neural network on the PC side.

[0263] As a possible implementation of S207, the association module samples the first feature map and the second feature map to obtain corresponding sampling features, respectively, and performs calculations based on the sampling features to obtain a correlation feature map.

[0264] For example, in step 2-a, the first feature map is sampled based on the association module to obtain the first sampling feature; in step 2-b, the second feature map is sampled based on the association module to obtain the second sampling feature; in step 2-c, the first sampling feature and the second sampling feature are calculated to obtain the correlation feature map between the first feature map and the second feature map.

[0265] As a possible implementation of step 2-a, step 3-a, determining a first target feature; step 3-b, taking the first target feature as a central block, sampling a sampling range of K×K blocks to obtain a first sampling feature, wherein K<7 and K is a positive integer.

[0266] Similarly, as a possible implementation of step 2-b, step 4-a determines the second target feature; step 4-b takes the second target feature as the center block and samples K×K blocks to obtain the second sampled feature. Where K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

[0267] It can be understood that each block of the first feature map and the second feature map is sampled, that is, each block in the first feature map can be the first target feature, and each block in the second feature map can be the second target feature.

[0268] Figure 7 A schematic diagram showing an example of the sampling area of ​​a neural network on the current PC.

[0269] Figure 7 In (a), (b) and (c), 9×9 blocks (including white blocks and dark gray blocks) are taken as examples to represent the features to be sampled, blocks including black dots are taken as examples to represent target features, white blocks are taken as examples to represent features within the sampling range, and dark gray blocks are taken as examples to represent features outside the sampling range.

[0270] like Figure 7 As shown in (a), sampling is performed with the target feature as the central block, the sampling range is 9×9, and the number of sampling blocks is 81.

[0271] like Figure 7 As shown in (b), sampling is performed with the target feature as the central block, the sampling range is 7×7, and the number of sampling blocks is 49.

[0272] like Figure 7 As shown in (c), sampling is performed with the target feature as the central block, the sampling range is 9×9, and the number of sampling blocks is 53.

[0273] It is understandable that a wider sampling area can obtain more sufficient information, so that the association module can obtain sufficient information, abstract the motion representation based on the association between features, and thus obtain motion information. However, it will also make the network model more complex and increase the amount of calculation, which will bring considerable pressure to the real-time mobile game scenes, making it difficult to achieve real-time reasoning on the mobile terminal and the power consumption is too high.

[0274] In addition, in some cases, compared with Figure 7The sampling method shown in (b) is as follows: Figure 7 The sampling method shown in (c) in Figure 1 has a higher accuracy of motion estimation on the PC side. However, experiments have shown that these two sampling methods have almost no effect on the accuracy of the motion model on the mobile side. Figure 7 The calculation of the sampling method shown in (c) is more complicated, the amount of calculation increases, and the power consumption is greater. Therefore, a sampling area with a regular shape is preferred on the mobile terminal.

[0275] Figure 8 A schematic diagram of an example of a sampling area of ​​a motion estimation model provided in an embodiment of the present application.

[0276] Figure 8 In the figure, a 9×9 block (including a white block and a dark gray block) is taken as an example to represent the features to be sampled, a block including black dots is taken as an example to represent the target features, a white block is taken as an example to represent the features within the sampling range, and a dark gray block is taken as an example to represent the features outside the sampling range.

[0277] For example, if the feature map to be sampled is a 12×12 block, each block in the feature map to be sampled is sampled as a target feature, that is, each block is used as the center point to sample the features within the sampling range. Figure 8 The schematic diagram given is not a schematic diagram of the feature map.

[0278] like Figure 8 As shown, sampling is performed with the target feature as the central block, the sampling range is 5×5, and the number of sampling blocks is 25.

[0279] As shown above, in this application, K<7 and K is a positive integer. Experiments have shown that using a narrower sampling range for sampling at the mobile terminal has almost no effect on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0280] It should be noted that K=5 is only an example, and K may have other values, such as 4 or 6 or other integers, without limitation. K=5 is the preferred value verified by experiments in this application. When K=5, the accuracy and performance of the motion estimation model achieve the best compromise.

[0281] It should be noted that the sampling area involved in the present application may also be referred to as a sampling range. The smaller or narrower the sampling area or sampling range is, the fewer the number of sampling features is.

[0282] Model structure three: Compared with the current PC-side neural network, the network structure of the motion estimation model removes the feature remapping (warp, also called distortion) operation.

[0283] In the current PC-side neural network, the warp operation is used to perform feature mapping in the neural network, that is, mapping the features of one image to the features of another image, and further mining the correlation information between the two images. Experiments have shown that removing this operation has almost no effect on the accuracy of the motion estimation model, but greatly reduces the complexity of the motion estimation model.

[0284] In summary, it can be seen from one or more of the above model structures one to three that the accuracy of the motion estimation model is guaranteed while reducing the complexity of the model, making the motion estimation model effective and concise, so that real-time inference can be performed in mobile game scenarios to estimate motion information.

[0285] The motion estimation model provided in the present application is described in detail below from the aspect of the computational logic between network layers within the motion estimation model.

[0286] The network layer includes a hidden layer and an associated module, wherein the hidden layer includes a convolutional layer and a pooling layer; and the associated module includes an associated layer and a decoder.

[0287] First, let's introduce the calculation logic between the hidden layer and the associated module. Fig.10 Reasoning process #2 in the figure for better understanding.

[0288] The number of the first hidden layer and the second hidden layer are both L, and the L first hidden layers correspond to the L second hidden layers one by one. The number of associated modules is P.

[0289] In a possible implementation of S207 and S204a, step 5-a, based on P association modules, processes the P first sub-feature maps output by part or all of the first hidden layers in the L layers, and the P second sub-feature maps output by the corresponding second hidden layers, to obtain motion information. The P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

[0290] It can be understood that when P=L, the first sub-feature map calculated by each first hidden layer in the L-layer first hidden layer and the second sub-feature calculated by the corresponding second hidden layer in the L-layer second hidden layer are both processed based on an association module.

[0291] That is to say, the calculation result of the first hidden layer of the lth layer in the first hidden layer of the Lth layer is not only output to the first hidden layer of the l+1th layer, but also to the lth associated module, 1≤l≤L and l is an integer.

[0292] Alternatively, when P<L, there are P first hidden layers in the L first hidden layers, and each first hidden layer in the P first hidden layers calculates a first sub-feature map, and the second sub-feature calculated by the corresponding second hidden layer in the L second hidden layer is processed based on an association module.

[0293] That is to say, the calculation result of the pth first hidden layer in the Pth first hidden layer is not only output to the next first hidden layer in the Lth first hidden layer, but also to the pth associated module, 1≤p≤P and p is an integer.

[0294] As a possible implementation of S207a, the correlation layer may calculate the correlation by referring to the following formula:

[0295]

[0296] Wherein, l represents the first hidden layer of the first hidden layer of L layers, or the second hidden layer of the second hidden layer of L layers; f1 l The feature of the first hidden layer of the second sample image at the lth layer, or the first sampling feature (including K×K blocks #1) obtained by sampling based on the K×K sampling range based on the feature calculated for the first hidden layer of the second sample image at the lth layer; represents the feature of the third sample image in the second hidden layer of the lth layer, or the feature calculated for the third sample image in the second hidden layer of the lth layer, and the second sampling feature (including K×K blocks #2) obtained by sampling based on the above K×K sampling range; N l is the characteristic length of the second hidden layer or the first hidden layer of the lth layer.

[0297] Understandable, It means that the K×K blocks #1 included in the first sampling feature are multiplied by the K×K blocks #2 included in the second sampling feature, and the feature map obtained by the multiplication includes K×K blocks #3.

[0298] Preferably, the feature graphs outputted by the shallower, middle and deepest hidden layers in the first hidden layer of the L layer or the second hidden layer of the L layer can be processed based on the association module. Experiments have shown that by calculating the correlation degree based on the feature graphs outputted by the shallower, middle and deepest hidden layers, the accuracy and power consumption of the motion estimation model can achieve a better compromise.

[0299] Exemplarily, when P=3, in one implementation of step 5-a, in step 6-a, the cth first sub-feature map output by the first hidden layer of the cth layer and the cth second sub-feature map output by the second hidden layer of the cth layer are input into the first association module to obtain the first intermediate feature; in step 6-b, the bth first sub-feature map output by the first hidden layer of the bth layer and the bth second sub-feature map output by the second hidden layer of the bth layer, and the first intermediate feature are input into the second association module to obtain the second intermediate feature; in step 6-c, the ath first sub-feature map output by the first hidden layer of the ath layer and the ath second sub-feature map output by the second hidden layer of the ath layer, and the second intermediate feature are input into the third association module to obtain motion information, 1≤a<b≤L-1, c=L, 2≤P≤L and P and L are all integers, and a, b and c are all integers.

[0300] Preferably, when L=6 and the first hidden layer of L includes 5 convolutional layers and 1 pooling layer, the first hidden layer of the ath layer is the second convolutional layer, the first hidden layer of the bth layer is the fifth convolutional layer, and the first hidden layer of the cth layer is the pooling layer. Fig. 9 and related descriptions.

[0301] The calculation logic between multiple related modules is given below.

[0302] In one implementation of step 5-a, case 1, for the first association module among the P association modules, includes the first association layer and the first decoder. In the first association module, the input of the first association layer is the first first sub-feature map among the P first sub-feature maps and the first second sub-feature map among the P second sub-feature maps, and the output is the first association feature map; the input of the first decoder is the first association feature map, and the output is the first intermediate feature.

[0303] Case 2: For the i-th association module among the P association modules, 2≤i≤P-1 and i is an integer, the i-th association module includes the i-th association layer and the i-th decoder. The i-th association layer calculates the association between the i-th first sub-feature map among the P first sub-feature maps and the i-th second sub-feature map among the P second sub-feature maps to obtain the i-th association feature map, and inputs the i-th association feature map into the i-th decoder; the input of the i-th decoder includes the i-th association feature map and the i-1th intermediate feature output by the i-1th decoder, the i-th decoder decodes the i-th association feature map and the i-1th intermediate feature to obtain the i-th intermediate feature, and inputs it into the i+1th decoder.

[0304] Case 3: For the Pth association module, the Pth association module includes a Pth association layer and a Pth decoder. The Pth association layer calculates the association between the Pth first sub-feature map and the Pth second sub-feature map to obtain a Pth association feature map, and inputs the Pth association feature map into the Pth decoder;

[0305] The Pth decoder decodes the Pth correlation feature map and the P-1th intermediate feature to obtain motion information.

[0306] For details, please refer to Fig. 9 A schematic diagram showing three associated modules as an example.

[0307] Fig. 9 The training flowchart 300 of the motion estimation model provided in the embodiment of the present application corresponds to the model training side.

[0308] like Fig. 9 As shown, the motion estimation module includes 6 first hidden layers and 6 second hidden layers, and 3 association modules.

[0309] Among them, taking L=6 as an example, the L-layer first hidden layer includes 5 first convolutional layers and one first pooling layer, and the L-layer second hidden layer includes 5 second convolutional layers and 1 second pooling layer.

[0310] The calculation logic between the hidden layer and the association module is taken as an example of P=3, a=2, b=5, c=6. The second first sub-feature map output by the first convolution layer of the second layer and the second second sub-feature map of the second convolution layer of the second layer are input to the association layer #3 of the third association module; the fifth first sub-feature map output by the first convolution layer of the fifth layer and the fifth second sub-feature map of the second convolution layer of the fifth layer are input to the association layer #2 of the second association module; the sixth first sub-feature map output by the first pooling layer and the sixth second sub-feature map of the second pooling layer are input to the association layer #1 of the first association module.

[0311] It should be noted that L=6 is only an example, and L may have other values, such as L=7 or 8. In the case of L=6, the values ​​of a, b and c are only examples, and a, b and c may have other values, such as a=1, b=4, c=6, or a=2, b=4, c=6, etc.

[0312] Among them, the calculation logic between multiple association modules takes P=3 as an example. The input of the decoder #1 of the first association module (it should be noted that the decoder #x involved in this application is also referred to as the xth decoder, x≥1 and x is an integer. Similarly, the association layer #x, the association feature map #x and other expressions are the same, and are uniformly explained here without repeating them) is the association feature map #1 output by the association layer #1, and the output is the first intermediate feature, and the first intermediate feature is input into the decoder #2 of the second association module; the input of the decoder #2 of the second association module is the association feature map #2 output by the association layer #2 and the first intermediate feature, and the output is the second intermediate feature, and the second intermediate feature is input into the decoder #3 of the third association module; the input of the decoder #3 of the third association module is the association feature map #3 output by the association layer #3 and the second intermediate feature, and the output is motion information.

[0313] S301a, inputting the second sample image into the first input layer, and processing the second sample image based on the first input layer to obtain first sample image data.

[0314] S301b, inputting the third sample image into the second input layer, and processing the third sample image based on the second input layer to obtain second sample image data.

[0315] S302a, input the first sample image data into the first convolution layer of the first layer, the calculation result of the first convolution layer of the first layer is input into the first convolution layer of the second layer, the calculation result of the first convolution layer of the second layer (the second first sub-feature map) is input into the first convolution layer of the third layer and the third association module, the calculation result of the first convolution layer of the third layer is input into the first convolution layer of the fourth layer, the calculation result of the first convolution layer of the fourth layer is input into the first convolution layer of the fifth layer, and the calculation result of the first convolution layer of the fifth layer (the fifth first sub-feature map) is input into the first pooling layer.

[0316] S302b, input the second sample image data into the second convolution layer of the first layer, the calculation result of the second convolution layer of the first layer is input into the second convolution layer of the second layer, the calculation result of the second convolution layer of the second layer (the second second sub-feature map) is input into the second convolution layer of the third layer and the third association module, the calculation result of the second convolution layer of the third layer is input into the second convolution layer of the fourth layer, the calculation result of the second convolution layer of the fourth layer is input into the second convolution layer of the fifth layer, and the calculation result of the second convolution layer of the fifth layer (the fifth second sub-feature map) is input into the second pooling layer.

[0317] S303a, the calculation result of the first pooling layer (the sixth first sub-feature map) is output to the first association module.

[0318] S303b, the calculation result of the second pooling layer (the sixth second sub-feature map) is output to the first association module.

[0319] S304, the correlation layer #1 calculates the correlation for the sixth first sub-feature graph and the sixth second sub-feature graph to obtain a correlation feature graph #1, and inputs it to the decoder #1.

[0320] S305, decoder #1 decodes the correlation feature map #1 to obtain the first intermediate feature, and inputs it to decoder #2.

[0321] S306a, the calculation result of the first convolutional layer of the fifth layer (the fifth first sub-feature map) is also input into the second association module.

[0322] S306b, the calculation result of the 5th second convolutional layer (the 5th second sub-feature map) is also input into the 2nd association module.

[0323] S307, the correlation layer #2 calculates the correlation between the fifth first sub-feature graph and the fifth second sub-feature graph to obtain a correlation feature graph #2, and inputs it to the decoder #2.

[0324] S308, decoder #2 decodes the correlation feature map #2 to obtain the second intermediate feature, and inputs it to decoder #2.

[0325] S309a, the calculation result of the first convolutional layer of the second layer (the second first sub-feature map) is also input into the third association module.

[0326] S309b, the calculation result of the second convolutional layer of the second layer (the second second sub-feature map) is also input into the third association module.

[0327] S310, the correlation layer #3 calculates the correlation between the second first sub-feature graph and the second second sub-feature graph to obtain a correlation feature graph #3, and inputs it to the decoder #3.

[0328] S311, decoder #3 decodes the correlation feature map #3 and outputs motion information.

[0329] For S312 and S313, please refer to the description related to S205 and S206.

[0330] The beneficial effects of the inference flowchart 300 can be found in the corresponding description of the method 200 and will not be elaborated herein.

[0331] Fig.10 A schematic diagram of a motion estimation method 400 based on a motion estimation model provided in an embodiment of the present application, corresponding to the model inference side.

[0332] It is understandable that the electronic device is deployed with a game application (which may be the same as or different from the game application mentioned in method 200, and is not limited in this application), and the electronic device is also deployed with a motion estimation model trained by method 200.

[0333] like Fig.10 As shown, the method 400 includes the following steps.

[0334] S401, acquiring two adjacent frames of images from a first application.

[0335] The two adjacent frames of images are images constituting the to-be-displayed screen of the first application.

[0336] Exemplarily, the first application is a game application, and the training data set is obtained from the game application in a game scenario; or, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; or, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

[0337] S402 , down-sampling two adjacent frames of images to obtain a first image and a second image.

[0338] It can be understood that each time the first application displays a frame of an image, the electronic device needs to draw, render, synthesize and display the frame of image before the image can be displayed on the display screen. When the application generates a rendering event, the CPU of the electronic device can obtain the image resource to be rendered to perform the drawing, rendering and synthesis and display operations of the current real image.

[0339] Exemplarily, the first image is a displayed non-predicted image (that is, a real image), the second image is a non-predicted image that is not displayed, and the first image is the previous non-predicted image of the second image in the display sequence. Alternatively, image #1 before the first image is down-sampled is a displayed non-predicted image, image #2 before the second image is down-sampled is a non-predicted image that is not displayed, and image #1 is the previous non-predicted image of image #2 in the display sequence.

[0340] S403: Input the first image and the second image into a motion estimation model.

[0341] S404: The motion estimation model outputs target motion information of two adjacent frames of images.

[0342] The reasoning process #1 of the motion estimation module includes S403 and S404. As a possible implementation of the reasoning process #1, the reasoning process #2 includes the following steps.

[0343] S403a, processing the first image based on the first input layer; S405a, the first input layer outputs the first image data.

[0344] S403b, processing the second image based on the second input layer; S405a, the second input layer outputs the second image data.

[0345] It can be understood that S403a and S403b are possible specific examples of S403.

[0346] S405a, further comprising inputting the first image data into a first hidden layer, and performing feature extraction on the first image data based on the first hidden layer; S406a, the first hidden layer outputs a first feature map.

[0347] For example, the first hidden layer includes one or more first convolutional layers and first pooling layers.

[0348] S405b, further comprising inputting the second image data into a second hidden layer, and performing feature extraction on the second image data based on the second hidden layer; S406b, the second hidden layer outputs a second feature map.

[0349] For example, the second hidden layer includes one or more layers of a second convolutional layer and a second pooling layer.

[0350] S407, processing the first feature map and the second feature map based on the association module; S404a, the association module outputs the target motion information.

[0351] It can be understood that S404a is a possible specific example of S404.

[0352] In a possible implementation, the association module includes an association layer and a decoder.

[0353] S407a, the correlation layer calculates the correlation between the first feature map and the second feature map, and outputs a correlation feature map.

[0354] S407b, the decoder is used to decode the correlation feature map; S404b, the decoder outputs the target motion information.

[0355] It can be understood that S404b is a possible specific example of S404, and S407a and S407b are possible specific examples of S407.

[0356] The above solution first downsamples the image obtained from the game application and then inputs it into the motion estimation model, which can speed up the inference speed and reduce power consumption so that the model can be deployed on the mobile terminal and can meet the frame rate and power consumption requirements of the mobile game scene, realize efficient and real-time motion estimation, and improve rendering effects.

[0357] During the training process, by setting the attention weight, the trained motion estimation model can have different training accuracies for different pixels in the input image. Moreover, by adjusting the attention weight, the training accuracy of the motion estimation model for different pixels can be adjusted. Therefore, the computing power can be allocated in a targeted manner during model inference, so that when the same computing power is used, the final displayed image user experience is better.

[0358] S401 to S407b in method 400 may correspond to the description of S201 to S207b in method 200 on the model training side, except that:

[0359] (1) The parameters in the motion estimation model on the model inference side are the parameters that have been updated after the loss function value is back-propagated, or the trained parameters. The parameters in the motion estimation model on the model training side are the parameters to be updated during the training process.

[0360] (2) The first feature map in method 200 is the first feature map of the second sample image, and the first feature map in method 400 is the first feature map of the first image, which are different. The feature map and other data obtained in the intermediate calculation process of the motion estimation model are similar and will not be described in detail below.

[0361] (3) Compared with method 200, in method 400, the first sample image, the second sample image, the first sample image data, the second sample image data and the motion information are replaced by the first image, the second image, the first image data, the second image data and the target motion information, respectively.

[0362] (4) Model inference side: refer to the description of model training side Fig. 9 Related content, replaced by see Fig.11 Related content (with Fig. 9 For example, refer to the description of the model training side. Fig. 9 The schematic diagram of 3 associated modules is replaced by, see Fig.11 The schematic diagram of 3 associated modules is taken as an example. Other replacements are similar and will not be described in detail.

[0363] Similarly, the structure of the motion estimation model may also refer to the corresponding description of the model training side above.

[0364] Similarly, the computational logic between network layers within the motion estimation model may also refer to the corresponding description of the model training side above.

[0365] Similarly, the calculation logic between multiple associated modules can also refer to the corresponding description of the model training side above.

[0366] Fig.11The inference flowchart 500 of the motion estimation model provided in the embodiment of the present application corresponds to the model inference side.

[0367] like Fig.11 As shown, the motion estimation module includes 6 first hidden layers and 6 second hidden layers, and 3 association modules.

[0368] Among them, taking L=6 as an example, the L-layer first hidden layer includes 5 first convolutional layers and one first pooling layer, and the L-layer second hidden layer includes 5 second convolutional layers and 1 second pooling layer.

[0369] The calculation logic between the hidden layer and the association module is taken as an example of P=3, a=2, b=5, c=6. The second first sub-feature map output by the first convolution layer of the second layer and the second second sub-feature map of the second convolution layer of the second layer are input to the association layer #3 of the third association module; the fifth first sub-feature map output by the first convolution layer of the fifth layer and the fifth second sub-feature map of the second convolution layer of the fifth layer are input to the association layer #2 of the second association module; the sixth first sub-feature map output by the first pooling layer and the sixth second sub-feature map of the second pooling layer are input to the association layer #1 of the first association module.

[0370] It should be noted that L=6 is only an example, and L may have other values, such as L=7 or 8. In the case of L=6, the values ​​of a, b and c are only examples, and a, b and c may have other values, such as a=1, b=4, c=6, or a=2, b=4, c=6, etc.

[0371] Among them, the calculation logic between multiple association modules takes P=3 as an example. The input of the decoder #1 of the first association module is the association feature map #1 output by the association layer #1, and the output is the first intermediate feature, and the first intermediate feature is input to the decoder #2 of the second association module; the input of the decoder #2 of the second association module is the association feature map #2 output by the association layer #2 and the first intermediate feature, and the output is the second intermediate feature, and the second intermediate feature is input to the decoder #3 of the third association module; the input of the decoder #3 of the third association module is the association feature map #3 output by the association layer #3 and the second intermediate feature, and the output is the target motion information.

[0372] S501a, inputting the first image into the first input layer, and processing the first image based on the first input layer; S502a, the first input layer outputs the first image data.

[0373] S501b, input the second image into the second input layer, and process the second image based on the second input layer; S502b, the first input layer outputs the second image data.

[0374] S502a also includes inputting the first image data into the first convolution layer of the first layer, inputting the calculation result of the first convolution layer of the first layer into the first convolution layer of the second layer, inputting the calculation result of the first convolution layer of the second layer (the second first sub-feature map) into the first convolution layer of the third layer and the third association module, inputting the calculation result of the first convolution layer of the third layer into the first convolution layer of the fourth layer, inputting the calculation result of the first convolution layer of the fourth layer into the first convolution layer of the fifth layer, and inputting the calculation result of the first convolution layer of the fifth layer (the fifth first sub-feature map) into the first pooling layer.

[0375] S502b also includes inputting the second image data into the first second convolution layer, inputting the calculation result of the first second convolution layer into the second second convolution layer, inputting the calculation result of the second second convolution layer (the second second sub-feature map) into the third second convolution layer and the third association module, inputting the calculation result of the third second convolution layer into the fourth second convolution layer, inputting the calculation result of the fourth second convolution layer into the fifth second convolution layer, and inputting the calculation result of the fifth second convolution layer (the fifth second sub-feature map) into the second pooling layer.

[0376] S503a, the calculation result of the first pooling layer (the sixth first sub-feature map) is output to the first association module.

[0377] S503b, the calculation result of the second pooling layer (the sixth second sub-feature map) is output to the first association module.

[0378] S504, the correlation layer #1 calculates the correlation for the sixth first sub-feature graph and the sixth second sub-feature graph to obtain a correlation feature graph #1, and inputs it to the decoder #1.

[0379] S505, decoder #1 decodes the correlation feature map #1 to obtain the first intermediate feature, and inputs it to decoder #2.

[0380] S506a, the calculation result of the first convolutional layer of the fifth layer (the fifth first sub-feature map) is also input into the second association module.

[0381] S506b, the calculation result of the 5th second convolutional layer (the 5th second sub-feature map) is also input into the 2nd association module.

[0382] S507, the correlation layer #2 calculates the correlation between the fifth first sub-feature graph and the fifth second sub-feature graph to obtain a correlation feature graph #2, and inputs it to the decoder #2.

[0383] S508, decoder #2 decodes the correlation feature map #2 to obtain the second intermediate feature, and inputs it to decoder #2.

[0384] S509a, the calculation result of the first convolutional layer of the second layer (the second first sub-feature map) is also input into the third association module.

[0385] S509b, the calculation result of the second convolutional layer of the second layer (the second second sub-feature map) is also input into the third association module.

[0386] S510, the correlation layer #3 calculates the correlation between the second first sub-feature graph and the second second sub-feature graph to obtain a correlation feature graph #3, and inputs it to the decoder #3.

[0387] S511, decoder #3 decodes the correlation feature map #3 and outputs the target motion information.

[0388] The beneficial effects of the inference flowchart 500 can be found in the corresponding description of the method 400 and will not be elaborated here.

[0389] Fig.12 The software structure block diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer (not shown in the figure), the system layer, and the kernel layer.

[0390] The application layer may include a series of application packages.

[0391] like Fig.12 As shown, the application package may include a first application, which may be a game, live broadcast, video, etc. The application package may also include other applications, such as camera, gallery, calendar, phone, map, WLAN, Bluetooth, music, short message, etc. (not shown in the figure).

[0392] Among them, games, videos, or live broadcast applications require certain rendering effects when playing pictures. Based on this requirement, the motion estimation model provided in the embodiment of the present application can realize efficient and real-time motion estimation in mobile game scenes, thereby improving the rendering effect.

[0393] The system layer may include multiple functional modules, such as image processing library, image resource processing module, motion estimation module, post-processing and UI rendering module, etc. It may also include image engine, media library, etc. (not shown in the figure).

[0394] The image resource processing module is used to obtain image resources (real image frames) and transmit the image resources to the motion estimation module.

[0395] The image processing library is used to draw, render, synthesize and display real images. The image processing library includes a 2D graphics processing library and a 3D graphics processing library. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0396] The motion estimation module is used to perform motion estimation on the image resources and transmit the motion estimation result to the post-processing and UI rendering module. The motion estimation module can call the motion estimation model provided in this application to perform motion estimation.

[0397] The functions of post-processing and UI rendering modules can be found in Figure 4 Related instructions for step (14) in .

[0398] The kernel layer is a layer between hardware and software. The kernel layer includes at least a display driver. It may also include a camera driver, a Wi-Fi driver, a communication driver, an audio driver, a sensor driver, a Bluetooth driver, etc. (not shown).

[0399] Combine the following Fig.12 , taking the first application as a game as an example, Fig.12 The rendering process is explained in the following section.

[0400] The game application generates an image rendering event and sends the image resource data to the image processing library. The three-dimensional graphics library in the image processing library performs image drawing, rendering and other operations based on the received image resource data. The image resource processing module intercepts the image resource data (current real image frame) sent by the game application to the image processing library, and transmits the intercepted image resource data to the prediction frame processing module. The motion estimation module obtains the previous real image frame of the current real image frame based on the image resource data (current real image frame), performs motion estimation based on the current real image frame and the previous real image frame, and obtains motion information. The post-processing and UI rendering unit performs post-processing and rendering processing on the current real image frame based on the motion information (for example, it can perform corresponding processing based on interpolation, super-resolution or anti-aliasing scenes) to obtain the image frame to be displayed, and sends the image frame to be displayed to the display driver for display.

[0401] The embodiment of the present application provides a chip system, which includes one or more processors for calling and running instructions stored in the memory from the memory, so that the method of the embodiment of the present application is executed. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0402] Among them, the chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0403] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0404] The computer program product can be stored in a memory, and is finally converted into an executable target file that can be executed by a processor after preprocessing, compiling, assembling and linking.

[0405] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0406] The computer-readable storage medium may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM).

[0407] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0408] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0409] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0410] It should be understood that the "plurality" mentioned in this application refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0411] The terms (or numbers) "first", "second", ..., etc. that appear in the embodiments of the present application are only used for descriptive purposes, that is, only to distinguish different objects, such as different "coordinates", etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", ..., etc. may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one of the following (items)" or similar expressions thereof refers to any combination of these items, including any combination of a single (item) or a plurality of (items).

[0412] In short, the above is only a preferred embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the protection scope of this application.

Claims

1. A motion estimation model training method, characterized in that: The method comprises: Acquire a training data set, the training data set comprising M adjacent frame image groups and M sample motion information, each adjacent frame image group comprising two adjacent frames of first sample images, the M sample motion information being motion information of the M adjacent frame image groups, M being a positive integer, the training data set being acquired from a first application on a mobile terminal, the first sample images being images constituting a to-be-displayed screen of the first application; Downsampling the two first sample images included in the m-th adjacent frame image group of the M adjacent frame image groups respectively to obtain corresponding second sample images and third sample images, 1≤m≤M and m is an integer, and the motion information of the m-th adjacent frame image group is the m-th sample motion information of the M sample motion information; Inputting the second sample image and the third sample image into the motion estimation model to obtain motion information; Obtaining a loss function value according to a difference between the motion information and the m-th sample motion information; The parameters that need to be trained in the motion estimation model are adjusted according to the loss function value.

2. The method according to claim 1, characterized in that The first application is a game application, and the training data set is obtained from the game application in a game scenario; Alternatively, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; Alternatively, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

3. The method according to claim 1 or 2, characterized in that The obtaining a loss function value according to the difference between the motion information and the m-th sample motion information includes: The loss function value is obtained according to the difference between the motion information and the mth sample motion information, and the attention weight, wherein the attention weight is the weight of the pixels at corresponding positions in the second sample image and the third sample image, and the attention weight is used to indicate the user's attention to the pixels at the corresponding positions.

4. The method according to claim 3, characterized in that Among the pixels at the corresponding positions, pixels located in a target focus area are determined as target pixels, and the focus weight of the target pixel is greater than the focus weights of other pixels, wherein the target focus area includes central areas of the second sample image and the third sample image.

5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining a training data set includes: Acquire the M adjacent frame image groups and image-related data from the first application program; Determining a target method for acquiring motion information according to the type of the image-related data; Determining the M sample motion information according to the image-related data in the target manner; Wherein, when the type of the image-related data includes three-dimensional information of the first sample image, the target method includes three-dimensional reprojection; or, when the type of the image-related data does not include three-dimensional information of the first sample image, the target method includes a method for motion estimation based on an optical flow method or a neural network on a personal computer.

6. The method according to claim 5, characterized in that In the case where the type of the image-related data includes three-dimensional information of the first sample image and the target mode includes three-dimensional reprojection, the three-dimensional information of the first sample image includes a scene depth value and a coordinate transformation matrix; The determining the M pieces of motion information according to the image-related data in the target manner includes: The motion information corresponding to each of the adjacent frame image groups is obtained based on the three-dimensional reprojection according to the scene depth values ​​and coordinate transformation matrices of the two first sample images in each of the adjacent frame image groups.

7. The method according to any one of claims 1 to 6, characterized in that Inputting the second sample image and the third sample image into the motion estimation model to obtain motion information includes: Processing the second sample image based on the first input layer to obtain first sample image data, and processing the third sample image based on the second input layer to obtain second sample image data; Extracting features from the first sample image data based on the first hidden layer to obtain a first feature map, and extracting features from the second sample image data based on the second hidden layer to obtain a second feature map; The first feature map and the second feature map are processed based on an association module to obtain the motion information, wherein the association module is used to calculate the association degree between the first feature map and the second feature map.

8. The method according to claim 7, characterized in that The processing of the first feature map and the second feature map based on the association module to obtain the motion information includes: Sampling the first feature map based on the association module to obtain a first sampling feature; Sampling the second feature map based on the association module to obtain a second sampling feature; Based on the association module, a correlation degree is calculated according to the first sampling feature and the second sampling feature to obtain a correlation degree feature graph of the first feature graph and the second feature graph; The correlation feature map is decoded to obtain the motion information.

9. The method according to claim 8, characterized in that The sampling of the first feature map based on the association module to obtain a first sampling feature includes: determining a first target feature based on the association module; taking the first target feature as a central block and sampling K×K blocks to obtain the first sampling feature; The sampling of the second feature map based on the association module to obtain a second sampling feature includes: determining a second target feature based on the association module; taking the second target feature as a central block and sampling K×K blocks to obtain the second sampling feature; Among them, K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

10. The method according to any one of claims 7 to 9, characterized in that The number of layers of the first hidden layer and the second hidden layer are both L, and L layers of the first hidden layer and L layers of the second hidden layer correspond one to one, and the number of the associated modules is P; The processing of the first feature map and the second feature map based on the association module to obtain the motion information includes: Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layers and the corresponding P second sub-feature maps output by the second hidden layer are processed respectively to obtain the motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

11. The method according to claim 10, characterized in that The first association module among the P association modules includes the first association layer and the first decoder, the i-th association module among the P association modules includes the i-th association layer and the i-th decoder, and the P-th association module includes the P-th association layer and the P-th decoder. The P-th association module calculates the P first sub-feature graphs output by part or all of the first hidden layers in the L layers and the corresponding P second sub-feature graphs output by the second hidden layer to obtain the motion information, including: The first correlation layer calculates the correlation between the first of the P first sub-feature graphs and the first of the P second sub-feature graphs to obtain a first correlation feature graph, and inputs the first correlation feature graph to the first decoder; the first decoder decodes the first correlation feature graph to obtain a first intermediate feature, and inputs the first intermediate feature to the second decoder; The i-th correlation layer calculates the correlation between the i-th first sub-feature graph and the i-th second sub-feature graph to obtain the i-th correlation feature graph, and inputs the i-th correlation feature graph into the i-th decoder; the i-th decoder decodes the i-th correlation feature graph to obtain the i-th intermediate feature, and inputs the i-th intermediate feature into the i+1-th decoder, 2≤i≤P-1 and i is an integer; The P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain a P-th correlation feature map, and inputs the P-th correlation feature map into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain the motion information.

12. The method according to claim 10, characterized in that When P=3, based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layer and the corresponding P second sub-feature maps output by the second hidden layer are processed to obtain the motion information, including: Inputting the cth first sub-feature map output by the cth first hidden layer and the cth second sub-feature map output by the cth second hidden layer into the first association module to obtain the first intermediate feature; Inputting the bth first sub-feature map output by the bth first hidden layer, the bth second sub-feature map output by the bth second hidden layer, and the first intermediate feature into a second association module to obtain a second intermediate feature; The ath first sub-feature map output by the ath first hidden layer, the ath second sub-feature map output by the ath second hidden layer, and the second intermediate feature are input into the third association module to obtain the motion information, 1≤a<b≤L-1, c=L, and a, b and c are all integers.

13. The method according to claim 10, characterized in that L<10, the first hidden layer of L layers includes L-1 first convolutional layers and 1 first pooling layer, and the second hidden layer of L layers includes L-1 second convolutional layers and 1 second pooling layer; Among them, when L=6, the 6 layers of the first hidden layers include 5 layers of the first convolutional layers and 1 layer of the first pooling layer, and the 6 layers of the second hidden layers include 5 layers of the second convolutional layers and 1 layer of the second pooling layer, the a-th layer of the first hidden layer is the 2nd layer of the first convolutional layer, the b-th layer of the first hidden layer is the 5th layer of the first convolutional layer, the c-th layer of the first hidden layer is the first pooling layer, the a-th layer of the second hidden layer is the 2nd layer of the second convolutional layer, the b-th layer of the second hidden layer is the 5th layer of the second convolutional layer, and the c-th layer of the second hidden layer is the second pooling layer.

14. A motion estimation method based on a motion estimation model, applied to electronic equipment, characterized in that: The electronic device is deployed with a first application, the motion estimation model is deployed on the electronic device, and the motion estimation model is trained by the motion estimation model training method according to any one of claims 1 to 5, the method comprising: Acquire two adjacent frames of images from the first application, and perform downsampling processing on the two adjacent frames of images to obtain a first image and a second image, wherein the two adjacent frames of images are images constituting a to-be-displayed screen of the first application; Inputting the first image and the second image into the motion estimation model to obtain target motion information of the two adjacent frames of images includes: Processing the first image based on the first input layer to obtain first image data, and processing the second image based on the second input layer to obtain second image data; Extracting features from the first image data based on the first hidden layer to obtain a first feature map, and extracting features from the second image data based on the second hidden layer to obtain a second feature map; The first feature map and the second feature map are processed based on an association module to obtain the target motion information, wherein the association module is used to calculate the association degree between the first feature map and the second feature map.

15. The method according to claim 14, characterized in that The first application is a game application, and the training data set is obtained from the game application in a game scenario; Alternatively, the first application is a video application, and the training data set is obtained from the video application in a video playback scenario; Alternatively, the first application is a live broadcast application, and the training data set is obtained from the live broadcast application in a live broadcast scenario.

16. The method according to claim 14 or 15, characterized in that The step of processing the first feature map and the second feature map based on the association module to obtain the target motion information includes: Sampling the first feature map based on the association module to obtain a first sampling feature; Sampling the second feature map based on the association module to obtain a second sampling feature; Calculating the first sampling feature and the second sampling feature to obtain a correlation feature graph of the first feature graph and the second feature graph; The correlation feature map is decoded to obtain the target motion information.

17. The method according to claim 16, characterized in that The sampling of the first feature map based on the association module to obtain a first sampling feature includes: determining a first target feature; taking the first target feature as a central block and sampling K×K blocks to obtain the first sampling feature; The sampling of the second feature map based on the association module to obtain a second sampling feature includes: determining a second target feature; taking the second target feature as a central block and sampling K×K blocks to obtain the second sampling feature; Among them, K<7 and K is a positive integer, and the position of the first target feature in the first feature map corresponds to the position of the second target feature in the second feature map.

18. The method according to any one of claims 14 to 17, characterized in that The number of layers of the first hidden layer and the second hidden layer are both L, and L layers of the first hidden layer and L layers of the second hidden layer correspond one to one, and the number of the associated modules is P; The step of processing the first feature map and the second feature map based on the association module to obtain the target motion information includes: Based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layers and the corresponding P second sub-feature maps output by the second hidden layer are processed respectively to obtain the target motion information, wherein the P first sub-feature maps are included in the first feature map, the P second sub-feature maps are included in the second feature map, 2≤P≤L, and P and L are both integers.

19. The method according to claim 18, characterized in that The first association module among the P association modules includes the first association layer and the first decoder, the i-th association module among the P association modules includes the i-th association layer and the i-th decoder, and the P-th association module includes the P-th association layer and the P-th decoder. The P-th association module calculates the P first sub-feature graphs output by part or all of the first hidden layers in the L layers and the corresponding P second sub-feature graphs output by the second hidden layer to obtain the target motion information, including: The first correlation layer calculates the correlation between the first of the P first sub-feature graphs and the first of the P second sub-feature graphs to obtain a first correlation feature graph, and inputs the first correlation feature graph to the first decoder; the first decoder decodes the first correlation feature graph to obtain a first intermediate feature, and inputs the first intermediate feature to the second decoder; The i-th correlation layer calculates the correlation between the i-th first sub-feature graph and the i-th second sub-feature graph to obtain the i-th correlation feature graph, and inputs the i-th correlation feature graph into the i-th decoder; the i-th decoder decodes the i-th correlation feature graph to obtain the i-th intermediate feature, and inputs the i-th intermediate feature into the i+1-th decoder, 2≤i≤P-1 and i is an integer; The P-th correlation layer calculates the correlation between the P-th first sub-feature map and the P-th second sub-feature map to obtain a P-th correlation feature map, and inputs the P-th correlation feature map into the P-th decoder; the P-th decoder decodes the P-th correlation feature map and the P-1-th intermediate feature to obtain the target motion information.

20. The method of claim 19, wherein: When P=3, based on the P association modules, the P first sub-feature maps output by part or all of the first hidden layers in the L layer and the corresponding P second sub-feature maps output by the second hidden layer are processed to obtain the target motion information, including: Inputting the cth first sub-feature map output by the cth first hidden layer and the cth second sub-feature map output by the cth second hidden layer into the first association module to obtain the first intermediate feature; Inputting the bth first sub-feature map output by the bth first hidden layer, the bth second sub-feature map output by the bth second hidden layer, and the first intermediate feature into a second association module to obtain a second intermediate feature; The ath first sub-feature map output by the ath first hidden layer, the ath second sub-feature map output by the ath second hidden layer, and the second intermediate feature are input into the third association module to obtain the target motion information, 1≤a<b≤L-1, c=L, and a, b and c are all integers.

21. The method of claim 20, wherein: L<10, the first hidden layer of L layers includes L-1 first convolutional layers and 1 first pooling layer, and the second hidden layer of L layers includes L-1 second convolutional layers and 1 second pooling layer; Among them, when L=6, the 6 layers of the first hidden layers include 5 layers of the first convolutional layers and 1 layer of the first pooling layer, and the 6 layers of the second hidden layers include 5 layers of the second convolutional layers and 1 layer of the second pooling layer, the a-th layer of the first hidden layer is the 2nd layer of the first convolutional layer, the b-th layer of the first hidden layer is the 5th layer of the first convolutional layer, the c-th layer of the first hidden layer is the first pooling layer, the a-th layer of the second hidden layer is the 2nd layer of the second convolutional layer, the b-th layer of the second hidden layer is the 5th layer of the second convolutional layer, and the c-th layer of the second hidden layer is the second pooling layer.

22. The method of claim 14, wherein: The first image is a displayed non-predicted image, the second image is a non-predicted image that has not been displayed, the first image is a previous non-predicted image of the second image in a display timing, and the method further includes: Shifting and filling pixels of the second image and the first image according to the target motion information to obtain one or more predicted images; After displaying the one or more predicted images, the second image is displayed.

23. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 13, or, to execute the method as described in any one of claims 14 to 22.

24. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 13, or executes the method as described in any one of claims 14 to 22.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 13, or executes the method as claimed in any one of claims 14 to 22.

Images