Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and belongs to the technical field of intelligent terminals, and the method comprises the steps: creating at least one second grid layer under the condition that a real frame image comprises a reconstruction region meeting a first preset condition and / or a second preset condition, and carrying out the reconstruction of the reconstruction region based on the reconstruction attribute of the reconstruction region, and respectively drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer. Therefore, no matter how the geometric edge of the discontinuous image area is complex, pixel points with different attributes in the discontinuous image area can be drawn on different grid layers through a grid reconstruction mode, so that the pixel area in each grid is a continuous pixel area. Therefore, when image distortion processing is carried out on the grids with continuous depths, serious deformation of grid graphs cannot be caused, so that the continuity of the images is ensured, the problem of distortion of discontinuous image areas of complex geometric edges is solved, and the game experience of users is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent terminal technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] Video interpolation technology aims to improve the frame rate and smoothness of videos, making them look smoother. By inserting a predicted frame (or intermediate frame, transition frame, etc.) between adjacent real frames (or original frames), the frame rate of the video can be doubled, and the quality of the predicted frame is directly related to the smoothness of the video. The moving direction and speed of each object in the image are calculated through the information of two adjacent frames, and the predicted frame is obtained by moving each object in the image accordingly.

[0003] In the current prediction frame generation technology, image warping is a way to better ensure image continuity. However, in some image areas, the image warping operation may cause local image distortion, making the difference between the predicted frame image and the real frame image obvious, greatly reducing the user's visual experience. Summary of the invention

[0004] The present application provides an image processing method, device and electronic device, which can solve the problem of local image distortion, especially the distortion problem of discontinuous image areas with complex geometric edges.

[0005] In a first aspect, the present application provides an image processing method, comprising: acquiring resource information, the resource information comprising a real frame image, depth information of the real frame image, and pixel attribute information of the real frame image; the pixel attribute information is used to characterize static pixels and dynamic pixels; creating a first grid layer of the real frame image; in a case where the real frame image includes a reconstruction area that satisfies a first preset condition and / or a second preset condition, creating at least one second grid layer, and based on the reconstruction attributes of the reconstruction area, drawing the pixels of the reconstruction area on the first grid layer and the at least one second grid layer, respectively; wherein the first preset condition is that the reconstruction area includes static pixels of different depth levels, and the second preset condition is that the reconstruction area includes both static pixels and dynamic pixels; The reconstruction attributes of the pixels drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction attributes include depth level and pixel attributes; a first motion vector corresponding to the first image in the first grid layer and a second motion vector corresponding to each second image in each second grid layer are obtained; the first image refers to a real frame image area included in the first grid layer; the second image refers to a real frame image area included in the second grid layer; based on the first motion vector, the first image is subjected to image distortion processing to generate a first distorted image; based on each second motion vector, each second image is subjected to image distortion processing to generate at least one second distorted image; the first distorted image and the at least one second distorted image are superimposed to generate a predicted frame image.

[0006] Thus, in the image processing method, the discontinuous image area in the real frame image, that is, the reconstruction area, is first identified. Then, according to the reconstruction attributes of the reconstruction area, multiple grid layers are constructed, and the pixels of different attributes in the reconstruction area are drawn respectively on different grid layers. Then, based on the corresponding motion vectors, image distortion processing is performed on each grid layer respectively. In this way, no matter how complex the geometric edges of the discontinuous image area are, the pixels of different attributes in the discontinuous image area can be drawn respectively on different grid layers by means of grid reconstruction to ensure that the pixel areas in each grid are continuous pixel areas. In this way, when the image distortion processing is performed on each depth-continuous grid, the grid graphics will not be severely deformed, thereby ensuring the continuity of the image, solving the problem of distortion of discontinuous image areas with complex geometric edges, and improving the user's gaming experience.

[0007] In an achievable manner, when the real frame image includes a reconstruction area that satisfies a first preset condition and / or a second preset condition, at least one second grid layer is created, and based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer, respectively, including: when the real frame image includes a reconstruction area that satisfies the first preset condition, a second grid layer is created; static pixel points of a first depth level in the reconstruction area are drawn on the first grid layer; static pixel points of a second depth level in the reconstruction area are drawn on the second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0008] Thus, for the reconstruction area, pixels with different reconstruction attributes can be drawn on the first grid layer and the second grid layer, respectively. For example, static pixels of the first depth level in the reconstruction area can be drawn on the first grid layer, and static pixels of the second depth level in the reconstruction area can be drawn on the second grid layer.

[0009] In an achievable manner, when the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, at least one second grid layer is created, and based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer, respectively, including: when the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of static pixel points of each reconstruction area are the same, a second grid layer is created; static pixel points in the reconstruction area are drawn on the first grid layer, and dynamic pixel points in the reconstruction area are drawn on the second grid layer; or dynamic pixel points in the reconstruction area are drawn on the first grid layer, and static pixel points in the reconstruction area are drawn on the second grid layer.

[0010] In this way, the static pixel points and the dynamic pixel points in the reconstruction area can be drawn on different grid layers respectively to obtain respective continuous pixel areas.

[0011] In an achievable manner, when the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, at least one second grid layer is created, and based on the reconstruction attribute of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer, respectively, including: when the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of static pixel points of each of the reconstruction areas are different, or when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition, two second grid layers are created; static pixel points of a first depth level in the reconstruction area are drawn on the first grid layer, and static pixel points of a second depth level in the first second grid layer are drawn on the second grid layer; Draw static pixels of the second depth level in the reconstructed area on the second second grid layer, and draw dynamic pixels of the reconstructed area on the second second grid layer; or, draw dynamic pixels of the reconstructed area on the first grid layer, draw static pixels of the first depth level in the reconstructed area on the first second grid layer, and draw static pixels of the second depth level in the reconstructed area on the second second grid layer; or, draw static pixels of the first depth level in the reconstructed area on the first grid layer, draw dynamic pixels of the reconstructed area on the first second grid layer, and draw static pixels of the second depth level in the reconstructed area on the second second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0012] In this way, static pixels and dynamic pixels at different depth levels in the reconstructed area can be drawn on different grid layers respectively to obtain respective continuous pixel areas.

[0013] In an achievable manner, the method further includes: marking reconstruction attribute information of each of the reconstructed areas to obtain a marking map; the reconstruction attribute information includes reconstruction type information and depth level information; based on the reconstruction type information and the depth level information, determining the number of the second grid layers and the reconstruction attributes of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer.

[0014] In this way, the depth level, pixel attributes and other information of each region in the real frame image can be determined based on the label map to further perform grid reconstruction processing.

[0015] In an achievable manner, the method further includes: dividing the real frame image into multiple working groups, each of the working groups including multiple pixels; based on the depth information of the real frame image and the pixel attribute information of the real frame image, determining that the working group including static pixels of different depth levels in the real frame image is the reconstruction area that satisfies the first preset condition; and / or, based on the pixel attribute information of the real frame image, determining that the working group including both static pixels and dynamic pixels in the real frame image is the reconstruction area that satisfies the second preset condition.

[0016] In an achievable manner, the determination of the working group including static pixels of different depth levels in the real frame image as the reconstructed area that meets the first preset condition based on the depth information of the real frame image and the pixel attribute information of the real frame image includes: determining a first working group including static pixels among the multiple working groups based on the pixel attribute information of the real frame image; determining a second working group including pixels whose depth values ​​are within a preset range and whose depth values ​​difference between the pixels and surrounding pixels exceeds a preset threshold in the first working group based on the depth information of the real frame image; determining a maximum depth value of each of the second working groups; determining a minimum value of each of the maximum depth values ​​as the depth threshold of the real frame image; determining a third working group including static pixels of a first depth level and static pixels of a second depth level among the multiple working groups, wherein the depth values ​​of the static pixels of the first depth level are greater than the depth threshold, and the depth values ​​of the static pixels of the second depth level are less than or equal to the depth threshold; and determining the third working group as the reconstructed area that meets the first preset condition.

[0017] In this way, the depth threshold can be calculated based on the depth information of the static pixel points in the real frame image, and then the reconstruction areas that meet the first preset condition are determined based on the depth threshold.

[0018] In one implementable manner, the image distortion processing is performed on the first image based on the first motion vector to generate a first distorted image, including: setting the depth information of each first grid in the first grid layer; determining the first motion vector of each vertex of the first grid for each first grid in the first grid layer, and moving the first grid according to the first motion vector of each vertex of the first grid; when each first grid moves to a target position, determining the coverage relationship between the first grid and other first grids at the target position according to the depth information of the first grid and the depth information of other first grids at the target position.

[0019] In this way, since the pixel depth information is referred to when a grid is moved based on the motion vector at the first grid layer, the correct depth order relationship between different objects at the first grid layer can be guaranteed.

[0020] In one achievable manner, the image distortion processing is performed on each second image based on each second motion vector to generate at least one second distorted image, including: setting the depth information of each second grid in the second grid layer; determining the second motion vector of each vertex of the second grid for each second grid in the second grid layer, and moving the second grid according to the second motion vector of each vertex of the second grid; when each second grid moves to a target position, determining the covering relationship between the second grid and other second grids at the target position according to the depth information of the second grid and the depth information of other second grids at the target position.

[0021] In this way, since the pixel depth information is referred to when a grid is moved based on the motion vector on the second grid layer, the correct depth order relationship between different objects on the second grid layer can be guaranteed.

[0022] In one implementable manner, the superimposing the first warped image and the at least one second warped image to generate a predicted frame image includes: when superimposing the first warped image and the at least one second warped image, determining an overlapping relationship between a first grid in the first grid layer and a second grid in each second grid layer according to depth information of each first grid and each second grid.

[0023] In this way, the covering relationship between the first grid in the first grid layer and the second grid in each second grid layer is determined according to the depth information of each first grid and each second grid, so as to ensure the correct depth order relationship between different objects after superposition.

[0024] In one achievable manner, the resource information is resource information in a game application.

[0025] In a second aspect, the present application provides an image processing device, the device comprising:

[0026] A resource information acquisition module, used to acquire resource information, wherein the resource information includes a real frame image, depth information of the real frame image, and pixel attribute information;

[0027] A first grid layer creation module, used to create a first grid layer of the real frame image;

[0028] A reconstruction module, configured to create at least one second grid layer when the real frame image includes a reconstruction area that satisfies a first preset condition and / or a second preset condition, and to draw pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on reconstruction properties of the reconstruction area; wherein the first preset condition is that the reconstruction area includes static pixel points with different depth levels, and the second preset condition is that the reconstruction area includes both static pixel points and dynamic pixel points; the reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties;

[0029] A motion vector acquisition module, used to acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image area included in the first grid layer; the second image refers to a real frame image area included in the second grid layer;

[0030] A first distorted image generating module, configured to perform image distortion processing on the first image based on the first motion vector to generate a first distorted image;

[0031] A second distorted image generating module, configured to perform image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image;

[0032] The superposition processing module is used to perform superposition processing on the first distorted image and the at least one second distorted image to generate a predicted frame image.

[0033] In a third aspect, the present application provides an electronic device comprising a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes a method as described in any one of the first aspects.

[0034] In a fourth aspect, the present application provides a computer storage medium, characterized in that a computer program or instructions are stored in the computer storage medium, and when the computer program or instructions are executed, the method as described in any one of the first aspects is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of a prediction frame generation process provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of image gridding provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of image grid distortion provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of a grid subdivision provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 7 An image processing flow chart provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of grid standardization provided in an embodiment of the present application;

[0044] Fig. 9 A flowchart for determining a reconstruction area that meets a first preset condition provided in an embodiment of the present application;

[0045] Fig. 10A A schematic diagram of pixel distribution provided in an embodiment of the present application;

[0046] Fig. 10B A marking diagram provided for an embodiment of the present application;

[0047] Fig.11 A schematic diagram of pixel distribution of each grid layer after grid reconstruction processing provided in an embodiment of the present application;

[0048] Fig.12 A schematic diagram of image distortion provided in an embodiment of the present application;

[0049] Fig.13A One of the schematic diagrams of an application scenario provided in an embodiment of the present application;

[0050] Fig. 13B One of the schematic diagrams of an application scenario provided in an embodiment of the present application;

[0051] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0053] With the development of electronic technology, the refresh rate of electronic devices (such as mobile phones) screens is getting higher and higher, and the frame rate of video sources is also getting higher and higher, which will increase the memory usage and rendering power consumption in electronic devices. However, due to the power consumption of electronic devices or the capacity limitations of CPU and GPU, high refresh rate and high frame rate often cause electronic devices to heat up or freeze, thus affecting the user experience.

[0054] Take mobile games as an example. With the development and promotion of large-scale mobile games, the rendering pipeline of modern mobile games is becoming more and more complex, and the resource load is also increasing. At the same time, the computing power growth of modern mobile phones does not meet the needs of modern large-scale mobile games, and mobile phones are limited by limited battery capacity and heat dissipation capabilities. Therefore, developers focus on how to reduce unnecessary rendering overhead, improve limited frame rate and fluency, and reduce heat generation while ensuring that the game quality is not significantly affected.

[0055] In order to improve the user experience, developers often reduce the frame rate of the video source, and then increase the video frame rate through video interpolation technology to achieve halving the rendering at the same frame rate, thereby significantly reducing the power consumption of the mobile phone and reducing the heat of the mobile phone.

[0056] Video interpolation technology aims to improve the frame rate and smoothness of the video, making the video look smoother. By inserting a predicted frame (or intermediate frame, transition frame, etc.) between adjacent real frames (or original frames, etc.), the frame rate of the video can be doubled. The image quality of the predicted frame is directly related to the smoothness of the video. Among them, the moving direction and speed of each object in the image are calculated through the image information of two adjacent real frames, and the objects in the image are moved accordingly to obtain the predicted frame.

[0057] As the name implies, real frames refer to image frames originally existing in the video source, not image frames generated by prediction. For example, real frames can be image frames drawn by developers when developing applications such as games, or image frames captured by image acquisition devices (such as cameras) when making videos.

[0058] Video interpolation methods can be divided into predicted frame interpolation and predicted frame extrapolation. In the predicted frame interpolation method, a predicted frame image is calculated based on two adjacent real frame images, and the predicted frame image is interpolated between the two real frame images to improve the video frame rate. In the predicted frame extrapolation method, a predicted frame image is calculated based on two adjacent real frame images, and the predicted frame image is interpolated after the two real frame images as the next frame image to improve the video frame rate.

[0059] The following briefly describes the generation process of the prediction frame by taking a game scene as an example. It should be noted that the relevant drawings shown in this embodiment are all shown in grayscale form.

[0060] like Figure 1 As shown, the first real frame and the second real frame are two adjacent image frames in a certain game application. It is assumed that the second real frame is the current image frame and the first real frame is the previous image frame of the second real frame. Therefore, the mobile phone can calculate the predicted frame based on the first real frame and the second real frame. Taking the predicted frame interpolation method as an example, the smart terminal can interpolate the calculated predicted frame between the two real frames for display, so as to increase the frame rate of the game video and achieve half the rendering at the same frame rate.

[0061] In this game application, the real frame includes image information and UI (User Interface) information. Image information can be understood as the picture information in the video source, and UI information refers to the overall design information of the software's human-computer interaction, operating logic, and interface aesthetics. When the mobile phone calculates the predicted frame based on adjacent real frames, since the interface UI information will not change, the mobile phone can first separate the image information and UI information, and only calculate the predicted frame based on the image information, thereby reducing the amount of image data processing and improving the accuracy of the predicted frame. Therefore, Figure 1 As shown, when calculating the predicted frame, the mobile phone can first extract the first real frame image 101 and the first UI 102 from the first real frame, extract the second real frame image 103 and the second UI 104 from the second real frame, and then perform an image prediction operation based on the first real frame image 101 and the second real frame image 103 to obtain the predicted frame image 107.

[0062] Continue to refer to Figure 1When calculating the predicted frame image 107, the mobile phone can first calculate the motion vector (MV) 105 of the first real frame image 101 and the second real frame image 103, and perform image warping on the second real frame image 103 based on the motion vector 105 to obtain the predicted frame image 106. The image warping process refers to changing the position of image pixels through a certain transformation, such as translation transformation, rotation transformation, scaling transformation, affine transformation, perspective transformation, columnar transformation, etc.

[0063] Among them, the image distortion processing can better ensure the continuous movement of each object in the image, but because the movement speeds of each object in the image are different, the predicted frame image 106 obtained after the image distortion processing may have image information overlap or missing. The mobile phone also needs to perform image completion (Blur) processing on the predicted frame image 106 to obtain the predicted frame image 107. At this time, the mobile phone fuses the second UI 104 and the predicted frame image 107 to obtain the predicted frame 108 calculated based on the first real frame and the second real frame.

[0064] It should be noted that, whether it is image distortion processing or fusion UI information processing, the information in the first real frame image or the second real frame image can be used as a reference, and this embodiment does not limit this.

[0065] In computer graphics, mesh geometry is usually used as the basic unit for image processing. Similarly, in the image distortion processing stage mentioned above, the mobile phone also uses mesh geometry as the basic unit for image distortion processing. In other words, the mobile phone performs image distortion processing by forming a geometric group of triangles or quadrilaterals in the image. Take a quadrilateral mesh as an example. Figure 2 As shown, the real frame image 20 is a global work group. A plurality of quadrilateral grids 201 can be divided in the real frame image 20, and each quadrilateral grid 201 includes a plurality of pixel points 2011. Each quadrilateral grid 201 is a local work group, which is a basic unit for the mobile phone to perform image distortion processing.

[0066] However, for more complex image scenes (for example, image scenes in games), dynamic objects and static objects may be included, and the depth difference of static objects is relatively large. In addition, since most games are virtual worlds, when users play mobile games, user operations often cause large-angle shaking of the game scene, unlike the physical world where object movement follows physical laws. Therefore, in the case of complex game scenes, large-angle scene shaking may only cause larger movement of some objects in the image, rather than larger movement of all objects in the image. In other words, in the game scene, the distribution of MV images calculated based on continuous real frame images is uneven, especially in discontinuous image areas, where the difference in MV values ​​is relatively large. Among them, discontinuous image areas include static image areas with large differences in depth information and / or image areas that include both dynamic pixels and static pixels.

[0067] In this way, if the same grid includes discontinuous image areas, using the grid as a basic unit for image distortion processing will cause local stretching of the image, making the predicted frame image distorted, which will in turn make the difference between the predicted frame and the real frame large, and the user's visual experience poor.

[0068] For example, Figure 3 (1) shows an example of a grayscale image of a motion vector frame calculated based on a game image. Figure 3 In the grayscale image shown in (1), the motion vector of the area with lighter grayscale (the grayscale value is closer to 1) is smaller, and the motion vector of the area with darker grayscale (the grayscale value is closer to 0) is larger. Figure 3 In (1), the motion vectors on the left and right sides of the edge area of ​​the static object in the image frame change greatly. Figure 3 In (1) and (2), taking region 301 as an example, sub-region 3011 corresponds to the column region in the image, and sub-region 3012 corresponds to the sky region in the image. The motion vector corresponding to the column is significantly smaller than the motion vector corresponding to the sky. In other words, the motion vector values ​​on the left and right sides of the column edge region vary greatly. In this case, once the motion vectors corresponding to different pixel regions in the same grid differ greatly, image distortion processing of the grid based on the motion vector will inevitably cause serious deformation of the grid graphics, and even the problem of partial grid graphics overlapping, and the accuracy of the depth information of the object in the image cannot be guaranteed. Figure 3As shown in (3), among the four vertices of mesh 201, the motion vectors corresponding to V0 and V1 are quite different from the motion vectors corresponding to V2 and V3. After moving each vertex according to its corresponding motion vector, mesh 201 is distorted into mesh 201', which not only severely deforms the mesh pattern but also may cover other meshes. This will cause distortion in some areas of the predicted frame image, especially in the discontinuous edge area. Figure 3 As shown in (4).

[0069] As a result, in the predicted frame image generated after the image distortion process, the discontinuous image area will have a local stretching problem, which is quite different from the real scene, resulting in a poor visual experience for the user.

[0070] To solve the above technical problems, in some embodiments, for image areas with discontinuous depth values, the mobile phone can subdivide the grid based on the depth information of the pixels, and perform image distortion processing using the subdivided geometric figures as units, thereby solving the problem of local stretching in image areas with discontinuous depth values ​​in the predicted frame image, so as to better retain continuous graphic information, thereby improving the picture quality of the predicted frame image and providing the user with a better visual experience.

[0071] For example, Figure 4As shown, if there are two intersection points Point1 and Point2 between the edge of the static object and the grid 201, and the two intersection points Point1 and Point2 are located on different sides of the grid 201, the grid 201 can be subdivided into multiple triangles based on the two intersection points Point1 and Point2, so that the depth information in each subdivided triangle will not be too different. In this way, when the image is distorted for these subdivided triangles, each subdivided triangle is moved according to the motion vector corresponding to its own vertex. For example, triangle S-Tringle1 (V0-V1-Point1) moves to triangle S-Tringle1 (V0'-V1'-Point1'), and triangle S-Tringle2 (V0-Point1-Point2) moves to triangle S-Tringle2 (V0'-Point1'-Point2'). Similarly, triangle S-Tringle1'(Point1-Point2-V2) moves to triangle S-Tringle1'(Point1"-Point2"-V2"), and triangle S-Tringle2'(V2-Point2-V3) moves to triangle S-Tringle2'(V2"-Point2"-V3"). Since the depth information of each triangle after subdivision is consistent, their movement directions are also relatively consistent, thus avoiding the problem of severe mesh deformation due to excessive differences in mesh depth information, thereby improving picture quality and making the predicted frame image closer to the real frame image.

[0072] However, the above solution can only subdivide the grid to achieve the corresponding technical effect when there are two intersections between the edge of the static object and the grid 201, and the two intersections are located on different sides of the grid 201. Therefore, the above solution is only applicable to solving the distortion problem of regular geometric edges, but cannot solve the distortion problem of complex geometric edges.

[0073] To solve the above-mentioned technical problems, in the image processing method provided by the embodiment of the present application, the discontinuous image area in the real frame image is first identified (the embodiment of the present application refers to such an area as a reconstruction area). Then, according to the reconstruction attributes of the reconstruction area, multiple grid layers are constructed, and the pixels of different attributes in the reconstruction area are drawn respectively in different grid layers. Based on the corresponding motion vectors, each grid layer is subjected to image distortion processing respectively. In this way, no matter how complex the geometric edge of the image area with discontinuous depth values ​​is, the pixels of different attributes in the image area with discontinuous depth values ​​can be drawn respectively in different grid layers by means of grid reconstruction, so as to ensure that the pixel areas in each grid are all pixel areas with continuous depth. In this way, when the grids with continuous depths are subjected to image distortion processing, the grid graphics will not be severely deformed, thereby ensuring the continuity of the image, solving the problem of distortion of the image area with discontinuous depth values ​​at the edges of complex geometry, and improving the user's gaming experience.

[0074] The image processing method provided in the embodiment of the present application can be applied to electronic devices. Optionally, the electronic device 100 can be a terminal, which can also be called a terminal device, a smart terminal, etc. The terminal can be a device such as a cellular phone or a tablet computer (pad), which is not limited in the present application.

[0075] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0076] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0077] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0078] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0079] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0080] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0081] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0082] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0083] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0084] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0085] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0086] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0087] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0088] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0089] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0090] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0091] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0093] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0094] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0095] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0096] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0097] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0098] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0099] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0100] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0101] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0102] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0103] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0104] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0105] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0106] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0107] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0108] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0109] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A. A plurality of speakers 170A can be provided in the electronic device 100. For example, a speaker 170A can be provided on the top of the electronic device 100, and a speaker 170A can also be provided on the bottom, and so on.

[0110] The receiver 170B, also called a "receiver", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the receiver 170B can be placed close to the human ear to receive the voice. In some embodiments, the speaker 170A and the receiver 170B can also be set as one component, which is not limited in the present application.

[0111] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0112] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0113] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0114] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0115] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0116] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0117] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0118] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0119] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0120] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0121] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0122] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0123] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0124] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0125] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0126] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0127] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0128] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0129] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0130] Figure 6 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0131] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0132] The application layer can include a series of application packages.

[0133] like Figure 6 As shown, the application package may include games, cameras, galleries, calendars, calls, maps, navigation, WLAN, Bluetooth, music, videos, short messages and other applications.

[0134] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0135] like Figure 6 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0136] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0137] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0138] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0139] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0140] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0141] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0142] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0143] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0144] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0145] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0146] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0147] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] A 2D graphics engine is a drawing engine for 2D drawings.

[0150] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0151] Next, an embodiment of the image processing method provided by the present application is described.

[0152] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the method may include the following steps:

[0153] Step 401, obtaining resource information.

[0154] Resource information refers to the image data of the game APP, which is intended to be displayed in the game APP interface. Among them, resource information can include real frame image information and UI information.

[0155] Among them, UI information refers to the overall design information of the software's human-computer interaction, operating logic, and interface aesthetics.

[0156] The real frame image information includes but is not limited to the real frame image, the depth information of the real frame image and the pixel attribute information of the real frame image. The pixel attribute information is used to characterize whether the pixel is a static pixel or a dynamic pixel.

[0157] Dynamic pixels are usually pixels that change due to dynamic objects, lighting, materials, etc. in the scene. For example, moving objects, flashing lights, changing textures, etc. Static pixels are usually composed of static scene elements, such as fixed terrain, stationary buildings, fixed textures, etc.

[0158] In some embodiments, taking the game interpolation scene as an example, the game APP can send resource information to the 3D graphics processing library, and the 3D image processing library renders the image of the game APP. In this way, the electronic device can intercept the real frame image information after the rendering process.

[0159] Step 402: Create a first grid layer of a real frame image.

[0160] In the embodiment of the present application, the real frame image may be any one of the two adjacent real frame images used to calculate the predicted frame image. Figure 1 , the real frame image can be the first real frame image or the second real frame image.

[0161] Creating the first grid layer of the real frame image, which may also be referred to as gridding the real frame image, is to divide the real frame image into geometric shapes that are easier to handle, such as polygons, triangles, or quadrilaterals.

[0162] The created first grid layer has the same size as the real frame image, so that each graphic filled in the first grid layer can cover a pixel area of ​​the real frame image.

[0163] Exemplarily, in the embodiment of the present application, each grid in the created first grid layer is a quadrilateral grid, that is, the real frame image is gridded according to the size of the quadrilateral, wherein the quadrilateral may be a square.

[0164] In some embodiments, the first mesh layer may be standardized. If each mesh in the first mesh layer is a non-triangular mesh, such as a quadrilateral mesh, the non-triangular mesh may be standardized so that each non-triangular mesh in the first mesh layer is divided into a plurality of triangles for facilitating graphic calculations.

[0165] Exemplarily, if the first mesh layer created is a quadrilateral mesh, each quadrilateral in the quadrilateral mesh may be divided into two triangles again to achieve mesh standardization.

[0166] like Figure 8 As shown in (1), each quadrilateral mesh 201 in the first mesh layer can be divided into triangle 201_1 and triangle 201_2. Figure 8 As shown in (2), taking the quadrilateral mesh Square (V0-V1-V2-V3) as an example, the quadrilateral mesh can be divided into a triangular mesh Tringle1 (V0-V1-V2) and a triangular mesh Tringle2 (V2-V3-V0).

[0167] It should be noted that the first grid layer can cover every pixel in the real frame image. Each grid in the first grid layer corresponds to a pixel area (including multiple pixels) in the real frame image, and the pixel area corresponding to each grid may be an area with continuous depth values, an area with discontinuous depth values, an area with all dynamic pixels, or an area including both dynamic pixels and static pixels. Based on the foregoing description, it can be seen that due to the continuous depth value area and the area including both dynamic pixels and static pixels, distortion and deformation are prone to occur. Therefore, the embodiment of the present application can first identify the grids that are prone to distortion and deformation, and then perform grid reconstruction processing on these grids, so that the pixels with different attributes in the same grid are respectively drawn in the grids corresponding to different grid layers, so that the attributes of the pixels in each grid after the grid reconstruction processing are consistent. For example, after the grid reconstruction, the depth level of each pixel in the same grid is the same, and each pixel in the same grid is a static pixel or a dynamic pixel. In this way, since the corresponding pixel points in each grid are continuous, when each grid layer is subjected to image distortion processing based on the motion vector corresponding to each grid, the continuous graphic information of each object in the real frame image can be better retained, thereby avoiding the problem of distortion in the edge areas of each object in the real frame image.

[0168] The following is an introduction to the specific implementation method of the grid reconstruction process provided in the embodiments of the present application.

[0169] Step 403: when the real frame image includes a reconstructed area that meets the first preset condition and / or the second preset condition, create at least one second grid layer.

[0170] Step 404 : based on the reconstruction properties of the reconstruction area, draw the pixel points of the reconstruction area on the first grid layer and at least one second grid layer respectively.

[0171] Image deformation is prone to occur in static areas with discontinuous depth and areas that include both dynamic and static pixels. Therefore, the embodiment of the present application needs to identify these areas that are prone to image deformation so as to perform mesh reconstruction on these areas that are prone to image deformation.

[0172] Among them, the grid that meets the first preset condition is a static area with discontinuous depth (i.e., a grid including static pixels with different depth levels), and the grid that meets the second preset condition is a grid including both static pixels and dynamic pixels. In the embodiment of the present application, the grid that meets the first preset condition and / or the second preset condition is referred to as a reconstruction area or a reconstruction grid.

[0173] In the embodiment of the present application, each grid that meets the first preset condition and / or the second preset condition corresponds to a reconstruction area, wherein the reconstruction area that meets the first preset condition can be called a static reconstruction area, and the reconstruction area that meets the second preset condition can be called a dynamic reconstruction area.

[0174] In some embodiments, determining whether a real frame image includes a static reconstruction area that meets a first preset condition can be achieved in the following manner: first determine a depth threshold of static pixels in the real frame image; then, based on the depth threshold, determine whether each static pixel in each grid including the static pixel is continuous; if not continuous, the grid is a static reconstruction area.

[0175] Specifically, Fig. 9 As shown, the method for determining whether the real frame image includes a static reconstruction area that meets the first preset condition can refer to the description of the following steps 4041 to 4047.

[0176] Step 4041, divide the real frame image into a plurality of working groups, each working group including a plurality of pixel points.

[0177] That is to say, the real frame image is a global work group, and each work group after division is a local work group. Each work group corresponds to the size of each grid in the first grid layer, and is a basic unit for the electronic device to perform image distortion processing (see Figure 2 ).

[0178] Step 4042: Based on the pixel attribute information of the real frame image, determine a first working group including static pixels from among multiple working groups.

[0179] The electronic device can identify a working group including static pixels, namely, the first working group, based on the pixel attribute information.

[0180] Step 4043, based on the depth information of the real frame image, determine in the first working group a second working group including a pixel whose depth value is within a preset range and a difference between the depth value of the pixel and surrounding pixels exceeds a preset threshold.

[0181] The preset depth range may be understood as a depth range of interest, within which edges of static objects are likely to be distorted when the image is distorted.

[0182] If the difference between the depth values ​​of any pixel in the current first working group and the surrounding pixels exceeds a preset threshold, it indicates that the depths of the static pixels in the current working group are discontinuous. Otherwise, it indicates that the depths of the static pixels in the current working group are consistent. In the embodiment of the present application, the working group in the first working group that has a depth discontinuity area in the depth range of interest is determined as the second working group.

[0183] Step 4044, determining the maximum depth value of each second working group.

[0184] Since the second working group is a depth discontinuous area, that is, the second working group includes static pixels with different depth values, the maximum depth value of the static pixels in each second working group can be determined.

[0185] Step 4045: determine the minimum value among the maximum depth values ​​as the depth threshold of the real frame image.

[0186] Step 4046: Determine a third working group including static pixels at the first depth level and static pixels at the second depth level from among the multiple working groups.

[0187] In the embodiment of the present application, the depth value of each static pixel point can be compared with the depth threshold. If the depth value of the static pixel point is greater than the depth threshold, the static pixel point belongs to a static pixel point of the first depth level, for example, the first depth level is far. If the depth value of the static pixel point is less than or equal to the depth threshold, the static pixel point belongs to a static pixel point of the second depth level, for example, the first depth level is near. It can be seen that the first depth level and the second depth level belong to different depth levels, and the depth difference between the first depth level and the second depth level is large.

[0188] In this way, the embodiment of the present application can determine the depth level of each work group based on the depth threshold. The depth level of each work group can include the following situations:

[0189] The first type is that the working group includes static pixels, and the depth value of each static pixel is greater than the depth threshold. In this case, all static pixels in the working group belong to the static pixels of the first depth level, that is, a single depth level working group.

[0190] The second type is that the working group includes static pixels, and the depth value of each static pixel is less than or equal to the depth threshold. In this case, all static pixels in the working group belong to the static pixels of the second depth level, that is, a single depth level working group.

[0191] The third type is that the working group includes static pixels, and the depth values ​​of some static pixels are greater than the depth threshold, and the depth values ​​of some static pixels are less than or equal to the depth threshold. In this case, the working group includes both static pixels of the first depth level and static pixels of the second depth level, which is a depth level discontinuous working group.

[0192] The fourth type is that the working group does not include static pixels, and all pixels are dynamic pixels.

[0193] It can be seen that among the above four situations, only the working group that meets the third situation belongs to a static area with discontinuous depth, and such a static area with discontinuous depth is prone to image deformation.

[0194] In this way, the embodiment of the present application refers to the working group that meets the third condition as the third working group.

[0195] Step 4047, determining the third working group as a reconstruction area that meets the first preset condition.

[0196] In the embodiment of the present application, the third working group refers to a working group that satisfies the third condition above, i.e., a working group including static pixels at different depth levels. In other words, the third working group is a static reconstruction area that satisfies the first preset condition. Each third working group corresponds to a static reconstruction area.

[0197] For example, Fig. 10A As shown, the real image frame 20 is divided into a plurality of quadrilateral grids 201, each quadrilateral grid 201 corresponds to a work group, and each work group includes a plurality of pixel points 2011. In this way, after the above steps 4031 to 4037 are processed, the depth level of the pixels in each work group can be determined. For example, work group A1-B1, work group A1-B3, work group A3-B3, work group A4-B2 and work group A4-B3 all belong to the first case above, that is, the above work groups are all single depth levels, and the depth levels are all the first depth levels. Work group A2-B2, work group A3-B1 and work group A3-B2 all belong to the second case above, that is, the above work groups are all single depth levels, and the depth levels are all the second depth levels. Work group A2-B1, work group A2-B3 and work group A4-B1 all belong to the third case above, that is, the above work groups are all work groups with discontinuous depth levels. Working group A1-B2, working group A5-B1, working group A5-B2 and working group A5-B3 all belong to the fourth case, that is, the above working groups do not include static pixels. In this way, it can be determined that working group A2-B1, working group A2-B3 and working group A4-B1 are all static reconstruction areas that meet the first preset condition.

[0198] It can be seen that the above embodiment can determine the static reconstruction area in the real frame image based on the depth information of the real frame image and the pixel attribute information of the real frame image.

[0199] In some embodiments, determining whether the real frame image includes a dynamic reconstruction area that meets the second preset condition can be achieved in the following manner: based on the pixel attribute information of the real frame image, determining the situation of dynamic pixels and static pixels in each working group; then, determining the working group including static pixels and dynamic pixels as the dynamic reconstruction area.

[0200] For example, please see Fig. 10A , the situations of dynamic pixels and static pixels in each working group can include the following situations:

[0201] The first type is that all pixels in the work group are dynamic, for example Fig. 10A Among them, Working Group A1-B2, Working Group A5-B1, Working Group A5-B2 and Working Group A5-B3.

[0202] The second type is that all pixels in the work group are static, for example Fig. 10A Working Group A1-B3, Working Group A2-B1, Working Group A3-B1, etc.

[0203] The third type is that the work group includes both dynamic pixels and static pixels, for example Fig. 10A Among them, Working Group A1-B1, Working Group A2-B2, Working Group A2-B3 and Working Group A4-B3.

[0204] It can be seen that among the above three situations, only the work group corresponding to the third situation is a dynamic reconstruction area that meets the second preset condition. Thus, the embodiment of the present application determines the work group that meets the third situation as a dynamic reconstruction area.

[0205] It should be noted that the same working group may also meet the first preset condition and the second preset condition at the same time. Fig. 10A The working group A2-B3 includes both static pixels and dynamic pixels, and the static pixels in the working group also include static pixels of the first depth level and static pixels of the second depth level. In this case, the working group A2-B3 is a reconstruction area that satisfies both the first preset condition and the second preset condition, that is, a static + dynamic reconstruction area.

[0206] It can be seen that each working group corresponding to the real frame image in the embodiment of the present application can be divided into a non-reconstruction area and a reconstruction area. Among them, the non-reconstruction area refers to an area that does not require grid reconstruction, for example, a grid that is all dynamic pixels, or a grid that is all static pixels and the static pixels are static pixels of a single depth level. The reconstruction area refers to an area that requires grid reconstruction, and the reconstruction area is divided into a static reconstruction area, a dynamic reconstruction area, and a static + dynamic reconstruction area.

[0207] In this way, the reconstruction properties of each reconstruction area can be determined, that is, the depth level and pixel properties of the pixels contained in each reconstruction area.

[0208] In some embodiments, each reconstructed region may be calculated using a compute shader.

[0209] In some embodiments, it is also possible to mark whether each work group is a reconstruction area, as well as reconstruction attribute information such as depth level and pixel attributes of each reconstruction area.

[0210] For example, Fig. 10B For Fig. 10A The corresponding labeled diagram. Fig. 10B As shown, the corresponding reconstruction attribute information, such as reconstruction type, depth level and other information, can be marked for the reconstruction area. For example, the reconstruction area A1-B1 can be marked as a dynamic reconstruction area + the first depth level, the reconstruction area A2-B1 can be marked as a static reconstruction area, the reconstruction area A2-B2 can be marked as a dynamic reconstruction area + the second depth level, the reconstruction area A2-B3 can be marked as a dynamic + static reconstruction area, the reconstruction area A4-B1 can be marked as a static reconstruction area, and the reconstruction area A4-B3 can be marked as a dynamic reconstruction area + the first depth level. Among them, the working group that does not meet the first preset condition and the second preset condition may not be marked, that is, the corresponding area may be blank.

[0211] In the embodiment of the present application, when the real frame image includes a reconstruction area, at least one second grid layer is created. Then, based on the reconstruction properties of the reconstruction area, the pixel points of the reconstruction area are drawn on the first grid layer and at least one second grid layer respectively. In this way, the pixel points of different reconstruction properties in each reconstruction area can be drawn on different grid layers respectively to ensure that the pixel areas in each grid after drawing are all pixel areas with continuous depth.

[0212] For example, Fig. 10A Take the real frame image shown in the figure for example to perform grid reconstruction. After determining that the real frame image includes a reconstruction area that meets the first preset condition and the second preset condition, it can be determined to create two second grid layers. Fig.11 As shown, Fig.11 (a) is the first grid layer M1, Fig.11 (b) is the second grid layer M2, Fig.11 (c) is the second grid layer M3.

[0213] After creating the second grid layer M2 and the second grid layer M3, it can be determined that each grid layer is used to draw pixels of which reconstruction attribute in the reconstruction area. For example, it can be determined that the first grid layer M1 is used to draw static pixels of the first depth level in the reconstruction area, the second grid layer M2 is used to draw static pixels of the second depth level in the reconstruction area, and the second grid layer M3 is used to draw dynamic pixels in the reconstruction area.

[0214] In this way, the pixels to be drawn in each grid layer can be determined based on the reconstruction attributes of each reconstruction area. Fig. 10BThe reconstruction properties of each reconstruction area in the marked image shown determine the pixels drawn in each grid layer.

[0215] Exemplary, combined Fig. 10B and Fig.11 As shown, for a reconstruction area whose reconstruction attribute is dynamic reconstruction + first depth level, the static pixel points of the first depth level in the reconstruction area can be retained in the first grid layer M1, and the dynamic pixel points in the reconstruction area can be drawn into the grid corresponding to the second grid layer M3. For a reconstruction area whose reconstruction attribute is static reconstruction, the static pixel points of the first depth level in the reconstruction area can be retained in the first grid layer M1, and the static pixel points of the second depth level in the reconstruction area can be drawn into the grid corresponding to the second grid layer M2. For a reconstruction area whose reconstruction attribute is dynamic + static reconstruction, the static pixel points of the first depth level in the reconstruction area can be retained in the first grid layer M1, the static pixel points of the second depth level in the reconstruction area can be drawn into the grid corresponding to the second grid layer M2, and the dynamic pixel points in the reconstruction area can be drawn into the grid corresponding to the second grid layer M3.

[0216] like Fig.11 As shown in (a), the first grid layer M1 created in the embodiment of the present application may include all pixels in the real frame. Therefore, the pixels of the first depth level in each reconstructed area may be retained in the first grid layer M1, and the pixels of the second depth level and dynamic pixels in each reconstructed area may be removed. Fig. 10B and Fig.11 As shown in (a), the reconstructed areas including static pixels of the first depth level are reconstructed areas A1-B1, reconstructed areas A2-B2, reconstructed areas A2-B3, reconstructed areas A4-B1 and reconstructed areas A4-B3. In this way, the static pixels of the first depth level in the above-mentioned reconstructed areas are retained in the first grid layer M1, and the static pixels and dynamic pixels of the second depth level in the above-mentioned reconstructed areas are removed. For the reconstructed areas that do not include static pixels of the first depth level (such as the reconstructed areas A2-B2), the first grid layer M1 does not retain any pixels in the reconstructed areas, that is, all pixels are removed.

[0217] Among them, since the pixel points in each non-reconstructed area in the first grid layer M1 are continuous, there is no need to process the non-reconstructed area in the first grid layer M1, that is, the pixel points in the non-reconstructed area in the first grid layer M1 are retained, and there is no need to draw the pixel points in the non-reconstructed area on other second grid layers.

[0218] like Fig.11 As shown in (b), the second grid layer M2 is used to draw static pixel points of the second depth level in each reconstruction area. Fig. 10B and Fig.11As shown in (b), the reconstructed areas including static pixels of the second depth level are the reconstructed areas A2-B1, the reconstructed areas A2-B2, the reconstructed areas A2-B3 and the reconstructed areas A4-B1. In this way, the corresponding static pixels of the second depth level are drawn in the grids corresponding to each reconstructed area in the second grid layer M2. For example, the static pixels of the second depth level in the reconstructed area A2-B1 are drawn in the grid A2-B1 of the second grid layer M2. Similarly, the static pixels of the second depth level in the reconstructed areas A2-B2, the reconstructed areas A2-B3 and the reconstructed areas A4-B1 are drawn in the grids A2-B2, the grid A2-B3 and the grid A4-B1 of the second grid layer M2, respectively. It can be seen that the second grid layer M2 only includes the static pixels of the second depth level in each reconstructed area.

[0219] like Fig.11 As shown in (c), the second grid layer M3 is used to draw dynamic pixels in each reconstruction area. Fig. 10B and Fig.11 As shown in (c), the reconstruction areas including dynamic pixels are reconstruction area A1-B1, reconstruction area A2-B2, reconstruction area A2-B3 and reconstruction area A4-B3. In this way, the corresponding dynamic pixels are drawn in the grids corresponding to each reconstruction area in the second grid layer M3. For example, the dynamic pixels in the reconstruction area A1-B1 are drawn in the grid A1-B1 of the second grid layer M3. Similarly, the dynamic pixels in the reconstruction area A2-B2, the reconstruction area A2-B3 and the reconstruction area A4-B3 are drawn in the grids A2-B2, the grid A2-B3 and the grid A4-B3 of the second grid layer M3, respectively. It can be seen that the second grid layer M3 only includes dynamic pixels in each reconstruction area.

[0220] In summary, for a static reconstruction area that meets the first preset condition, it is necessary to reconstruct the static reconstruction area twice, that is, to draw the pixels at different depth levels in the static reconstruction area in the first grid layer M1 and the second grid layer M2 respectively. For a dynamic reconstruction area that meets the second preset condition, it is also necessary to reconstruct the dynamic reconstruction area twice, that is, to draw the static pixels in the dynamic reconstruction area in the first grid layer M1 or the second grid layer M2, and to draw the dynamic pixels in the dynamic reconstruction area in the second grid layer M3. For a reconstruction area that meets both the first preset condition and the second preset condition, it is necessary to reconstruct the reconstruction area three times, that is, to draw the static pixels at different depth levels in the reconstruction area in the first grid layer M1 and the second grid layer M2, and to draw the dynamic pixels in the reconstruction area in the second grid layer M3.

[0221] like Fig.11As shown, after the above-mentioned grid reconstruction processing, an image is obtained on the first grid layer M1, the second grid layer M2 and the second grid layer M3 respectively (in the embodiment of the present application, the image obtained on the first grid layer may be referred to as the first image, and the image obtained on the second grid layer may be referred to as the second image). Among them, the first image and each second image are local images in the real frame image. For example, compared with the complete real frame image, the first image obtained on the first grid layer M1 does not include static pixels and dynamic pixels at the second depth level in the reconstructed area, the second image obtained on the second grid layer M2 only includes static pixels at the second depth level in the reconstructed area, and the second image obtained on the second grid layer M3 only includes dynamic pixels in the reconstructed area.

[0222] It should be noted that the grids in the first grid layer M1, the second grid layer M2 and the third grid layer M3 correspond to each other one by one, so that after superimposing the image obtained by the first grid layer M1, the image obtained by the second grid layer M2 and the image obtained by the third grid layer M3, a complete real frame image can be obtained.

[0223] It should also be noted that the above embodiment is only illustrative of creating two second grid layers, and does not limit the number of created second grid layers. The number of created second grid layers can be determined based on the reconstruction attributes.

[0224] In some embodiments, when the real frame image only includes the reconstruction area that meets the first preset condition (i.e., only includes the static reconstruction area), a second grid layer can be created. In this way, for the reconstruction area, pixels with different reconstruction attributes can be drawn on the first grid layer and the second grid layer, respectively. For example, static pixels of the first depth level in the reconstruction area can be drawn on the first grid layer; static pixels of the second depth level in the reconstruction area can be drawn on the second grid layer.

[0225] In some embodiments, when the real frame image only includes the reconstruction area that meets the second preset condition (i.e., only includes the dynamic reconstruction area), and the depth level of the static pixels in each reconstruction area is the same, a second grid layer can be created. In this way, the static pixels in the reconstruction area can be drawn in the first grid layer, and the dynamic pixels in the reconstruction area can be drawn in the second grid layer; or the dynamic pixels in the reconstruction area can be drawn in the first grid layer, and the static pixels in the reconstruction area can be drawn in the second grid layer.

[0226] In some embodiments, when the real frame image only includes the reconstruction area that meets the second preset condition (i.e., only includes the dynamic reconstruction area), and the depth levels of the static pixels of each reconstruction area are different, two second grid layers can be created. In this way, the static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, the static pixels of the second depth level in the reconstruction area can be drawn in the first second grid layer, and the dynamic pixels of the reconstruction area can be drawn in the second second grid layer; or, the dynamic pixels of the reconstruction area can be drawn in the first grid layer, the static pixels of the first depth level in the reconstruction area can be drawn in the first second grid layer, and the static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer; or, the static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, the dynamic pixels of the reconstruction area can be drawn in the first second grid layer, and the static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer.

[0227] In some embodiments, when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition (i.e., includes dynamic + static reconstruction areas), two second grid layers can be created. In this way, static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, static pixels of the second depth level in the reconstruction area can be drawn in the first second grid layer, and dynamic pixels of the reconstruction area can be drawn in the second second grid layer; or, dynamic pixels of the reconstruction area can be drawn in the first grid layer, static pixels of the first depth level in the reconstruction area can be drawn in the first second grid layer, and static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer; or, static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, dynamic pixels of the reconstruction area can be drawn in the first second grid layer, and static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer.

[0228] In the above different embodiments, the method of drawing different pixel points in different grid layers based on different reconstruction attributes can refer to Fig.11 The description is not repeated here.

[0229] In this way, by drawing the pixels of different reconstruction attributes in each reconstruction area on different grid layers, the pixel areas in each grid in each grid layer are all pixel areas with continuous depth. Fig.11 In (a), each reconstructed area of ​​the first grid layer M1 includes only static pixels of the first depth level or no pixels, each non-reconstructed area of ​​the first grid layer M1 includes only dynamic pixels or only static pixels, and all static pixels in the same non-reconstructed area have the same depth level. In other words, the pixel areas in each grid in the first grid layer are all continuous pixel areas. Fig.11The second grid layer M2 shown in (b) only includes static pixels at the second depth level, so that the pixel areas in each grid in the second grid layer M2 are also continuous pixel areas. Fig.11 The second grid layer M3 shown in (c) includes only dynamic pixel points, so the pixel areas in each grid in the second grid layer M3 are also continuous pixel areas.

[0230] Step 405 , obtaining a first motion vector corresponding to the first image and a second motion vector corresponding to each second image.

[0231] The resource information acquired in step 401 may include two adjacent real frame images. In this way, the motion vector corresponding to the real frame image may be calculated based on the two adjacent real frame images.

[0232] It should be understood that the motion vector corresponding to the real frame image includes the motion vector of each area in the real frame image. Since the embodiment of the present application disassembles the real frame image into different grid layers by means of grid reconstruction, the motion vector of the real frame image area corresponding to each grid layer can be obtained based on the motion vector corresponding to the real frame image. That is, the first motion vector corresponding to the first image in the first grid layer and the second motion vector corresponding to the second image in each second grid layer are obtained.

[0233] Among them, the first image corresponds to multiple grids in the first grid layer, so that the first motion vector includes motion vectors corresponding to the multiple grids in the first grid layer. Similarly, the second image corresponds to multiple grids in the second grid layer, so that the second motion vector includes motion vectors corresponding to the multiple grids in the second grid layer. Specifically, the motion vector corresponding to the image in the grid can be represented by the motion vectors of the four vertices of each grid. Because in the embodiment of the present application, all pixel areas in each grid are pixel areas with consistent depth after the grid reconstruction process. Therefore, the motion vectors corresponding to each grid in each grid layer are basically the same.

[0234] For example, Fig. 10A Take the motion vector of the reconstructed area A2-B3 as an example. Fig.12 As shown in (a), the reconstructed area A2-B3 includes pixels of three reconstruction attributes, and the motion vectors of the four vertices of the reconstructed area A2-B3 are V0, V1, V2 and V3 respectively. Among them, the motion vectors V0 and V3 are the same, and the motion vectors V0 and V3 represent the motion vectors of the static pixels of the first depth level in the reconstructed area A2-B3, the motion vector V1 represents the motion vector of the dynamic pixels in the reconstructed area A2-B3, and the motion vector V2 represents the motion vector of the static pixels of the second depth level in the reconstructed area A2-B3.

[0235] It can be seen that the motion vectors corresponding to static pixel areas at different depth levels in the real frame image are quite different, and the motion vectors corresponding to static pixel areas and dynamic pixel areas are also quite different.

[0236] Since, in the embodiment of the present application, all pixel regions in each grid are pixel regions with the same depth after the grid reconstruction process, the motion vectors of the four vertices of each grid are basically the same after the process. Fig.12 As shown in (b1), the grid (V0-V0-V0-V0) is the reconstructed grid corresponding to the reconstructed area A2-B3 in the first grid layer M1. The reconstructed grid only includes static pixels of the first depth level. Therefore, the motion vector corresponding to the reconstructed grid is the motion vector corresponding to the static pixels of the first depth level in the reconstructed grid. In this way, the running vectors of the four vertices of the reconstructed grid are all motion vectors V0. Similarly, Fig.12 As shown in (c1), the grid (V2-V2-V2-V2) is the reconstructed grid corresponding to the reconstructed area A2-B3 in the second grid layer M2. The reconstructed grid only includes static pixels at the second depth level. Therefore, the motion vector corresponding to the reconstructed grid is the motion vector corresponding to the static pixels at the second depth level in the reconstructed grid. In this way, the running vectors of the four vertices of the reconstructed grid are all motion vectors V2. Similarly, Fig.12 As shown in (d1), the grid (V1-V1-V1-V1) is the reconstructed grid corresponding to the reconstructed area A2-B3 in the second grid layer M3. The reconstructed grid only includes dynamic pixels. Therefore, the motion vector corresponding to the reconstructed grid is the motion vector corresponding to the dynamic pixel in the reconstructed grid. In this way, the running vectors of the four vertices of the reconstructed grid are all motion vector V0 or motion vector V3.

[0237] It should be understood that the embodiment of the present application can obtain the motion vector corresponding to each grid in each grid layer according to the depth level or pixel attribute of the pixel point in each grid, and the motion vectors of the four vertices of each grid are substantially the same. The above is only an exemplary description of the motion vectors of the grids corresponding to the reconstructed area A2-B3 on each grid layer. The motion vectors of other grids can refer to the description of this example, which will not be repeated here.

[0238] In an embodiment of the present application, a matching calculation method can be selected to calculate the motion vector corresponding to each image area according to the pixel attributes (such as static objects, dynamic objects) and depth information corresponding to each image area in the real frame image, so as to obtain the motion vector corresponding to the real frame image.

[0239] In an optional implementation, some objects in the image (such as people, etc.) are dynamic objects (i.e., objects in motion), and some objects in the image (such as the sky, pillars, etc.) are static objects.

[0240] For example, for the image area corresponding to the static object, the corresponding motion vector can be calculated by the reprojection method; for the image area corresponding to the dynamic object, the corresponding motion vector can be calculated by the optical flow method. The steps of the method for calculating the motion vector by the reprojection method and the optical flow method can refer to the existing technology and will not be repeated here.

[0241] Step 306: Based on the first motion vector, perform image distortion processing on the first image in the first grid layer to generate a first distorted image.

[0242] Step 307 : Based on each second motion vector, perform image distortion processing on each first image in each second grid layer to generate at least one second distorted image.

[0243] When performing image distortion processing on the first image in the first grid layer, each vertex of each grid (also referred to as the first grid) corresponding to the first image can be moved to a corresponding position according to the corresponding motion vector, so as to obtain a first distorted image after image distortion.

[0244] Similarly, when performing image distortion processing on the second image in the second grid layer, each vertex of each grid (also referred to as the second grid) corresponding to the second image can be moved to a corresponding position according to the corresponding motion vector, so as to obtain a second distorted image after the image distortion.

[0245] It should be pointed out that the image distortion processing based on the motion vector refers to moving the position of each mesh vertex according to the motion vector corresponding to each mesh vertex. Whether it is the first mesh layer or the second mesh layer, whether it is to move a static object or a dynamic object, the processing method is the same, the only difference is that the processing is based on different motion vectors at different mesh layers.

[0246] For example, Fig.12 As shown, Fig.12 In (b2), (c2) and (d2), the dotted grids are the positions before the movement, and the solid grids are the positions after the movement. Fig.12 As shown in (b1) and (b2), when the image of the grid (V0-V0-V0-V0) in the first grid layer M1 is distorted, the motion vectors of the four vertices of the grid (V0-V0-V0-V0) are all V0. Therefore, each vertex moves to the corresponding position according to the corresponding motion vector V0, and the grid (V0'-V0'-V0'-V0') is obtained. Similarly, Fig.12As shown in (c1) and (c2), when the image of the grid (V2-V2-V2-V2) in the second grid layer M2 is distorted, the motion vectors of the four vertices of the grid (V2-V2-V2-V2) are all V2. Therefore, each vertex moves to the corresponding position according to the corresponding motion vector V2, and the grid (V2'-V2'-V2'-V2') is obtained. Fig.12 As shown in (d1) and (d2), when the image of the grid (V1-V1-V1-V1) in the second grid layer M3 is distorted, the motion vectors of the four vertices of the grid (V1-V1-V1-V1) are all V1. Therefore, each vertex moves to the corresponding position according to the corresponding motion vector V1, and the grid (V1'-V1'-V1'-V1') is obtained.

[0247] In some embodiments, if the mesh layer is mesh normalized (i.e., each mesh is divided into two triangles, see Figure 8 ), each vertex of each triangle can be moved to a corresponding position according to the motion vector corresponding thereto, so as to obtain a first distorted image and each second distorted image after the image is distorted.

[0248] Since each grid in the embodiment of the present application is a continuous pixel area, the motion vectors of each vertex of each grid or triangle are substantially the same. Thus, when image distortion is performed on each grid or triangle based on the motion vector, each grid or triangle will not be severely deformed, thus ensuring the continuity of the image and avoiding distortion problems.

[0249] In addition, for each mesh (or triangle), the depth information of the mesh can also be set according to the depth values ​​of the mesh vertices. Exemplarily, interpolation processing is performed according to the depth values ​​of each mesh vertex to obtain the depth value of each pixel in the mesh as the depth information of the mesh.

[0250] In this way, when each grid layer moves a grid based on a motion vector, its coverage relationship with other grids in the grid layer can be determined based on the depth information of the grid. For example, when the first grid is moved based on the motion vector in the first grid layer, when each first grid moves to the target position, the coverage relationship between the first grid and other first grids at the target position is determined based on the depth information of the first grid and the depth information of other first grids at the target position. For another example, when the second grid is moved based on the motion vector in the second grid layer, when each second grid moves to the target position, the coverage relationship between the second grid and other second grids at the target position is determined based on the depth information of the second grid and the depth information of other second grids at the target position.

[0251] Therefore, when moving the grid (or triangle) in each grid layer, the pixel depth information is referred to, so that the correct depth order relationship between different objects on each grid layer can be guaranteed.

[0252] Step 408: Superimpose the first distorted image and at least one second distorted image to generate a predicted frame image.

[0253] When the first distorted image and the at least one second distorted image are superimposed, the grid layers may be aligned and then superimposed to ensure the correctness of the position of each object in the predicted frame image after superposition.

[0254] Exemplary, combined Fig.11 and Fig.12 As shown, since the first grid layer M1, the second grid layer M2 and the second grid layer M3 have the same size, the outer edges of the first grid layer M1, the second grid layer M2 and the second grid layer M3 can be aligned during the superposition process. In this way, images belonging to the same grid in each grid layer can be superimposed (for example, Fig.12 (b2), (c2) and (d2) are superimposed) to obtain a complete and continuous image.

[0255] When the first distorted image and at least one second distorted image are superimposed, the covering relationship between the first grid in the first grid layer and the second grid in each second grid layer can also be determined according to the depth information of each first grid and each second grid, so as to ensure the correct depth order relationship between different objects after superposition.

[0256] Fig.13A An example of a predicted frame image is shown as an example. Fig.13A In the predicted frame image shown, region 601 uses the existing technical solution to perform image distortion processing, and region 602 uses the solution provided by this embodiment to perform image distortion processing. Among them, region 601 and region 602 are only regions where the depth information of static objects is discontinuous. By comparing region 601 and region 602, it can be seen that the image of region 601 is distorted, especially the edge area of ​​the static object is severely deformed, while the picture effect of region 602 is better, the edge area of ​​the static object is clear, and there is no deformation of the edge area of ​​the static object.

[0257] Fig. 13B An exemplary comparison diagram of a predicted frame image is shown. Fig. 13B The predicted frame image shown in (1) is generated by performing image distortion processing using the solution provided by this embodiment, and Fig. 13B The predicted frame image shown in (2) is generated by image distortion processing using existing technical solutions. Fig. 13B Region 603 in (1) and Fig. 13BRegion 604 in (2) is the same image region, which is a region where the depth information of static objects is discontinuous. Fig. 13B From (1) and (2), it can be seen that the image in area 604 is distorted, especially the edge area of ​​the static object is severely deformed, while the image effect in area 603 is better, the edge area of ​​the static object is clear, and there is no deformation of the edge area of ​​the static object.

[0258] In this way, the image processing method provided by the embodiment of the present application can identify the discontinuous image area by the above-mentioned grid reconstruction method, no matter how complex the geometric edges of the discontinuous image area are (for example, the hair of the characters, leaves, etc. in the real frame image), and draw the pixel points of different reconstruction attributes in the discontinuous image area on different grid layers to ensure that the pixel areas in each grid are continuous pixel areas. In this way, when the continuous pixel areas in each grid are subjected to image distortion processing based on the corresponding motion vectors, the grid will not be severely deformed, thereby ensuring the continuity of the image. It can be seen that the image processing method provided by the embodiment of the present application can solve the problem of distortion of discontinuous image areas with complex geometric edges and enhance the user's gaming experience.

[0259] It should be noted that the above embodiment is only an illustrative description of the scheme of creating two second grid layers when the real frame image includes a reconstructed area that satisfies both the first preset condition and the second preset condition, and does not represent a limitation on the image processing method provided in the embodiment of the present application.

[0260] In some embodiments, when the real frame image includes a reconstructed area that satisfies both the first preset condition and the second preset condition, a second grid layer may also be created. In this case, the dynamic pixels in the real frame image and the static pixels of the first depth level may be retained in the first grid layer, and then the static pixels of the second depth level in the reconstructed area are drawn in the second grid layer. In this way, when the first grid layer is subjected to image distortion processing, the first grid layer may be subjected to image distortion processing based on the motion vectors of the static pixels of the first depth level in the first grid layer; and then, the first grid layer may be subjected to image distortion processing based on the motion vectors of the dynamic pixels in the first grid layer. The image distortion processing method for the second grid layer is the same as that in the above-mentioned embodiment and will not be described in detail here. In this way, the scheme can also solve the problem of image area distortion with discontinuous depth values ​​of complex geometric edges.

[0261] It should also be noted that after the image distortion phase operation and the superposition processing phase operation are completed, the related operations of the rasterization phase and the pixel processing phase can be continued to be completed to generate a predicted frame image. Among them, the parts of the predicted frame image generation process that are not fully explained in this embodiment can refer to the existing technology and will not be repeated here.

[0262] Since the moving speeds of various objects in the original video are different, the predicted frame images calculated by the electronic device based on the motion vector may have image information overlap or image information missing. That is, the predicted frame images obtained above are the predicted frame images to be completed.

[0263] Therefore, it is necessary to perform image completion on the predicted frame image obtained above to obtain a completed predicted frame image.

[0264] Furthermore, the completed prediction frame image may be synthesized with the UI information to obtain a target prediction frame, and the target prediction frame is sent for display.

[0265] In this way, after the prediction frame image is completed, it is synthesized with the UI information to obtain the target prediction frame, and then the target prediction frame can be output to the electronic device display screen for display.

[0266] It should be pointed out that the above process is only explained by processing one prediction frame as an example, and the process for processing each prediction frame is the same, so it will not be repeated here. The order in which the target prediction frames are sent and displayed is related to the interpolation method of the prediction frames. You can refer to the existing technology and will not repeat it here. You can also refer to the existing technology for the parts that are not explained in detail in this process, so it will not be repeated here.

[0267] Games with high refresh rates on mobile platforms are often prone to heating or lag due to power consumption or CPU and GPU capability limitations, thus affecting the user experience. To enhance the smoothness of the game experience, game developers often lower the original refresh rate and then generate predicted frames in the image space to halve the rendering operations at the same frame rate, thereby significantly reducing the power consumption of smart terminals and reducing the heating of smart terminals.

[0268] In the case of image distortion in the predicted frame image, especially the problem of complex geometric edge deformation with discontinuous depth information, the image processing method provided by the embodiment of the present application can be used to solve the problem well, thereby improving the image quality of the predicted frame image. In this way, the smoothness of the game can be ensured to remain unchanged when the number of rendering frames is halved, the power consumption of the smart terminal is reduced, and the heat generation of the smart terminal is reduced, thereby improving the user experience.

[0269] In other interpolation application scenarios, if there are areas in the image with discontinuous depth information (or large depth differences), the image processing method provided in the embodiment of the present application can also be used to perform image distortion processing to improve the image quality of the predicted frame image, which will not be elaborated here.

[0270] The various method embodiments described in this document may be independent solutions or may be combined according to internal logic, and these solutions all fall within the protection scope of this application.

[0271] The above embodiments introduce the image processing method provided by the present application. It is understandable that, in order to realize the above functions, the image processing device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0272] The embodiment of the present application can divide the functional modules of the image processing device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0273] Above, combined Figures 1 to 13B The method provided by the embodiment of the present application is described in detail. Fig.14 The device provided in the embodiment of the present application is described in detail. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment, so the contents not described in detail can be referred to the method embodiment above, and for the sake of brevity, they will not be repeated here.

[0274] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.14 As shown, the device 500 includes:

[0275] A resource information acquisition module 510, configured to acquire resource information, wherein the resource information includes a real frame image, depth information of the real frame image, and pixel attribute information;

[0276] A first grid layer creation module 520, used to create a first grid layer of the real frame image;

[0277] The reconstruction module 530 is used to create at least one second grid layer when the real frame image includes a reconstruction area that meets the first preset condition and / or the second preset condition, and draw pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction properties of the reconstruction area; wherein the first preset condition is that the reconstruction area includes static pixel points with different depth levels, and the second preset condition is that the reconstruction area includes both static pixel points and dynamic pixel points; the reconstruction properties of the pixels drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties;

[0278] The motion vector acquisition module 540 is used to acquire a first motion vector corresponding to the first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image area included in the first grid layer; the second image refers to a real frame image area included in the second grid layer;

[0279] A first distorted image generating module 550, configured to perform image distortion processing on the first image based on the first motion vector to generate a first distorted image;

[0280] A second distorted image generating module 560, configured to perform image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image;

[0281] The superposition processing module 570 is used to perform superposition processing on the first distorted image and the at least one second distorted image to generate a predicted frame image.

[0282] In one possible implementation, the reconstruction module 530 is specifically used to create a second grid layer when the real frame image includes a reconstructed area that satisfies the first preset condition; draw static pixel points of a first depth level in the reconstructed area in the first grid layer; and draw static pixel points of a second depth level in the reconstructed area in the second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0283] In one possible implementation, the reconstruction module 530 is specifically used to create a second grid layer when the real frame image includes a reconstruction area that meets the second preset condition and the depth levels of static pixels in each of the reconstruction areas are the same; draw the static pixels in the reconstruction area in the first grid layer, and draw the dynamic pixels in the reconstruction area in the second grid layer; or draw the dynamic pixels in the reconstruction area in the first grid layer, and draw the static pixels in the reconstruction area in the second grid layer.

[0284] In a possible implementation, the reconstruction module 530 is specifically used to create two second grid layers when the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of static pixels of each of the reconstruction areas are different, or when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition; draw static pixels of a first depth level in the reconstruction area in the first grid layer, draw static pixels of a second depth level in the reconstruction area in the first second grid layer, and draw dynamic pixels of the reconstruction area in the second second grid layer; or draw dynamic pixels of the reconstruction area in the first grid layer, draw static pixels of a first depth level in the reconstruction area in the first second grid layer, and draw static pixels of a second depth level in the reconstruction area in the second second grid layer; or draw static pixels of a first depth level in the reconstruction area in the first grid layer, draw dynamic pixels of the reconstruction area in the first second grid layer, and draw static pixels of a second depth level in the reconstruction area in the second second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0285] In a possible implementation, it also includes a marking module, which is used to mark the reconstruction information of each of the reconstruction areas to obtain a marking map; the reconstruction information includes reconstruction type information, depth level information and pixel attribute information; based on the reconstruction type information, depth level information and pixel attribute information, the number of the second grid layers and the reconstruction attributes of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are determined.

[0286] In a possible implementation, the method further includes a reconstruction area determination module, which is used to divide the real frame image into multiple working groups, each of which includes multiple pixel points; based on the depth information of the real frame image and the pixel attribute information of the real frame image, determine that the working group including static pixel points of different depth levels in the real frame image is the reconstruction area that meets the first preset condition; and / or, based on the pixel attribute information of the real frame image, determine that the working group including both static pixel points and dynamic pixel points in the real frame image is the reconstruction area that meets the second preset condition.

[0287] In one possible implementation, the reconstruction area determination module is specifically used to determine a first working group including static pixels in the multiple working groups based on the pixel attribute information of the real frame image; determine a second working group in the first working group including pixels whose depth values ​​are within a preset range and whose depth value difference with surrounding pixels exceeds a preset threshold based on the depth information of the real frame image; determine the maximum depth value of each of the second working groups; determine the minimum value of each of the maximum depth values ​​as the depth threshold of the real frame image; determine a third working group including static pixels of a first depth level and static pixels of a second depth level in the multiple working groups, wherein the depth value of the static pixels of the first depth level is greater than the depth threshold, and the depth value of the static pixels of the second depth level is less than or equal to the depth threshold; and determine the third working group as the reconstruction area that meets the first preset condition.

[0288] In a possible implementation, the first warped image generating module 550 is specifically used to set the depth information of each first mesh in the first mesh layer; for each first mesh in the first mesh layer, determine the first motion vector of each vertex of the first mesh, and move the first mesh according to the first motion vector of each vertex of the first mesh; when each first mesh moves to a target position, determine the coverage relationship between the first mesh and other first meshes at the target position according to the depth information of the first mesh and the depth information of other first meshes at the target position.

[0289] In a possible implementation, the second warped image generating module 560 is specifically used to set the depth information of each second mesh in the second mesh layer; for each second mesh in the second mesh layer, determine the second motion vector of each vertex of the second mesh, and move the second mesh according to the second motion vector of each vertex of the second mesh; when each second mesh moves to the target position, determine the coverage relationship between the second mesh and other second meshes at the target position according to the depth information of the second mesh and the depth information of other second meshes at the target position.

[0290] In a possible implementation, the overlay processing module 570 is specifically used to determine the covering relationship between the first grid in the first grid layer and the second grid in each second grid layer according to the depth information of each first grid and each second grid when overlaying the first distorted image and the at least one second distorted image.

[0291] According to the method provided by an embodiment of the present application, an embodiment of the present application also provides an electronic device, including a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the image processing method.

[0292] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0293] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0294] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0295] According to the method provided in the embodiments of the present application, the embodiments of the present application also provide a computer program product, which includes: a computer program or instructions, when the computer program or instructions are run on a computer, the computer executes the method of any one of the embodiments of the method.

[0296] According to the method provided in the embodiments of the present application, the embodiments of the present application also provide a computer storage medium, which stores a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the method of any one of the embodiments of the method.

[0297] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0298] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0299] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0300] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0301] In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit.

[0302] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0303] The devices, computer storage media, and computer program products provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the methods provided above, and will not be repeated here.

[0304] It should be understood that in each embodiment of the present application, the execution order of each step should be determined by its function and internal logic, and the size of the sequence number of each step does not mean the order of execution and does not limit the implementation process of the embodiment.

[0305] Each part of this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, computer storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0306] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0307] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire resource information, the resource information including a real frame image, depth information of the real frame image, and pixel attribute information of the real frame image; The pixel attribute information is used to characterize static pixels and dynamic pixels; Creating a first grid layer of the real frame image; In the case where the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, at least one second grid layer is created, and based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer respectively; wherein the first preset condition is that the reconstruction area includes static pixel points with different depth levels, and the second preset condition is that the reconstruction area includes both static pixel points and dynamic pixel points; the reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties; Acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image region included in the first grid layer; the second image refers to a real frame image region included in the second grid layer; Based on the first motion vector, performing image distortion processing on the first image to generate a first distorted image; Based on each of the second motion vectors, performing image distortion processing on each of the second images respectively to generate at least one second distorted image; The first warped image and the at least one second warped image are superimposed to generate a predicted frame image.

2. The method according to claim 1, characterized in that When the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, creating at least one second grid layer, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction attribute of the reconstruction area, comprises: In the case where the real frame image includes a reconstruction area that satisfies the first preset condition, creating a second grid layer; Draw static pixel points of a first depth level in the reconstruction area in the first grid layer; Static pixel points of a second depth level in the reconstructed area are drawn in the second grid layer; wherein the first depth level and the second depth level are different depth levels.

3. The method according to claim 1, characterized in that When the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, creating at least one second grid layer, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction attribute of the reconstruction area, comprises: In the case where the real frame image includes a reconstructed area that meets the second preset condition and the depth levels of static pixels of each of the reconstructed areas are the same, creating a second grid layer; Draw static pixel points in the reconstruction area on the first grid layer, and draw dynamic pixel points in the reconstruction area on the second grid layer; or, Dynamic pixel points in the reconstruction area are drawn in the first grid layer, and static pixel points in the reconstruction area are drawn in the second grid layer.

4. The method according to claim 1, characterized in that: When the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, creating at least one second grid layer, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction attribute of the reconstruction area, comprises: When the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of static pixels of the reconstruction areas are different, or when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition, two second grid layers are created; Drawing static pixel points of a first depth level in the reconstructed area on the first grid layer, drawing static pixel points of a second depth level in the reconstructed area on the first second grid layer, and drawing dynamic pixel points in the reconstructed area on the second second grid layer; or, Draw dynamic pixels in the reconstructed area on the first grid layer, draw static pixels at a first depth level in the reconstructed area on the first second grid layer, and draw static pixels at a second depth level in the reconstructed area on the second second grid layer; or, Drawing static pixel points of a first depth level in the reconstructed area on the first grid layer, drawing dynamic pixel points in the reconstructed area on a first second grid layer, and drawing static pixel points of a second depth level in the reconstructed area on a second second grid layer; The first depth level and the second depth level are different depth levels.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Marking the reconstruction attribute information of each of the reconstruction areas to obtain a marking map; the reconstruction attribute information includes reconstruction type information and depth level information; Based on the reconstruction type information and the depth level information, the number of the second grid layers and reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are determined.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Dividing the real frame image into a plurality of working groups, each of the working groups comprising a plurality of pixel points; Based on the depth information of the real frame image and the pixel attribute information of the real frame image, determining that the working group including static pixels of different depth levels in the real frame image is the reconstruction area that meets the first preset condition; and / or, Based on the pixel attribute information of the real frame image, the working group including both static pixel points and dynamic pixel points in the real frame image is determined as the reconstruction area that meets the second set condition.

7. The method according to claim 6, characterized in that The determining, based on the depth information of the real frame image and the pixel attribute information of the real frame image, that the working group including static pixels of different depth levels in the real frame image is the reconstruction area satisfying the first preset condition comprises: Based on the pixel attribute information of the real frame image, determining a first working group including static pixels among the multiple working groups; Based on the depth information of the real frame image, determining in the first working group a second working group including a pixel whose depth value is within a preset range and a difference between the depth value of the pixel and surrounding pixels exceeds a preset threshold; Determine a maximum depth value of each of the second working groups; Determine the minimum value among the maximum depth values ​​as the depth threshold of the real frame image; Determine a third working group including static pixels of a first depth level and static pixels of a second depth level from among the multiple working groups, wherein the depth values ​​of the static pixels of the first depth level are greater than the depth threshold, and the depth values ​​of the static pixels of the second depth level are less than or equal to the depth threshold; The third working group is determined to be the reconstruction area that meets the first preset condition.

8. The method according to claim 1, characterized in that The step of performing image distortion processing on the first image based on the first motion vector to generate a first distorted image includes: Setting depth information of each first grid in the first grid layer; For each first mesh in the first mesh layer, determine a first motion vector of each vertex of the first mesh, and move the first mesh according to the first motion vector of each vertex of the first mesh; When each of the first grids moves to a target position, the covering relationship between the first grid and other first grids at the target position is determined according to the depth information of the first grid and the depth information of other first grids at the target position.

9. The method according to claim 8, characterized in that The step of performing image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image includes: Setting depth information of each second grid in the second grid layer; For each second mesh in the second mesh layer, determine a second motion vector of each vertex of the second mesh, and move the second mesh according to the second motion vector of each vertex of the second mesh; When each second grid moves to the target position, the covering relationship between the second grid and other second grids at the target position is determined according to the depth information of the second grid and the depth information of other second grids at the target position.

10. The method according to claim 9, characterized in that The superimposing the first distorted image and the at least one second distorted image to generate a predicted frame image comprises: When the first distorted image and the at least one second distorted image are superimposed, the covering relationship between the first grid in the first grid layer and the second grid in each second grid layer is determined according to the depth information of each first grid and each second grid.

11. The method according to claim 1, characterized in that: The resource information is resource information in the game application.

12. An image processing device, characterized in that: The device comprises: A resource information acquisition module, used to acquire resource information, wherein the resource information includes a real frame image, depth information of the real frame image, and pixel attribute information; the pixel attribute information is used to characterize static pixel points and dynamic pixel points; A first grid layer creation module, used to create a first grid layer of the real frame image; A reconstruction module, configured to create at least one second grid layer when the real frame image includes a reconstruction area that satisfies a first preset condition and / or a second preset condition, and to draw pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on reconstruction properties of the reconstruction area; wherein the first preset condition is that the reconstruction area includes static pixel points with different depth levels, and the second preset condition is that the reconstruction area includes both static pixel points and dynamic pixel points; the reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties; A motion vector acquisition module, used to acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image area included in the first grid layer; the second image refers to a real frame image area included in the second grid layer; A first distorted image generating module, configured to perform image distortion processing on the first image based on the first motion vector to generate a first distorted image; A second distorted image generating module, configured to perform image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image; The superposition processing module is used to perform superposition processing on the first distorted image and the at least one second distorted image to generate a predicted frame image.

13. An electronic device, characterized in that: It comprises a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1-11.

14. A computer storage medium, characterized in that: The computer storage medium stores a computer program or instruction. When the computer program or instruction is executed, the method according to any one of claims 1 to 11 is executed.

Images