Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and belongs to the technical field of intelligent terminals, and the method comprises the steps: creating at least one second grid layer under the condition that a real frame image comprises a reconstruction region meeting a first preset condition and / or a second preset condition, and carrying out the reconstruction of the reconstruction region based on the reconstruction attribute of the reconstruction region, and respectively drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer. Therefore, no matter how the geometric edge of the discontinuous image area is complex, pixel points with different attributes in the discontinuous image area can be drawn on different grid layers through a grid reconstruction mode, so that the pixel area in each grid is a continuous pixel area. Therefore, when image distortion processing is carried out on the grids with continuous depths, serious deformation of grid graphs cannot be caused, so that the continuity of the images is ensured, the problem of distortion of discontinuous image areas of complex geometric edges is solved, and the game experience of users is improved.

Description

Image processing method, device and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 3, 2023, with application number 202310809777.1 and invention name “A method, device and electronic device for image processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of smart terminal technology, and in particular to an image processing method, device and electronic device. Background Art

[0003] Video interpolation technology aims to improve the frame rate and smoothness of videos, making them appear smoother. By inserting a predicted frame (also known as an intermediate frame or transition frame) between adjacent real frames (or original frames), the frame rate of a video can be doubled. The quality of the predicted frame is directly related to the smoothness of the video. The movement direction and speed of each object in the image are calculated from the information of two adjacent frames, and the objects in the image are moved accordingly to generate the predicted frame.

[0004] In current prediction frame generation technology, image warping is a good way to ensure image continuity. However, in some image regions, image warping may cause local image distortion, making the difference between the predicted frame image and the real frame image obvious, greatly reducing the user's visual experience.

[0005] Summary of the Invention

[0006] The present application provides an image processing method, device and electronic device, which can solve the problem of local image distortion, especially the distortion problem of discontinuous image areas with complex geometric edges.

[0007] In a first aspect, the present application provides an image processing method, comprising: acquiring resource information, the resource information comprising a real frame image, depth information of the real frame image, and pixel attribute information of the real frame image; the pixel attribute information being used to characterize static pixel points and dynamic pixel points; creating a first grid layer of the real frame image; creating at least one second grid layer when the real frame image comprises a reconstruction area that satisfies a first preset condition and / or a second preset condition, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer, respectively, based on the reconstruction attributes of the reconstruction area; wherein the first preset condition is that the reconstruction area comprises static pixel points of different depth levels, and the second preset condition is that the reconstruction area comprises both static pixel points and dynamic pixel points; The reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth level and pixel properties; a first motion vector corresponding to the first image in the first grid layer and a second motion vector corresponding to each second image in each second grid layer are obtained; the first image refers to the real frame image area included in the first grid layer; the second image refers to the real frame image area included in the second grid layer; based on the first motion vector, the first image is subjected to image warping processing to generate a first distorted image; based on each second motion vector, the second image is subjected to image warping processing to generate at least one second distorted image; the first distorted image and the at least one second distorted image are superimposed to generate a predicted frame image.

[0008] Thus, in this image processing method, the discontinuous image area in the real frame image, i.e., the reconstruction area, is first identified. Then, according to the reconstruction attributes of the reconstruction area, multiple grid layers are constructed, and the pixel points of different attributes in the reconstruction area are drawn respectively on different grid layers. Then, based on the corresponding motion vectors, image distortion processing is performed on each grid layer respectively. In this way, no matter how complex the geometric edges of the discontinuous image area are, the pixel points of different attributes in the discontinuous image area can be drawn respectively on different grid layers by means of grid reconstruction to ensure that the pixel areas in each grid are continuous pixel areas. In this way, when the image distortion processing is performed on each depth-continuous grid, the grid graphics will not be severely deformed, thereby ensuring the continuity of the image, solving the problem of distortion of discontinuous image areas with complex geometric edges, and improving the user's gaming experience.

[0009] In one feasible manner, when the real frame image includes a reconstruction area that satisfies a first preset condition and / or a second preset condition, at least one second grid layer is created, and based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer, respectively, including: when the real frame image includes a reconstruction area that satisfies the first preset condition, a second grid layer is created; static pixel points of a first depth level in the reconstruction area are drawn on the first grid layer; static pixel points of a second depth level in the reconstruction area are drawn on the second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0010] Thus, for the reconstruction area, pixels with different reconstruction attributes can be drawn on the first and second grid layers, respectively. For example, static pixels at the first depth level in the reconstruction area can be drawn on the first grid layer, and static pixels at the second depth level in the reconstruction area can be drawn on the second grid layer.

[0011] In one feasible manner, when the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, at least one second grid layer is created, and based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer, respectively, including: when the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of the static pixel points of each of the reconstruction areas are the same, creating a second grid layer; drawing the static pixel points in the reconstruction area in the first grid layer, and drawing the dynamic pixel points in the reconstruction area in the second grid layer; or drawing the dynamic pixel points in the reconstruction area in the first grid layer, and drawing the static pixel points in the reconstruction area in the second grid layer.

[0012] In this way, the static pixels and the dynamic pixels in the reconstructed area can be drawn on different grid layers respectively to obtain respective continuous pixel areas.

[0013] In an achievable manner, when the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, at least one second grid layer is created, and based on the reconstruction attributes of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer, respectively, including: when the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of the static pixel points of each of the reconstruction areas are different, or when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition, two second grid layers are created; static pixel points of the first depth level of the reconstruction area are drawn on the first grid layer, and static pixel points of the second depth level of the reconstruction area are drawn on the first second grid layer; Draw the static pixel points of the second depth level in the reconstructed area on the second second grid layer, and draw the dynamic pixel points in the reconstructed area on the second second grid layer; or draw the dynamic pixel points in the reconstructed area on the first grid layer, draw the static pixel points of the first depth level in the reconstructed area on the first second grid layer, and draw the static pixel points of the second depth level in the reconstructed area on the second second grid layer; or draw the static pixel points of the first depth level in the reconstructed area on the first grid layer, draw the dynamic pixel points in the reconstructed area on the first second grid layer, and draw the static pixel points of the second depth level in the reconstructed area on the second second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0014] In this way, static pixels and dynamic pixels at different depth levels in the reconstructed area can be drawn on different grid layers respectively to obtain respective continuous pixel areas.

[0015] In one feasible manner, the method further includes: marking the reconstruction attribute information of each of the reconstructed areas to obtain a marking map; the reconstruction attribute information includes reconstruction type information and depth level information; based on the reconstruction type information and depth level information, determining the number of the second grid layers and the reconstruction attributes of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer.

[0016] In this way, the depth level, pixel attributes and other information of each region in the real frame image can be determined based on the label map to further perform grid reconstruction processing.

[0017] In one feasible manner, the method further includes: dividing the real frame image into multiple working groups, each working group including multiple pixel points; based on the depth information of the real frame image and the pixel attribute information of the real frame image, determining that the working group including static pixel points of different depth levels in the real frame image is the reconstruction area that meets the first preset condition; and / or, based on the pixel attribute information of the real frame image, determining that the working group including both static pixel points and dynamic pixel points in the real frame image is the reconstruction area that meets the second preset condition.

[0018] In one feasible manner, the working group including static pixels of different depth levels in the real frame image is determined as the reconstructed area that meets the first preset condition based on the depth information of the real frame image and the pixel attribute information of the real frame image, including: based on the pixel attribute information of the real frame image, determining a first working group including static pixels among the multiple working groups; based on the depth information of the real frame image, determining a second working group including pixels in the first working group whose depth values ​​are within a preset range and the difference between the depth values ​​of the pixels and the surrounding pixels exceeds a preset threshold; determining the maximum depth value of each of the second working groups; determining the minimum value of each of the maximum depth values ​​as the depth threshold of the real frame image; determining a third working group including static pixels of a first depth level and static pixels of a second depth level among the multiple working groups, wherein the depth values ​​of the static pixels of the first depth level are greater than the depth threshold, and the depth values ​​of the static pixels of the second depth level are less than or equal to the depth threshold; and determining the third working group as the reconstructed area that meets the first preset condition.

[0019] In this way, a depth threshold can be calculated based on the depth information of static pixels in the real frame image, and then the reconstruction areas that meet the first preset condition are determined based on the depth threshold.

[0020] In one feasible manner, the image warping processing is performed on the first image based on the first motion vector to generate a first distorted image, including: setting the depth information of each first grid in the first grid layer; determining the first motion vector of each vertex of the first grid for each first grid in the first grid layer, and moving the first grid according to the first motion vector of each vertex of the first grid; when each first grid moves to a target position, determining the coverage relationship between the first grid and other first grids at the target position according to the depth information of the first grid and the depth information of other first grids at the target position.

[0021] In this way, since the pixel depth information is referred to when one grid is moved based on the motion vector at the first grid layer, the correct depth order relationship between different objects at the first grid layer can be guaranteed.

[0022] In one feasible manner, the image warping processing is performed on each second image based on each second motion vector to generate at least one second warped image, including: setting depth information of each second grid in the second grid layer; determining, for each second grid in the second grid layer, the second motion vector of each vertex of the second grid, and moving the second grid according to the second motion vector of each vertex of the second grid; when each second grid moves to a target position, determining the overlapping relationship between the second grid and other second grids at the target position based on the depth information of the second grid and the depth information of other second grids at the target position.

[0023] In this way, since the pixel depth information is referred to when one grid is moved based on the motion vector on the second grid layer, the correct depth order relationship between different objects on the second grid layer can be guaranteed.

[0024] In one feasible manner, the superimposing the first warped image and the at least one second warped image to generate a predicted frame image includes: when superimposing the first warped image and the at least one second warped image, determining, based on depth information of each first grid and each second grid, an overlapping relationship between a first grid in the first grid layer and a second grid in each second grid layer.

[0025] In this way, the overlapping relationship between the first grid in the first grid layer and the second grid in each second grid layer is determined based on the depth information of each first grid and each second grid, which can ensure the correct depth order relationship between different objects after superposition.

[0026] In one achievable manner, the resource information is resource information in a game application.

[0027] In a second aspect, the present application provides an image processing device, comprising:

[0028] A resource information acquisition module, configured to acquire resource information, wherein the resource information includes a real frame image, depth information of the real frame image, and pixel attribute information;

[0029] A first grid layer creation module, configured to create a first grid layer of the real frame image;

[0030] a reconstruction module, configured to, when the real frame image includes a reconstruction region that satisfies a first preset condition and / or a second preset condition, create at least one second grid layer, and based on reconstruction properties of the reconstruction region, draw pixel points of the reconstruction region on the first grid layer and the at least one second grid layer, respectively; wherein the first preset condition is that the reconstruction region includes static pixels of different depth levels, and the second preset condition is that the reconstruction region includes both static pixels and dynamic pixels; and the reconstruction properties of the pixels drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties;

[0031] a motion vector acquisition module, configured to acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image region included in the first grid layer; the second image refers to a real frame image region included in the second grid layer;

[0032] a first distorted image generating module, configured to perform image distortion processing on the first image based on the first motion vector to generate a first distorted image;

[0033] a second distorted image generating module, configured to perform image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image;

[0034] The superposition processing module is used to perform superposition processing on the first distorted image and the at least one second distorted image to generate a predicted frame image.

[0035] In a third aspect, the present application provides an electronic device comprising a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes a method as described in any one of the first aspects.

[0036] In a fourth aspect, the present application provides a computer storage medium, characterized in that a computer program or instruction is stored in the computer storage medium, and when the computer program or instruction is executed, the method as described in any one of the first aspects is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] FIG1 is a schematic diagram of a process for generating a prediction frame according to an embodiment of the present application;

[0039] FIG2 is a schematic diagram of an image gridding method provided in an embodiment of the present application;

[0040] FIG3 is a schematic diagram of an image grid distortion provided by an embodiment of the present application;

[0041] FIG4 is a schematic diagram of a grid subdivision provided in an embodiment of the present application;

[0042] FIG5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0043] FIG6 is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0044] FIG7 is a flowchart of an image processing method provided by an embodiment of the present application;

[0045] FIG8 is a schematic diagram of grid standardization provided in an embodiment of the present application;

[0046] FIG9 is a flowchart of determining a reconstruction area that meets a first preset condition according to an embodiment of the present application;

[0047] FIG10A is a schematic diagram of pixel distribution provided in an embodiment of the present application;

[0048] FIG10B is a marking diagram provided in an embodiment of the present application;

[0049] FIG11 is a schematic diagram of pixel distribution of each grid layer after grid reconstruction processing provided by an embodiment of the present application;

[0050] FIG12 is a schematic diagram of image distortion provided by an embodiment of the present application;

[0051] FIG13A is a schematic diagram of one of the application scenarios provided in an embodiment of the present application;

[0052] FIG13B is a schematic diagram of one of the application scenarios provided in an embodiment of the present application;

[0053] FIG14 is a schematic structural diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0055] With the advancement of electronic technology, the refresh rates of electronic devices (such as mobile phones) are increasing, and the frame rates of video sources are also increasing. This has led to an increase in memory usage and rendering power consumption in electronic devices. However, due to the power consumption of electronic devices or the limitations of CPU and GPU capabilities, high refresh and frame rates often cause electronic devices to overheat or experience lag, thus affecting the user experience.

[0056] Take mobile gaming as an example. With the development and popularity of large-scale mobile games, the rendering pipelines of modern mobile games are becoming increasingly complex, and resource loads are also increasing. At the same time, the increased computing power of modern mobile phones has not been able to meet the needs of modern large-scale mobile games, and mobile phones are limited by limited battery capacity and heat dissipation capabilities. Therefore, developers focus on how to reduce unnecessary rendering overhead, improve the limited frame rate and smoothness, and reduce heat generation without significantly affecting game quality.

[0057] In order to improve the user experience, developers often reduce the frame rate of the video source, and then increase the video frame rate through video interpolation technology, so as to achieve half the rendering at the same frame rate, thereby significantly reducing the power consumption of the mobile phone and reducing the heat of the mobile phone.

[0058] Video interpolation technology aims to improve the frame rate and smoothness of videos, making them appear smoother. By inserting a predicted frame (also known as an intermediate frame or transition frame) between adjacent real frames (or original frames, etc.), the frame rate of the video can be doubled. The image quality of the predicted frame is directly related to the smoothness of the video. The image information of two adjacent real frames is used to calculate the movement direction and speed of each object in the image, and the objects in the image are moved accordingly to obtain the predicted frame.

[0059] As the name implies, real frames refer to the original image frames in the video source, not the image frames generated by prediction methods. For example, real frames can be image frames drawn by developers when developing applications such as games, or image frames captured by image acquisition devices (such as cameras) during video production.

[0060] Video interpolation methods can be categorized as predicted frame interpolation and predicted frame extrapolation. In the predicted frame interpolation method, a predicted frame image is calculated based on two adjacent real-world frames and inserted between them to increase the video frame rate. In the predicted frame extrapolation method, a predicted frame image is calculated based on two adjacent real-world frames and inserted after them as the next frame to increase the video frame rate.

[0061] The following briefly describes the process of generating a prediction frame using a game scene as an example. It should be noted that the relevant drawings shown in this embodiment are all shown in grayscale form.

[0062] As shown in Figure 1, the first real frame and the second real frame are two adjacent image frames in a game application. Assume that the second real frame is the current image frame and the first real frame is the image frame immediately preceding the second real frame. Therefore, the mobile phone can calculate a predicted frame based on the first and second real frames. Using predicted frame interpolation as an example, the smart terminal can insert the calculated predicted frame between the two real frames for display, thereby increasing the frame rate of the game video and achieving half the rendering speed at the same frame rate.

[0063] In this game application, the real frame includes image information and UI (User Interface) information. Among them, image information can be understood as the picture information in the video source, and UI information refers to the overall design information of the software's human-computer interaction, operation logic, and interface aesthetics. When the mobile phone calculates the predicted frame based on adjacent real frames, since the interface UI information will not change, the mobile phone can first separate the image information and UI information, and only calculate the predicted frame based on the image information, thereby reducing the amount of image data processing and improving the accuracy of the predicted frame. Therefore, as shown in Figure 1, when calculating the predicted frame, the mobile phone can first extract the first real frame image 101 and the first UI 102 in the first real frame, and extract the second real frame image 103 and the second UI 104 in the second real frame, and then perform an image prediction operation based on the first real frame image 101 and the second real frame image 103 to obtain the predicted frame image 107.

[0064] Continuing with FIG1 , when calculating the predicted frame image 107, the mobile phone may first calculate the motion vector (MV) 105 of the first real frame image 101 and the second real frame image 103, and perform image warping on the second real frame image 103 based on the motion vector 105 to obtain the predicted frame image 106. Image warping refers to changing the position of image pixels through certain transformations, such as translation, rotation, scaling, affine, perspective, and cylindrical transformations.

[0065] Image warping can effectively ensure the continuous movement of objects in the image. However, due to the different movement speeds of objects in the image, image information may be overlapped or missing in the predicted frame image 106 obtained through image warping. The mobile phone also needs to perform image completion (Blur) on the predicted frame image 106 to obtain the predicted frame image 107. At this point, the mobile phone fuses the second UI 104 with the predicted frame image 107 to obtain the predicted frame 108 calculated based on the first and second real frames.

[0066] It should be noted that both the image distortion processing and the UI information fusion processing can be based on the information in the first real frame image or the second real frame image, and this embodiment does not limit this.

[0067] Among them, in computer graphics, grid geometry is usually used as the basic unit of image processing. Similarly, in the image distortion processing stage mentioned above, the mobile phone also uses grid geometry as the basic unit for image distortion processing. That is to say, the mobile phone performs image distortion processing by forming a geometric group of triangles or quadrilaterals in the image. Taking the quadrilateral grid as an example, as shown in Figure 2, the real frame image 20 is a global working group. A plurality of quadrilateral grids 201 can be divided in the real frame image 20, and each quadrilateral grid 201 includes a plurality of pixel points 2011. Among them, each quadrilateral grid 201 is a local working group, which is a basic unit for the mobile phone to perform image distortion processing.

[0068] However, for more complex image scenes (for example, image scenes in games), dynamic objects and static objects may be included, and the depth difference of static objects is relatively large. In addition, since most games are virtual worlds, when users play mobile games, user operations often cause large-angle shaking of the game scene, unlike the physical world where the movement of objects follows the laws of physics. Therefore, in the case of more complex game scenes, large-angle scene shaking may only cause part of the objects in the image to move more, rather than causing all objects in the image to move more. That is to say, in the game scene, the distribution of MV images calculated based on continuous real frame images is uneven, especially in discontinuous image areas, the difference in MV values ​​is relatively large. Among them, discontinuous image areas include static image areas with large differences in depth information and / or image areas that include both dynamic pixels and static pixels.

[0069] In this way, if the same grid includes discontinuous image areas, using the grid as a basic unit for image distortion processing will cause local stretching of the image, resulting in distortion of the predicted frame image, which in turn will cause a large difference between the predicted frame and the real frame, resulting in a poor visual experience for users.

[0070] For example, FIG3 (1) shows a grayscale image of a motion vector frame calculated based on a game image. In the grayscale image shown in FIG3 (1), the lighter the grayscale (the closer the grayscale value is to 1), the smaller the motion vector is, and the darker the grayscale (the closer the grayscale value is to 0), the larger the motion vector is. Continuing to refer to FIG3 (1), the motion vectors on the left and right sides of the edge area of ​​the static object in the image frame vary greatly. Combining FIG3 (1) and (2), taking area 301 as an example, sub-area 3011 corresponds to the column area in the image, and sub-area 3012 corresponds to the sky area in the image. The motion vector corresponding to the column is significantly smaller than the motion vector corresponding to the sky. In other words, the motion vector values ​​on the left and right sides of the column edge area vary greatly. In this case, once the motion vectors corresponding to different pixel areas in the same grid differ greatly, image distortion processing of the grid based on the motion vector will inevitably cause the grid graphic to be severely deformed, and even cause the problem of partial grid graphic overlap, and the accuracy of the object depth information in the image cannot be guaranteed. As shown in (3) in FIG3 , among the four vertices of grid 201, the motion vectors corresponding to V0 and V1 are significantly different from the motion vectors corresponding to V2 and V3. After moving each vertex according to its corresponding motion vector, grid 201 is distorted into grid 201'. Not only is the grid pattern severely deformed, but it may also cover other grids. This will cause distortion in some areas of the predicted frame image, especially in the discontinuous edge areas, as shown in (4) in FIG3 , such as discontinuous edge areas 310 and discontinuous edge areas 320 shown in (4) in FIG3 .

[0071] As a result, in the predicted frame image generated after image distortion processing, discontinuous image areas will have local stretching problems, which are quite different from the real scene, resulting in a poor visual experience for users.

[0072] To solve the above technical problems, in some embodiments, for image areas with discontinuous depth values, the mobile phone can subdivide the grid based on the depth information of the pixel points, and perform image distortion processing using the subdivided geometric figures as units, so as to solve the problem of local stretching in image areas with discontinuous depth values ​​in the predicted frame image, so as to better retain continuous graphic information, thereby improving the picture quality of the predicted frame image and providing the user with a better visual experience.

[0073] For example, as shown in FIG4 , if there are two intersection points Point1 and Point2 between the edge of a static object and the mesh 201, and the two intersection points Point1 and Point2 are located on different sides of the mesh 201, the mesh 201 can be subdivided into multiple triangles based on the two intersection points Point1 and Point2, so that the depth information in each subdivided triangle will not differ too much. In this way, when image distortion is performed on these subdivided triangles, each subdivided triangle is moved according to the motion vector corresponding to its own vertex. For example, triangle S-Tringle1 (V0-V1-Point1) moves to triangle S-Tringle1 (V0'-V1'-Point1'), and triangle S-Tringle2 (V0-Point1-Point2) moves to triangle S-Tringle2 (V0'-Point1'-Point2'). Similarly, triangle S-Tringle1'(Point1-Point2-V2) moves to triangle S-Tringle1'(Point1"-Point2"-V2"), and triangle S-Tringle2'(V2-Point2-V3) moves to triangle S-Tringle2'(V2"-Point2"-V3"). Since the depth information of each triangle after subdivision is consistent, their movement direction is also relatively consistent. This can avoid the problem of severe mesh deformation caused by excessive differences in mesh depth information, thereby improving image quality and making the predicted frame image closer to the real frame image.

[0074] However, since the above solution can only subdivide the grid to achieve the corresponding technical effect when there are two intersection points between the edge of the static object and the grid 201, and these two intersection points are located on different sides of the grid 201, the above solution is only applicable to solving the distortion problem of regular geometric edges, but cannot solve the distortion problem of complex geometric edges.

[0075] To solve the above-mentioned technical problems, in the image processing method provided in the embodiment of the present application, the discontinuous image area in the real frame image is first identified (the embodiment of the present application refers to such an area as a reconstruction area). Then, according to the reconstruction attributes of the reconstruction area, a plurality of grid layers are constructed, and the pixels of different attributes in the reconstruction area are drawn respectively in different grid layers. Based on the motion vectors corresponding to each other, the image distortion processing is performed on each grid layer respectively. In this way, no matter how complex the geometric edges of the image area with discontinuous depth values ​​are, the pixels of different attributes in the image area with discontinuous depth values ​​can be drawn respectively in different grid layers by means of grid reconstruction to ensure that the pixel areas in each grid are all pixel areas with continuous depth. In this way, when the grids with continuous depth are subjected to image distortion processing, the grid graphics will not be severely deformed, thereby ensuring the continuity of the image, solving the problem of distortion of the image area with discontinuous depth values ​​of complex geometric edges, and improving the user's gaming experience.

[0076] The image processing method provided in the embodiment of the present application can be applied to electronic devices. Optionally, the electronic device 100 can be a terminal, which can also be called a terminal device, a smart terminal, etc. The terminal can be a device such as a cellular phone or a tablet computer, which is not limited in this application.

[0077] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in Figure 5, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0078] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0080] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0081] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0082] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0083] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.

[0084] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0085] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0086] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0087] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0088] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0089] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0090] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0091] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0092] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0093] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0094] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0095] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0096] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0097] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0098] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0099] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0100] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0101] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0102] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0103] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0104] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0105] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0106] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0107] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0108] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0109] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0110] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0111] Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Electronic device 100 can use speaker 170A to listen to music or make hands-free calls. Multiple speakers 170A can be provided in electronic device 100. For example, one speaker 170A can be provided on the top of electronic device 100, another speaker 170A can be provided on the bottom, and so on.

[0112] Receiver 170B, also known as an "earpiece," is used to convert audio signals into sound signals. When electronic device 100 receives a call or voice message, the user can hold receiver 170B close to their ear to listen to the voice. In some embodiments, speaker 170A and receiver 170B may be integrated into one component, although this is not a limitation of the present invention.

[0113] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0114] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0115] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0116] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0117] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0118] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0119] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0120] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0121] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0122] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0123] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0124] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0125] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0126] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0127] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0128] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0129] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0130] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0131] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0132] FIG6 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0133] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0134] The application layer can include a series of application packages.

[0135] As shown in FIG6 , the application package may include applications such as games, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0136] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0137] As shown in FIG6 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0138] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0139] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0140] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0141] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0142] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0143] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0144] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0145] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0146] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0147] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0148] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0149] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0150] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0151] A 2D graphics engine is a drawing engine for 2D drawings.

[0152] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0153] Below, an embodiment of the image processing method provided by this application is described.

[0154] 405 is a flow chart of an image processing method provided in an embodiment of the present application. As shown in FIG7 , the method may include the following steps:

[0155] Step 401: Obtain resource information.

[0156] Resource information refers to the image data of the game app, which is intended to be displayed in the game app interface. Among them, resource information can include real frame image information and UI information.

[0157] Among them, UI information refers to the overall design information of the software's human-computer interaction, operating logic, and interface aesthetics.

[0158] The real frame image information includes but is not limited to the real frame image, depth information of the real frame image, and pixel attribute information of the real frame image. The pixel attribute information is used to characterize whether a pixel is a static pixel or a dynamic pixel.

[0159] Dynamic pixels are pixels that change due to factors such as dynamic objects, lighting, and materials in the scene. For example, moving objects, flickering lights, and changing textures. Static pixels are usually composed of static scene elements, such as fixed terrain, stationary buildings, and fixed textures.

[0160] In some embodiments, taking a game interpolation scenario as an example, a game app can send resource information to a 3D graphics processing library, which then renders the game app's image. This allows the electronic device to capture the rendered real-world frame image information.

[0161] Step 402: Create a first grid layer of the real frame image.

[0162] In the embodiment of the present application, the real frame image can be any one of the two adjacent real frame images used to calculate the predicted frame image. For example, referring to FIG1 , the real frame image can be the first real frame image or the second real frame image.

[0163] Creating the first grid layer of the real frame image, which may also be referred to as meshing the real frame image, is for dividing the real frame image into geometric shapes that are easier to process, such as polygons, triangles, or quadrilaterals.

[0164] The created first grid layer is consistent with the size of the real frame image, so that each graphic filled in the first grid layer can cover a pixel area of ​​the real frame image.

[0165] For example, in the embodiment of the present application, each grid in the first grid layer is a quadrilateral grid, that is, the real frame image is gridded according to the size of the quadrilateral, wherein the quadrilateral can be a square.

[0166] In some embodiments, the first mesh layer may be normalized. If each mesh in the first mesh layer is a non-triangular mesh, such as a quadrilateral mesh, the non-triangular mesh may be normalized so that each non-triangular mesh in the first mesh layer is divided into multiple triangles to facilitate graphics calculations.

[0167] For example, if the first mesh layer created is a quadrilateral mesh, each quadrilateral in the quadrilateral mesh may be divided into two triangles to achieve mesh standardization.

[0168] As shown in (1) of Figure 8 , each quadrilateral mesh 201 in the first mesh layer can be divided into triangle 201_1 and triangle 201_2. As shown in (2) of Figure 8 , taking the quadrilateral mesh Square (V0-V1-V2-V3) as an example, the quadrilateral mesh can be divided into triangular mesh Tringle1 (V0-V1-V2) and triangular mesh Tringle2 (V2-V3-V0).

[0169] It should be noted that the first grid layer can cover every pixel in the real frame image. Each grid in the first grid layer corresponds to a pixel area (including multiple pixels) in the real frame image, and the pixel area corresponding to each grid may be an area with continuous depth values, an area with discontinuous depth values, an area that is entirely dynamic pixels, or an area that includes both dynamic pixels and static pixels. Based on the foregoing description, it can be seen that due to the continuous depth value area and the area that includes both dynamic pixels and static pixels, distortion is prone to occur. Therefore, the embodiment of the present application can first identify the grids that are prone to distortion, and then perform grid reconstruction processing on these grids, so as to draw the pixels with different attributes in the same grid into the grids corresponding to different grid layers, so that the attributes of the pixels in each grid after the grid reconstruction processing are consistent. For example, after grid reconstruction, the depth level of each pixel in the same grid is the same, and each pixel in the same grid is either a static pixel or a dynamic pixel. In this way, since the corresponding pixel points in each grid are continuous, when the image distortion processing is performed on each grid layer based on the motion vector corresponding to each grid, the continuous graphic information of each object in the real frame image can be better retained, so as to avoid the distortion problem in the edge area of ​​each object in the real frame image.

[0170] The following describes a specific implementation method for grid reconstruction processing provided in an embodiment of the present application.

[0171] Step 403 : When the real frame image includes a reconstructed region that meets the first preset condition and / or the second preset condition, create at least one second grid layer.

[0172] Step 404 : Based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and at least one second grid layer.

[0173] Image deformation is prone to occur in static areas with discontinuous depth, as well as in areas containing both dynamic and static pixels. Therefore, embodiments of the present application need to identify these areas prone to image deformation and perform mesh reconstruction on these areas.

[0174] The grid that meets the first preset condition is a static region with discontinuous depth (i.e., a grid including static pixels at different depth levels), and the grid that meets the second preset condition is a grid including both static pixels and dynamic pixels. In this embodiment of the application, the grid that meets the first and / or second preset conditions is referred to as a reconstruction region or a reconstruction grid.

[0175] In the embodiment of the present application, each grid that meets the first preset condition and / or the second preset condition corresponds to a reconstruction area. The reconstruction area that meets the first preset condition can be called a static reconstruction area, and the reconstruction area that meets the second preset condition can be called a dynamic reconstruction area.

[0176] In some embodiments, determining whether a real frame image includes a static reconstruction area that meets a first preset condition can be achieved in the following manner: first, determining a depth threshold of static pixel points in the real frame image; then, based on the depth threshold, determining whether each static pixel point in each grid including the static pixel point is continuous; if not, the grid is a static reconstruction area.

[0177] Specifically, as shown in FIG9 , the method for determining whether the real frame image includes a static reconstruction area that meets the first preset condition can refer to the description of the following steps 4041 to 4047 .

[0178] Step 4041: Divide the real frame image into multiple working groups, each working group including multiple pixels.

[0179] That is, the real frame image is a global workgroup, and the divided workgroups are local workgroups. Each workgroup corresponds to the size of each grid in the first grid layer and is a basic unit for image distortion processing by the electronic device (see Figure 2).

[0180] Step 4042: Based on the pixel attribute information of the real frame image, determine a first working group including static pixels in multiple working groups.

[0181] The electronic device can identify a work group including static pixels, ie, the first work group, based on the pixel attribute information.

[0182] Step 4043 , based on the depth information of the real frame image, determine in the first working group a second working group including a pixel whose depth value is within a preset range and a depth difference between the pixel and surrounding pixels exceeds a preset threshold.

[0183] The preset depth range may be understood as a depth range of interest, within which edges of static objects are likely to be distorted when the image is distorted.

[0184] If the depth difference between any pixel in the current first working group and the surrounding pixels exceeds a preset threshold, it indicates that the depths of the static pixels in the current working group are discontinuous. Otherwise, it indicates that the depths of the static pixels in the current working group are consistent. In this embodiment of the present application, the working group in the first working group that has a depth discontinuity area in the depth range of interest is determined as the second working group.

[0185] Step 4044: Determine the maximum depth value of each second working group.

[0186] Since the second working group is a depth-discontinuous area, that is, the second working group includes static pixels with different depth values, the maximum depth value of the static pixels in each second working group can be determined.

[0187] Step 4045: Determine the minimum value among the maximum depth values ​​as the depth threshold of the real frame image.

[0188] Step 4046 : Determine a third working group from the plurality of working groups, including static pixels at the first depth level and static pixels at the second depth level.

[0189] In the embodiment of the present application, the depth value of each static pixel can be compared with a depth threshold. If the depth value of the static pixel is greater than the depth threshold, the static pixel belongs to a static pixel of the first depth level, for example, the first depth level is far. If the depth value of the static pixel is less than or equal to the depth threshold, the static pixel belongs to a static pixel of the second depth level, for example, the first depth level is near. It can be seen that the first depth level and the second depth level belong to different depth levels, and the depth difference between the first depth level and the second depth level is relatively large.

[0190] In this way, the embodiment of the present application can determine the depth level of each work group based on the depth threshold. The depth level of each work group can include the following situations:

[0191] The first type is that the work group includes static pixels, and the depth value of each static pixel is greater than the depth threshold. In this case, all static pixels in the work group belong to the first depth level, that is, a single depth level work group.

[0192] The second type is that the work group includes static pixels, and the depth value of each static pixel is less than or equal to the depth threshold. In this case, all static pixels in the work group belong to the second depth level, that is, a single depth level work group.

[0193] The third type is when the work group includes static pixels, and the depth values ​​of some static pixels are greater than the depth threshold, while the depth values ​​of some static pixels are less than or equal to the depth threshold. In this case, the work group includes both static pixels of the first depth level and static pixels of the second depth level, which is a depth-discontinuous work group.

[0194] The fourth type is that the working group does not include static pixels, and all pixels are dynamic pixels.

[0195] It can be seen that among the above four situations, only the work group that meets the third situation belongs to a static area with discontinuous depth. Such a static area with discontinuous depth is prone to image deformation.

[0196] In this way, the embodiment of the present application refers to the working group that meets the above third condition as the third working group.

[0197] Step 4047: Determine the third working group as a reconstruction area that meets the first preset condition.

[0198] In this embodiment of the present application, the third workgroup refers to a workgroup that satisfies the third condition described above, i.e., a workgroup that includes static pixels at different depth levels. In other words, the third workgroup is a static reconstruction area that satisfies the first preset condition. Each third workgroup corresponds to a static reconstruction area.

[0199] For example, as shown in FIG10A , a real image frame 20 is divided into multiple quadrilateral grids 201 , each of which corresponds to a workgroup. Each workgroup includes multiple pixels 2011 . Thus, after processing through steps 4031 to 4037 , the depth level of the pixels within each workgroup can be determined. For example, workgroups A1-B1, A1-B3, A3-B3, A4-B2, and A4-B3 all fall into the first scenario, meaning they all have a single depth level and are all at the first depth level. Workgroups A2-B2, A3-B1, and A3-B2 all fall into the second scenario, meaning they all have a single depth level and are all at the second depth level. Workgroups A2-B1, A2-B3, and A4-B1 all fall into the third scenario, meaning they all have discontinuous depth levels. Working groups A1-B2, A5-B1, A5-B2, and A5-B3 all fall into the fourth scenario described above, meaning they do not include static pixels. Thus, working groups A2-B1, A2-B3, and A4-B1 can be determined to be static reconstruction areas that meet the first pre-determined condition.

[0200] It can be seen that the above embodiment can determine the static reconstruction area in the real frame image based on the depth information of the real frame image and the pixel attribute information of the real frame image.

[0201] In some embodiments, determining whether the real frame image includes a dynamic reconstruction area that meets the second preset condition can be achieved in the following manner: based on the pixel attribute information of the real frame image, determining the situation of dynamic pixel points and static pixel points in each working group; then, determining the working group including static pixel points and dynamic pixel points as the dynamic reconstruction area.

[0202] For example, please continue to refer to FIG10A , the situations of dynamic pixels and static pixels in each work group may include the following situations:

[0203] The first type is that all the pixels in the working group are dynamic pixels, such as the working group A1-B2, the working group A5-B1, the working group A5-B2 and the working group A5-B3 in Figure 10A.

[0204] The second type is that all the pixels in the working group are static pixels, such as the working group A1-B3, working group A2-B1, working group A3-B1, etc. in Figure 10A.

[0205] The third type is that the working group includes both dynamic pixels and static pixels, such as the working group A1-B1, working group A2-B2, working group A2-B3 and working group A4-B3 in Figure 10A.

[0206] It can be seen that, among the above three situations, only the work group corresponding to the third situation is a dynamic reconstruction area that meets the second preset condition. Thus, the embodiment of the present application determines the work group that meets the third situation as a dynamic reconstruction area.

[0207] It should be noted that the same workgroup can also meet both the first and second preset conditions. For example, workgroup A2-B3 in Figure 10A includes both static and dynamic pixels, and the static pixels within this workgroup also include static pixels at the first depth level and static pixels at the second depth level. In this case, workgroup A2-B3 is a reconstruction area that meets both the first and second preset conditions, i.e., a static + dynamic reconstruction area.

[0208] It can be seen that the working groups corresponding to the real frame images in the embodiments of the present application can be divided into non-reconstruction areas and reconstruction areas. Among them, the non-reconstruction area refers to an area that does not require grid reconstruction, for example, a grid consisting entirely of dynamic pixels, or a grid consisting entirely of static pixels, and the static pixels are static pixels of a single depth level. The reconstruction area refers to an area that requires grid reconstruction, and the reconstruction area is further divided into static reconstruction area, dynamic reconstruction area, and static + dynamic reconstruction area.

[0209] In this way, the reconstruction properties of each reconstruction area can be determined, that is, the depth level and pixel properties of the pixels contained in each reconstruction area.

[0210] In some embodiments, each reconstructed region may be obtained by calculation using a compute shader.

[0211] In some embodiments, it is also possible to mark whether each work group is a reconstruction area, as well as reconstruction attribute information such as depth level and pixel attributes of each reconstruction area.

[0212] Exemplarily, FIG10B is a marking diagram corresponding to FIG10A. As shown in FIG10B , corresponding reconstruction attribute information, such as reconstruction type, depth level and other information, can be marked for the reconstruction area. For example, the reconstruction area A1-B1 can be marked as a dynamic reconstruction area + first depth level, the reconstruction area A2-B1 can be marked as a static reconstruction area, the reconstruction area A2-B2 can be marked as a dynamic reconstruction area + second depth level, the reconstruction area A2-B3 can be marked as a dynamic + static reconstruction area, the reconstruction area A4-B1 can be marked as a static reconstruction area, and the reconstruction area A4-B3 can be marked as a dynamic reconstruction area + first depth level. Among them, the working group that does not meet the first preset condition and the second preset condition may not be marked, that is, the corresponding area may be blank.

[0213] In an embodiment of the present application, when a real frame image includes a reconstruction region, at least one second grid layer is created. Then, based on the reconstruction properties of the reconstruction region, pixels of the reconstruction region are drawn on the first grid layer and at least one second grid layer. In this way, pixels of different reconstruction properties in each reconstruction region can be drawn on different grid layers, ensuring that the pixel regions within each grid are depth-continuous after drawing.

[0214] For example, let's take the mesh reconstruction process for the real frame image shown in Figure 10A as an example. After determining that the real frame image includes a reconstructed region that satisfies the first and second preset conditions, it can be determined that two second mesh layers are created. As shown in Figure 11, Figure 11 (a) shows the first mesh layer M1, Figure 11 (b) shows the second mesh layer M2, and Figure 11 (c) shows the second mesh layer M3.

[0215] After creating the second grid layer M2 and the second grid layer M3, it can be determined which grid layer is used to draw pixels of which reconstruction attributes in the reconstruction area. For example, the first grid layer M1 can be used to draw static pixels of the first depth level in the reconstruction area, the second grid layer M2 can be used to draw static pixels of the second depth level in the reconstruction area, and the second grid layer M3 can be used to draw dynamic pixels in the reconstruction area.

[0216] In this way, the pixels to be drawn in each grid layer can be determined based on the reconstruction attributes of each reconstruction area. Specifically, the pixels to be drawn in each grid layer can be determined based on the reconstruction attributes of each reconstruction area in the labeling diagram shown in FIG10B.

[0217] For example, in combination with FIG10B and FIG11 , for a reconstruction area whose reconstruction attribute is dynamic reconstruction + first depth level, the static pixel points of the first depth level in the reconstruction area can be retained in the first grid layer M1, and the dynamic pixel points in the reconstruction area can be drawn into the grid corresponding to the second grid layer M3. For a reconstruction area whose reconstruction attribute is static reconstruction, the static pixel points of the first depth level in the reconstruction area can be retained in the first grid layer M1, and the static pixel points of the second depth level in the reconstruction area can be drawn into the grid corresponding to the second grid layer M2. For a reconstruction area whose reconstruction attribute is dynamic + static reconstruction, the static pixel points of the first depth level in the reconstruction area can be retained in the first grid layer M1, the static pixel points of the second depth level in the reconstruction area can be drawn into the grid corresponding to the second grid layer M2, and the dynamic pixel points in the reconstruction area can be drawn into the grid corresponding to the second grid layer M3.

[0218] As shown in (a) of FIG11 , the first grid layer M1 created in an embodiment of the present application may include all pixels in the real frame. Therefore, the pixels of the first depth level in each reconstruction area may be retained in the first grid layer M1, and the pixels and dynamic pixels of the second depth level in each reconstruction area may be removed. In combination with FIG10B and FIG11 (a), the reconstruction areas including static pixels of the first depth level are reconstruction area A1-B1, reconstruction area A2-B2, reconstruction area A2-B3, reconstruction area A4-B1, and reconstruction area A4-B3. In this way, the static pixels of the first depth level in the above-mentioned reconstruction areas are retained in the first grid layer M1, and the static pixels and dynamic pixels of the second depth level in the above-mentioned reconstruction areas are removed. For the reconstruction area that does not include static pixels of the first depth level (such as reconstruction area A2-B2), the first grid layer M1 does not retain any pixels in the reconstruction area, that is, all pixels are removed.

[0219] Among them, since the pixel points in each non-reconstructed area in the first grid layer M1 are continuous, there is no need to process the non-reconstructed area in the first grid layer M1, that is, the pixel points in the non-reconstructed area in the first grid layer M1 are retained, and there is no need to draw the pixel points in the non-reconstructed area on other second grid layers.

[0220] As shown in FIG11(b), the second grid layer M2 is used to draw static pixels at the second depth level in each reconstruction area. Combining FIG10B and FIG11(b), the reconstruction areas including static pixels at the second depth level are reconstruction area A2-B1, reconstruction area A2-B2, reconstruction area A2-B3, and reconstruction area A4-B1. Thus, static pixels at the second depth level are drawn in the grids corresponding to each reconstruction area in the second grid layer M2. For example, static pixels at the second depth level in reconstruction area A2-B1 are drawn within grid A2-B1 of the second grid layer M2. Similarly, static pixels at the second depth level in reconstruction area A2-B2, reconstruction area A2-B3, and reconstruction area A4-B1 are drawn within grids A2-B2, A2-B3, and A4-B1 of the second grid layer M2, respectively. It can be seen that the second grid layer M2 only includes static pixels at the second depth level in each reconstruction area.

[0221] As shown in (c) of Figure 11, the second grid layer M3 is used to draw dynamic pixel points in each reconstruction area. In combination with Figure 10B and Figure 11 (c), the reconstruction areas including dynamic pixel points are reconstruction area A1-B1, reconstruction area A2-B2, reconstruction area A2-B3, and reconstruction area A4-B3. In this way, the corresponding dynamic pixel points are drawn in the grids corresponding to each reconstruction area in the second grid layer M3. For example, the dynamic pixel points in the reconstruction area A1-B1 are drawn in the grid A1-B1 of the second grid layer M3. Similarly, the dynamic pixel points in the reconstruction area A2-B2, reconstruction area A2-B3, and reconstruction area A4-B3 are drawn in the grids A2-B2, grid A2-B3, and grid A4-B3 of the second grid layer M3, respectively. It can be seen that the second grid layer M3 only includes dynamic pixel points in each reconstruction area.

[0222] In summary, for a static reconstruction area that meets the first preset condition, it is necessary to reconstruct the static reconstruction area twice, that is, to draw pixel points at different depth levels in the static reconstruction area in the first grid layer M1 and the second grid layer M2 respectively. For a dynamic reconstruction area that meets the second preset condition, it is also necessary to reconstruct the dynamic reconstruction area twice, that is, to draw static pixel points in the dynamic reconstruction area in the first grid layer M1 or the second grid layer M2, and to draw dynamic pixel points in the dynamic reconstruction area in the second grid layer M3. For a reconstruction area that meets both the first preset condition and the second preset condition, it is necessary to reconstruct the reconstruction area three times, that is, to draw static pixel points at different depth levels in the reconstruction area in the first grid layer M1 and the second grid layer M2, and to draw dynamic pixel points in the reconstruction area in the second grid layer M3.

[0223] As shown in Figure 11, after the above-mentioned grid reconstruction processing, an image is obtained on the first grid layer M1, the second grid layer M2, and the second grid layer M3 respectively (in this embodiment of the application, the image obtained on the first grid layer may be referred to as the first image, and the image obtained on the second grid layer may be referred to as the second image). The first image and each second image are local images in the real frame image. For example, compared with the complete real frame image, the first image obtained on the first grid layer M1 does not include static pixels and dynamic pixels at the second depth level in the reconstructed area, the second image obtained on the second grid layer M2 only includes static pixels at the second depth level in the reconstructed area, and the second image obtained on the second grid layer M3 only includes dynamic pixels in the reconstructed area.

[0224] It should be noted that the grids in the first grid layer M1, the second grid layer M2 and the third grid layer M3 correspond to each other one by one. In this way, after superimposing the image obtained by the first grid layer M1, the image obtained by the second grid layer M2 and the image obtained by the third grid layer M3, a complete real frame image can be obtained.

[0225] It should also be noted that the above embodiment is merely an example of creating two second grid layers, and does not limit the number of second grid layers to be created. The number of second grid layers to be created can be determined based on the reconstruction attributes.

[0226] In some embodiments, when the real frame image only includes a reconstructed region that meets the first preset condition (i.e., only includes a static reconstructed region), a second grid layer can be created. Thus, for the reconstructed region, pixels with different reconstruction attributes can be drawn on the first grid layer and the second grid layer, respectively. For example, static pixels at a first depth level in the reconstructed region can be drawn on the first grid layer, while static pixels at a second depth level in the reconstructed region can be drawn on the second grid layer.

[0227] In some embodiments, when the real frame image only includes reconstructed areas that meet the second preset condition (i.e., only includes dynamic reconstructed areas), and the depth levels of the static pixels in each reconstructed area are the same, a second grid layer can be created. In this way, the static pixels in the reconstructed area can be drawn in the first grid layer, and the dynamic pixels in the reconstructed area can be drawn in the second grid layer; or the dynamic pixels in the reconstructed area can be drawn in the first grid layer, and the static pixels in the reconstructed area can be drawn in the second grid layer.

[0228] In some embodiments, when the real frame image only includes the reconstruction area that meets the second preset condition (i.e., only includes the dynamic reconstruction area), and the depth levels of the static pixels of each reconstruction area are different, two second grid layers can be created. In this way, the static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, the static pixels of the second depth level in the reconstruction area can be drawn in the first second grid layer, and the dynamic pixels of the reconstruction area can be drawn in the second second grid layer; or the dynamic pixels of the reconstruction area can be drawn in the first grid layer, the static pixels of the first depth level in the reconstruction area can be drawn in the first second grid layer, and the static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer; or the static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, the dynamic pixels of the reconstruction area can be drawn in the first second grid layer, and the static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer.

[0229] In some embodiments, when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition (i.e., includes dynamic + static reconstruction areas), two second grid layers can be created. In this way, static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, static pixels of the second depth level in the reconstruction area can be drawn in the first second grid layer, and dynamic pixels of the reconstruction area can be drawn in the second second grid layer; or, dynamic pixels of the reconstruction area can be drawn in the first grid layer, static pixels of the first depth level in the reconstruction area can be drawn in the first second grid layer, and static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer; or, static pixels of the first depth level in the reconstruction area can be drawn in the first grid layer, dynamic pixels of the reconstruction area can be drawn in the first second grid layer, and static pixels of the second depth level in the reconstruction area can be drawn in the second second grid layer.

[0230] Among them, in the above different embodiments, the method of drawing different pixel points in different grid layers based on different reconstruction attributes can refer to the description of Figure 11, which will not be repeated here.

[0231] In this way, by drawing the pixels of different reconstruction attributes in each reconstruction area on different grid layers, the pixel areas in each grid in each grid layer are all pixel areas with continuous depth. For example, each reconstruction area of ​​the first grid layer M1 shown in (a) of Figure 11 only includes static pixels of the first depth level or does not include pixels, each non-reconstruction area of ​​the first grid layer M1 only includes dynamic pixels or only includes static pixels, and the depth level of all static pixels in the same non-reconstruction area is the same. In other words, the pixel areas in each grid in the first grid layer are all continuous pixel areas. The second grid layer M2 shown in (b) of Figure 11 only includes static pixels of the second depth level, so that the pixel areas in each grid in the second grid layer M2 are also continuous pixel areas. Similarly, the second grid layer M3 shown in (c) of Figure 11 only includes dynamic pixels, so that the pixel areas in each grid in the second grid layer M3 are also continuous pixel areas.

[0232] Step 405 : Obtain a first motion vector corresponding to the first image and a second motion vector corresponding to each second image.

[0233] The resource information acquired in step 401 may include two adjacent real frame images. In this way, the motion vector corresponding to the real frame image can be calculated based on the two adjacent real frame images.

[0234] It should be understood that the motion vector corresponding to the real frame image includes the motion vectors of each region in the real frame image. Because the embodiments of the present application decompose the real frame image into different grid layers through grid reconstruction, the motion vector of the real frame image region corresponding to each grid layer can be obtained based on the motion vector corresponding to the real frame image. In other words, the first motion vector corresponding to the first image in the first grid layer and the second motion vector corresponding to the second image in each second grid layer are obtained.

[0235] In which, the first image corresponds to multiple grids in the first grid layer, so that the first motion vector includes motion vectors corresponding to multiple grids in the first grid layer. Similarly, the second image corresponds to multiple grids in the second grid layer, so that the second motion vector includes motion vectors corresponding to multiple grids in the second grid layer. Specifically, the motion vector corresponding to the image in the grid can be represented by the motion vectors of the four vertices of each grid. Since in the embodiment of the present application, after the grid reconstruction process, all pixel areas in each grid are pixel areas with consistent depth. Therefore, the motion vectors corresponding to each grid in each grid layer are basically the same.

[0236] For example, taking the motion vectors of the reconstructed area A2-B3 in Figure 10A as an example, as shown in Figure 12(a), the reconstructed area A2-B3 includes pixels of three reconstruction attributes, and the motion vectors of the four vertices of the reconstructed area A2-B3 are V0, V1, V2, and V3, respectively. Among them, motion vectors V0 and V3 are the same. Motion vectors V0 and V3 represent the motion vectors of static pixels at the first depth level in the reconstructed area A2-B3, motion vector V1 represents the motion vector of dynamic pixels in the reconstructed area A2-B3, and motion vector V2 represents the motion vector of static pixels at the second depth level in the reconstructed area A2-B3.

[0237] It can be seen that the motion vectors corresponding to static pixel areas at different depth levels in the real frame image are quite different, and the motion vectors corresponding to static pixel areas and dynamic pixel areas are also quite different.

[0238] Since, in the embodiment of the present application, all pixel regions within each grid are pixel regions of the same depth after the grid reconstruction process, the motion vectors of the four vertices of each grid after the processing are substantially the same. As shown in (b1) in FIG12 , the grid (V0-V0-V0-V0) is the reconstructed grid corresponding to the reconstructed area A2-B3 in the first grid layer M1. The reconstructed grid only includes static pixels at the first depth level. Therefore, the motion vector corresponding to the reconstructed grid is the motion vector corresponding to the static pixels at the first depth level within the reconstructed grid. Thus, the running vectors of the four vertices of the reconstructed grid are all motion vector V0. Similarly, as shown in (c1) in FIG12 , the grid (V2-V2-V2-V2) is the reconstructed grid corresponding to the reconstructed area A2-B3 in the second grid layer M2. The reconstructed grid only includes static pixels at the second depth level. Therefore, the motion vector corresponding to the reconstructed grid is the motion vector corresponding to the static pixels at the second depth level within the reconstructed grid. Thus, the running vectors of the four vertices of the reconstructed grid are all motion vector V2. Similarly, as shown in (d1) in Figure 12, the grid (V1-V1-V1-V1) is the reconstructed grid corresponding to the reconstructed area A2-B3 in the second grid layer M3. The reconstructed grid only includes dynamic pixels. Therefore, the motion vector corresponding to the reconstructed grid is the motion vector corresponding to the dynamic pixel in the reconstructed grid. In this way, the running vectors of the four vertices of the reconstructed grid are all motion vector V0 or motion vector V3.

[0239] It should be understood that, in embodiments of the present application, motion vectors corresponding to each grid in each grid layer can be obtained based on the depth level or pixel attributes of the pixels within each grid, and the motion vectors of the four vertices of each grid are substantially the same. The above description only uses the motion vectors corresponding to the grids in the reconstructed area A2-B3 on each grid layer as an example. The motion vectors of other grids can refer to the description of this example and will not be repeated here.

[0240] In an embodiment of the present application, a matching calculation method can be selected to calculate the motion vector corresponding to each image area separately based on the pixel attributes (such as static objects, dynamic objects) and depth information corresponding to each image area in the real frame image, so as to obtain the motion vector corresponding to the real frame image.

[0241] In an optional embodiment, some objects in the image (such as people, etc.) are dynamic objects (ie, objects in motion), and some objects in the image (such as the sky, pillars, etc.) are static objects.

[0242] For example, for image regions corresponding to static objects, the corresponding motion vectors can be calculated using the reprojection method; for image regions corresponding to dynamic objects, the corresponding motion vectors can be calculated using the optical flow method. The steps for calculating motion vectors using the reprojection and optical flow methods can be referenced in existing technologies and will not be further described here.

[0243] Step 406 : Perform image warping on the first image in the first grid layer based on the first motion vector to generate a first warped image.

[0244] [Corrected 08.11.2024 according to Rule 91] Step 407: Based on each second motion vector, image warping processing is performed on each second image in each second grid layer to generate at least one second warped image.

[0245] When performing image distortion processing on the first image in the first grid layer, each vertex of each grid (also referred to as the first grid) corresponding to the first image can be moved to a corresponding position according to the corresponding motion vector, so as to obtain a first distorted image after image distortion.

[0246] Similarly, when performing image distortion processing on the second image in the second grid layer, each vertex of each grid (also referred to as the second grid) corresponding to the second image can be moved to a corresponding position according to the corresponding motion vector, so as to obtain a second distorted image after the image distortion.

[0247] It's important to note that image warping based on motion vectors involves moving the position of each mesh vertex according to the motion vector corresponding to that vertex. Whether warping at the first or second mesh layer, and whether moving static or dynamic objects, the processing is consistent; the only difference is that different motion vectors are used at different mesh layers.

[0248] For example, as shown in Figure 12, the dotted grids in (b2), (c2), and (d2) in Figure 12 represent the positions before the movement, and the solid grids represent the positions after the movement. As shown in (b1) and (b2) in Figure 12, when the image of the grid (V0-V0-V0-V0) in the first grid layer M1 is warped, the motion vector of the four vertices of the grid (V0-V0-V0-V0) is all V0. Therefore, each vertex is moved to the corresponding position according to its corresponding motion vector V0, resulting in the grid (V0'-V0'-V0'-V0'). Similarly, as shown in (c1) and (c2) in Figure 12, when the image of the grid (V2-V2-V2-V2) in the second grid layer M2 is warped, the motion vector of the four vertices of the grid (V2-V2-V2-V2) is all V2. Therefore, each vertex is moved to the corresponding position according to its corresponding motion vector V2, resulting in the grid (V2'-V2'-V2'-V2'). As shown in (d1) and (d2) in Figure 12, when the image of the grid (V1-V1-V1-V1) in the second grid layer M3 is warped, the motion vector of the four vertices of the grid (V1-V1-V1-V1) is all V1. Therefore, each vertex is moved to the corresponding position according to its corresponding motion vector V1, resulting in the grid (V1'-V1'-V1'-V1').

[0249] In some embodiments, if the grid layer is a grid that has undergone grid normalization processing (i.e., each grid is divided into two triangles, see Figure 8 for details), each vertex of each triangle can be moved to a corresponding position according to its corresponding motion vector, so as to obtain a first distorted image and each second distorted image after image distortion.

[0250] Because each grid in the present embodiment is a continuous pixel region, the motion vectors of each vertex in each grid or triangle are substantially the same. Therefore, when image distortion processing is performed on each grid or triangle based on the motion vectors, the grids or triangles will not be severely deformed, ensuring image continuity and avoiding distortion issues.

[0251] In addition, for each mesh (or triangle), the depth information of the mesh can also be set according to the depth values ​​of the mesh vertices. For example, interpolation processing is performed based on the depth values ​​of each mesh vertex to obtain the depth value of each pixel in the mesh as the depth information of the mesh.

[0252] In this way, when each grid layer moves a grid based on a motion vector, its coverage relationship with other grids in the grid layer can be determined based on the depth information of the grid. For example, when the first grid is moved based on the motion vector in the first grid layer, when each first grid moves to the target position, the coverage relationship between the first grid and other first grids at the target position is determined based on the depth information of the first grid and the depth information of other first grids at the target position. For another example, when the second grid is moved based on the motion vector in the second grid layer, when each second grid moves to the target position, the coverage relationship between the second grid and other second grids at the target position is determined based on the depth information of the second grid and the depth information of other second grids at the target position.

[0253] Therefore, when moving meshes (or triangles) in each mesh layer, pixel depth information is referred to, which can ensure the correct depth order relationship between different objects on each mesh layer.

[0254] Step 408 : Superimpose the first warped image and at least one second warped image to generate a predicted frame image.

[0255] When the first warped image and the at least one second warped image are superimposed, the grid layers may be aligned and then superimposed to ensure the correctness of the position of each object in the predicted frame image after superposition.

[0256] For example, as shown in Figures 11 and 12 , since the first grid layer M1, the second grid layer M2, and the second grid layer M3 are of the same size, the outer edges of the first grid layer M1, the second grid layer M2, and the second grid layer M3 can be aligned during the overlay process. In this way, images within the same grid in each grid layer can be overlaid (for example, (b2), (c2), and (d2) in Figure 12 ) to obtain a complete and continuous image.

[0257] When the first distorted image and at least one second distorted image are superimposed, the overlapping relationship between the first grid in the first grid layer and the second grid in each second grid layer can be determined based on the depth information of each first grid and each second grid, so as to ensure the correct depth order relationship between different objects after superposition.

[0258] FIG13A exemplarily shows an example of a predicted frame image. In the predicted frame image shown in FIG13A , region 601 is subjected to image distortion processing using an existing technical solution, and region 602 is subjected to image distortion processing using the solution provided in this embodiment. Regions 601 and 602 are only regions where the depth information of static objects is discontinuous. By comparing regions 601 and 602, it can be seen that the image in region 601 is distorted, especially the edge regions of static objects are severely deformed, while the image effect in region 602 is better, the edge regions of static objects are clear, and there is no deformation of the edge regions of static objects.

[0259] FIG13B exemplarily shows a comparative schematic diagram of a predicted frame image. Among them, the predicted frame image shown in (1) in FIG13B is generated by performing image distortion processing using the solution provided in this embodiment, while the predicted frame image shown in (2) in FIG13B is generated by performing image distortion processing using the existing technical solution. Region 603 in (1) in FIG13B and region 604 in (2) in FIG13B are the same image regions, both of which are regions where the depth information of static objects is discontinuous. By comparing (1) and (2) in FIG13B , it can be seen that the image in region 604 is distorted, especially the edge region of the static object is severely deformed, while the picture effect of region 603 is better, the edge region of the static object is clear, and there is no deformation of the edge region of the static object.

[0260] In this way, the image processing method provided by the embodiment of the present application can identify the discontinuous image area by the above-mentioned grid reconstruction method, no matter how complex the geometric edges of the discontinuous image area are (for example, the hair of a person, leaves, etc. in a real frame image), and draw the pixel points of different reconstruction attributes in the discontinuous image area on different grid layers to ensure that the pixel areas in each grid are continuous pixel areas. In this way, when the continuous pixel areas in each grid are subjected to image distortion processing based on their respective corresponding motion vectors, the grid will not be severely deformed, thereby ensuring the continuity of the image. It can be seen that the image processing method provided by the embodiment of the present application can solve the problem of distortion of discontinuous image areas with complex geometric edges and enhance the user's gaming experience.

[0261] It should be noted that the above embodiment is only an illustrative description of the scheme of creating two second grid layers when the real frame image includes a reconstructed area that satisfies both the first preset condition and the second preset condition, and does not represent a limitation on the image processing method provided in the embodiment of the present application.

[0262] In some embodiments, when the real frame image includes a reconstructed area that satisfies both the first preset condition and the second preset condition, a second grid layer may also be created. In this case, the dynamic pixels in the real frame image and the static pixels of the first depth level may be retained in the first grid layer, and then the static pixels of the second depth level in the reconstructed area are all drawn in the second grid layer. In this way, when performing image distortion processing on the first grid layer, the image distortion processing may be performed on the first grid layer based on the motion vector of the static pixels of the first depth level in the first grid layer; and then, the image distortion processing may be performed on the first grid layer based on the motion vector of the dynamic pixels in the first grid layer. The image distortion processing method for the second grid layer is the same as that in the above embodiment and will not be repeated here. In this way, this solution can also solve the problem of distortion of image areas with discontinuous depth values ​​at complex geometric edges.

[0263] It should also be noted that after the image warping and overlay processing operations are completed, the rasterization and pixel processing operations can be continued to generate a predicted frame image. Regarding the predicted frame image generation process, any details not fully explained in this embodiment can be referred to the prior art and will not be repeated here.

[0264] Since the movement speeds of various objects in the original video are different, the predicted frame images calculated by the electronic device based on the motion vector may have image information overlap or image information missing. In other words, the predicted frame images obtained above are the predicted frame images to be completed.

[0265] Therefore, it is necessary to perform image completion on the predicted frame image obtained above to obtain a completed predicted frame image.

[0266] Furthermore, the completed predicted frame image can be synthesized with the UI information to obtain a target predicted frame, and the target predicted frame is displayed.

[0267] In this way, after completing the complement processing of the prediction frame image, it is synthesized with the UI information to obtain the target prediction frame, and then the target prediction frame can be output to the electronic device display screen for display.

[0268] It should be noted that the above process is only explained using the example of processing a single predicted frame. The process for processing each predicted frame is also similar and will not be repeated here. The order in which the target predicted frames are displayed is related to the method of interpolating the predicted frames. You can refer to existing technologies and will not repeat them here. Any details not fully explained in this process can also be referred to existing technologies and will not be repeated here.

[0269] High-refresh-rate games on mobile platforms are often prone to overheating or lag due to power consumption or CPU and GPU limitations, thus affecting the user experience. To enhance the smoothness of the game experience, game developers often lower the original refresh rate and then generate predicted frames in the image space, reducing the number of rendering operations by half at the same frame rate, thereby significantly reducing the power consumption of smart terminals and reducing the heat generation of smart terminals.

[0270] The image processing method provided by the embodiments of this application can effectively address image distortion in predicted frames, especially complex geometric edge deformations with discontinuous depth information, thereby improving the quality of predicted frame images. This ensures the smoothness of the game while halving the number of rendered frames, reduces power consumption and heat generation in smart terminals, and thus enhances the user experience.

[0271] In other interpolation application scenarios, if there are areas in the image with discontinuous depth information (or large depth differences), the image processing method provided in the embodiment of the present application can also be used to perform image distortion processing to improve the image quality of the predicted frame image, which will not be elaborated here.

[0272] The various method embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the scope of protection of this application.

[0273] The above embodiments introduce the image processing method provided by the present application. It is understandable that, in order to realize the above functions, the image processing device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0274] The embodiment of the present application can divide the image processing device into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0275] The method provided in the embodiment of the present application is described in detail above with reference to Figures 1 to 13B . Below, the apparatus provided in the embodiment of the present application is described in detail with reference to Figure 14 . It should be understood that the description of the apparatus embodiment corresponds to the description of the method embodiment. Therefore, for matters not described in detail, reference can be made to the method embodiment above, and for the sake of brevity, no further description will be given here.

[0276] FIG14 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. As shown in FIG14 , the device 500 includes:

[0277] A resource information acquisition module 510 is configured to acquire resource information, wherein the resource information includes a real frame image, depth information of the real frame image, and pixel attribute information;

[0278] A first grid layer creation module 520, configured to create a first grid layer of the real frame image;

[0279] The reconstruction module 530 is configured to, when the real frame image includes a reconstruction region that satisfies a first preset condition and / or a second preset condition, create at least one second grid layer, and based on reconstruction properties of the reconstruction region, draw pixel points of the reconstruction region on the first grid layer and the at least one second grid layer, respectively; wherein the first preset condition is that the reconstruction region includes static pixels of different depth levels, and the second preset condition is that the reconstruction region includes both static pixels and dynamic pixels; and the reconstruction properties of the pixels drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties.

[0280] A motion vector acquisition module 540 is configured to acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image region included in the first grid layer; and the second image refers to a real frame image region included in the second grid layer;

[0281] A first distorted image generating module 550 is configured to perform image distortion processing on the first image based on the first motion vector to generate a first distorted image;

[0282] A second distorted image generating module 560 is configured to perform image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image;

[0283] The superposition processing module 570 is configured to perform superposition processing on the first warped image and the at least one second warped image to generate a predicted frame image.

[0284] In one possible implementation, the reconstruction module 530 is specifically used to create a second grid layer when the real frame image includes a reconstructed area that meets the first preset condition; draw static pixel points of a first depth level in the reconstructed area in the first grid layer; and draw static pixel points of a second depth level in the reconstructed area in the second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0285] In one possible implementation, the reconstruction module 530 is specifically used to create a second grid layer when the real frame image includes a reconstructed area that meets the second preset condition and the depth level of the static pixel points in each reconstructed area is the same; draw the static pixel points in the reconstructed area in the first grid layer, and draw the dynamic pixel points in the reconstructed area in the second grid layer; or draw the dynamic pixel points in the reconstructed area in the first grid layer, and draw the static pixel points in the reconstructed area in the second grid layer.

[0286] In one possible implementation, the reconstruction module 530 is specifically configured to, when the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of the static pixels of each of the reconstruction areas are different, or when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition, create two second grid layers; draw the static pixels of the first depth level in the reconstruction area in the first grid layer, draw the static pixels of the second depth level in the reconstruction area in the first second grid layer, and draw the dynamic pixels of the reconstruction area in the second second grid layer; or draw the dynamic pixels of the reconstruction area in the first grid layer, draw the static pixels of the first depth level in the reconstruction area in the first second grid layer, and draw the static pixels of the second depth level in the reconstruction area in the second second grid layer; or draw the static pixels of the first depth level in the reconstruction area in the first grid layer, draw the dynamic pixels of the reconstruction area in the first second grid layer, and draw the static pixels of the second depth level in the reconstruction area in the second second grid layer; wherein the first depth level and the second depth level are different depth levels.

[0287] In one possible implementation, a marking module is further included, which is used to mark the reconstruction information of each of the reconstruction areas to obtain a marking map; the reconstruction information includes reconstruction type information, depth level information and pixel attribute information; based on the reconstruction type information, depth level information and pixel attribute information, the number of the second grid layers and the reconstruction attributes of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are determined.

[0288] In one possible implementation, the method further includes a reconstruction area determination module for dividing the real frame image into multiple working groups, each of which includes multiple pixel points; based on the depth information of the real frame image and the pixel attribute information of the real frame image, determining that the working group in the real frame image that includes static pixel points with different depth levels is the reconstruction area that meets the first preset condition; and / or, based on the pixel attribute information of the real frame image, determining that the working group in the real frame image that includes both static pixel points and dynamic pixel points is the reconstruction area that meets the second preset condition.

[0289] In one possible implementation, the reconstruction area determination module is specifically used to determine a first workgroup including static pixels in the multiple workgroups based on pixel attribute information of the real frame image; determine a second workgroup in the first workgroup including pixel points whose depth values ​​are within a preset range and the difference between the depth values ​​of the pixel points and surrounding pixel points exceeds a preset threshold based on the depth information of the real frame image; determine the maximum depth value of each of the second workgroups; determine the minimum value of each of the maximum depth values ​​as the depth threshold of the real frame image; determine a third workgroup including static pixel points of a first depth level and static pixel points of a second depth level in the multiple workgroups, wherein the depth value of the static pixel points of the first depth level is greater than the depth threshold, and the depth value of the static pixel points of the second depth level is less than or equal to the depth threshold; and determine the third workgroup as the reconstruction area that meets the first preset condition.

[0290] In one possible implementation, the first warped image generation module 550 is specifically configured to set the depth information of each first mesh in the first mesh layer; determine, for each first mesh in the first mesh layer, a first motion vector of each vertex of the first mesh, and move the first mesh according to the first motion vector of each vertex of the first mesh; and when each first mesh moves to a target position, determine, based on the depth information of the first mesh and the depth information of other first meshes at the target position, an overlapping relationship between the first mesh and other first meshes at the target position.

[0291] In one possible implementation, the second warped image generation module 560 is specifically configured to set depth information for each second mesh in the second mesh layer; determine, for each second mesh in the second mesh layer, a second motion vector for each vertex of the second mesh, and move the second mesh according to the second motion vector for each vertex of the second mesh; and, when each second mesh moves to a target position, determine an overlapping relationship between the second mesh and the other second meshes at the target position based on the depth information of the second mesh and the depth information of the other second meshes at the target position.

[0292] In one possible implementation, the overlay processing module 570 is specifically configured to determine, when overlaying the first distorted image and the at least one second distorted image, an overlay relationship between a first grid in the first grid layer and a second grid in each second grid layer based on depth information of each first grid and each second grid.

[0293] According to the method provided in an embodiment of the present application, an embodiment of the present application also provides an electronic device, including a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device performs the image processing method.

[0294] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0295] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0296] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0297] According to the method provided in the embodiment of the present application, the embodiment of the present application also provides a computer program product, which includes: a computer program or instructions, which, when the computer program or instructions are run on a computer, enables the computer to execute the method of any one of the method embodiments.

[0298] According to the method provided in the embodiment of the present application, the embodiment of the present application also provides a computer storage medium, which stores a computer program or instruction. When the computer program or instruction is run on a computer, the computer executes the method of any one of the embodiments of the method.

[0299] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0300] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0301] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0302] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0303] In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit.

[0304] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0305] The devices, computer storage media, and computer program products provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the methods provided above, and will not be repeated here.

[0306] It should be understood that in each embodiment of the present application, the execution order of each step should be determined by its function and internal logic. The size of the sequence number of each step does not mean the order of execution and does not limit the implementation process of the embodiment.

[0307] The various sections of this specification are described in a progressive manner. Similar portions between embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the embodiments of the apparatus, computer storage medium, and computer program product are generally similar to the method embodiments, so their descriptions are simplified. For relevant details, refer to the descriptions of the method embodiments.

[0308] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0309] The above-described embodiments of the present application do not constitute a limitation on the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire resource information, the resource information including a real frame image, depth information of the real frame image, and pixel attribute information of the real frame image; The pixel attribute information is used to characterize static pixels and dynamic pixels; Creating a first grid layer of the real frame image; In the case where the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, at least one second grid layer is created, and based on the reconstruction properties of the reconstruction area, pixel points of the reconstruction area are drawn on the first grid layer and the at least one second grid layer respectively; wherein the first preset condition is that the reconstruction area includes static pixel points with different depth levels, and the second preset condition is that the reconstruction area includes both static pixel points and dynamic pixel points; the reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties; Acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image region included in the first grid layer; the second image refers to a real frame image region included in the second grid layer; Based on the first motion vector, performing image distortion processing on the first image to generate a first distorted image; Based on each of the second motion vectors, performing image distortion processing on each of the second images respectively to generate at least one second distorted image; The first warped image and the at least one second warped image are superimposed to generate a predicted frame image.

2. The method according to claim 1, characterized in that When the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, creating at least one second grid layer, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction attribute of the reconstruction area, comprises: In the case where the real frame image includes a reconstruction area that satisfies the first preset condition, creating a second grid layer; Draw static pixel points of a first depth level in the reconstruction area in the first grid layer; Static pixel points of a second depth level in the reconstructed area are drawn in the second grid layer; wherein the first depth level and the second depth level are different depth levels.

3. The method according to claim 1, characterized in that When the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, creating at least one second grid layer, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction attribute of the reconstruction area, comprises: In the case where the real frame image includes a reconstructed area that meets the second preset condition and the depth levels of static pixels of each of the reconstructed areas are the same, creating a second grid layer; Draw static pixel points in the reconstruction area on the first grid layer, and draw dynamic pixel points in the reconstruction area on the second grid layer; or, Dynamic pixel points in the reconstruction area are drawn in the first grid layer, and static pixel points in the reconstruction area are drawn in the second grid layer.

4. The method according to claim 1, characterized in that: When the real frame image includes a reconstruction area that satisfies the first preset condition and / or the second preset condition, creating at least one second grid layer, and drawing pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on the reconstruction attribute of the reconstruction area, comprises: When the real frame image includes a reconstruction area that satisfies the second preset condition and the depth levels of static pixels of the reconstruction areas are different, or when the real frame image includes a reconstruction area that satisfies the first preset condition and the second preset condition, two second grid layers are created; Drawing static pixel points of a first depth level in the reconstructed area on the first grid layer, drawing static pixel points of a second depth level in the reconstructed area on the first second grid layer, and drawing dynamic pixel points in the reconstructed area on the second second grid layer; or, Draw dynamic pixels in the reconstructed area on the first grid layer, draw static pixels at a first depth level in the reconstructed area on the first second grid layer, and draw static pixels at a second depth level in the reconstructed area on the second second grid layer; or, Drawing static pixel points of a first depth level in the reconstructed area on the first grid layer, drawing dynamic pixel points in the reconstructed area on a first second grid layer, and drawing static pixel points of a second depth level in the reconstructed area on a second second grid layer; The first depth level and the second depth level are different depth levels.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Marking the reconstruction attribute information of each of the reconstruction areas to obtain a marking map; the reconstruction attribute information includes reconstruction type information and depth level information; Based on the reconstruction type information and the depth level information, the number of the second grid layers and reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are determined.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Dividing the real frame image into a plurality of working groups, each of the working groups comprising a plurality of pixel points; Based on the depth information of the real frame image and the pixel attribute information of the real frame image, determining that the working group including static pixels of different depth levels in the real frame image is the reconstruction area that meets the first preset condition; and / or, Based on the pixel attribute information of the real frame image, the working group including both static pixel points and dynamic pixel points in the real frame image is determined as the reconstruction area that meets the second set condition.

7. The method according to claim 6, characterized in that The determining, based on the depth information of the real frame image and the pixel attribute information of the real frame image, that the working group including static pixels of different depth levels in the real frame image is the reconstruction area satisfying the first preset condition comprises: Based on the pixel attribute information of the real frame image, determining a first working group including static pixels among the multiple working groups; Based on the depth information of the real frame image, determining in the first working group a second working group including a pixel whose depth value is within a preset range and a difference between the depth value of the pixel and surrounding pixels exceeds a preset threshold; Determine a maximum depth value of each of the second working groups; Determine the minimum value among the maximum depth values ​​as the depth threshold of the real frame image; Determine a third working group including static pixels of a first depth level and static pixels of a second depth level from among the multiple working groups, wherein the depth values ​​of the static pixels of the first depth level are greater than the depth threshold, and the depth values ​​of the static pixels of the second depth level are less than or equal to the depth threshold; The third working group is determined to be the reconstruction area that meets the first preset condition.

8. The method according to claim 1, characterized in that: The step of performing image distortion processing on the first image based on the first motion vector to generate a first distorted image includes: Setting depth information of each first grid in the first grid layer; For each first mesh in the first mesh layer, determine a first motion vector of each vertex of the first mesh, and move the first mesh according to the first motion vector of each vertex of the first mesh; When each of the first grids moves to a target position, the covering relationship between the first grid and other first grids at the target position is determined according to the depth information of the first grid and the depth information of other first grids at the target position.

9. The method according to claim 8, characterized in that The step of performing image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image includes: Setting depth information of each second grid in the second grid layer; For each second mesh in the second mesh layer, determine a second motion vector of each vertex of the second mesh, and move the second mesh according to the second motion vector of each vertex of the second mesh; When each second grid moves to the target position, the covering relationship between the second grid and other second grids at the target position is determined according to the depth information of the second grid and the depth information of other second grids at the target position.

10. The method according to claim 9, characterized in that The superimposing the first distorted image and the at least one second distorted image to generate a predicted frame image includes: When the first distorted image and the at least one second distorted image are superimposed, the covering relationship between the first grid in the first grid layer and the second grid in each second grid layer is determined according to the depth information of each first grid and each second grid.

11. The method according to claim 1, characterized in that The resource information is resource information in the game application.

12. An image processing device, characterized in that: The device comprises: A resource information acquisition module, used to acquire resource information, wherein the resource information includes a real frame image, depth information of the real frame image, and pixel attribute information; the pixel attribute information is used to characterize static pixel points and dynamic pixel points; A first grid layer creation module, used to create a first grid layer of the real frame image; A reconstruction module, configured to create at least one second grid layer when the real frame image includes a reconstruction area that satisfies a first preset condition and / or a second preset condition, and to draw pixel points of the reconstruction area on the first grid layer and the at least one second grid layer respectively based on reconstruction properties of the reconstruction area; wherein the first preset condition is that the reconstruction area includes static pixel points with different depth levels, and the second preset condition is that the reconstruction area includes both static pixel points and dynamic pixel points; the reconstruction properties of the pixel points drawn on different grid layers in the first grid layer and the at least one second grid layer are different, and the reconstruction properties include depth levels and pixel properties; A motion vector acquisition module, used to acquire a first motion vector corresponding to a first image in the first grid layer, and a second motion vector corresponding to each second image in each second grid layer; the first image refers to a real frame image area included in the first grid layer; the second image refers to a real frame image area included in the second grid layer; A first distorted image generating module, configured to perform image distortion processing on the first image based on the first motion vector to generate a first distorted image; A second distorted image generating module, configured to perform image distortion processing on each of the second images based on each of the second motion vectors to generate at least one second distorted image; The superposition processing module is used to perform superposition processing on the first distorted image and the at least one second distorted image to generate a predicted frame image.

13. An electronic device, characterized in that: It comprises a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1-11.

14. A computer storage medium, characterized in that: The computer storage medium stores a computer program or instruction. When the computer program or instruction is executed, the method according to any one of claims 1 to 11 is executed.

Images