Image processing method and electronic device

The image frame segmentation process is performed through uneven blocking technology, which improves the matching accuracy and rendering effect of video interpolation frames, solves the problem of low grid block matching accuracy in the prior art, and improves the user experience.

CN116740241BActive Publication Date: 2024-11-01HONOR DEVICE CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202211215657.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-30
Publication Date
2024-11-01
Estimated Expiration
2042-09-30

AI Technical Summary

Technical Problem

When the existing video interpolation technology calculates the motion vector of multiple frame images before and after, the matching accuracy of grid blocks is low, resulting in poor rendering of transition frames and affecting user experience.

Method used

The image frame display area is segmented using uneven blocking method, and a more refined segmentation method is adopted for nearby objects based on projection scaling information to improve matching accuracy and ensure that the position matching of grid blocks in front and back multi-frame images is accurate.

Benefits of technology

Improve the rendering effect of transition frames, reduce algorithm overhead, avoid image block distortion, and improve user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116740241B_ABST
    Figure CN116740241B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides an image processing method and an electronic device, which are applied to the field of image processing technology. In the method, the electronic device obtains a near dynamic object from a real frame, and subdivides the image block corresponding to the instruction for drawing the near dynamic object, so that the image block where the near dynamic object is located is divided into small grid blocks, so that the grid block matching accuracy is high, the calculated motion vector of the grid block is more refined, and therefore the matching accuracy is higher, and the rendering effect of the predicted frame is better. Since the electronic device does not subdivide all the image blocks of the real frame, the algorithm overhead is also smaller.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] Image interpolation technology inserts transition frames between two frames to increase the frame rate from 45 or 60 to 90, 120 or even more. Therefore, this technology has improved the problem of shortage of high frame rate images in current terminals to a certain extent.

[0003] In the process of predicting the transition frame, it is necessary to first divide the picture into blocks, and then determine the motion vector diagram of the two frames before and after in units of grid blocks. Then, the luminosity, color and other information of each grid block of the two frames before and after the transition frame are matched based on the motion vector diagram.

[0004] When matching information such as luminance and chrominance, the degree of subdivision of the grid block determines the matching accuracy. However, the existing video interpolation technology obtains multiple grid blocks by dividing the blocks. When calculating the motion vector diagram of the previous and next multiple frames, the matching accuracy of the same grid block of the previous and next multiple frames is low, resulting in the distortion of the position of the grid block in the previous and next multiple frames. Therefore, the rendering effect of the rendered transition frame is poor, affecting the user's viewing experience. Summary of the invention

[0005] Based on this, the embodiments of the present application provide an image processing method and an electronic device, which can segment the image frame display area in an uneven manner, such as adopting a more refined segmentation method for the near objects in the image than for the far objects according to the projection scaling information, so as to improve the matching accuracy of the image details of the near objects. The positions of the grid blocks in the previous and next frames of images are not easy to match incorrectly, and the image of the rendered transition frame is displayed normally, and it is not easy for some image blocks to be distorted. Therefore, the rendering effect of the predicted frame image based on image prediction is good, and the beneficial effect of low algorithm overhead can be achieved.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, in which, when the electronic device performs a rendering operation of an image frame, the electronic device obtains projection scaling information corresponding to at least one drawing instruction of the image frame, wherein the projection scaling information indicates the scaling degree of the drawing object corresponding to the drawing instruction in the area to be displayed. Then, the electronic device determines the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information, wherein the coarse subdivided area is divided into a plurality of grid blocks, and the subdivided area is divided into a plurality of first-level blocks. At least one first-level block of the subdivided area is divided into a plurality of second-level blocks, so that the size of the second-level block in the subdivided area is smaller than the size of the grid block in the coarse subdivided area, wherein the grid blocks obtained by segmenting the subdivided area and the grid blocks obtained by segmenting the coarse subdivided area are used to determine the object position of at least one drawing object in the image frame.

[0007] In the present application, the electronic device uses an uneven block division method to segment the image frame display area. Specifically, during the image frame rendering process, the electronic device obtains the scaling degree of the drawing object corresponding to the drawing instruction in the display area, and based on the scaling degree, the scaling of the drawing object mapped from the model space to the camera perspective can be determined. If the scaling degree is large, it means that the drawing object is far from the camera; conversely, if the scaling degree is small, it means that the drawing object is close to the camera. Therefore, based on the projection scaling information, the subdivided area and the coarse area of ​​the display area can be distinguished. A finer segmentation method is adopted for the subdivided area than for the coarse area, so that the number of grid blocks obtained by segmenting the subdivided area will be greater than the number of grid blocks obtained by segmenting the coarse area. The more grid blocks there are, the higher the matching accuracy of the grid blocks, and the higher the object position of the obtained drawing object in the image frame. That is, the present application can achieve the beneficial effect of improving the matching accuracy of the grid blocks obtained by segmentation by segmenting the image frame display area by using an uneven block division method.

[0008] In another possible design of the first aspect, after dividing at least one first-level block of the subdivided area into a plurality of second-level blocks, the method further includes: determining an object position of at least one drawing object in each two adjacent image frames according to the coarse-divided area and the first-level blocks and the second-level blocks in the subdivided area, obtaining a motion vector of the drawing object based on the object position of the drawing object in each two adjacent image frames, and predicting a predicted frame between each two adjacent image frames based on the motion vector.

[0009] Under this design, the electronic device can match the object position of the drawn object in two adjacent image frames based on the grid blocks obtained by segmenting the coarse area and the subdivided area. Since the matching accuracy of the subdivided area is higher, the subdivided area is not easy to match incorrectly when matching the positions of the previous and subsequent image frames. Then, the transition frame obtained by rendering the motion vector of the drawn object obtained by matching is not prone to partial image block distortion, so the image rendering effect of the transition frame is good.

[0010] In another possible design manner of the first aspect, obtaining projection scaling information corresponding to at least one drawing instruction of an image frame includes: obtaining a camera viewing angle parameter of at least one drawing instruction of the image frame, and physical position information of a drawing object corresponding to at least one drawing instruction. Based on the camera viewing angle parameter and the physical position information, obtaining projection scaling information of the drawing object under the camera viewing angle.

[0011] Under this design method, the electronic device can obtain projection scaling information by acquiring camera viewing angle parameters and position information, and adopt a more refined segmentation method for nearby objects in the image than for distant objects based on the projection scaling information, so as to improve the matching accuracy of the image details of nearby objects.

[0012] In another possible design manner of the first aspect, obtaining camera viewing angle parameters of at least one drawing instruction of an image frame and physical position information of a drawing object corresponding to at least one drawing instruction includes: detecting that the drawing instruction is a dynamic drawing instruction, obtaining camera viewing angle parameters of at least one dynamic drawing instruction of the image frame and physical position information of a drawing object corresponding to at least one dynamic drawing instruction, wherein the dynamic drawing instruction is used to instruct the drawing of a dynamic object.

[0013] In this design, the electronic device only obtains the projection scaling information of the dynamic object, and determines whether the drawing area of ​​the dynamic object is a coarse area or a fine area based on the projection scaling information. In other words, the electronic device only subdivides the image block where the nearby dynamic object is located to reduce algorithm overhead and improve processing speed.

[0014] In another possible design of the first aspect, determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information includes: detecting that the drawing instruction is a dynamic drawing instruction, and determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information. Detecting that the drawing instruction is a static drawing instruction, determining the area drawn by the static drawing instruction as the coarse subdivided area, wherein the static drawing instruction is used to instruct drawing of a static object.

[0015] Under this design, when the electronic device detects that the drawing instruction is a static drawing instruction, it does not obtain the projection scaling information of the static drawing instruction, and directly determines the area of ​​the static drawing instruction as the coarse area. When the drawing instruction is detected as a dynamic drawing instruction, it determines whether the area corresponding to the dynamic drawing instruction needs to be subdivided based on the projection scaling information. The use of uneven block division for static drawing instructions and dynamic drawing instructions can reduce the algorithm overhead.

[0016] In another possible design manner of the first aspect, the projection scaling information includes a projection scaling factor, and determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information includes: if the projection scaling factor is greater than a first threshold, determining that the area drawn by the drawing instruction corresponding to the projection scaling factor is the coarse subdivided area. If the projection scaling factor is less than or equal to the first threshold, determining that the area drawn by the drawing instruction corresponding to the projection scaling factor is the subdivided area.

[0017] Under this design, the electronic device can determine the coarse area and the fine area by the size relationship between the projection scaling factor and the first threshold. The projection scaling factor represents the scaling degree of the drawing object under the camera's perspective after it is mapped to the camera's perspective. If the projection scaling factor is greater than the first threshold, it means that the area of ​​the drawing object is small after being mapped to the camera's perspective, then it is determined that the drawing object is a distant object, and the area drawn by the drawing instruction for drawing the drawing object is determined as the coarse area. On the contrary, if the projection scaling factor is less than or equal to the first threshold, that is, the area of ​​the drawing object is large after being mapped to the camera's perspective, then it is determined that the drawing object is a nearby object, and the area drawn by the drawing instruction for drawing the drawing object is determined as the fine area. In this way, by setting the projection scaling factor and the first threshold, the coarse area and the fine area can be quickly determined in the area to be displayed.

[0018] In another possible design manner of the first aspect, the subdivided area is a grid area obtained by gridding the drawing object corresponding to the drawing instruction, and the display block of the drawing object in the area to be displayed is included in the subdivided area; after dividing at least one first-level block of the subdivided area into multiple second-level blocks, the method also includes: detecting that the second-level block contains an area outside the display block in the area to be displayed, and dividing the second-level block into multiple third-level blocks.

[0019] Under this design, when the electronic device determines that the second-level block of the subdivided area needs to be further segmented, the second-level block is segmented into multiple third-level blocks. Specifically, when segmenting, the grid block is generally subdivided into n*n grid blocks. Therefore, after segmentation, the grid block may contain other areas except the image block corresponding to the drawing instruction. Therefore, the image block may contain multiple chromaticities and luminosities, which are prone to matching errors. Therefore, for the second-level blocks that also contain other drawing objects, the electronic device further subdivides them, so that the positions of the grid blocks obtained by further segmentation in the previous and next multiple frames of images are not easy to match incorrectly, so that the matching accuracy is better.

[0020] In another possible design of the first aspect, dividing the second-level block into a plurality of third-level blocks includes: if the size of the second-level block does not meet the preset size condition, dividing the second-level block into a plurality of third-level blocks. The method also includes: if the size of the second-level block meets the preset size condition, stopping the division operation of the second-level block.

[0021] In this design, the preset size condition includes: the size of the grid block is a preset minimum size. When the size of the second-level block of the electronic device is the minimum size, it means that the second-level block cannot be further subdivided, so it is not divided. When the size of the second-level block is not the minimum size, when the second-level block contains other drawing objects, it is divided into a third-level block. By setting the preset condition, the process can be ended when the second-level block cannot be further subdivided, so as to improve the efficiency of the processing process and avoid occupying GPU computing resources.

[0022] In another possible design of the first aspect, after determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information, the method further includes: marking a drawing instruction for drawing the subdivided area to obtain a first marking message, wherein the first marking message is used to mark a prompt character in an image block rendered by the drawing instruction of the subdivided area. Dividing at least one first-level block of the subdivided area into a plurality of second-level blocks includes: dividing at least one first-level block of the area where the prompt character is located in the subdivided area into a plurality of second-level blocks.

[0023] In this design, the first marking message can mark the marked drawing area with a prompt character during the process of the electronic device rendering the image frame, so that the electronic device can determine the subdivided area that needs to be subdivided according to the prompt information. In other words, the electronic device sends the first marking message before rendering the image frame, so that the rendered image frame is marked with a prompt character. In this way, the electronic device can directly perform subdivision operations on the marked area.

[0024] In another possible design of the first aspect, after determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information, the method further includes: marking the drawing instruction for drawing the subdivided area to obtain a second marking message, wherein the second marking message of the subdivided area corresponding to each drawing instruction is different, wherein the second marking message is used to mark the prompt character in the image block rendered by the drawing instruction of the subdivided area, and to determine whether to segment the second-level block. After segmenting at least one first-level block of the subdivided area into multiple second-level blocks, the method further includes: if the second-level block includes multiple second marking messages, segmenting the second-level block into multiple third-level blocks. If the second-level block includes one second marking message, stopping the segmentation operation of the second-level block.

[0025] In this design, different prompt characters are marked on different drawing objects, so whether to further perform a segmentation operation on the second-level block can be directly determined according to whether the second-level block includes multiple prompt characters, thereby improving the efficiency of the segmentation operation.

[0026] In another possible design manner of the first aspect, detecting that the drawing instruction is a dynamic drawing instruction includes: if physical position information corresponding to two adjacent drawing instructions is different, determining that the drawing instruction is a dynamic drawing instruction.

[0027] In another possible design manner of the first aspect, the subdivision area is a grid area obtained by meshing the drawing object corresponding to the drawing instruction, and the physical position information includes coordinate information of a grid vertex in the subdivision area where the drawing object is located, wherein the grid vertex is a vertex of the grid area.

[0028] In this design, the projection scaling information of the image block drawn by the drawing instruction is determined by approximately obtaining the projection scaling information of a mesh vertex, which reduces the amount of calculation and has higher calculation efficiency.

[0029] In a second aspect, an embodiment of the present application provides an image processing method, which can be applied to an electronic device, and to an image frame containing multiple drawing objects, the drawing objects including a first object and a second object, and when the difference between the volume of the first object in the model space and the volume of the second object in the model space is greater than or equal to a threshold, the image frame is divided into blocks; in this method, when the electronic device performs a rendering operation of the image frame, the object screen ratio corresponding to at least one drawing instruction of the image frame is obtained, wherein the object screen ratio is used to indicate the occupancy of the drawing object corresponding to the drawing instruction in the area to be displayed. If the object screen ratio is greater than or equal to the second threshold, the area drawn by the drawing instruction corresponding to the object screen ratio is determined to be a subdivided area. If the object screen ratio is less than the second threshold, the area drawn by the drawing instruction corresponding to the object screen ratio is determined to be a coarse subdivided area, wherein the coarse subdivided area is divided into a plurality of grid blocks, and the subdivided area is divided into a plurality of first-level subdivided blocks. At least one first-level block of the subdivided area is divided into multiple second-level blocks so that the size of each second-level block obtained by the subdivided area segmentation is smaller than the size of each grid block obtained by the coarse area segmentation, wherein the grid blocks obtained by the subdivided area segmentation and the grid blocks obtained by the coarse area segmentation are used to determine the object position of at least one drawing object in the image frame.

[0030] In the present application, when the volume difference between the first object and the second object is large, the subdivided area and the coarse subdivided area are not determined based on the projection scaling information, but are determined based on the occupancy of the drawing object in the area to be displayed. A high occupancy means that if the position matching error of the drawing object is more obvious, the user experience will be worse, so it is necessary to subdivide the area where the drawing object with a high occupancy is located, so as to achieve the beneficial effect of improving the matching accuracy of the image detail part of the drawing object.

[0031] In another possible design manner of the second aspect, the occupancy includes a ratio of an area of ​​the drawing object in the to-be-displayed area to a total area of ​​the to-be-displayed area.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and one or more processors. The memory and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method of the first aspect and its possible design method, and the method of the second aspect and its possible design method.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect and possible design thereof.

[0034] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and its possible design method, and the method of the second aspect and its possible design method.

[0035] It can be understood that the beneficial effects that can be achieved by the method of the second aspect and any possible design thereof provided above, the electronic device described in the third aspect, the computer storage medium described in the fourth aspect, and the computer program product described in the fifth aspect can be referred to the beneficial effects in the first aspect and any possible design thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a frame insertion technology provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 4 An interactive diagram of an image processing method provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a template test operation provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a marked dynamic drawing instruction rendering process provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of a block operation provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of an operation of matching grid blocks provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of the effect corresponding to an image processing method provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of a resource interception process provided in an embodiment of the present application;

[0046] Fig.11 A flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0048] Before introducing the image processing method provided in the embodiment of the present application, the image interpolation technology is first introduced here.

[0049] Figure 1 This is a schematic diagram of a frame insertion technology provided in an embodiment of the present application. Figure 1 As shown, real frames N-1, N, and N+1 are image frames displayed by the game application in time sequence before interpolation. The small number of real frames leads to low game fluency. Since each real frame includes at least one object, each object can be obtained by executing the drawing instructions corresponding to the object by the graphics processing unit (GPU), and the drawing instructions for the same object are the same, so the movement trajectory of the object between adjacent real frames is predictable. For example Figure 1 In the game, the character is a dynamic object, and the tree and car are static objects. The tree and car themselves are stationary and only change with the viewing angle, so the movement trajectory of the tree and car can be easily obtained based on the change in viewing angle. The character itself will move in the world space (the world space can be understood as the total space that the game application can display). For example, in the real frame N-1, the position of the character on the screen (referred to as position) is on the left side of the tree and the car. In the real frame N, the position of the character is on the right side of the tree and on the left side of the car. Then it can be predicted that the position of the character will change from left to right. Therefore, a predicted frame can be obtained based on the prediction of two adjacent real frames and a predicted frame (also called a transition frame) can be inserted between the two real frames. For example Figure 1 A predicted frame M is inserted between the real frame N-1 and the real frame N, and a predicted frame M+1 is inserted between the real frame N and the real frame N+1 to achieve the transition of the object motion trajectory.

[0050] In the related art, the motion trajectory of a moving object between two frames of images is usually calculated by block matching. Specifically, the space of the screen is divided into multiple grid blocks, and the grid blocks with the same luminosity and color of the two frames are matched, and the motion trajectory of the object in the image is accurately determined by the motion vector of the block. In the process of grid segmentation, the greater the degree of subdivision of the grid block, the smaller the grid block, but the more matching times, so the greater the algorithm overhead; and the smaller the degree of subdivision of the block, the larger the block grid, then the lower the matching accuracy, and the worse the rendering effect of the transition frame. Therefore, neither the uniformly subdivided grid nor the uniformly coarse grid can achieve the purpose of taking into account both low algorithm overhead and good rendering effect.

[0051] In order to improve the rendering effect of the inserted prediction frame without increasing the algorithm overhead, in the image processing method provided in the embodiment of the present application, the electronic device obtains the near dynamic object from the real frame, and subdivides the image block corresponding to the instruction to draw the near dynamic object, so that the image block where the near dynamic object is located is divided into small grid blocks, so that the grid block matching accuracy is high, the calculated motion vector of the grid block is more refined, and the matching accuracy is higher, so the rendering effect of the prediction frame is better. Since the electronic device does not subdivide all the image blocks of the real frame, the algorithm overhead is also smaller.

[0052] Exemplarily, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, an augmented reality (AR) or virtual reality (VR) device, etc., which has a display screen. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0053] Figure 2 The hardware structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, taking the electronic device 200 as a mobile phone as an example, the mobile phone may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0054] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0055] The processor 210 may include one or more processing units, for example, the processor 210 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processor GPU, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0056] The controller can be the nerve center and command center of the mobile phone. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0057] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0058] The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 can receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 240 can receive wireless charging input through a wireless charging coil of the mobile phone. While the charging management module 240 is charging the battery 242, it can also power the mobile phone through the power management module 241.

[0059] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0060] The wireless communication function of the mobile phone can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0061] The mobile phone implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0062] The display screen 294 is used to display images, videos, etc. The display screen 294 is the above-mentioned folding screen (such as a flexible folding screen or a multi-screen folding screen). The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc.

[0063] The mobile phone can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and APP processor.

[0064] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, media files such as music and videos are stored in the external memory card.

[0065] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional APPs and data processing of the mobile phone by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0066] It should be noted that the above Figure 2 The description thereof is merely an example of an application carrier of the solution provided in the embodiment of the present application. Figure 2 The composition does not constitute a limitation on the solution described in the embodiments of the present application. In other embodiments, the electronic device may also have Figure 2 More or fewer components may be shown.

[0067] exist Figure 2 In the example shown, the hardware composition of the electronic device is provided. In some embodiments, the electronic device can also be configured by its various hardware components (such as Figure 2 The hardware composition shown in FIG. 1 is used to run an operating system. In the operating system, different software layers may be provided to implement the running of different programs.

[0068] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of a mobile phone.

[0069] like Figure 3As shown, the layered architecture can divide the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer (framework), the native method layer (native), and the kernel layer.

[0070] The application layer may include a series of application packages. For example, the application package may include applications such as camera, calendar, map, video, music, short message, gallery, call and navigation. In the embodiment of the present application, the application layer also includes game applications.

[0071] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer to provide support for the operation of the applications in the application layer.

[0072] Among them, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like.

[0073] The window manager provides window management services (WMS). WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0074] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0075] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0076] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0077] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages. It can stay for a short time and then disappear automatically without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0078] The activity manager can provide activity management services (ActivityManagerService, AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, broadcast receivers) as well as manage and schedule application processes.

[0079] The input manager can provide input management services (IMS), which can be used to manage system input, such as touch screen input, key input, sensor input, etc. IMS takes events from input device nodes and distributes them to appropriate windows through interaction with WMS.

[0080] In the embodiment of the present application, one or more functional modules may be provided in the application framework layer to implement the rendering scheme provided in the embodiment of the present application. Exemplarily, the application framework layer may be provided with an interception module, a separation module, an identification module, a marking module, a block module, etc. In subsequent examples, the functions of the above modules will be described in detail.

[0081] The native layer is used to provide system services, location services, open graphics libraries (such as opengl open graphics library, vulkan open graphics library), touch services, display services, etc. for applications in the application layer. First of all, it should be noted that the Android operating system is a Linux-based operating system mainly used for portable devices. The development of upper-layer applications (such as application layer and framework layer) in the Android system is generally completed based on Java. Since some low-level tasks are not easy to implement in Java, when it comes to tasks such as local services, link libraries or hardware drivers, it is usually necessary to allow C programs to implement them, and C programs run in the system Native library. The system Native library includes an interface for Java to call C++ code.

[0082] The kernel layer is the layer between hardware and software. The kernel layer contains at least camera driver, audio driver, sensor driver, etc.

[0083] exist Figure 3 In the example of , the electronic device may also include a hardware layer, which may include a processor (such as a CPU and a GPU). In some implementations, the CPU may be used to control each module in the application framework layer to implement its respective functions, and the GPU may be used to perform corresponding rendering processing according to the API in the graphics library called by the instructions processed by each module in the application framework layer.

[0084] The image processing method provided in the embodiment of the present application can be implemented in a mobile phone having the above-mentioned hardware structure and software structure.

[0085] The following is an example of the workflow of mobile phone software and hardware, combined with the insertion scenario.

[0086] When the game application at the application layer needs to render an image, it issues a rendering command. In response to the rendering command, the application framework layer converts the rendering command into a drawing instruction and issues the drawing instruction. During the application layer's issuance process, the application framework layer calls the native layer's interface to intercept the drawing instruction and identifies whether the drawing instruction is used to draw a nearby dynamic object. If so, the drawing instruction is marked. In the block segmentation module, the image blocks corresponding to the marked drawing instructions are subdivided, and the image blocks corresponding to the unmarked drawing instructions are roughly divided. The GPU at the hardware layer performs rendering based on the drawing instructions, and matches the previous and next frames of the roughly divided and subdivided blocks to obtain a motion vector map of the full-screen object. Based on the drawing instructions and the motion vector map, the GPU generates a transition frame, and inserts the transition frame into the real frame rendered by the previous and next frames to end the frame insertion process.

[0087] Before describing in detail the image processing method provided in the embodiments of the present application, some professional technical terms that may be involved in the method are first introduced here.

[0088] Graphics library: Also known as a drawing library, the graphics library is used to define an application programming interface (API) that is cross-programming language and cross-platform, which contains many functions for processing graphics. Taking OpenGL (Open Graphics Library) as an example, the API defined by OpenGL includes an interface for drawing two-dimensional images or three-dimensional images, such as the drawing function glDrawElements(); the API defined by OpenGL also includes an interface for presenting the image drawn by the drawing function to the display interface, such as the rendering function eglSwapBuffers(). The embodiments of this application do not give examples one by one here. Among them, the functions in OpenGL can be called through instructions. For example, when developing a game application, the developer writes a drawing instruction based on the function in the graphics library. Through the drawing instruction, the drawing function can be called to draw a two-dimensional image or a three-dimensional image.

[0089] Image frame: The two-dimensional or three-dimensional image drawn by the drawing function called by the drawing instruction mentioned above is called an image frame. Specifically, during the game running or video playing process, it is displayed by continuously rendering and quickly playing one frame after another. Each image frame is a static image displayed in the game application or video. Each static image frame is composed of objects, and each object can be obtained by executing the drawing instructions corresponding to the object by the GPU. Objects in the image frame can be divided into static objects and dynamic objects according to whether they move. Among them, dynamic objects can be further divided into distant dynamic objects and near dynamic objects according to whether they are objects far away from the lens (or screen) or objects close to the lens.

[0090] Drawing instructions: Each object in an image frame is obtained by drawing an electronic device through software or hardware executing drawing instructions. An object can be drawn by one or more drawing instructions. Generally, an object corresponds to a drawing instruction one by one. Each drawing instruction also includes specific parameters it carries, such as vertex information, etc. When the electronic device executes the drawing instruction, the object corresponding to the drawing instruction is drawn based on its specific parameters. It should be noted that the drawing instructions corresponding to the same object in different frames are consistent. For example, a large tree displayed in multiple consecutive game image frames of the game screen is drawn by the electronic device hardware based on the same drawing instruction. However, the present application does not limit this "same" to no difference. In some embodiments, the same can also be "approximate" or "similar".

[0091] The following will introduce the image processing method provided by the embodiment of the present application in detail with reference to the accompanying drawings and game scenes. It should be understood that the image processing method provided by the embodiment of the present application is not limited to game scenes. For example, drawing instructions issued by non-game applications can also be processed by the method provided by the embodiment of the present application. Figure 4 As shown, the image processing method includes:

[0092] S401: The game application sends a drawing instruction to the graphics library.

[0093] Before S401, the method further includes: the mobile phone starts the game application in response to the operation of starting the game application.

[0094] Then, the game engine in the game application can draw image frames by calling the drawing function in the graphics library through the graphics library interface based on the drawing instructions issued by the game application.

[0095] That is, this embodiment is executed during the screen rendering process of the game application. When the CPU transmits dynamic instructions to the GPU, the mobile phone intercepts the drawing instructions and determines how to divide different areas of the screen into grid blocks of different sizes based on the information included in the drawing instructions. For specific interception technology, see S403 below.

[0096] It should be noted that what the CPU transmits to the GPU is a drawing instruction stream, and a drawing instruction stream is generally used to draw a frame of an image. A drawing instruction stream includes one or more drawing instructions, and each drawing instruction is used to draw an object. However, it may take multiple drawing instructions to draw an object completely, such as Figure 1 The character may be drawn by multiple drawing instructions, where the character's head corresponds to a drawing instruction, the character's hand corresponds to a drawing instruction, the character's torso corresponds to a drawing instruction, etc. The embodiments of the present application do not give examples one by one here.

[0097] After the CPU transmits the drawing instruction stream to the GPU, the GPU runs one or more drawing instructions in the drawing instruction stream to call one or more interfaces of the graphics library to achieve drawing of one or more objects in the image frame.

[0098] In this embodiment, one drawing instruction is used for illustration. It should be understood that in actual operation, each drawing instruction included in the drawing instruction stream can be processed by the present solution.

[0099] S402: The separation module determines whether the drawing instruction is a dynamic drawing instruction. If so, S403 is executed; if not, the separation module executes S1.

[0100] Before intercepting the drawing instructions, the dynamic drawing instructions and the static drawing instructions are separated by a separation module. Specifically, as mentioned above, the static object itself is stationary and only changes due to the viewing angle, so the motion trajectory of the static object can be easily obtained according to the change of the viewing angle. The dynamic object itself moves in the world space, so the motion vector of the dynamic object needs to be calculated in a refined manner. Therefore, this embodiment subdivides the image blocks corresponding to the drawing instructions for drawing dynamic objects, which can reduce the algorithm overhead and improve the processing efficiency.

[0101] In some embodiments, the separation module determines whether the drawing instruction is a dynamic drawing instruction, which may include S2-S4, that is, S402 includes S2-S4.

[0102] S2: The separation module obtains context parameter information of the drawing instruction, wherein the context parameter information includes first model position information of the previous drawing instruction and second model position information of the drawing instruction.

[0103] S3: Determine whether the second model position information is updated relative to the first model position information.

[0104] S4: If the second model position information is updated, determining that the drawing instruction is a dynamic drawing instruction.

[0105] Specifically, the model position information (also called physical position information) of the static object drawn by the static drawing instruction remains static, that is, the position of the static object itself will not change, so the model position information of the static drawing instruction will only be updated once. The physical position information of the dynamic object drawn by the dynamic drawing instruction can be moved, that is, the position of the dynamic object itself will change, so the model position information of the dynamic drawing instruction will be updated every time the CPU transmits the drawing instruction to the GPU. Based on this, the separation module can determine whether the model position information is updated according to the context parameter information. If updated, the drawing instruction is determined to be a dynamic drawing instruction. Otherwise, if not updated, the drawing instruction is determined to be a static drawing instruction.

[0106] Exemplarily, in the related technology, the drawing instruction can update the model position information through the glBuffersubData() function. Therefore, if the separation module recognizes the glBuffersubData() function, it means that the drawing instruction is a dynamic drawing instruction. Then the separation module sends a first message indicating that the drawing instruction is a dynamic drawing instruction to the interception module. After receiving the first message, the interception module executes S403.

[0107] S403: The interception module intercepts vertex information and camera viewing matrix of the dynamic drawing instruction and sends them to the recognition module.

[0108] The interception means that the interception module can direct the drawing instruction directed to the graphics library to the recognition module outside the graphics library at the same time. That is, the interception operation of the interception module intercepting the drawing instruction does not affect the drawing instruction being sent to the graphics library.

[0109] In this step, the purpose of intercepting the drawing instruction is to use the intercepted resources to obtain the depth information of the image block corresponding to the drawing instruction. According to the depth information, it can be determined whether the drawing object is a near-view object or a distant object, so as to adopt different precision block operations for dynamic objects with different depth information.

[0110] Specifically, the intercepted resources include vertex information and camera perspective matrix. Among them, vertex information is used to determine the vertex space coordinates of the dynamic object drawn by the dynamic drawing instruction in the world space; the camera perspective matrix is ​​used to determine the position of the camera in the world space. Among them, vertex information and camera perspective matrix can be intercepted in the process of the CPU passing the drawing instruction stream to the GPU. By converting the vertex information to the camera space, the vertex perspective coordinates based on the camera perspective can be obtained, and the vertex perspective coordinates can be used to obtain the depth information of the dynamic object.

[0111] Therefore, the interception module sends the vertex information and the camera viewing matrix to the recognition module, so that the recognition module executes the following S404 to obtain the depth information of the dynamic object.

[0112] S404, the recognition module determines whether the dynamic drawing instruction is a dynamic drawing instruction for drawing a near-view object according to the vertex information and the camera viewing angle matrix. If so, the recognition module executes S406, and if not, the recognition module executes S5.

[0113] Among them, the near-field object refers to the object that the camera captures at a close distance. In the game screen, the screen displays the image from the camera's perspective, so when the object is close to the camera, the object is also close to the screen / user's line of sight. In contrast to the near-field object, the dynamic drawing instruction can also draw the far-field object, which refers to the object that the camera captures at a long distance. When the object is far away from the camera, the object is also far away from the screen / user's line of sight.

[0114] The display clarity of near-field objects and far-field objects in the game screen is different. The near-field objects close to the user's line of sight are displayed more clearly, while the far-field images far from the user's line of sight are displayed more blurred. Therefore, in this step, near-field objects are identified so that only the dynamic drawing instructions for drawing near-field objects are processed in blocks. This can improve the matching accuracy of the foreground details of the screen while reducing the algorithm overhead and improving the processing efficiency.

[0115] As mentioned above, the vertex information can be converted to the camera space to obtain the vertex view coordinates based on the camera view. Therefore, in the recognition module, the vertex view coordinates are first obtained according to the vertex information and the camera view matrix, and then the vertex view coordinates are used to determine whether the dynamic drawing instruction draws a close-up object.

[0116] Exemplarily, taking the case where the vertex information includes information of a vertex in the drawing instruction as an example, the detailed process of determining whether the dynamic drawing instruction is a near-view object in this step is introduced.

[0117] P f Represents vertex information, MVP 4x4 Represents the camera perspective matrix, MVP 4x4 *P f Get (x, y, z, w), where (x, y, z) represents the vertex view coordinates of a vertex. w represents the projection scaling factor. If the projection scaling factor is small, it means that the dynamic object is in the near view; if the projection scaling factor is large, it means that the dynamic object is in the distant view. Therefore, a threshold A is set to 1.0 to determine whether the dynamic object is in the near view. When w>1.0, the dynamic object is in the distant view; when w≤1, the dynamic object is in the near view. After the recognition module recognizes that the dynamic drawing instruction is a dynamic drawing instruction for drawing a near view object, the recognition module sends a second message marking the dynamic drawing instruction for drawing the near view object to the marking module, and the marking module executes the following S406.

[0118] S405: The marking module receives a second message, where the second message is used to instruct to mark a dynamic drawing instruction for drawing a near-view object. In response to the second message, the dynamic drawing instruction is marked, and the marked dynamic drawing instruction is sent to the rendering module.

[0119] The purpose of the marking module marking the dynamic drawing instructions for drawing the near-view object is to mark the instructions so that the drawing area can be marked with the content marked by the marking module during the drawing process of the game screen, so that the partitioning module can directly determine the image blocks that need to be partitioned from the drawing area. In other words, the marking module marks the instructions so that the GPU can display the marked image blocks when drawing the area based on the instructions, so that the partitioning module can directly divide the marked image blocks into blocks.

[0120] Specifically, the CPU transmits dynamic instructions to the GPU, which is equivalent to transmitting the vertex information, vertex color information and other parameters of the image frame to be drawn to the GPU so that the GPU can draw. The parameters transmitted include the color buffer, the depth buffer, and the stencil buffer. Among them, the color buffer represents the color information of each grid point, such as the RGB value. The depth buffer represents the depth information of each grid point. The template buffer is used to record some marking values ​​so that the marked part can be processed accordingly during subsequent rendering. Therefore, in this step, the stencil buffer is obtained, and the marked content is written in the stencil buffer. If the content of the written mark is 1, then the image blocks of the drawing instructions, such as the characters in the image frame, are marked as 1, indicating that the characters need to be further meshed.

[0121] For example, Figure 5 As shown, the marking module receives the dynamic drawing instruction and starts the template test, that is, writes the marked content such as the mark "1" in the stencil buffer to generate the GLstencil Mask (1) instruction. In the subsequent rendering process, the GLstencil Mask (1) instruction is inserted to mark the drawn image block with 1. Then the marking module sends the marked dynamic drawing instruction to the block module, that is, sends the GLstencil Mask (1) and the dynamic drawing instruction to the rendering module.

[0122] S406: The rendering module receives the marked dynamic drawing instruction and renders to obtain an image block, wherein the mark information is displayed in the image block.

[0123] The rendering module is in the GPU, and the GPU's off-screen rendering operation is performed in the screen buffer currently used for display. When the off-screen rendering is completed, the image block corresponding to the dynamic drawing instruction of the inserted mark can display the mark information. For the specific method of rendering, please refer to the implementation of off-screen rendering technology in the relevant technology, and no cumbersome explanation is given here.

[0124] The rendering process of the marked dynamic drawing instructions is as follows Figure 6As shown. First, the initial mesh of the model is skeletonized, and the bone matrix is ​​obtained after skeletonization. The purpose is to imitate the bone changes of natural objects and make the dynamic object movements more natural. Then the GPU converts the bone matrix into the MVP matrix. Finally, it is processed in the vertex shader to obtain the Normalized Device Coordinates (NDC) vertices. Among the bone matrix, MVP matrix, and NDC vertices obtained during the GPU rendering process, the bone matrix can only indicate the bone position, but cannot distinguish the distance of the dynamic object. After being converted into the MVP matrix, through MVP*P f Therefore, in the GPU rendering process, the recognition module intercepts the MPV matrix after the GPU obtains the MVP matrix.

[0125] After the rendering is completed, the block division module receives the image block displaying the marking information and executes the following S407.

[0126] S407: The segmentation module segments the image blocks displaying the mark.

[0127] Among them, subdivision is a kind of block accuracy relative to coarse division. Figure 7 As shown, Figure 7 (b) and Figure 7 (c) relative Figure 7 For (a), it is a rough classification method.

[0128] In some embodiments, the number of block divisions is used as the difference between fine division and coarse division. If the number of block divisions is greater than or equal to a threshold value B, it indicates a fine division operation. If the number of block divisions is less than or equal to a threshold value B, it indicates a coarse division operation.

[0129] In some other embodiments, the size of the divided grid blocks is used as the subdivided and coarsely divided blocks. If the grid block volume ≥ threshold C, it indicates a coarse division operation. If the grid block volume < threshold C, it indicates a subdivided operation.

[0130] Combine the following Figure 7 The subdivision process of this step is introduced. For each image block corresponding to a drawing instruction, whether it has a display mark or not, the subdivision module performs subdivision operations according to the number of grids initially set, such as Figure 7 As shown in (a), the block where the dynamic object is located is divided into a 5*5 grid B after the first block operation. If the image block has a display mark, the block module Figure 7 The image is further divided into blocks based on (a). Before the division, the division module first determines the area to be divided this time. Otherwise, if the image block does not have a display mark, the division is terminated.

[0131] After the first block operation, the block module performs a second block operation (i.e., subdivision) on the display marked area. As shown in (a) of Figure 7, the middle area of ​​the circle occupies an entire grid. When matching the middle area of ​​the circle, the luminosity and chromaticity of the entire grid of the frame image are matched with the luminosity and chromaticity of the entire grid of the previous frame image, and the correct grid can be matched. The part outside the middle area of ​​the circle does not occupy an entire grid. Part of the grid B displays the luminosity and chromaticity of the circle, while the other part displays the luminosity and chromaticity of other objects. Therefore, it is not easy to match the correct grid when performing grid matching. Based on this, the grid B where the object does not occupy the entire grid is subjected to a second block operation, and the grid B is subdivided to obtain a 2*2 grid C.

[0132] Similarly, before dividing the grid C, the area of ​​this subdivision must be determined first. After the subdivision area is determined, the grid C where the object does not occupy the entire grid is divided for the third time, and the grid C is subdivided to obtain a 2*2 grid D.

[0133] In some embodiments, in the above S406, each dynamic drawing instruction for drawing a near-view object inserts different tag information, so that in S407, it can be determined whether to perform a secondary blocking operation based on the tag information.

[0134] Specifically, when dividing into blocks, the grid block is generally subdivided into n*n grid blocks. Therefore, after dividing into blocks, the grid block may contain other areas except the image block corresponding to the drawing instruction. For example Figure 7 (a), the four grid blocks on the edge of the circle include not only the area where the circle is located (i.e., the area of ​​the graphic block), but also the area outside the circle. Therefore, other marking information different from the marking information of the circle will be inserted into the area outside the circle. For example, the marking information of the circle is 1, and the marking information outside the circle is 2. Then the four grid blocks on the original edge are marked with two marking information, 1 and 2. In other words, the grid B where the object mentioned above does not occupy an entire grid will be marked with multiple marking information. Then when the blocking module detects that the grid block includes multiple marking information, it performs a re-blocking operation. Determining whether to re-block the grid block through different marking information can improve the efficiency of blocking.

[0135] When the size of the subdivided grid reaches the preset minimum grid size, the subdivision stops. The final subdivided grid diagram is as follows: Figure 7 As shown in (c), it can be seen that most areas of the image block are subdivided into a complete grid, so as to facilitate the subsequent execution of S408 for accurate grid block matching.

[0136] S408 , the matching module receives the subdivided image blocks, matches the same grid blocks of two adjacent image frames, and obtains motion vector maps of all grid blocks between two adjacent image frames.

[0137] The above S407 introduces the subdivision operation of an image block, and according to the drawing instruction stream, the subdivision operation can be performed on all the image blocks that need to be subdivided in an image frame, and the image blocks that do not need to be subdivided can be subdivided by the initial set number of grids as described in S407. In this way, an image frame after the image frame is divided into blocks can be obtained. This operation is repeated many times to obtain multiple image frames. Then the matching module performs a matching operation, such as Figure 8 As shown, image frame B is the next image frame of image frame A. A first grid block 801 is selected in image frame A, and the color of the first grid block 801 is identified as red; the matching module matches the grid block with the color red in image frame B, and matches it to the second grid block 802 in image frame B. Then, the displacement difference between the first grid block 801 and the second grid block 802 is obtained to obtain a motion vector. After performing a matching operation on each grid block in image frame A, the motion vector of each grid block can be obtained, and the motion vector is represented in the same figure to obtain a motion vector diagram.

[0138] Based on the motion vector map, electronic devices including mobile phones can perform image frame prediction processing, temporal anti-aliasing (TAA) motion blur processing, super-resolution processing, etc.

[0139] The following describes the effect that can be achieved by subdividing image blocks using a scenario in which image frames are predicted based on motion vector maps.

[0140] The prediction of the image frame is achieved by inserting a prediction frame C between the image frame A and the image frame B, thereby increasing the number of picture frames.

[0141] like Fig. 9In (a), in image frame A, the hand image block is divided into two grid blocks, denoted as grid block 901, RGB value (252, 224, 203), and grid block 902, RGB value (248, 197, 183). The scene block around the hand image block is divided into one grid block, denoted as grid block 903, RGB value (124, 252,). In image frame B, the hand image block is divided into grid block 904 and grid block 905, and the scene block is divided into grid block 906, and the RGB values ​​are the same as above. The correct matching result should be: grid block 901 matches grid block 904, grid block 902 matches grid block 905, and grid block 903 matches grid block 906. Then the motion vector of grid block 901 can be denoted as x4-x1, the motion vector of grid block 902 is denoted as x5-x2, and the motion vector of grid block 903 is denoted as x6-x3. In this way, the motion vectors of the image blocks of the prediction frame C inserted between the image frames A and B can be approximately recorded as follows: (x4-x1) / 2 for grid block 907, (x5-x2) / 2 for grid block 908, and (x6-x3) / 2 for grid block 909. Then, the RGB values ​​are filled into grid blocks 907 to 909. The prediction frame C with accurate prediction results is obtained.

[0142] However, due to the low block precision, grid block 902 is matched to grid block 906, resulting in the motion vector of grid block 902 being incorrectly recorded as x6-x2. Then the motion vector of grid block 908 is incorrectly predicted as (x6-x2) / 2. Fig. 9 As shown in (a), the position of the grid block 908 in the rendered prediction frame C is distorted.

[0143] and Fig. 9 In (b), in the image frame D, the hand image block is divided into eight grid blocks 911, two grid blocks 912, and one grid block 913, and the scene block is divided into one grid block 914. The grid block has high block division accuracy, so the grid block is not easy to match incorrectly, so the motion vector and RGB value of the predicted grid block are more accurate, such as Fig. 9 The rendering effect of the predicted frame F in (b) is better than Fig. 9 The rendering effect of the predicted frame C in (a) is better.

[0144] In the above embodiments, the processing of dynamic drawing instructions for drawing near-view objects is introduced.

[0145] In some embodiments, the above method further comprises:

[0146] S1: The separation module determines that the drawing instruction is a static drawing instruction, and does not send the first message to the interception module.

[0147] In this step, the interception module receives the first message and performs the operation of intercepting resources. If the first message is not received, the resources are not intercepted accordingly. Then the subsequent marking module will not mark the static drawing instructions, and the block module will not subdivide the static drawing instructions. In this way, electronic devices including mobile phones can perform targeted block processing with different precisions for different drawing instructions, so as to improve the matching accuracy of the image details with low algorithm overhead.

[0148] In some embodiments, the above method further comprises:

[0149] S5: The recognition module determines that the dynamic drawing instruction is a dynamic drawing instruction for drawing a distant object, and does not send the second message to the marking module.

[0150] In this step, if the identification module does not send the second message to the marking module, the marking module will not mark the dynamic drawing instruction for drawing the distant view. Then the subsequent block segmentation module will not segment the image blocks corresponding to the dynamic drawing instruction for drawing the distant view, so as to achieve the beneficial effect of reducing the algorithm overhead.

[0151] In some embodiments, the vertex information in S404 may be a grid starting point. The grid starting point indicates the first vertex of the grid where the block image is located. During off-screen rendering, vertices are sequentially written into the memory, so the value of the first vertex written into the memory is the grid starting point.

[0152] For example, each dynamic drawing instruction includes a mesh vertex identifier MeshVertexId, which is a resource handle corresponding to a buffer storing vertices in the GPU. Its data format can be determined by the glVertexAttribPointer function.

[0153] void glVertexAttribPointer(GLuint index,GLint size,GLenum type,GLboolean normalized,G Lsizei stride,const void*pointer)

[0154] In the function, type represents the data type, and pointer represents the offset position of the grid information in the buffer, that is, pointer represents the offset of the grid information in the memory address.

[0155] The starting point of the mesh can be obtained according to the Mesh Vertex Id and pointer.

[0156] In this embodiment, a mapping table of Mesh Vertex Id+pointer and mesh starting point can be maintained. First, find out whether there is a mapping table record for the mesh drawn by the current drawing instruction. If it exists, record the mesh starting point. If it does not exist, map the Mesh Vertex GPU memory to the CPU accessible memory space through glMapBuffer (a function to obtain a cache object pointer) to obtain a pointer. Then record the mesh starting point according to the pointer offset value and add it to the mapping table for subsequent use.

[0157] like Fig.10 As shown, S403 is further explained in combination with an example of obtaining a grid starting point and a camera viewing angle matrix.

[0158] Dynamic drawing instructions include resources such as mesh vertex identifiers, transformation parameters, and camera view matrix. Among them, the camera view matrix can be directly intercepted. The mesh vertex needs to first determine whether there is a mapping table for the mesh vertex identifier based on the mesh vertex identifier. If so, get the corresponding mesh starting point from the mapping table. If there is no mapping table, first get the pointer based on the function of the cache object pointer, then record the mesh starting point according to the pointer offset value, and add it to the mapping table.

[0159] The advantage of only obtaining the grid starting point is that the projection scaling factor of the image block drawn by the drawing instruction can be approximately determined by the projection scaling factor obtained from the grid starting point, which reduces the amount of calculation and has higher calculation efficiency.

[0160] In summary, the embodiment of the present application provides an image processing method, which can obtain dynamic drawing instructions from drawing instructions, and obtain the depth information of dynamic objects in the dynamic drawing instructions. According to the depth information, it can be determined whether the dynamic object is a close-up object or a distant object, so as to adopt high-precision subdivision operation for the close-up object. The more subdivided the close-up object is, the more refined the calculation result of the motion vector of the object is, and the higher the accuracy of the result is. Therefore, based on the motion vector, the subsequent image frame prediction processing can obtain a prediction frame with better rendering effect and more realistic picture, thereby improving the user experience.

[0161] To facilitate the understanding of the above embodiments, a complete process example is used below to illustrate the present application scheme and its effects. Fig.11 This is a flowchart of an image processing method in a specific example of the present application.

[0162] Reference Fig.11 , the image processing method provided in the embodiment of the present application may include:

[0163] S1101, issuing a drawing instruction stream. In this step, an electronic device including a mobile phone responds to the operation of starting a game application and starts the game application. Then, a game engine in the game application can render a full-screen image (i.e., a frame of image) based on the drawing instruction stream issued by the game application.

[0164] S1102, judging the movement of the drawing instruction. This step is equivalent to the above S402, and will not be described here in detail.

[0165] S1103. If static, draw normally.

[0166] S1104. If dynamic, intercept the camera viewing angle matrix in the uniform variable (uniform).

[0167] S1105: Map the mesh vertex identifier to the mesh starting point.

[0168] S1102 - S1105 are repeated to render each drawing instruction in the drawing instruction stream.

[0169] S1106, intercepting the camera viewing angle matrix and the grid starting point. This step is equivalent to the above S403.

[0170] S1107: Determine whether the dynamic grid is close to the camera. This step is equivalent to the above S404.

[0171] S1108: If yes, start the template test to obtain the template mask map. The purpose of starting the template test is to insert the marking instruction. During the rendering process, the image block where the marking instruction is inserted will be marked with the marked content, thereby obtaining the template mask map. The template mask map is equivalent to the image block showing the mark in the above S407.

[0172] S1109: Coarsely divide or subdivide the grid based on the template mask image. This step is equivalent to the above S407.

[0173] S1110, calculate the full-screen motion vector. This step is equivalent to the above S408.

[0174] In summary, by adopting the method of the embodiment of the present application, electronic devices including mobile phones can effectively solve the problem of low accuracy of current image segmentation, which leads to poor matching accuracy of image details.

[0175] Some other embodiments of the present application provide an electronic device, which may include: the above-mentioned display screen (such as a touch screen), a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above-mentioned method embodiment. The structure of the electronic device can refer to Figure 3 The structure of the electronic device 200 is shown.

[0176] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0177] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0178] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0179] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0181] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0183] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: When the electronic device performs a rendering operation of an image frame, obtaining projection scaling information corresponding to at least one drawing instruction of the image frame, wherein the projection scaling information indicates a scaling degree of a drawing object corresponding to the drawing instruction in a to-be-displayed area; Determine, according to the projection scaling information, a subdivided area and a coarse subdivided area of ​​the image frame in the area to be displayed, wherein the coarse subdivided area includes a plurality of grid blocks, and the subdivided area includes a plurality of first-level subdivided blocks; Divide at least one first-level block of the subdivided area into a plurality of second-level blocks, so that the size of each second-level block obtained by segmenting the subdivided area is smaller than the size of each grid block obtained by segmenting the coarse-divided area; Determine, according to the coarse-divided area and the first-level blocks and the second-level blocks in the subdivided area, an object position of at least one of the drawing objects in each of two adjacent image frames, and obtain a motion vector of the drawing object based on the object position of the drawing object in each of two adjacent image frames; Predicting between every two adjacent image frames based on the motion vector to obtain a predicted frame; The projection scaling information includes a projection scaling factor, and determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information includes: If the projection scaling factor is greater than a first threshold, determining that the area drawn by the drawing instruction corresponding to the projection scaling factor is the coarsely divided area; If the projection scaling factor is less than or equal to the first threshold, it is determined that the area drawn by the drawing instruction corresponding to the projection scaling factor is the subdivided area.

2. The method according to claim 1, characterized in that The obtaining projection scaling information corresponding to at least one drawing instruction of the image frame includes: Acquire a camera viewing angle parameter of at least one of the drawing instructions of the image frame, and physical position information of a drawing object corresponding to at least one of the drawing instructions; Based on the camera viewing angle parameter and the physical position information, the projection scaling information of the drawing object under the camera viewing angle is obtained.

3. The method according to claim 2, characterized in that The acquiring of the camera viewing angle parameter of at least one drawing instruction of the image frame and the physical position information of the drawing object corresponding to at least one drawing instruction includes: It is detected that the drawing instruction is a dynamic drawing instruction, and a camera viewing angle parameter of at least one of the dynamic drawing instructions of the image frame and physical position information of a drawing object corresponding to at least one of the dynamic drawing instructions are obtained, wherein the dynamic drawing instruction is an instruction for drawing a dynamic object.

4. The method according to any one of claims 1 to 3, characterized in that The subdivided area is a grid area obtained by gridding the drawing object corresponding to the drawing instruction, and a display block of the drawing object in the area to be displayed is included in the subdivided area; After dividing at least one of the first-level blocks of the subdivided area into a plurality of second-level blocks, the method further comprises: It is detected that the second-level block contains an area outside the display block in the to-be-displayed area, and the second-level block is divided into a plurality of third-level blocks.

5. The method according to any one of claims 1 to 3, characterized in that: After dividing at least one of the first-level blocks of the subdivided area into a plurality of second-level blocks, the method further includes: If the size of the second-level blocks meets the preset size condition, the segmentation operation on the second-level blocks is stopped.

6. The method according to any one of claims 1 to 3, characterized in that After determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information, the method further includes: Marking a drawing instruction for drawing the subdivided area to obtain a first marking message, wherein the first marking message is used to mark a prompt character in an image block rendered by the drawing instruction for the subdivided area; The step of dividing at least one first-level block of the subdivided area into a plurality of second-level blocks comprises: At least one of the first-level blocks in the area where the prompt character is located in the subdivided area is divided into a plurality of second-level blocks.

7. The method according to any one of claims 1 to 3, characterized in that Before acquiring the projection scaling information corresponding to at least one drawing instruction of the image frame, the method further includes: It is detected that the drawing instruction is a static drawing instruction, and the area drawn by the static drawing instruction is determined as the coarse-divided area, wherein the static drawing instruction is an instruction for drawing a static object.

8. The method according to any one of claims 1 to 3, characterized in that After determining the subdivided area and the coarse subdivided area of ​​the image frame in the area to be displayed according to the projection scaling information, the method further includes: Marking a drawing instruction for drawing the subdivided area to obtain a second marking message, wherein the second marking message of the subdivided area corresponding to each drawing instruction is different, wherein the second marking message is used to mark a prompt character in an image block rendered by the drawing instruction of the subdivided area, and is used to determine whether to segment the second-level block; After dividing the at least one first-level block of the subdivided area into a plurality of second-level blocks, the method further includes: If the second-level block includes a plurality of the second-level marking messages, dividing the second-level block into a plurality of third-level blocks; If the second-level block includes one second marking message, the segmentation operation on the second-level block is stopped.

9. The method according to claim 3, characterized in that: The detecting that the drawing instruction is a dynamic drawing instruction includes: If the physical position information corresponding to two adjacent drawing instructions is different, it is determined that the drawing instruction is the dynamic drawing instruction.

10. The method according to claim 2 or 3, characterized in that: The subdivided area is a grid area obtained by meshing the drawing object corresponding to the drawing instruction, and the physical position information includes coordinate information of a grid vertex in the subdivided area where the drawing object is located, wherein the grid vertex is a vertex of the grid area.

11. An image processing method, characterized in that: The method is applied to an electronic device, and is applied to an image frame containing a plurality of drawing objects, wherein the drawing objects include a first object and a second object, and when the difference between the volume of the first object in a model space and the volume of the second object in the model space is greater than or equal to a threshold, the image frame is subjected to block processing; the method comprises: When the electronic device performs a rendering operation of the image frame, obtaining an object screen ratio corresponding to at least one drawing instruction of the image frame, wherein the object screen ratio is used to indicate an occupation degree of the drawing object corresponding to the drawing instruction in the area to be displayed; If the screen proportion of the object is greater than or equal to a second threshold, determining that the area drawn by the drawing instruction corresponding to the screen proportion of the object is a subdivided area; If the screen proportion of the object is less than a second threshold, determining that the area drawn by the drawing instruction corresponding to the screen proportion of the object is a coarse area, wherein the coarse area includes a plurality of grid blocks, and the subdivided area includes a plurality of first-level blocks; Divide at least one first-level block of the subdivided area into a plurality of second-level blocks, so that the size of each second-level block obtained by segmenting the subdivided area is smaller than the size of each grid block obtained by segmenting the coarse-divided area; Determine, according to the coarse-divided area and the first-level blocks and the second-level blocks in the subdivided area, an object position of at least one of the drawing objects in each of two adjacent image frames, and obtain a motion vector of the drawing object based on the object position of the drawing object in each of two adjacent image frames; A predicted frame is obtained by predicting between every two adjacent image frames based on the motion vector.

12. The method according to claim 11, characterized in that The occupancy includes the proportion of the area of ​​the drawing object in the to-be-displayed area to the total area of ​​the to-be-displayed area.

13. An electronic device, characterized in that: include: A communication module, a memory and one or more processors; the communication module, the memory and the processor are coupled; wherein the memory is used to store computer program code, the computer program code includes computer instructions; when the computer instructions are executed by the processor, When the processor executes the computer instructions, the electronic device performs the method according to any one of claims 1 to 10 or performs the method according to claim 11 or 12.

14. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on the electronic device, enable the electronic device to execute the method according to any one of claims 1 to 10 or the method according to claim 11 or 12.

Citation Information

Patent Citations

  • Image segmentation method and device based on multi-level grid

    CN103714529A

  • Virtual reality image processing system based on mobile terminal

    CN108921050A

  • Video image data frame insertion processing method and related equipment

    CN110933497A

  • Image processing method and related equipment

    CN113066001A

  • Video super-division method and device, electronic equipment and readable storage medium

    CN114782251A