A picture processing method based on OpenGL ES architecture

Abstract

The present invention relates to an image processing method based on the OpenGL ES architecture. By constructing a virtual cache dedicated to image processing in the running memory (RAM) of a mobile terminal, the technical problem that the mobile terminal GPU does not have a dedicated frame cache capacity, which makes it difficult to efficiently process image effects, can be solved. It is not necessary to improve the performance of the mobile terminal GPU to achieve multi-layer multi-effect superimposed image beautification and modification effects in the terminal, which greatly improves the efficiency and quality of user image editing and image beautification using a mobile terminal, thereby improving the user experience. The present invention has a wide range of application scenarios. As long as it is a portable mobile terminal device, the technology can be applied to achieve multi-layer multi-effect image beautification operations without improving GPU performance.

Description

Technical Field

[0001] The present invention relates to a method for processing an image, and in particular to a method for processing an image by superimposing multiple layers and multiple effects under an OpenGL ES architecture. Background Art

[0002] Traditional desktop computers or laptops are equipped with powerful graphics processors (GPUs), supplemented by large-capacity graphics card memory (also known as frame buffers), which enable traditional computers to process huge amounts of graphic files, edit pictures, overlay different filter styles, and render and display them on output devices at a relatively fast speed. However, on mobile terminals such as mobile phones and tablets, the performance and power of GPUs are limited due to the size, weight, battery capacity and other conditions of the terminal devices. If the traditional OpenGL architecture is directly applied to mobile terminals to process graphic information, it will be difficult for the mobile terminals to complete data processing. In view of the characteristics of mobile terminal GPUs, the existing technology generally uses an ES subset of the OpenGL architecture to provide image processing solutions for mobile terminals.

[0003] Since the screen size of early mobile terminals was not large and the resolution was generally not high, people did not have high requirements for the quality and color of the pictures displayed on mobile terminals. OpenGL ES can meet the requirements of picture editing with a single effect or multiple effects superimposed on a single layer. As the size of mobile terminals gradually increases and mobile applications (APPs) become more abundant, the number of scenes where people take photos and modify pictures on mobile terminals is increasing, and the requirements for picture modification and beautification effects are gradually increasing. However, the corresponding GPU performance improvement is not obvious, especially for the system-on-chip (SoC) based on the Advanced RISC Machine (ARM) architecture that is currently widely used in mobile terminals. The GPU is integrated into the SoC and is not configured with a separate frame buffer unit, which limits the GPU's ability to process image files with large data volumes. Therefore, how to achieve richer picture modification and beautification effects on limited GPU performance has become a research direction worthy of attention. Summary of the invention

[0004] The purpose of the present invention is to provide a method for image processing based on the OpenGL ES architecture in view of the performance characteristics of the GPU of a mobile terminal and the user's expectations for image processing in a mobile terminal, which is implemented by the following technical solutions:

[0005] A method for processing images based on the OpenGL ES architecture comprises the following steps:

[0006] Apply for a virtual cache space in the OpenGL ES architecture;

[0007] Importing a target image from a storage device into a virtual cache space, using an edge detection algorithm to identify image object boundaries in the target image and capture key pixel points, and layering the target image according to the image object boundary density to generate multiple layered boundary group images;

[0008] Performing RGB color image layering on multiple layered boundary group images, generating multiple layered boundary color group images and corresponding multiple layered boundary color group image labels, and establishing a mapping relationship between the layered boundary color group images and the layered boundary color group image labels, and writing the multiple layered boundary color group images into a storage and clearing them from a virtual cache space;

[0009] In the virtual cache space, the target texture is superimposed on the layered boundary color group map label according to the target effect and rendered to generate a new layered boundary color group map label, and the target texture data is written into the layered boundary color group map corresponding to the storage through a mapping relationship and superimposed to form a new layered boundary color group map;

[0010] Repeat the previous step until multiple target textures and multiple layered boundary color group map labels are completely superimposed.

[0011] Generate an effect picture label according to the superposition result and render it, write the rendering result into the storage through the mapping relationship to generate an effect picture and output it to the display module;

[0012] Clear multiple layered boundary color group maps in storage and multiple layered boundary color group map tags in virtual buffer space, and close and release virtual buffer space in the OpenGL ES architecture.

[0013] Compared with the prior art, the image processing method of the present invention, by utilizing the characteristics of the ES subset of the OpenGL architecture, enables the mobile terminal to build a virtual cache dedicated to image processing in the running memory (RAM), thereby solving the technical problem that the mobile terminal GPU does not have a dedicated frame cache, resulting in difficulty in efficiently processing image effects. At the same time, the group image label containing key pixels is used as the rendering material in the virtual cache space, thereby reducing the defect of the prior art that the original image needs to be used as the rendering material and occupies a large amount of running memory space, so that the existing mobile terminal can achieve the beautification and modification effect of multi-layer and multi-effect superposition in the terminal without increasing the running memory capacity and GPU performance, which greatly improves the efficiency and quality of users using mobile terminals for image editing and beautification, thereby improving the user experience.

[0014] Furthermore, the edge detection algorithm in step S2 can be one or more of Canny edge detection, Sobel edge detection, Prewitt edge detection or Roberts edge detection. After being processed by one or more of the above edge algorithms, multiple layered boundary group images with the same key pixel points and different image object boundary densities are generated.

[0015] Furthermore, the multiple layered boundary group images generated in step S2 are specifically a foreground layered boundary group image, a portrait layered boundary group image and a background layered boundary group image.

[0016] Furthermore, in step S3, the RGB color image is layered into a binary layered boundary color group image, a red layered boundary color group image, a green layered boundary color group image and a blue layered boundary color group image.

[0017] Furthermore, the layered boundary color group image label established in step S3 includes the size data, key pixel points, and image object boundaries of the layered boundary color group image.

[0018] Furthermore, the mapping relationship established in step S3 is a conversion relationship between the size data, key pixel points, and image object boundaries in the layered boundary color group map and multiple layered boundary color group map labels.

[0019] Through the above steps, the present invention can realize superimposing a target texture on a target image and generating an effect image based on the OpenGL ES architecture while occupying the least RAM resources.

[0020] In one embodiment, according to the requirements of the effect image, the superimposed target texture may also be the following elements:

[0021] The target texture may be one or more of the elements such as clarity, brightness, contrast, sharpness, saturation, hue, texture, text label, etc. corresponding to the layered boundary color group map.

[0022] The target texture may be one or more of the layered boundary color group map labels processed in the above steps.

[0023] Through the above technical scheme, the present invention solves the technical problem that it is difficult to complete multiple layers and multiple effects in a mobile terminal, realizes the technical effect of superimposing multiple different effects on a picture, and further enriches the picture beautification effects that can be achieved by users.

[0024] In one embodiment, according to the user's operation requirements, the layered edge color group map tag that completes texture overlay and rendering completes the overlay of texture and layered edge color group map in the GPU and renders it to the display module of the mobile terminal for display to the user.

[0025] Through the above technical scheme, the present invention solves the technical problem that users cannot view the rendering effects of part of the layers as they wish when beautifying pictures using a mobile terminal, and achieves the technical effect of facilitating timely adjustment of parameters such as the method, quantity, and orientation of texture overlay to obtain pictures with better beautification effects.

[0026] In one embodiment, according to the actual overlay needs of textures and layers, when the target texture to be overlaid in step S4 is to offset the corresponding layered boundary color group map label, the layered boundary color group map and the corresponding label after the target texture is overlaid are deleted, specifically, the layered boundary color group map label after the target texture is overlaid is cleared in the virtual cache space, and the layered boundary color group map is cleared in the storage at the same time.

[0027] Through the above-mentioned technical scheme, the present invention application solves the technical problem of needing to superimpose invalid layered boundary color group image labels and invalid layered boundary color group images when rendering effect images, thereby achieving the technical effect of reducing the number of layers and the amount of data required to be processed by the mobile terminal for rendering effect images, thereby improving the efficiency of beautifying images in the mobile terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the image processing method based on OpenGL ES architecture of the present invention. DETAILED DESCRIPTION

[0029] Example 1

[0030] A method for processing images based on the OpenGL ES architecture comprises the following steps:

[0031] S111, applying for a virtual cache space in the OpenGL ES architecture;

[0032] S211, importing the target image from the storage into the virtual cache space, using Roberts edge detection and Sobel edge detection algorithms to respectively identify the image object boundaries in the target image and capture key pixel points, and layering the target image according to the image object boundary density to generate a portrait layered boundary group image and a background layered boundary group image;

[0033] S311, performing RGB color image layering on the portrait layered boundary group image and the background layered boundary group image generated in the previous step, and generating corresponding portrait binary layered boundary color group image, portrait red layered boundary color group image, portrait green layered boundary color group image, portrait blue layered boundary color group image, background binary layered boundary color group image, background red layered boundary color group image, background green layered boundary color group image and background blue layered boundary color group image; establishing labels for the above-mentioned groups of images in the virtual cache space according to the size data information of the above-mentioned groups of images, the coordinate information of the key pixel points in the groups of images, and the coordinate information of the image object boundaries in the groups of images, and establishing a mapping relationship between the size data, key pixel points, and image object boundaries in each layered boundary color group image and the corresponding layered boundary color group image labels between the above-mentioned groups of images and the group image labels; then writing the above-mentioned groups of images into a storage for storage and clearing them from the virtual cache space, and the virtual cache space only retains the labels and mapping relationships of the above-mentioned groups of images;

[0034] S411, in the virtual cache space, according to the target effect, superimpose contrast and brightness improvement data on the portrait binary layered boundary color group image label, and render in the virtual cache space to generate a new portrait binary layered boundary color group image label, the contrast and brightness improvement data are written into the portrait binary layered boundary color group image corresponding to the storage through a mapping relationship and updated to the portrait binary layered boundary color group image after contrast and brightness improvement; in the virtual cache space, according to the target effect, superimpose saturation and sharpness reduction data on the background blue layered boundary color group image label, and render in the virtual cache space to generate a new background blue layered boundary color group image label, the saturation and sharpness reduction data are written into the background blue layered boundary color group image corresponding to the storage through a mapping relationship and updated to the background blue layered boundary color group image after saturation and sharpness reduction;

[0035] S511, repeating step S401 to overlay and render each layered boundary color group map label in the virtual cache space according to the target effect of the effect image, and writing the rendered result into the corresponding layered boundary color group map in the storage through a mapping relationship;

[0036] S611, superimposing and rendering the labels of each layered boundary color group image superimposed in the virtual cache space in step S501, writing the rendering result into the storage through the mapping relationship to generate an effect image, and outputting the rendering result to the display module;

[0037] S711 , clearing each layer boundary color group map in the storage and multiple layer boundary color group map labels in the virtual buffer space, and closing and releasing the virtual buffer space in the OpenGL ES architecture.

[0038] The OpenGL ES architecture refers to the OpenGL ES instructions running in the RAM (or running memory) of the mobile terminal;

[0039] The virtual cache space refers to a dedicated cache field established in the RAM (or running memory) of the mobile terminal, which is used to store the data of the image file required for processing;

[0040] The image object boundary is a set of pixel points in the image that are detected and marked as edges by the edge detection algorithm;

[0041] The key pixel points include grayscale mutation points, intersection points of image aspect ratio dividing lines, intersection points of image aspect ratio dividing lines and image object boundaries, intersection points of image object boundaries marked by different edge detection algorithms, and 8-neighborhood pixel points of the above pixel points;

[0042] The target effect refers to the beautification effect that the user expects the target image to achieve;

[0043] The superposition refers to the process of modifying the key pixel points according to the corresponding values ​​of the target texture;

[0044] Through the above steps, this embodiment can use fewer memory units in the RAM as virtual cache space of the GPU frame cache unit, and use the virtual cache space to complete the superposition of multiple layers and multiple effects.

[0045] Compared with the prior art, the present invention provides an image processing method based on the OpenGL ES architecture, which uses an edge detection algorithm and RGB color image layers in a virtual cache space to generate each group of images, establishes corresponding labels and mapping relationships, and uses group image labels with small data volumes to perform texture overlays, thereby greatly reducing the storage resources required for the virtual cache space during the overlay and rendering process, so that multi-effect overlays of images can be achieved on mobile terminals, meeting the needs of users to beautify and modify images anytime and anywhere.

[0046] Example 2

[0047] Embodiment 2 is an improvement on Embodiment 1, and mainly includes the following steps when the target texture is a layer:

[0048] S121, applying for a virtual cache space in the OpenGL ES architecture;

[0049] S221, importing the target image from the storage into the virtual cache space, using the Canny edge detection algorithm to identify the image object boundary in the target image and capture key pixel points, and layering the target image according to the image object boundary density to generate a foreground layered boundary group image and a background layered boundary group image;

[0050] S321, performing RGB color image layering on the foreground layered boundary group image and the background layered boundary group image generated in the previous step, and generating corresponding foreground binary layered boundary color group image, foreground red layered boundary color group image, foreground green layered boundary color group image, foreground blue layered boundary color group image, background binary layered boundary color group image, background red layered boundary color group image, background green layered boundary color group image and background blue layered boundary color group image; establishing labels for the above-mentioned groups of images in the virtual cache space according to the size data information of the above-mentioned groups of images, the coordinate information of the key pixel points in the groups of images, and the coordinate information of the image object boundaries in the groups of images, and establishing a mapping relationship between the size data, key pixel points, and image object boundaries in each layered boundary color group image and the corresponding labels of each layered boundary color group image between the above-mentioned groups of images and the group image labels; then writing the above-mentioned groups of images into the storage for storage and clearing them from the virtual cache space, and the virtual cache space only retains the labels and mapping relationships of the above-mentioned groups of images;

[0051] S421, in the virtual cache space, superimposing the hue and brightness reduction data on the foreground binary layered boundary color group image label according to the target effect, and rendering in the virtual cache space to generate a new foreground binary layered boundary color group image label, the hue and brightness reduction data are written into the foreground binary layered boundary color group image corresponding to the storage through a mapping relationship and updated to the foreground binary layered boundary color group image after the hue and brightness are reduced; in the virtual cache space, superimposing the clarity and text label data on the background blue layered boundary color group image label according to the target effect, and rendering in the virtual cache space to generate a new background blue layered boundary color group image label, the clarity and text label data are written into the background blue layered boundary color group image corresponding to the storage through a mapping relationship and updated to the background blue layered boundary color group image after the clarity and text labels are superimposed;

[0052] S422, in the virtual cache space, two group image tags to which target textures have been added are superimposed according to the target effect, specifically, corresponding data is added to each other as the target texture of the group image tags in the virtual cache space, and new group image tags are respectively rendered to generate the new group image tags, the two new group image tag information should be consistent, and the two new composite group image tag data are written into the corresponding groups of the storage through the mapping relationship and updated into the composite group image in the storage;

[0053] S521, repeating steps S421 and S422 to overlay and render the labels of each layered boundary color group image in the virtual cache space according to the target effect of the effect image, and writing the rendered results into the corresponding layered boundary color group image in the storage through a mapping relationship;

[0054] S621, superimposing and rendering the labels of the layered boundary color group images superimposed in the virtual cache space in step S521, writing the rendering results into the storage through the mapping relationship to generate an effect image, and outputting the rendering results to the display module;

[0055] S721 . Clear each layer boundary color group map in the storage and multiple layer boundary color group map labels in the virtual buffer space, and close and release the virtual buffer space in the OpenGL ES architecture.

[0056] The difference between Example 2 and Example 1 is that Example 2 can realize the superposition of group image labels as target textures and generate two composite group image labels with consistent information. When the group image labels are used as target textures, the amount of data is large. Therefore, retaining two group image labels with consistent information facilitates mutual verification in subsequent steps to prevent errors and omissions in the superposition process, which may cause deviations in the superposition of layers.

[0057] This embodiment calls the group image tag to which the target texture has been added as the target texture for overlaying the group images, thereby greatly reducing the amount of rendering data for overlaying the group images. At the same time, by retaining two group image tags with consistent information for subsequent verification, it is possible to overlay multiple layers and multiple effects on images on mobile terminals, thereby increasing the means available to users to beautify images.

[0058] Example 3

[0059] Embodiment 3 is an improvement on Embodiment 1, and is mainly aimed at outputting rendering results to a display module during the rendering process for users or developers to view the superposition effect of the target texture, and includes the following steps:

[0060] S131, applying for a virtual cache space in the OpenGL ES architecture;

[0061] S231, importing the target image from the storage into the virtual cache space, using the Prewitt edge detection algorithm and the Sobel edge detection algorithm to respectively identify the image object boundaries in the target image and capture key pixel points, and layering the target image according to the image object boundary density to generate a portrait layered boundary group image and a background layered boundary group image;

[0062] S331, performing RGB color image layering on the portrait layered boundary group image and the background layered boundary group image generated in the previous step, and generating corresponding portrait binary layered boundary color group image, portrait red layered boundary color group image, portrait green layered boundary color group image, portrait blue layered boundary color group image, background binary layered boundary color group image, background red layered boundary color group image, background green layered boundary color group image and background blue layered boundary color group image; establishing labels for the above-mentioned groups of images in the virtual cache space according to the size data information of the above-mentioned groups of images, the coordinate information of the key pixel points in the groups of images, and the coordinate information of the image object boundaries in the groups of images, and establishing a mapping relationship between the size data, key pixel points, and image object boundaries in each layered boundary color group image and the corresponding layered boundary color group image labels between the above-mentioned groups of images and the group image labels; then writing the above-mentioned groups of images into the storage for storage and clearing them from the virtual cache space, and the virtual cache space only retains the labels and mapping relationships of the above-mentioned groups of images;

[0063] S431, in the virtual cache space, superimposing contrast and saturation enhancement data on the portrait red layered boundary color group image label according to the target effect, and rendering in the virtual cache space to generate a new portrait red layered boundary color group image label, the contrast and saturation enhancement data is written into the portrait red layered boundary color group image corresponding to the storage through a mapping relationship and updated to the portrait red layered boundary color group image after contrast and brightness enhancement;

[0064] S432, according to the needs of the user or developer, rendering the portrait red layered boundary color group image label with enhanced contrast and brightness in the virtual cache space, and outputting it to the display module;

[0065] S433, in the virtual cache space, according to the target effect, superimpose brightness and contrast reduction data on the background binary layered boundary color group image label, the background red layered boundary color group image label, the background green layered boundary color group image label and the background blue layered boundary color group image label, and perform a clearing operation when the superimposed brightness and contrast reduction data achieves the effect of offsetting the key pixels in the above group image labels and all the pixels in the group images, specifically: deleting the above group image labels and mapping relationships in the virtual cache space, and deleting the above group images in the storage;

[0066] S531, repeating steps S431, S432, and S433 to overlay and render the labels of each layered boundary color group image in the virtual cache space according to the target effect of the effect image, and writing the rendered results into the corresponding layered boundary color group image in the storage through a mapping relationship;

[0067] S631, superimposing and rendering the labels of each layered boundary color group image superimposed in the virtual cache space in step S531, writing the rendering result into the storage through the mapping relationship to generate an effect image, and outputting the rendering result to the display module;

[0068] S731 , clearing each layer boundary color group map in the storage and multiple layer boundary color group map labels in the virtual buffer space, and closing and releasing the virtual buffer space in the OpenGL ES architecture.

[0069] The difference between Example 3 and Example 1 is that Example 3 can reduce the amount of data required for rendering in subsequent steps of multi-layer overlay by clearing the group image label in the virtual cache space and the group image in the storage when the purpose of the overlay texture is to offset the image of a group image. At the same time, any texture overlay result of image processing can be rendered and output to the display module according to the operation of the user or developer.

[0070] This embodiment deletes the group image labels and mapping relationships whose image effects are offset in clearing the virtual cache space, so that the amount of data required for rendering in the subsequent steps of multi-layer overlay is greatly reduced, the efficiency of processing image data by the mobile terminal is improved, and the required cache resources are reduced. At the same time, any group image in the process of beautifying the image can be rendered and displayed according to the operation of the user or developer, so that the user or developer can know the processing status of the current image and adjust or modify the subsequent steps according to the target effect.

[0071] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

[0072] If the functions described in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present invention that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A method for image processing based on OpenGL ES architecture, characterized in that: The main steps include: S1. Apply for a virtual cache space in the OpenGL ES architecture; S2, importing the target image from the storage into the virtual cache space, using the edge detection algorithm to identify the image object boundary in the target image and capture the key pixel points, and layering the target image according to the image object boundary density to generate multiple layered boundary group images; S3, performing RGB color image layering on the multiple layered boundary group images, generating multiple layered boundary color group images and corresponding multiple layered boundary color group image labels, and establishing a mapping relationship between the layered boundary color group images and the layered boundary color group image labels, and writing the multiple layered boundary color group images into a storage and clearing them from the virtual cache space; S4, in the virtual cache space, according to the target effect, superimpose the target texture on the layered boundary color group map label and render to generate a new layered boundary color group map label, specifically, the group map labels in the virtual cache space add corresponding data to each other as the target texture, respectively render to generate new group map labels, and the data of the target texture is written into the layered boundary color group map corresponding to the storage through a mapping relationship and superimposed to form a new layered boundary color group map; S5, repeating step S4 until all the target textures and the layered boundary color group labels are superimposed; S6, generating an effect picture label according to the superposition result of S5 and rendering it, and writing the rendering result into the storage through the mapping relationship to generate an effect picture and output it to the display module; S7. Clear multiple layered boundary color group maps in the storage and multiple layered boundary color group map labels in the virtual cache space, and close and release the virtual cache space in the OpenGL ES architecture.

2. The image processing method according to claim 1, characterized in that: The edge detection algorithm is Canny edge detection, Sobel edge detection, Prewitt edge detection or Roberts edge detection.

3. The image processing method according to claim 2, characterized in that: The plurality of layered boundary group images are divided into a foreground layered boundary group image, a portrait layered boundary group image and a background layered boundary group image according to boundary density.

4. The image processing method according to claim 1, characterized in that: The multiple layered boundary color group images are divided into a binary layered boundary color group image, a red layered boundary color group image, a green layered boundary color group image and a blue layered boundary color group image according to the RGB color image layering.

5. The image processing method according to claim 4, characterized in that: The multiple layered boundary color group map labels include size data, key pixel points, and image object boundaries of the multiple layered boundary color group maps.

6. The image processing method according to claim 5, characterized in that: The mapping relationship includes the conversion relationship between the size data, key pixel points, and image object boundaries in the layered boundary color group map and multiple layered boundary color group map labels.

7. The image processing method according to claim 1, characterized in that: The target texture in step S4 includes clarity, brightness, contrast, sharpness, saturation, hue, texture, and text label.

8. The image processing method according to claim 7, characterized in that: The target texture in step S4 includes the new layered boundary color group map label.

9. The image processing method according to claim 1, characterized in that: In step S4, a new layered boundary color group map label is generated by rendering and the rendering result is output to the display module.

10. The image processing method according to claim 1, characterized in that: The method further comprises the following steps: In step S4, after superimposing the target texture, the corresponding layered boundary color group map tag is offset, the layered boundary color group map tag is cleared in the virtual cache space, and the layered boundary color group map is cleared in the storage.

Images