Image processing method and device, electronic equipment and storage medium

Abstract

The application relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: a central processor calculates the pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing; the central processor calculates the pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing; the central processor respectively imports, as input textures, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates into a shader of a graphics processor, with the original image as an input texture; and the central processor obtains output textures output by the shader from the graphics processor as the result image after image processing. The method can improve the image processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, and in particular to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Image processing technology can process an original image to obtain a result image with a desired effect.

[0003] For example, ultra-wide-angle camera modules have been widely used in terminal devices such as mobile phones, which can obtain a wider field of view than conventional camera modules, but the original images captured by the ultra-wide-angle camera modules usually have image distortion. As shown in FIG. 1, it is an original image captured by an ultra-wide-angle camera module of an embodiment. As can be seen from FIG. 1, the four peripheral edge regions of the original image are obviously curved and deformed, i.e., image distortion, and the farther away from the center of the image, the more serious the distortion. Figure 1 Figure 1 In order to eliminate the image distortion in the original image, the camera intrinsic parameters and distortion parameters can be obtained by calibrating the camera module, which are used for distortion correction processing of the original image captured by the camera to eliminate the image distortion phenomenon in the original image and obtain a result image after distortion correction. The result image obtained after the distortion correction processing is shown in FIG. 2. Figure 2

[0004] However, in practical applications, the resolution of the original image is getting larger and larger, for example, resolutions of 2K or above are becoming more and more common, and some application functions have high requirements for the real-time performance of the operation result image, for example, for the preview function, it is generally required to reach 20FPS or above without obvious lag. Using a central processing unit (CPU) to realize the above image processing has the problems of high occupation of operation resources and difficulty in meeting the real-time performance requirements of the application. SUMMARY

[0005] Therefore, it is necessary to provide an image processing method, device, electronic equipment and storage medium capable of more quickly and efficiently realizing image processing of an original image to solve the above technical problems.

[0006] ​​An image processing method, comprising: a central processing unit calculating pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing; the central processing unit calculating pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing; the central processing unit inputting, into a shader of a graphics processing unit, the original image as an input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates; and the central processing unit obtaining, from the graphics processing unit, an output texture output by the shader as the result image after image processing.

[0007] In one embodiment, the central processing unit calculating pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing comprises: the central processing unit establishing a corresponding result image grid according to a size of an expected result image, the result image grid comprising a plurality of first grid points distributed in a matrix on the result image; and the central processing unit determining pixel coordinates of each first grid point in the result image.

[0008] In one embodiment, the result image has a width of W and a height of H, the plurality of first grid points distributed in a matrix comprises a plurality of first grid points distributed in a square matrix, each square grid in the square matrix comprising four adjacent first grid points, and a side length of each square grid being d; and the central processing unit determines that pixel coordinates of a first grid point i with a width direction index of x and a height direction index of y in the plurality of first grid points in the result image are (u i ′,v i ′), where, and where,

[0009] In one embodiment, the central processing unit calculating pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing comprises: the central processing unit obtaining a coordinate mapping relationship between pixel coordinates in the result image and pixel coordinates in the original image; and the central processing unit converting the pixel coordinates of each first grid point to obtain pixel coordinates of each second grid point before image processing according to the coordinate mapping relationship.

[0010] In one embodiment, the coordinate mapping relationship is determined according to one or more combinations of a distortion correction conversion relationship, an affine transformation conversion relationship, and a spherical projection conversion relationship.

[0011] In one embodiment, when the result image has a resolution different from that of the original image, the coordinate mapping relationship is further determined according to the resolution of the result image and the resolution of the original image.

[0012] In one embodiment, the central processor imports the original image as an input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates into a shader of a graphics processor, including: the central processor imports the original image in the central processor as an input texture, the pixel coordinates of each first grid point in the central processor as vertex coordinates, and the pixel coordinates of each second grid point in the central processor as texture coordinates into a shader of a graphics processor by using a platform shared memory technology.

[0013] In one embodiment, the shader is configured to map an input texture from texture coordinates to vertex coordinates to obtain an output texture.

[0014] In one embodiment, the central processor obtains the output texture output by the shader from the graphics processor as the result image after image processing, including: the central processor obtains the output texture output by the shader from the graphics processor as the result image after image processing by using a platform shared memory technology.

[0015] In one embodiment, the original image is an image captured by a camera module, and the central processor displays the result image as a preview image on a display screen after obtaining the output texture as the result image.

[0016] An image processing apparatus, comprising:

[0017] A first grid point coordinate calculation module configured to cause a central processor to calculate pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing;

[0018] A second grid point coordinate calculation module configured to cause the central processor to calculate pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing;

[0019] A shader import module configured to cause the central processor to import the original image as an input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates into a shader of a graphics processor; and

[0020] A shader export module configured to cause the central processor to obtain an output texture output by the shader from the graphics processor as the result image after image processing.

[0021] An electronic device comprises a memory, a central processing unit and a graphic processing unit, the memory stores a computer program, the central processing unit implements the following steps when executing the computer program: the central processing unit calculates pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing; the central processing unit calculates pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing; the central processing unit respectively imports the original image as an input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates into a shader of the graphic processing unit; and the central processing unit obtains an output texture output by the shader from the graphic processing unit as the result image after image processing.

[0022] A computer readable storage medium stores a computer program, the computer program implements the following steps when executed by a central processing unit: the central processing unit calculates pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing; the central processing unit calculates pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing; the central processing unit respectively imports the original image as an input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates into a shader of a graphic processing unit; and the central processing unit obtains an output texture output by the shader from the graphic processing unit as the result image after image processing.

[0023] The image processing method, device, electronic device and storage medium calculate pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing, and calculate pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing, thereby obtaining a grid point mapping relationship before and after image processing. The central processing unit only needs to calculate the coordinate mapping relationship between the grid points, without calculating all coordinate mapping relationships of all pixels in the original image, which can greatly reduce the operation load of the central processing unit. Meanwhile, the calculated grid point mapping relationship represented by the pixel coordinates of each first grid point and the pixel coordinates of each second grid point, and the original image are imported into the shader of the graphic processing unit for operation, thereby ingeniously utilizing the multi-thread parallel processing feature of the shader of the graphic processing unit and the feature of combining the bilinear interpolation sampling operation with the triangular rendering operation, and efficiently realizing the drawing of the result image after image processing through the shader. Therefore, the operation load of the central processing unit can be reduced while the image processing efficiency is improved, and the real-time performance of the output result image of image processing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Here is the original image as an example;

[0025] Figure 2 Here is an example image of the distortion-corrected result;

[0026] Figure 3 This is a schematic diagram of an electronic device in one embodiment;

[0027] Figure 4 This is a flowchart illustrating an image processing method in one embodiment;

[0028] Figure 5 This is a schematic diagram of the resulting image grid and the original image grid in one embodiment;

[0029] Figure 6 This is a schematic diagram illustrating the execution of triangle partitioning in one embodiment;

[0030] Figure 7 This is a schematic block diagram of an image processing apparatus in one embodiment;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] The image processing method provided in this application can be applied to, for example... Figure 3 The electronic device 300 shown can be, but is not limited to, various smartphones, digital cameras, personal computers, laptops, tablets, etc. The electronic device 300 may be equipped with a camera 301, which can capture raw images in real time and perform the image processing method of this application embodiment on the raw image to obtain a corrected result image. In an alternative embodiment, the raw image may also be captured by an external device other than the electronic device 300. The electronic device 300 can download the raw image from the network or receive and store it from an external device. Then, the electronic device 300 can perform the image processing method of this application embodiment on the raw image obtained from the external device to obtain a corrected result image. The electronic device 300 may also be equipped with a display screen 302, so that the obtained corrected result image can be displayed on the display screen 302 in real time for the user to view.

[0034] In one embodiment, as shown in Figure 4 An image processing method is provided, to which the method is applied to Figure 3 The method can include the following steps S410-S440.

[0035] Step S410, the central processor calculates the pixel coordinates of each first grid point in the result image grid corresponding to the result image after image processing.

[0036] In one embodiment, step S410 can include: the central processor establishes a corresponding result image grid according to the size of the expected result image, the result image grid including a plurality of first grid points distributed in a matrix on the result image; and the central processor determines the pixel coordinates of each first grid point in the result image.

[0037] For example, according to the size of the result image required, for example, according to the aspect ratio of the result image, a result image grid of M rows x N columns of M x N first grid points can be established, which can represent the coordinate positions of the pixel points in each region of the result image. The M row x N column result image grid can uniformly cover the result image.

[0038] As shown in Figure 5 The value of each first grid point i in the result image grid can be represented as (u i ′,v i ′). Where u i ′ is the horizontal coordinate of the first grid point i in the pixel coordinate system, and v i ′ is the vertical coordinate of the first grid point i in the pixel coordinate system.

[0039] For example, taking the aspect ratio of the required result image as 4:3 (for example, the resolution is 1024 x 768), M can be 80 and N can be 60. In other examples, the values of M and N can be determined according to actual needs, for example, according to the aspect ratio, resolution, etc. of the result image. The greater the values of M and N, the higher the processing accuracy of the image, but the more processing resources required and the lower the processing rate.

[0040] After establishing the corresponding result image grid according to the size of the result image, the pixel coordinates (u i ′,v i ′) of each first grid point in the result image grid can be determined accordingly.

[0041] For example, in one embodiment, the result image has a width of W and a height of H, and the plurality of first grid points distributed in a matrix includes a plurality of first grid points distributed in a square matrix, each square grid formed by every four adjacent first grid points in the square matrix has a side length of d, then the number of first grid points in the result image grid along the width direction is and the number along the height direction is Correspondingly, the index value (x, y) of each first grid point i can be constructed, x is the index along the width direction, and y is the index along the height direction, The index value (x, y) represents the relative position of the first grid point i in the result image grid, that is, it can indicate that it is the first grid point in the xth row and yth column (x and y are counted from 0), and each first grid point can have a unique index value. Correspondingly, when determining the pixel coordinates of each first grid point in the result image, the central processing unit can determine that the first grid point i with the width direction index x and the height direction index y in the plurality of first grid points has the pixel coordinates (u i ′,v i ′) in the result image. Wherein:

[0042] In step S420, the central processing unit calculates the pixel coordinates of each second grid point in the original image grid corresponding to the original image before image processing.

[0043] Wherein, the original image is an unprocessed image taken by a camera, and image distortion exists in the original image. The camera can include various devices capable of capturing images, such as a camera, a camera module, etc.

[0044] In one embodiment, step S420 includes: the central processing unit obtains a coordinate mapping relationship between the pixel coordinates in the result image and the pixel coordinates in the original image; and the central processing unit converts the pixel coordinates of each first grid point to obtain the pixel coordinates of each second grid point before image processing according to the coordinate mapping relationship.

[0045] Referring to Figure 5As shown, the plurality of second grid points in the original image grid have one-to-one correspondence with the plurality of first grid points in the result image grid. For example, in the case that the result image grid is composed of first grid points distributed in M rows by N columns, the original image grid is also composed of second grid points distributed in M rows by N columns. After the result image grid and the coordinate mapping relationship are determined, the pixel coordinates of each first grid point in the result image grid are converted according to the coordinate mapping relationship to correspondingly determine the pixel coordinates of each second grid point in the original image grid. Thus, for a given result image grid, what kind of original image grid will be determined depends on the setting of the coordinate mapping relationship. Therefore, different coordinate mapping relationships can be designed according to different application requirements to achieve different image processing effects.

[0046] Taking the need for distortion correction of the original image as an example, in this embodiment, in order to achieve the distortion correction processing, the coordinate mapping relationship can be determined according to the distortion correction conversion relationship. In other embodiments, the coordinate mapping relationship can alternatively or additionally be determined according to an affine transformation conversion relationship, a spherical projection conversion relationship, etc., to correspondingly achieve affine transformation processing, spherical projection processing, etc. of the original image.

[0047] In the case that the original image and the desired result image have the same resolution, generally only basic image mapping processing such as distortion correction, affine transformation, spherical projection, etc. of the original image is needed to obtain the corresponding image processing effect, in which case the coordinate mapping relationship can be determined according to the application requirements according to one or more combinations of the distortion correction conversion relationship, the affine transformation conversion relationship, and the spherical projection conversion relationship.

[0048] However, in some cases, for example in the preview function, the desired result image can have a different resolution from the original image, which requires more processing of the image to ensure that the obtained result image has the desired result image resolution.

[0049] In one embodiment, when the result image has a different resolution from the original image, a scaling and / or cropping processing of the original image can be further included before step S410 to obtain a processed original image having the desired result image resolution, and then steps S410-S440 are executed to determine the result image having the desired result image resolution, or steps S410-S440 can be executed first to obtain a result image having the original image resolution, and then a scaling and / or cropping processing of the result image having the original image resolution is further included after step S440 to determine the result image having the desired result image resolution.

[0050] Alternatively, in another embodiment, when the result image has a different resolution from the original image, the coordinate mapping relationship can be determined on the basis of one or more combinations of a distortion correction conversion relationship, an affine transformation conversion relationship, a spherical projection conversion relationship, and also according to the resolution of the result image and the resolution of the original image. For example, it is also determined according to the scaling ratio between the resolution of the result image and the resolution of the original image, so that the original image grid obtained by conversion through the coordinate mapping relationship in step S420 matches the size of the original image. Specifically, the original image grid can cover the original image as much as possible without exceeding the boundary of the original image. Thus, the coordinate mapping relationship performs image scaling processing in addition to basic image mapping processing, which can further improve the image processing efficiency.

[0051] Taking the original image and the result image having the same resolution and the distortion correction of the original image as an example, correspondingly, the coordinate mapping relationship can be the distortion correction conversion relationship.

[0052] The distortion in the ultra-wide-angle camera module is generally composed of two parts, i.e., the radial distortion caused by the lens shape and the tangential distortion caused by the camera assembly deviation. Assuming that the pixel coordinates of any first pixel point in the result image after distortion correction are (u i ′,v i ′), the pixel coordinates of the corresponding second pixel point in the original image before distortion correction are (u i ,v i ), the coordinates of the first pixel point and the coordinates of the corresponding second pixel point can satisfy a predetermined distortion correction conversion relationship {(u i ,v i ),(u i ′,v i ′)}, and the distortion correction conversion relationship {(u i ,v i ),(u i ′,v i ′)} can be determined according to the camera intrinsic parameters and the distortion parameters of the camera module. The distortion correction conversion relationship will determine how to correct the distortion of the original image to determine the corrected result image. The camera intrinsic parameters and the distortion parameters can be determined by calibrating the camera, and the camera intrinsic parameters and the distortion parameters, as inherent parameters of the camera, can also be obtained from the manufacturer or other ways.

[0053] In one embodiment, the pixel coordinates (u i ′,v i ′) of the first pixel point after distortion correction and the pixel coordinates (u i ,v i ) of the corresponding second pixel point before distortion correction can satisfy the following distortion correction conversion relationship:

[0054]

[0055] wherein f x , f y , c x , c y are camera intrinsic parameters, f x = a f, f y = b f, f is the focal length of the imaging camera, a is the number of pixels per unit distance of the focal length f on the x-axis of the pixel coordinate system, b is the number of pixels per unit distance of the focal length f on the y-axis of the pixel coordinate system, c x is the coordinate of the optical center on the x-axis of the pixel coordinate system, c y is the coordinate of the optical center on the y-axis of the pixel coordinate system. After the camera intrinsic parameters are determined through calibration or other means, f x , f y , c x , c y may also be determined accordingly.

[0056] wherein k1, k2, k3, k4, k5, k6 are radial distortion parameters, p1, p2 are tangential distortion parameters. These parameters can also be determined through calibration of the camera or other means.

[0057] Thus, in the case where the camera intrinsic parameters and the distortion parameters are determined, the above distortion correction conversion relationship can be determined as the coordinate mapping relationship.

[0058] The above describes the setting of the distortion correction conversion relationship by way of example, and those skilled in the art can understand that the affine transformation conversion relationship and the spherical projection conversion relationship can also be set in a similar manner. Thus, different coordinate mapping relationships can be designed according to different image processing requirements.

[0059] In this step S420, using the set coordinate mapping relationship, the central processing unit can calculate the pixel coordinates (u i ′, v i ′) of each second grid point corresponding to the pixel coordinates (u i , v i ) of each first grid point determined in step S410. Accordingly, the original image grid composed of these second grid points is also determined.

[0060] In step S430, the central processor inputs the original image as the input texture, the pixel coordinates of each first grid point as the vertex coordinates, and the pixel coordinates of each second grid point as the texture coordinates into the shader of the graphic processing unit (GPU) respectively.

[0061] The shader can realize the rendering of the image. In an embodiment, the shader is configured to map the input texture from the texture coordinates to the vertex coordinates to obtain an output texture. Specifically, after receiving the input texture, the shader creates an output texture corresponding to the input texture for saving the result image to be rendered and outputted, and the output texture has the same resolution as the result image. Then, referring to FIG. 4B, the shader divides the vertex coordinates corresponding to the output texture into a plurality of small triangles, calculates the texture coordinates corresponding to each pixel point of the output texture inside each triangle by using the triangle interpolation feature when the graphic processing unit samples, and combines the bilinear interpolation algorithm of the graphic processing unit to calculate the color value corresponding to each texture coordinate according to the input texture, so as to determine the color value of each pixel point in the output texture, that is, the output texture is rendered. Figure 6

[0062] The data can be transmitted from the central processor to the graphic processing unit in various ways.

[0063] In an embodiment, step S430 can include that the central processor inputs the original image in the central processor as the input texture, the pixel coordinates of each first grid point in the central processor as the vertex coordinates, and the pixel coordinates of each second grid point in the central processor as the texture coordinates into the shader of the graphic processing unit respectively by using the platform shared memory technology. The platform shared memory technology allows fast data transmission between the central processor and the graphic processing unit without copying data. Taking the Android platform as an example, the platform shared memory technology can be realized by AHardwareBuffer cooperating with EGLImageKHR, so that the data in the central processor can be quickly inputted into the graphic processing unit by the platform shared memory technology.

[0064] In another embodiment, step S430 can also include that the central processor inputs the original image as the input texture, the pixel coordinates of each first grid point in the central processor as the vertex coordinates, and the pixel coordinates of each second grid point in the central processor as the texture coordinates into the video memory, and the shader of the graphic processing unit reads the input texture, the vertex coordinates, and the texture coordinates from the video memory respectively.

[0065] ​In other embodiments, the central processor can also transmit part of the data to the graphics processor via the platform shared memory technology, and transmit another part of the data to the graphics processor by copying to the video memory.

[0066] In step S440, the central processor obtains the output texture output by the shader as the result image after image processing from the graphics processor.

[0067] In step S430, the central processor can also obtain the output texture output by the shader as the result image after image processing from the graphics processor in a plurality of different ways.

[0068] In one embodiment, step S440 can include that the central processor obtains the output texture output by the shader as the result image after image processing from the graphics processor via the platform shared memory technology. For example, the output texture after rendering by the graphics processor can be read into an output buffer in the central processor, so that the data in the graphics processor can be quickly exported to the central processor via the platform shared memory technology.

[0069] In another embodiment, step S440 can also include that the graphics processor copies the output texture after rendering to the video memory, and the central processor reads the output texture from the video memory as the result image after image processing.

[0070] In other embodiments, the graphics processor can also transmit part of the data to the central processor via the platform shared memory technology, and transmit another part of the data to the central processor by copying to the video memory.

[0071] After obtaining the corrected result image, the central processor can also output the result image, for example, control the display to display the result image, for different application requirements. For example, in the case of being applied to the preview function, in one embodiment, the original image is an image obtained by a camera module, and after obtaining the output texture as the result image, the central processor also displays the result image on the display as a preview image. In the preview function, the result image can have a smaller resolution than that of the original image. Taking the example of performing distortion correction processing on the original image to obtain the result image for the preview function, the inventors found in actual tests that, on an MTK / Qualcomm mid-end mobile platform, the operation time of performing distortion correction processing on an original image with an input resolution of 2K using the method of the present application can be within 30 ms, meeting the real-time requirement of the preview function.

[0072] In the image processing method, the central processing unit calculates pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing, and calculates pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing, so that a grid point mapping relationship corresponding to the original image and the result image is obtained. The central processing unit only needs to calculate the coordinate mapping relationship between the grid points, without calculating all coordinate mapping relationships of all pixels in the original image, so that the operation load of the central processing unit can be greatly reduced. Meanwhile, the grid point mapping relationship represented by the calculated pixel coordinates of each first grid point and the pixel coordinates of each second grid point, and the original image are imported into a shader of the graphics processing unit for operation, so that the multi-thread parallel processing feature of the shader of the graphics processing unit and the feature of combining the bilinear interpolation sampling operation with the triangular rendering operation are ingeniously utilized, and the drawing of the result image after image processing can be efficiently realized by the shader. Therefore, the operation load of the central processing unit can be reduced, and the image processing efficiency and the real-time performance of the output result image of image processing can be improved.

[0073] It should be understood that although Figure 4 The steps in the flowchart of the image processing method are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 4 At least part of the steps in the image processing method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0074] In one embodiment, as shown in Figure 7 An image processing apparatus 700 is provided, including a first grid point coordinate calculation module 710, a second grid point coordinate calculation module 720, a shader import module 730, and a shader export module 740, wherein:

[0075] The first grid point coordinate calculation module 710 is configured to enable the central processing unit to calculate pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing.

[0076] The second grid point coordinate calculation module 720 is configured to enable the central processing unit to calculate pixel coordinates of each second grid point in an original image grid corresponding to an original image before image processing.

[0077] Shader import module 730 is used to enable the central processing unit to import the original image as input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates into the graphics processor's shaders respectively; and

[0078] Shader export module 740 is used to enable the central processing unit to obtain the output texture of the shader from the graphics processor as the result image after image processing.

[0079] In one embodiment, the image processing apparatus 700 further includes a shader rendering module 750, which is used to enable the shader to map the input texture from texture coordinates to vertex coordinates to obtain the output texture.

[0080] In one embodiment, the image processing apparatus 700 further includes a display module 760, which is configured to, after the central processing unit acquires the output texture as a result image, also display the result image on a display screen as a preview image.

[0081] Specific limitations regarding the image processing apparatus 700 can be found in the limitations of the image processing method described above, and will not be repeated here. Each module in the image processing apparatus 700 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the electronic device, or stored in software in the memory of the electronic device, so that the processor can call and execute the operations corresponding to each module.

[0082] In one embodiment, such as Figure 8 As shown, an electronic device 300 is provided, including a memory 303, a central processing unit 304, and a graphics processing unit 305. The central processing unit 304 and the graphics processing unit 305 can be communicatively connected, for example, through platform shared memory technology and / or through video memory. The memory 303 stores a computer program, and when the central processing unit 304 executes the computer program, it performs the following steps:

[0083] The central processing unit 304 calculates the pixel coordinates of each first grid point in the result image grid corresponding to the result image after image processing;

[0084] The central processing unit 304 calculates the pixel coordinates of each second grid point in the original image grid corresponding to the original image before image processing;

[0085] The central processing unit 304 takes the original image as the input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates, and imports them into the shader of the graphics processing unit 305 respectively; and

[0086] The central processing unit 304 obtains the output texture of the shader from the graphics processor 305 as the result image after image processing.

[0087] In one embodiment, such as Figure 8 As shown, the electronic device 300 may further include a camera 301 coupled to the electronic device 300. The term "coupled" is intended to include both a form where the camera 301 is directly housed within the electronic device 300, and a form where the camera 301 is located outside the electronic device 300 (e.g., the camera 804 is located on another terminal device), and a communicative connection is established between the camera 301 and the electronic device 300. The camera 301 is used to capture raw images and transmit the raw images to the central processing unit 304; the central processing unit 304 receives the raw images from the camera 301.

[0088] In one embodiment, such as Figure 8 As shown, the electronic device 300 may also include a display 302; when the central processing unit 304 executes a computer program, it also controls the display 302 to display the result image.

[0089] In other embodiments, the central processing unit 304, when executing a computer program, also implements the steps of the method described in any of the above embodiments.

[0090] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a central processing unit, performs the following steps:

[0091] The central processing unit calculates the pixel coordinates of each first grid point in the result image grid corresponding to the result image after image processing;

[0092] The central processing unit calculates the pixel coordinates of each second grid point in the original image grid corresponding to the original image before image processing;

[0093] The central processing unit (CPU) takes the original image as the input texture, the pixel coordinates of each first grid point as vertex coordinates, and the pixel coordinates of each second grid point as texture coordinates, and imports them into the graphics processor's shaders respectively; and

[0094] The central processing unit obtains the output texture from the shader from the graphics processing unit, which is used as the resulting image after image processing.

[0095] In other embodiments, when the computer program is executed by the central processing unit, it also implements the steps of the method described in any of the above embodiments.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0097] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0098] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. An image processing method, the method comprising: calculating, by a central processing unit, pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing, wherein the result image is a preview image; obtaining, by the central processing unit, a coordinate mapping relationship between the pixel coordinates in the result image and pixel coordinates in an original image, wherein the coordinate mapping relationship is determined according to a corresponding conversion relationship of a basic image mapping process and resolutions of the result image and the original image, and the result image has a resolution smaller than that of the original image; converting, by the central processing unit, the pixel coordinates of the each first grid point to obtain pixel coordinates of each second grid point before image processing according to the coordinate mapping relationship; inputting, by the central processing unit, the original image as input texture, the pixel coordinates of the each first grid point as vertex coordinates, and the pixel coordinates of the each second grid point as texture coordinates into a shader of a graphics processing unit respectively; obtaining, by the central processing unit, output texture output by the shader from the graphics processing unit as the result image after image processing; and displaying, by the central processing unit, the result image on a display screen as the preview image.

2. The method of claim 1, wherein, The calculating, by the central processing unit, pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing comprises: establishing, by the central processing unit, a corresponding result image grid according to a size of an expected result image, wherein the result image grid comprises a plurality of first grid points distributed in a matrix on the result image; determining, by the central processing unit, pixel coordinates of each first grid point in the result image.

3. The method of claim 2, wherein, The result image has a width of , and a height of , a plurality of first grid points in the matrix distribution includes a plurality of first grid points in a square matrix distribution, and a side length of a square grid formed by every four adjacent first grid points in the square matrix is ; The central processing unit determines a first grid point in the plurality of first grid points with a width direction index of and a height direction index of in the result image with a pixel coordinate of wherein , , and wherein: 。 4. The method of claim 1, wherein, The corresponding conversion relationship of the basic image mapping process comprises one or more combinations of a distortion correction conversion relationship, an affine transformation conversion relationship, and a spherical projection conversion relationship.

5. The method according to any one of claims 1 to 4, characterized in that, The inputting, by the central processing unit, the original image as input texture, the pixel coordinates of the each first grid point as vertex coordinates, and the pixel coordinates of the each second grid point as texture coordinates into a shader of a graphics processing unit respectively comprises: inputting, by the central processing unit, the original image in the central processing unit as input texture, the pixel coordinates of the each first grid point in the central processing unit as vertex coordinates, and the pixel coordinates of the each second grid point in the central processing unit as texture coordinates into a shader of a graphics processing unit respectively by using a platform shared memory technology.

6. The method according to any one of claims 1 to 4, characterized in that, The shader maps input texture from texture coordinates to vertex coordinates to obtain output texture.

7. The method according to any one of claims 1 to 4, characterized in that, The obtaining, by the central processing unit, output texture output by the shader from the graphics processing unit as the result image after image processing comprises: obtaining, by the central processing unit, output texture output by the shader from the graphics processing unit as the result image after image processing by using a platform shared memory technology.

8. The method according to any one of claims 1 to 4, characterized in that, The original image is an image captured by a camera module.

9. An image processing apparatus characterized by comprising: The apparatus comprises: The first grid point coordinate calculation module is configured to cause the central processing unit to calculate pixel coordinates of each first grid point in a result image grid corresponding to a result image after image processing; wherein the result image is a preview image. The second grid point coordinate calculation module is configured to cause the central processing unit to obtain a coordinate mapping relationship between pixel coordinates in the result image and pixel coordinates in an original image; wherein the coordinate mapping relationship is determined according to a corresponding conversion relationship of basic image mapping processing and resolutions of the result image and the original image, and the result image has a resolution smaller than that of the original image; and the second grid point coordinate calculation module is further configured to cause the central processing unit to convert the pixel coordinates of the each first grid point to obtain pixel coordinates of each second grid point before image processing according to the coordinate mapping relationship. The shader import module is configured to cause the central processing unit to import, into a shader of a graphics processing unit, the original image as input texture, the pixel coordinates of the each first grid point as vertex coordinates, and the pixel coordinates of the each second grid point as texture coordinates. The shader export module is configured to cause the central processing unit to obtain, from the graphics processing unit, output texture output by the shader as the result image after image processing. The display module is configured to cause the central processing unit to display the result image on a display screen as the preview image.

10. An electronic device comprising a memory, a central processing unit and a graphics processing unit, said memory storing a computer program, characterized in that, The central processing unit, when executing the computer program, implements the steps of the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the central processing unit, implements the steps of the method of any one of claims 1 to 8.

Images