Image processing method and device, equipment and medium

Abstract

The embodiment of the invention relates to an image processing method and device, equipment and a medium, and the method comprises the steps: obtaining a to-be-processed first image, and obtaining a first grid; determining a first mapping relation between the first image and the first grid, and determining vertex pixel information of the first grid based on the first mapping relation and the first image; obtaining a target distortion parameter, and adjusting the position of the vertex of the first grid in the three-dimensional space based on the vertex pixel information of the first grid and the target distortion parameter to obtain a second grid; and generating a target image based on the second grid. According to the embodiment of the invention, the first image can be converted into the target image with the three-dimensional distortion effect, and a three-dimensional distortion special effect is provided for a user, so that the image special effect requirement of the user is met.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, equipment and medium. Background Art

[0002] Nowadays, more and more ordinary users, professional editors and other people need to use multimedia editing software to edit multimedia materials such as images obtained by shooting. Most of the existing multimedia editing software can provide special effects processing functions, such as adding special effects to images uploaded by users, thereby enhancing the appeal and fun of the video. However, the inventor has found through research that the types of existing image special effects are limited and it is difficult to meet the diverse needs of users. Therefore, new image special effects are urgently needed. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image processing method, apparatus, device and medium.

[0004] An embodiment of the present disclosure provides an image processing method, which includes: obtaining a first image to be processed and obtaining a first mesh; determining a first mapping relationship between the first image and the first mesh, and determining vertex pixel information of the first mesh based on the first mapping relationship and the first image; obtaining a target distortion parameter, and adjusting the position of the vertices of the first mesh in a three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh; and generating a target image based on the second mesh.

[0005] Optionally, obtaining the first grid includes: using multiple processes to respectively construct corresponding sub-grids; wherein the sub-grid is a plane grid of M rows and N columns, and M and N are both preset positive integers; and merging multiple sub-grids to obtain the first grid.

[0006] Optionally, the first grid is a plane grid, and determining a first mapping relationship between the first image and the first grid includes: determining the first mapping relationship based on ratio information between a size of the first image and a size of the first grid.

[0007] Optionally, determining the vertex pixel information of the first mesh based on the first mapping relationship and the first image includes: determining a sampling position corresponding to the vertex of the first mesh, and determining a pixel point in the first image corresponding to the sampling position based on the first mapping relationship, and using the pixel information of the pixel point as the vertex pixel information of the first mesh; the pixel information of the pixel point includes brightness information and / or color information of the pixel point.

[0008] Optionally, the horizontal coordinate of the sampling position corresponding to the vertex is: the horizontal coordinate of the vertex or the horizontal coordinate of the specified position of the fragment to which the vertex belongs in the first grid; the vertical coordinate of the sampling position corresponding to the vertex is: the vertical coordinate of the vertex or the vertical coordinate of the specified position of the fragment to which the vertex belongs in the first grid.

[0009] Optionally, the target distortion parameter is a preset adjustable parameter, and the target distortion parameter includes a distortion axis and a maximum distortion angle; wherein the maximum distortion angle is the maximum angle at which vertices in the first mesh rotate around the distortion axis; adjusting the positions of the vertices of the first mesh in three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh includes: determining a target angle at which the vertices of the first mesh rotate around the distortion axis based on the vertex pixel information of the first mesh and the maximum distortion angle; and adjusting the positions of the vertices of the first mesh in three-dimensional space based on the distortion axis and the target angle to obtain a second mesh.

[0010] Optionally, the vertex pixel information includes vertex brightness values ​​or vertex color values; determining the target angle of rotation of the vertices of the first mesh around the warp axis based on the vertex pixel information of the first mesh and the maximum warp angle includes: based on the maximum warp angle, obtaining a second mapping relationship between the vertex pixel information of the first mesh and the angle of rotation of the vertices of the first mesh around the warp axis; the second mapping relationship is represented by a positive correlation function or a negative correlation function; based on the vertex pixel information of the first mesh and the second mapping relationship, determining the target angle of rotation of the vertices of the first mesh around the warp axis.

[0011] Optionally, adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the distortion axis and the target angle to obtain the second mesh includes: adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the distortion axis and the target angle to obtain vertex position information of the second mesh; determining the vertex pixel information of the second mesh based on the vertex position information of the second mesh and the vertex pixel information of the first mesh; determining the fragment pixel information of the first mesh based on the first mapping relationship and / or the vertex pixel information of the first mesh; performing a depth adjustment operation and a fragment edge feathering operation on the fragment pixel information of the first mesh based on the vertex pixel information of the second mesh to obtain the fragment pixel information of the second mesh; obtaining the second mesh based on the vertex pixel information and the fragment pixel information of the second mesh.

[0012] Optionally, obtaining the second mesh based on the vertex pixel information and fragment pixel information of the second mesh includes: obtaining preset stylization parameters; wherein the stylization parameters include a first parameter and / or a second parameter, the first parameter indicates transparency processing for target edge lines of the second mesh, and the second parameter is used to indicate transparency processing for target fragments of the second mesh; adjusting the transparency in the vertex pixel information and fragment pixel information of the second mesh based on the stylization parameters; obtaining the second mesh based on the vertex pixel information and fragment pixel information of the second mesh after the transparency adjustment.

[0013] Optionally, generating a target image based on the second grid includes: acquiring parameter information of a projection camera; wherein the parameter information includes an observation angle and a focal length, the observation angle is used to indicate a viewing angle of the projection camera relative to the second grid, and the focal length is used to indicate a distance between the projection camera and the second grid; and performing screen display processing on the second grid based on the parameter information of the projection camera to obtain a target image.

[0014] The disclosed embodiment also provides an image processing device, including: a first mesh acquisition module, used to acquire a first image to be processed and to acquire a first mesh; a vertex pixel determination module, used to determine a first mapping relationship between the first image and the first mesh, and to determine vertex pixel information of the first mesh based on the first mapping relationship and the first image; a second mesh acquisition module, used to acquire a target distortion parameter, and to adjust the position of the vertices of the first mesh in three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh; and a target image generation module, used to generate a target image based on the second mesh.

[0015] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the image processing method provided by the embodiment of the present disclosure.

[0016] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image processing method provided by the embodiment of the present disclosure.

[0017] The above technical solution provided by the embodiment of the present disclosure can first determine a first mapping relationship between the first image and the first mesh, thereby determining vertex pixel information of the first mesh based on the first mapping relationship and the first image, and then adjust the positions of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh distorted in the three-dimensional space, and obtain a target image with a three-dimensional distortion effect based on the second mesh, thereby providing users with three-dimensional distortion special effects and meeting users' image special effects needs.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of constructing a first grid provided in an embodiment of the present disclosure;

[0023] Figure 3 A second grid schematic diagram provided for an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of an image processing flow provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of an image processing effect provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] Figure 1 The following is a flow chart of an image processing method provided by an embodiment of the present disclosure. The method can be executed by an image processing device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0031] Step S102, obtaining a first image to be processed and obtaining a first grid.

[0032] The disclosed embodiment does not limit the content of the first image, such as any first image that needs to be processed with three-dimensional (3D) distortion special effects. For example, the first image can be a plane image, that is, a two-dimensional (2D) image. In the disclosed embodiment, a first grid can be obtained, and the first grid can be a pre-constructed plane grid. In some embodiments, the grid size, number of grids, etc. of the first grid can be flexibly set according to demand. In other words, the parameters of the first grid can be directly preset without establishing the first grid based on the first image.

[0033] Step S104: determine a first mapping relationship between the first image and the first mesh, and determine vertex pixel information of the first mesh based on the first mapping relationship and the first image.

[0034] In order to present the pixel information of the first image through the first grid, in other words, to map the first image onto the first grid, a first mapping relationship between the first image and the first grid can be determined first, and an association can be established between the first image and the first grid. Then, based on the first mapping relationship, pixel information corresponding to the vertices of the first grid can be determined from the first image, thereby obtaining vertex pixel information of the first grid.

[0035] Step S106 , obtaining a target distortion parameter, and adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter, so as to obtain a second mesh.

[0036] Since the first grid is a planar grid mapped with pixel information of the first image, in order to achieve a three-dimensional distortion effect, the embodiment of the present disclosure can adjust the positions of the vertices of the first grid in the three-dimensional space based on the target distortion parameters, thereby obtaining a second grid in the three-dimensional space. In other words, the second grid is a three-dimensional solid grid.

[0037] Step S108: generating a target image based on the second grid.

[0038] The second grid is a three-dimensional grid obtained by adjusting the vertex positions of the first grid, and the pixel information of the first image is also mapped on it, that is, the three-dimensional effect of the first image can be obtained efficiently and conveniently through the above method. The second grid is projected by a projection camera or the like, and a target image with a three-dimensional distortion effect can be obtained. Specifically, the first image contains a target object, and the target image contains a target object with a three-dimensional distortion effect. And the target distortion parameters are different, the three-dimensional distortion effect of the second grid presenting the content of the first image is different, and the projection method is different, such as the observation angle and focal length of the projection camera. The distortion perception of the target object presented in the projected target image is also different, so that users can be provided with rich, interesting and visually appealing three-dimensional distortion special effects to meet the user's image special effects needs.

[0039] In some implementations, the step of obtaining the first grid may be performed with reference to the following steps 1 and 2:

[0040] Step 1, use multiple processes to respectively construct corresponding sub-grids; wherein the sub-grid is a plane grid of M rows and N columns, and M and N are both preset positive integers. In practical applications, multiple processes can be used to simultaneously construct corresponding sub-grids, thereby ensuring the efficiency of grid construction. The sub-grid of M rows and N columns constructed by each process can be called an instance. For ease of implementation, in some implementation examples, M is a value greater than 1, and N is a value equal to 1; in other implementation examples, M is a value equal to 1, and N is a value greater than 1. In other implementation examples, M and N are both values ​​greater than 1, and M and N can be the same or different, and can be flexibly set.

[0041] Step 2: merge multiple subgrids to obtain the first grid. The first grid is a plane grid. The grid can be merged based on rows or columns, which can be set flexibly. For easy understanding, please refer to Figure 2 A schematic diagram of constructing a first grid is shown in FIG. 1 , and multiple separate instances can be created by using a multi-instance technology, each instance corresponding to a sub-grid. Figure 2 In the example, a subgrid is a column of grids. By merging multiple subgrids constructed at the same time, the first grid can be obtained efficiently and reliably. Figure 2As shown, the intersection of the lines in the first mesh is the vertex of the first mesh. The squares in the first mesh can be regarded as the fragments of the first mesh, which can also be called facets. In practical applications, the first mesh can be constructed in a triangle manner, where a square is composed of two triangles, or each triangle can be directly regarded as a fragment of the first mesh. The above method can efficiently construct the first mesh with strong mesh construction performance.

[0042] On the basis that the first grid is a plane grid, the above-mentioned determination of the first mapping relationship between the first image and the first grid includes: determining the first mapping relationship based on the ratio information between the size of the first image and the size of the first grid. Specifically, the above-mentioned sizes can be represented by length and width, such as the ratio information includes: the ratio between the length of the first image and the length of the first grid, the ratio between the width of the first image and the width of the first grid, and then the first mapping relationship between the first image and the first grid can be determined based on the above ratios, so that the pixel information of the first image can be accurately mapped to the first grid later, so that the first grid presents the pixel information of the first image.

[0043] On the basis of acquiring the first mapping relationship, the disclosed embodiment provides an implementation method of determining the vertex pixel information of the first mesh based on the first mapping relationship and the first image, including: determining the sampling position corresponding to the vertex of the first mesh, and determining the pixel point corresponding to the sampling position in the first image based on the first mapping relationship, and using the pixel information of the pixel point as the vertex pixel information of the first mesh; the pixel information of the pixel point includes the brightness information and / or color information of the pixel point. The sampling position is also the pixel position in the first image required for the vertex, and the pixel information obtained by sampling the position is the vertex pixel information of the first mesh. In the above manner, the pixel information in the first image corresponding to each vertex in the first mesh can be determined quickly and conveniently, and used as the vertex pixel information of the first mesh, thereby presenting the pixel information of the first image through the first mesh.

[0044] Exemplarily, the horizontal coordinate of the sampling position corresponding to the vertex is: the horizontal coordinate of the vertex or the horizontal coordinate of the specified position of the fragment to which the vertex belongs in the first grid; the vertical coordinate of the sampling position corresponding to the vertex is: the vertical coordinate of the vertex or the vertical coordinate of the specified position of the fragment to which the vertex belongs in the first grid. Taking the square in the first grid as a fragment as an example, the above-mentioned specified position can be the center position of the square. Exemplarily, taking the example that the pixel information includes brightness information, for each vertex in the first grid, the X position of the center of the square to which it belongs and the Y position of the vertex are used as the sampling position corresponding to the vertex, so as to sample the corresponding brightness value from the first image as the pixel information of the vertex.

[0045] In some embodiments, the target distortion parameter is a preset adjustable parameter, and the target distortion parameter includes a distortion axis and a maximum distortion angle; wherein the maximum distortion angle is the maximum angle of rotation of the vertices in the first mesh around the distortion axis; the distortion axis is a preset axis, and the mesh vertices need to rotate around the axis. On this basis, the above step of adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain the second mesh can be performed as shown in the following steps A and B:

[0046] Step A, based on the vertex pixel information of the first mesh and the maximum distortion angle, determine the target angle of rotation of the vertices of the first mesh around the distortion axis. That is, the target angle has a certain correlation with the vertex pixel information, and the target angle does not exceed the maximum distortion angle. Through the above method, the position of the vertices of the first mesh can be reasonably adjusted based on the vertex pixel information, which is more in line with the visual perception. In some specific implementation examples, the vertex pixel information includes the vertex brightness value or the vertex color value; step A can be performed with reference to the following steps A1 to A2:

[0047] Step A1, based on the maximum distortion angle, obtain the second mapping relationship between the vertex pixel information of the first mesh and the angle of rotation of the vertex of the first mesh around the distortion axis; the second mapping relationship is represented by a positive correlation function or a negative correlation function. Taking the vertex pixel information as a brightness value as an example, the angle of rotation of the vertex around the distortion axis can be determined based on the vertex brightness value. For example, if the second mapping relationship is represented by a positive correlation function, the larger the vertex brightness value, the larger the rotation angle of the vertex; if the second mapping relationship is represented by a positive correlation function, the smaller the vertex brightness value, the larger the rotation angle of the vertex. The above positive correlation function or negative correlation function can be flexibly set according to demand, and the maximum distortion angle can be used as an endpoint parameter of the positive correlation function or the negative correlation function to constrain the positive correlation function or the negative correlation function to represent the second mapping relationship between 0 degrees and the maximum distortion angle.

[0048] Step A2, based on the vertex pixel information of the first mesh and the second mapping relationship, determine the target angle of rotation of the vertices of the first mesh around the distortion axis. The target angle is positively or negatively correlated with the vertex pixel information. For example, if the vertex pixel information is vertex brightness value, three-dimensional rotation can be performed around the distortion axis based on the brightness value in this way, thereby presenting a distortion effect with a three-dimensional sense.

[0049] Step B, adjusts the positions of the vertices of the first mesh in the three-dimensional space based on the distortion axis and the target angle to obtain the second mesh. The vertices of the first mesh are rotated from the current position around the distortion axis by the target angle to obtain the rotated positions of the vertices of the first mesh. The mesh after the vertex positions are adjusted is the second mesh. In other words, the second mesh is the mesh obtained after the vertex positions of the first mesh are adjusted. Since the above positions are adjusted in the three-dimensional space, the adjusted vertex positions are not all located in the same plane. Through the above method provided by the embodiment of the present disclosure, the vertex positions of the initially preset plane mesh can be distorted based on the pixel information of the first image, thereby obtaining a three-dimensional mesh (second mesh) presenting the pixel information of the first image.

[0050] In some specific implementation examples, step B may be performed with reference to the following steps B1 to B5:

[0051] Step B1, adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the twist axis and the target angle to obtain the vertex position information of the second mesh.

[0052] Step B2: determining vertex pixel information of the second mesh based on the vertex position information of the second mesh and the vertex pixel information of the first mesh.

[0053] Since the first mesh is a planar mesh, the vertex pixel information of the first mesh usually does not include depth information, and the vertices of the second mesh are the vertices of the first mesh adjusted in three-dimensional space. Therefore, the vertex pixel information of the second mesh additionally adds depth information on the basis of the original vertex pixel information of the first mesh, and the depth information can be determined based on the vertex position information.

[0054] Step B3, based on the first mapping relationship and / or the vertex pixel information of the first grid, determine the fragment pixel information of the first grid. In some embodiments, the pixel information corresponding to the fragment of the first grid can be directly searched from the first image based on the first mapping relationship, and it can be used as the fragment pixel information of the first grid. In other embodiments, interpolation processing can be performed based on the vertex pixel information of the first grid to obtain the fragment pixel information of the first grid, such as for a square of the first grid, the pixel information of the square is obtained by interpolation processing through the pixel information of the four vertices of the square. In addition, the fragment pixel information of the first grid can be determined by combining the first mapping relationship and the vertex pixel information of the first grid at the same time, such as the central area of ​​the square determines the corresponding pixel information through the first mapping relationship, and the edge position of the square is obtained by interpolation of the vertex pixel information. The specific setting can be flexibly set and is not limited here.

[0055] Step B4: Based on the vertex pixel information of the second mesh, perform a depth adjustment operation and a fragment edge feathering operation on the fragment pixel information of the first mesh to obtain the fragment pixel information of the second mesh.

[0056] It is understandable that after adjusting the vertex position of the first mesh, the three-dimensional second mesh may have problems such as occlusion and jagged edges. To improve this problem, the depth adjustment operation and the fragment edge feathering operation can be performed on the fragment pixel information of the first mesh based on the vertex pixel information of the second mesh. Specifically, the depth in the fragment pixel information of the first mesh can be adjusted based on the vertex depth indicated by the vertex pixel information of the second mesh, and the transparency of the fragment edge can be adjusted based on the preset feathering parameters, and finally the fragment pixel information of the second mesh is obtained. The fragment pixel information of the second mesh may include depth information, brightness information, RGBA color information, etc. The above method can make the second mesh have more reasonable and realistic fragment pixel information. In practical applications, a vertex shader can be used to process mesh vertices, such as adjusting mesh vertex positions, determining vertex pixel information, etc.; a fragment shader can be used to process mesh fragments, such as performing depth adjustment operations and fragment edge feathering operations, determining fragment pixel information, etc.

[0057] Step B5, based on the vertex pixel information and fragment pixel information of the second mesh, obtain the second mesh. Based on the first mesh, combined with the vertex pixel information and fragment pixel information of the second mesh, it can be directly converted into the second mesh. For ease of understanding, please refer to Figure 3 A second grid schematic diagram is shown, which illustrates an example of grid distortion that does not display the original image information. Compared with the first grid, the second grid is more three-dimensional, and the distortion effect is related to the pixel information in the first image, thereby better presenting the features in the first image, which helps to further achieve the effect of three-dimensional distortion of the target object in the first image.

[0058] In some implementation examples, step B5 may be performed with reference to the following steps B5.1 to B5.3:

[0059] Step B5.1, obtaining preset stylization parameters; wherein the stylization parameters include a first parameter and / or a second parameter, the first parameter indicates that transparency processing is performed on the target edge lines of the second grid, and the second parameter is used to indicate that transparency processing is performed on the target fragments of the second grid. The target edge lines and target fragments can be determined based on a preset screen grid (also referred to as a preset stylized grid), the number of target edge lines and target fragments can be multiple, the size and number of grids in the screen grid can determine the size and number of target fragments, and the edge lines in the screen grid can determine the target edge lines. In practical applications, the stylization parameters can be flexibly set, such as setting the grid size and number of grids of the above-mentioned screen grid, so as to determine the target edge lines and target fragments of the second grid, and then based on whether the stylization parameters contain the first parameter or the second parameter, it is determined whether to perform transparency processing on the target edge lines of the second grid or on the target fragments of the second grid.

[0060] Step B5.2, adjusting the transparency of the vertex pixel information and the fragment pixel information of the second mesh based on the stylized parameters. By performing transparency processing on the target edge lines and / or target fragments, the second mesh can present a mesh scale-like appearance that only displays mesh fragments, or a mesh line-like appearance that only displays mesh edge lines.

[0061] Step B5.3, based on the vertex pixel information and fragment pixel information of the second grid after transparency adjustment, obtain the second grid. The second grid can present a stylized effect in which at least some edge lines are transparent or at least some fragments are transparent. In actual applications, a function option of whether to use the grid style can be provided to the user. If the user triggers this option, the above steps B5.1 to B5.3 are executed, otherwise the above steps do not need to be executed. The stylized parameters can be set by the user, such as the first parameter and / or the second parameter can be set directly by the user, or the parameters of the screen grid can be set by the user, and the first parameter and / or the second parameter are generated based on the parameters of the screen grid and the stylized effect selected by the user (only displaying edge lines or only displaying grid fragments).

[0062] On the basis of obtaining the second grid, the embodiment of the present disclosure provides an implementation example of generating a target image based on the second grid, which can be performed with reference to the following steps a and b:

[0063] Step a, obtaining parameter information of the projection camera; wherein the parameter information includes an observation angle and a focal length, wherein the observation angle is used to indicate the viewing angle of the projection camera relative to the second grid, and the focal length is used to indicate the distance between the projection camera and the second grid. The projection camera refers to a virtual camera used to capture a three-dimensional scene and render it onto a two-dimensional screen, and can be used to define specific parameters of the projection transformation such as the observation angle and the focal length.

[0064] Step b: performing screen display processing on the second grid based on parameter information of the projection camera to obtain a target image.

[0065] Based on the parameter information of the projection camera and the MVP transformation processing, the second mesh can be rendered and displayed on a two-dimensional screen to obtain a target image. Specifically, the MVP transformation represents a combination of three steps: Model, View, and Projection. These three transformations together determine how a three-dimensional model (second mesh) is rendered on a two-dimensional screen. Specifically, the model transformation (Model) is responsible for converting the second mesh from its local coordinate system to the world coordinate system. It usually includes rotation, scaling, and translation operations to adjust the position, direction, and size of the second mesh. The view transformation (View), also known as the camera transformation or observation transformation, can simulate the position and orientation of the camera and transform the coordinate system of the entire world relative to the camera, so as to achieve the observation of the second mesh from the camera's perspective. The projection transformation (Projection) projects points in the three-dimensional space onto a two-dimensional plane through orthogonal projection or perspective projection. In practical applications, a projection matrix can be constructed based on the parameter information of the projection camera, and then multiplied with the model matrix and the view matrix to generate the final MVP matrix. The MVP matrix can be used to transform each vertex of the second mesh, thereby completing the mapping from the three-dimensional space to the two-dimensional screen and generating the target image.

[0066] In practical applications, the parameter information of the projection camera can be flexibly set, such as changing the observation angle and focal length of the projection camera to change the viewing angle, thereby achieving a multi-angle presentation effect of the target object in the first image.

[0067] For ease of understanding, see also Figure 4 The image processing flow diagram shown in the figure shows that a first mapping relationship can be determined based on a first image (plane image) and a constructed first mesh (plane mesh), and then the mesh pixel information of the first mesh can be determined based on the first mapping relationship and the first image. Specifically, the vertex pixel information can be determined by a vertex shader, and the fragment pixel information can be determined by a fragment shader. Combined with the target distortion parameters (such as distortion axis, maximum distortion angle and other parameters) and the mesh pixel information, processing such as vertex position adjustment, fragment depth adjustment, feathering operation, etc. is performed, and then combined with the stylized parameters, a second mesh can be obtained, which is a distorted three-dimensional mesh. Then, the second mesh in the three-dimensional space can be mapped and displayed on a two-dimensional screen through a projection camera, thereby displaying a target image with a three-dimensional distortion effect.

[0068] See also Figure 5The schematic diagram of an image processing effect shown in FIG. 1 illustrates that a first image is processed by a three-dimensional distortion special effect (implemented by the image processing method provided by the embodiment of the present disclosure). Different parameters such as target distortion parameters, projection camera parameters, and stylization parameters can form different three-dimensional distortion effects. Figure 5 The figure simply illustrates four effects, namely, target image 1 to target image 4. Target image 1 and target image 2 do not use stylized parameters. They are mainly based on the different target distortion parameters and projection camera parameters, so as to present the user with a distorted three-dimensional flower from different perspectives. Target image 3 and target image 4 use stylized parameters. Target image 3 mainly presents grid edge lines, and the grid elements are processed with transparency and no longer present the corresponding pixel colors, so as to present the overall grid line appearance. Target image 4 mainly presents grid elements, and the grid edge lines are processed with transparency and no longer present the corresponding pixel colors, so as to present the overall grid scale-like appearance. Through the above-mentioned three-dimensional distortion special effects, a target image with strong interest, appeal and visual impact can be obtained, thereby meeting the user's needs for three-dimensional distortion special effects.

[0069] It is understandable that the related art basically only performs plane distortion on a plane image, while the above method provided by the embodiment of the present disclosure can first project the pixel information of the plane image onto a plane grid, and then adjust the grid vertices in combination with the distortion parameters to obtain a three-dimensional grid, and then project a target image with a three-dimensional distortion effect. The above method is easy to implement, and by adjusting the distortion parameters, stylization parameters and other parameters, the objects in the plane image can be distorted in different forms in the three-dimensional space to obtain a variety of target images with three-dimensional distortion effects, and the viewing angle can be changed in the three-dimensional space by adjusting the projection camera parameters, etc., and the objects in the plane image can be presented in three dimensions with multiple viewing angles, thereby providing users with three-dimensional distortion effects with rich visual effects.

[0070] Corresponding to the aforementioned image processing method, the embodiment of the present disclosure further provides an image processing device, Figure 6 FIG. 1 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 6 As shown, the image processing device includes:

[0071] A first grid acquisition module 602 is used to acquire a first image to be processed and acquire a first grid;

[0072] A vertex pixel determination module 604 is used to determine a first mapping relationship between the first image and the first mesh, and determine vertex pixel information of the first mesh based on the first mapping relationship and the first image;

[0073] A second mesh obtaining module 606 is used to obtain a target distortion parameter and adjust the position of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh;

[0074] The target image generating module 608 is configured to generate a target image based on the second grid.

[0075] The above-mentioned device provided by the embodiment of the present disclosure can first determine a first mapping relationship between the first image and the first mesh, thereby determining vertex pixel information of the first mesh based on the first mapping relationship and the first image, and then adjust the positions of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh distorted in the three-dimensional space, and obtain a target image with a three-dimensional distortion effect based on the second mesh, thereby providing users with three-dimensional distortion special effects and meeting users' image special effects needs.

[0076] In some embodiments, the first grid acquisition module 602 is specifically used to: use multiple processes to respectively construct corresponding sub-grids; wherein the sub-grid is a plane grid with M rows and N columns, and M and N are both preset positive integers; merge the multiple sub-grids to obtain the first grid.

[0077] In some implementations, the first grid is a plane grid, and the vertex pixel determination module 604 is specifically used to determine a first mapping relationship based on ratio information between a size of the first image and a size of the first grid.

[0078] In some embodiments, the vertex pixel determination module 604 is specifically used to: determine the sampling position corresponding to the vertex of the first mesh, and determine the pixel point corresponding to the sampling position in the first image based on the first mapping relationship, and use the pixel information of the pixel point as the vertex pixel information of the first mesh; the pixel information of the pixel point includes brightness information and / or color information of the pixel point.

[0079] In some embodiments, the horizontal coordinate of the sampling position corresponding to the vertex is: the horizontal coordinate of the vertex or the horizontal coordinate of the specified position of the fragment to which the vertex belongs in the first grid; the vertical coordinate of the sampling position corresponding to the vertex is: the vertical coordinate of the vertex or the vertical coordinate of the specified position of the fragment to which the vertex belongs in the first grid.

[0080] In some embodiments, the target distortion parameter is a preset adjustable parameter, and the target distortion parameter includes a distortion axis and a maximum distortion angle; wherein the maximum distortion angle is the maximum angle at which the vertices in the first mesh rotate around the distortion axis; the second mesh obtaining module 606 is specifically used to: determine the target angle at which the vertices of the first mesh rotate around the distortion axis based on the vertex pixel information of the first mesh and the maximum distortion angle; and adjust the positions of the vertices of the first mesh in three-dimensional space based on the distortion axis and the target angle to obtain the second mesh.

[0081] In some embodiments, the vertex pixel information includes vertex brightness values ​​or vertex color values; the second mesh obtaining module 606 is specifically used to: based on the maximum distortion angle, obtain a second mapping relationship between the vertex pixel information of the first mesh and the angle at which the vertices of the first mesh rotate around the distortion axis; the second mapping relationship is represented by a positive correlation function or a negative correlation function; based on the vertex pixel information of the first mesh and the second mapping relationship, determine the target angle at which the vertices of the first mesh rotate around the distortion axis.

[0082] In some embodiments, the second mesh obtaining module 606 is specifically used to: adjust the positions of the vertices of the first mesh in the three-dimensional space based on the distortion axis and the target angle to obtain the vertex position information of the second mesh; determine the vertex pixel information of the second mesh based on the vertex position information of the second mesh and the vertex pixel information of the first mesh; determine the fragment pixel information of the first mesh based on the first mapping relationship and / or the vertex pixel information of the first mesh; perform a depth adjustment operation and a fragment edge feathering operation on the fragment pixel information of the first mesh based on the vertex pixel information of the second mesh to obtain the fragment pixel information of the second mesh; obtain the second mesh based on the vertex pixel information and the fragment pixel information of the second mesh.

[0083] In some embodiments, the second mesh obtaining module 606 is specifically used to: obtain preset stylization parameters; wherein the stylization parameters include a first parameter and / or a second parameter, the first parameter indicates transparency processing for target edge lines of the second mesh, and the second parameter is used to indicate transparency processing for target fragments of the second mesh; adjust the transparency in vertex pixel information and fragment pixel information of the second mesh based on the stylization parameters; obtain the second mesh based on the vertex pixel information and fragment pixel information of the second mesh after transparency adjustment.

[0084] In some embodiments, the target image generation module 608 is specifically used to: obtain parameter information of the projection camera; wherein the parameter information includes an observation angle and a focal length, the observation angle is used to indicate the viewing angle of the projection camera relative to the second grid, and the focal length is used to indicate the distance between the projection camera and the second grid; based on the parameter information of the projection camera, the second grid is subjected to screen display processing to obtain a target image.

[0085] The image processing device provided in the embodiments of the present disclosure can execute the image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0087] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0088] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0090] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0092] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0093] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the image processing method provided by the embodiment of the present disclosure.

[0094] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0095] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the image processing method in the embodiment of the present disclosure when the computer program / instruction is executed by a processor.

[0096] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0097] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0098] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0099] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0100] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0101] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: include: Acquire a first image to be processed, and acquire a first grid; Determine a first mapping relationship between the first image and the first mesh, and determine vertex pixel information of the first mesh based on the first mapping relationship and the first image; Acquire a target distortion parameter, and adjust positions of vertices of the first mesh in three-dimensional space based on vertex pixel information of the first mesh and the target distortion parameter to obtain a second mesh; A target image is generated based on the second grid.

2. The method according to claim 1, characterized in that The obtaining of the first grid comprises: Using multiple processes to construct corresponding sub-grids respectively; wherein the sub-grid is a plane grid with M rows and N columns, and M and N are both preset positive integers; The plurality of sub-grids are merged to obtain a first grid.

3. The method according to claim 1, characterized in that The first grid is a plane grid, and determining a first mapping relationship between the first image and the first grid includes: A first mapping relationship is determined based on ratio information between a size of the first image and a size of the first grid.

4. The method according to claim 1 or 3, characterized in that: The determining the vertex pixel information of the first mesh based on the first mapping relationship and the first image includes: Determine the sampling position corresponding to the vertex of the first mesh, and determine the pixel point corresponding to the sampling position in the first image based on the first mapping relationship, and use the pixel information of the pixel point as the vertex pixel information of the first mesh; the pixel information of the pixel point includes brightness information and / or color information of the pixel point.

5. The method according to claim 1, characterized in that The horizontal coordinate of the sampling position corresponding to the vertex is: the horizontal coordinate of the vertex or the horizontal coordinate of the specified position of the fragment to which the vertex belongs in the first grid; The ordinate of the sampling position corresponding to the vertex is: the ordinate of the vertex or the ordinate of the designated position of the fragment to which the vertex belongs in the first grid.

6. The method according to claim 1, characterized in that The target distortion parameter is a preset adjustable parameter, and the target distortion parameter includes a distortion axis and a maximum distortion angle; wherein the maximum distortion angle is a maximum angle at which vertices in the first mesh rotate around the distortion axis; The adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter to obtain the second mesh includes: Determining a target angle at which the vertices of the first mesh rotate around the warp axis based on vertex pixel information of the first mesh and the maximum warp angle; The positions of the vertices of the first mesh in the three-dimensional space are adjusted based on the twist axis and the target angle to obtain a second mesh.

7. The method according to claim 6, characterized in that The vertex pixel information includes a vertex brightness value or a vertex color value; and determining a target angle at which the vertices of the first mesh rotate around the warp axis based on the vertex pixel information of the first mesh and the maximum warp angle includes: Based on the maximum distortion angle, obtaining a second mapping relationship between vertex pixel information of the first mesh and an angle at which the vertices of the first mesh rotate around the distortion axis; the second mapping relationship is represented by a positive correlation function or a negative correlation function; Based on the vertex pixel information of the first mesh and the second mapping relationship, a target angle for rotating the vertices of the first mesh around the twist axis is determined.

8. The method according to claim 1 or 6, characterized in that: The adjusting the positions of the vertices of the first mesh in the three-dimensional space based on the twist axis and the target angle to obtain the second mesh includes: Adjusting positions of vertices of the first mesh in three-dimensional space based on the twist axis and the target angle to obtain vertex position information of a second mesh; Determine vertex pixel information of the second mesh based on vertex position information of the second mesh and vertex pixel information of the first mesh; Determine fragment pixel information of the first mesh based on the first mapping relationship and / or vertex pixel information of the first mesh; Based on the vertex pixel information of the second mesh, perform a depth adjustment operation and a fragment edge feathering operation on the fragment pixel information of the first mesh to obtain the fragment pixel information of the second mesh; A second mesh is obtained based on the vertex pixel information and fragment pixel information of the second mesh.

9. The method according to claim 8, characterized in that The obtaining of the second mesh based on the vertex pixel information and the fragment pixel information of the second mesh comprises: Obtaining preset stylization parameters; wherein the stylization parameters include a first parameter and / or a second parameter, the first parameter indicates that transparency processing is performed on the target edge lines of the second grid, and the second parameter is used to indicate that transparency processing is performed on the target fragments of the second grid; Adjusting transparency in vertex pixel information and fragment pixel information of the second mesh based on the stylization parameter; A second mesh is obtained based on the vertex pixel information and fragment pixel information of the second mesh after transparency adjustment.

10. The method according to claim 1, characterized in that The generating a target image based on the second grid comprises: Acquire parameter information of a projection camera; wherein the parameter information includes an observation angle and a focal length, wherein the observation angle is used to indicate a viewing angle of the projection camera relative to the second grid, and the focal length is used to indicate a distance between the projection camera and the second grid; The second grid is processed for screen display based on the parameter information of the projection camera to obtain a target image.

11. An image processing device, characterized in that: include: A first grid acquisition module, used to acquire a first image to be processed and to acquire a first grid; a vertex pixel determination module, configured to determine a first mapping relationship between the first image and the first mesh, and determine vertex pixel information of the first mesh based on the first mapping relationship and the first image; A second mesh obtaining module, used for obtaining a target distortion parameter, and adjusting the position of the vertices of the first mesh in the three-dimensional space based on the vertex pixel information of the first mesh and the target distortion parameter, so as to obtain a second mesh; A target image generating module is used to generate a target image based on the second grid.

12. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the image processing method described in any one of claims 1 to 10.

Images