Image processing method and apparatus, as well as electronic equipment, storage medium and computer program

Abstract

The embodiments of the present disclosure relate to an image processing method, apparatus, device, and medium, which include: obtaining an original image to be processed and a perspective correction direction of the original image, including a preset direction and / or a custom direction; obtaining image transformation information corresponding to the original image according to the perspective correction direction, the image transformation information including one or more of image scaling information, image rotation information, and target point position information; and performing a correction process on the original image based on the image transformation information to obtain a corrected image. The embodiments of the present disclosure can effectively improve the perspective correction effect.

Description

Technical Field

[0001] (Reference to Related Applications) This application claims the priority of a Chinese patent application filed with the China National Intellectual Property Administration on December 13, 2022, with the application number 202211602553.5 and the invention title "Image Processing Method, Apparatus, Device and Medium", and all its contents are incorporated herein by reference.

[0002] (Technical Field) This disclosure relates to the field of image processing technology, and in particular, to an image processing method and apparatus, and electrons device , memory medium and computer programs related thereto.

Background Art

[0003] More and more users are performing post-processing such as optimization on the images obtained by shooting. Also, for example, the optimization needs such as beauty, filter change, sticker addition, etc. are various. One of the optimization needs is to change the perspective effect (perspective effect) of the image. For example, the user is not satisfied with the image obtained by shooting object A from the horizontal direction and hopes to obtain the image obtained by shooting object A in the forward direction. That is, by changing the shooting angle of the image in the post-image processing method, it is hoped that an image with a perspective effect that meets the needs can be obtained. However, the conventional perspective correction effect is not excellent, and it is difficult to meet the needs of users.

Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, device and medium to solve the above technical problems or at least partially solve them.

[0005] In a first embodiment, an embodiment of the present disclosure provides an image processing method comprising: obtaining a source image to be processed, a perspective correction direction of the source image including a preset direction and / or a custom direction; obtaining image conversion information corresponding to the source image, including one or more of image scaling information, image rotation information, and target point position information, according to the perspective correction direction; and performing a correction process on the source image based on the image conversion information to obtain a corrected image.

[0006] In a second embodiment, an embodiment of the present disclosure further provides an image processing apparatus comprising: a direction acquisition module for acquiring a source image to be processed and a perspective correction direction of the source image, including a preset direction and / or a custom direction; an information acquisition module for acquiring image conversion information corresponding to the source image, including one or more of image scaling information, image rotation information, and target point position information, according to the perspective correction direction; and a correction module for performing a correction process on the source image based on the image conversion information to obtain a corrected image.

[0007] In a third embodiment, an embodiment of the present disclosure further provides an electronic device comprising a processor and a memory for storing instructions that the processor can execute, wherein the processor reads the executable instructions from the memory and executes the instructions to realize an image processing method provided by the embodiment of the present disclosure.

[0008] In a fourth embodiment, an embodiment of the present disclosure further provides a computer-readable storage medium containing a computer program for performing an image processing method provided by an embodiment of the present disclosure.

[0009] The above-described technical solution provided in the embodiments of this disclosure can determine image transformation information corresponding to the perspective correction direction of the original image, and since the image transformation information corresponding to different perspective correction directions may be different, the perspective correction processing can be more focused on the original image, thereby effectively improving the perspective correction effect.

[0010] It should be understood that the contents described in this section are not intended to identify any key or essential features of the embodiments of this disclosure, nor are they intended to limit the scope of this disclosure. Other features of this disclosure will be readily apparent from the following description. [Brief explanation of the drawing]

[0011] The drawings here are incorporated into the specification and constitute part of this specification, illustrating embodiments conforming to this disclosure and are intended to be used together with the specification to interpret the principles of this disclosure. It is .

[0012] Hereinafter, in order to more clearly explain the embodiments of this disclosure or the technical proposals in the prior art, the drawings necessary for describing the embodiments or the prior art will be briefly described, and it will be apparent to those skilled in the art that other drawings can be obtained from these drawings without any creative work.

[0013] [Figure 1] This is a schematic diagram of the flow of the image processing method provided in the embodiments of this disclosure. [Figure 2] This is a schematic diagram of the planar mapping relationship provided by the embodiments of this disclosure. [Figure 3] This is a schematic diagram of the perspective transformation process provided by the embodiments of this disclosure. [Figure 4] This is a schematic diagram of the mask generation provided in the embodiments of this disclosure. [Figure 5] This is a schematic diagram of a black and white mask provided in the embodiments of this disclosure. [Figure 6] This is a schematic diagram of the gradual decrease in transparency provided by the embodiments of this disclosure. [Figure 7] This is a flowchart of the image processing provided in the embodiments of this disclosure. [Figure 8] This is a schematic diagram of the configuration of an image processing apparatus provided in the embodiments of this disclosure. [Figure 9] This is a schematic diagram of the configuration of the electronic device provided in the embodiments of this disclosure. [Modes for carrying out the invention]

[0014] To more clearly understand the above objects, features, and advantages of the present disclosure, the technical solutions of the present disclosure will be further described below. Unless there is a contradiction, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0015] In the following description, many specific details are set forth for a thorough understanding of the present disclosure. However, the present disclosure may be implemented in other manners different from those described herein, and the examples in the specification are only a part of the embodiments of the present disclosure, not all of the embodiments.

[0016] FIG. 1 is a schematic diagram of the flow of an image processing method provided by an embodiment of the present disclosure. This method may be executed by an image processing apparatus, where the apparatus may be implemented by software and / or hardware and may generally be integrated into an electronic device. As shown in FIG. 1, this method mainly includes the following steps S102 to S106.

[0017] Step S102: Obtain the original image to be processed and the perspective correction direction of the original image including a preset direction and / or a custom direction.

[0018] In actual applications, the original image may be an image uploaded or specified by the user. Here, there is no restriction on the source of the original image, and the embodiments of the present disclosure do not restrict the content in the original image. The perspective correction direction can directly affect the perspective effect. Usually, the shooting angle / direction of the camera can be directly used to change the perspective effect of the subject. The intuitive feeling of the perspective effect is that the near is large and the far is small. It can be understood that the effect of the near being large and the far being small represented by the subject varies with the shooting angle. For example, the perspective effect obtained when the camera shoots an object from the front is different from the perspective effect obtained when the camera shoots the object from above. In many cases, due to various factors such as space constraints and shooting levels, the perspective effect of the image obtained by the user's shooting is different from the perspective effect that the user really desires. Therefore, the user hopes to change the perspective effect of the image by post-processing the image. Specifically, the user can set the perspective correction direction to be adjusted for the original image. The perspective correction direction may be a preset direction provided by the image editing application or a custom direction of the user, thereby meeting the diverse needs of the user.

[0019] In some specific embodiments, the preset direction includes the horizontal direction and / or the vertical direction. The perspective effect that can be displayed in the horizontal direction means that the camera shoots an object along the horizontal direction of the object. For example, it can be understood that the camera is located on the left side of the object and shoots the object towards the right side of the object at a certain deflection angle. The perspective effect that can be displayed in the vertical direction means that the camera shoots an object along the vertical direction of the object. For example, it can be understood that the camera shoots the object from above at a certain pitch angle or shoots the object from below upwards. Also, the above horizontal direction and vertical direction should not be understood as being completely horizontal or completely vertical. In fact, there may be a deviation within a certain angle range, which is not limited here.

[0020] Step S104: According to the perspective correction direction, obtain image conversion information corresponding to the original image, including one or more of image scaling information, image rotation information, and target point position information.

[0021] In order to perform image correction more effectively, the embodiments of this disclosure take into full consideration the characteristics of the preset direction and the custom direction, and the image transformation information for perspective correction may differ for different directions. That is, first, the image transformation information necessary for perspective correction may be narrowed down and acquired according to the perspective correction direction, and thus different processing measures may be adopted. The processing measures corresponding to the preset direction and the custom direction will differ, and the processing measures are used to show a method for performing perspective correction on the original image, thereby achieving both accuracy and efficiency in processing.

[0022] In some specific embodiments, the image transformation information corresponding to the pre-set direction and the custom direction differs, and the spatial dimensions of the processing measures adopted differ accordingly. This spatial dimension is the spatial angle when processing the image, for example, a 3D spatial angle, a 2D spatial angle, etc., and therefore the method of processing the image also differs accordingly. For example, if the pre-set direction includes the horizontal and / or vertical direction, and the perspective correction direction includes the pre-set direction, it is determined that the image transformation information corresponding to the perspective correction direction includes image scaling information and / or image rotation information, and further, the target processing measure corresponding to the perspective correction direction includes performing perspective correction processing on the original image based on 3D space. If the perspective correction direction includes the custom direction, it is determined that the image transformation information corresponding to the perspective correction direction includes image scaling information and / or image rotation information, and the image transformation information includes target point position information, and further, the target processing measure corresponding to the perspective correction direction includes performing perspective correction processing on the original image based on 2D space. In the embodiments of this disclosure, since it is taken into full consideration that horizontal / vertical perspective correction is relatively similar to a simulation in which the camera is translated left / right / up / down in three-dimensional space, it is more convenient to process image correction in three-dimensional space. Therefore, image scaling information and / or image rotation information may be acquired and correction processing performed. However, processing perspective correction in any direction in three-dimensional space requires extremely complex calculations. Therefore, in the embodiments of this disclosure, a two-dimensional spatial viewing angle is adopted. This allows for the acquisition and processing of target point position information, including the planar position information of the target points corresponding to the original image and the corrected image, respectively. By determining the positional transformation correspondence between pixels in the planar image, custom direction perspective correction processing can be easily and efficiently realized.

[0023] According to the above method, it is possible to determine the necessary image transformation information in different directions and to acquire the image transformation information. In practical applications, the image transformation information may be determined based on user operations, for example, the image transformation information may be acquired by operating a specified control on the user interface, or the above image transformation information may be acquired in response to the user's operations on the image, but this is not limited to the above.

[0024] Step S106: Based on the image conversion information, a correction process is performed on the original image to obtain a corrected image.

[0025] Once the perspective correction direction corresponding to the original image is determined, the desired image transformation information is determined accordingly, that is, the target processing measures to be adopted are determined, and thus, perspective correction processing is performed on the original image using the target processing measures to obtain a corrected image.

[0026] The above-described technical solution provided in the embodiments of this disclosure can determine image transformation information corresponding to the perspective correction direction of the original image, and since the image transformation information corresponding to different perspective correction directions may be different, the perspective correction processing can be more precisely targeted to the original image, thereby improving the perspective correction effect.

[0027] To facilitate understanding, the following describes the image conversion information corresponding to the pre-set direction and the custom direction, as well as the method (abbreviated as a processing measure) for correcting the original image based on the image conversion information.

[0028] (1) Processing measures corresponding to a predetermined direction:

[0029] If the perspective correction direction includes a preset direction, and the preset direction includes the horizontal and / or vertical directions, the image conversion information includes image scaling information and / or image rotation information.

[0030] In practical implementation, image scaling information and / or image rotation information corresponding to a pre-set direction may be obtained first. In actual application, both image scaling information and image rotation information may be set by the user through designated controls on the interface. That is, image scaling information and / or image rotation information may be determined based on user operation. For example, horizontal trigger controls and vertical trigger controls may be set on the interface, and adjustment controls, i.e., the designated controls mentioned above, may be set simultaneously. For example, if the horizontal trigger control is active, when the user triggers the adjustment control, it is considered that the user has set the image scaling information and / or image rotation information corresponding to the horizontal direction. The embodiments of this disclosure are not limited to the form of the specified control. For example, the specified control may be a slider bar control, which is similar to a displayable scale on a measuring rod. Each scale has a mapping relationship with information such as image scaling information and image rotation information, allowing the user to change the perspective effect by moving the position of the slider bar's index point on the slider bar. In some specific embodiments, the position of the slider bar's index point may also have a mapping relationship with the image scaling information and image rotation information simultaneously. In other words, the user can change the image scaling information and image rotation information simultaneously by changing only the position of the index point, which is very simple and quick.

[0031] After obtaining image conversion information, correction processing can be performed on the original image based on the image conversion information. In some embodiments, this can be achieved by referring to the following steps a to c.

[0032] Step a: Determine camera movement information according to image rotation information. Since the movement of an object is relative, that is, if you want the object to move to the left in the field of view, it is equivalent to the camera moving to the right, so you can determine camera movement information according to image rotation information.

[0033] Step b: Determine the pixel point position transformation information of the original image based on the image scaling information and camera movement information.

[0034] In some specific embodiments, image scaling information may be represented by a model transformation matrix (model_matrix) to represent the scaling of the image itself, and camera movement information may be represented by a camera movement view matrix (which may be simply abbreviated as a view matrix, view_matrix) to represent the movement of the camera. Furthermore, an image transformation matrix (mvp_matrix) capable of representing the pixel point position transformation information of the original image may be determined by further combining a predetermined perspective projection matrix (projection_matrix). Specifically, the image transformation matrix may be obtained based on the product of the model transformation matrix, the camera movement view matrix, and the perspective projection matrix. In addition, in embodiments of this disclosure, fixed parameters of the perspective projection matrix may be set in advance, and it should be explained that the above matrix can be determined by determining the image scaling information and image rotation information set by the user for the original image, mainly in response to user operations, without knowing information such as camera parameters at the time of original image capture. Furthermore, image scaling may also involve changing the aspect ratio and should not be limited to proportional scaling.

[0035] Taking the example of a user performing positive horizontal correction on the original image, the embodiments of this disclosure may perform the following operations on the original image: keeping the left edge of the original image stationary, slightly increasing the width of the original image, i.e., changing the model_matrix matrix, rotating the left edge of the original image around the Y-axis, i.e., changing the view_matrix, then further combining the perspective projection matrix, and multiplying the three matrices to obtain an image transformation matrix corresponding to the pixel point position transformation information. In this, the method of adjusting the width and the rotation angle can both be determined based on the user's operation results on controls such as sliders.

[0036] Step c: Based on the pixel point position transformation information of the original image, a correction process is performed on the original image. This correction process is equivalent to performing perspective correction on the original image.

[0037] Specifically, pixel point position transformation information represented by an image transformation matrix may be sent to the GPU as a parameter for rendering, thereby enabling horizontal or vertical correction.

[0038] In the embodiments of this disclosure, the perspective correction in a preset direction such as horizontal or vertical is given full consideration to the fact that it is relatively similar to a simulation in which the camera is translated left / right or up / down in three-dimensional space, and the perspective correction processing in a preset direction can be easily and reliably realized by the above method.

[0039] (ii) Processing measures to support custom directions:

[0040] When the perspective correction direction includes a custom direction, the image transformation information includes target point position information, and when actually performing the transformation, the target point position information corresponding to the custom direction may be obtained first. For example, the target point position information includes the planar position information of the target points corresponding to the original image and the corrected image, where the corrected image is the image after perspective correction processing has been performed on the original image according to the custom direction. The planar position information of the target points corresponding to the original image and the corrected image is determined based on the user dragging a touchpoint on the original image. The point that the user touches on the original image is the touchpoint, and in some examples, the target point is determined based on the touchpoint, and the touchpoint may be used directly as the target point, or a point that has a predetermined association relationship with the touchpoint may be used as the target point, but this is not limited here. For example, the target point may correspond to the four vertices of the original image, and the user can set the custom perspective correction direction by touching one or more vertices of the original image and changing the vertex position by dragging the vertex in any direction as needed. Based on user input, the position of target points in both the original image and the corrected image can be determined, that is, the correspondence between identical points in the original image and the corrected image can be determined, thus facilitating the further establishment of the correspondence between the original image and the corrected image.

[0041] The step of obtaining image conversion information and then performing correction processing on the original image based on the image conversion information may be implemented in some embodiments by referring to steps A to C below.

[0042] Step A: Determine a first matrix based on planar position information. The first matrix represents the pixel transformation correspondence between the original image and the corrected image. The first matrix may also be understood as a transformation matrix, and in some specific examples, the first matrix may be implemented using a homography matrix.

[0043] For ease of understanding, you may refer to the schematic diagram of the plane mapping relationship shown in Figure 2, which briefly shows the mapping relationship between the original image plane and the new image plane when the perspective effect is changed. For specific methods of determining the correspondence, you may refer to related technologies. Note that Figure 2 is merely an example of the principle and shows that the mapping relationship between two planes can be processed based on two-dimensional space, and the specific principle will not be explained repeatedly here. In some specific embodiments, given that the plane position information of the target points in the original image and the plane position information of the target points in the corrected image are known, a homography matrix can be obtained by determining eight unknown quantities in the homography matrix according to the four target points. In this, the homography matrix is ​​a transformation matrix that can be used for image correction, and the correspondence between two images can be determined using four pairs of corresponding points, and the size of the homography matrix in a two-dimensional plane is 3*3, and specifically, each parameter in the homography matrix can be determined in the following manner according to the positional relationship of the target points in the two images, however a 33 is mainly a 11 ~a 32 This is a fixed value used to determine [something].

number

[0044] However, x', y' and u, v may be determined based on the planar position information of the target point in the original image and the planar position information of the target point in the corrected image, respectively, and w' and w may be considered as predetermined parameters. According to the above method, a homography matrix Mat can be obtained.

[0045] Step B: Find the inverse matrix of the first matrix to obtain the second matrix. The second matrix may also be the inverse matrix.

[0046] The first matrix describes the pixel offset method, meaning that by applying the first matrix to each pixel in the original image, the position of that pixel in the corrected image can be obtained. The formalized expression is as follows: Dst(x,y)=Mat*(x',y'), where (x',y') corresponds to the pixel coordinates in the original image and Dst(x,y) corresponds to the pixel coordinates in the corrected image. However, the above formula is more suitable for serial operations and contradicts the operation method of the GPU. The characteristics of the GPU itself indicate that it is suitable for parallel operations, and in actual application, it can only obtain Dst(x,y) of the corrected image, and cannot obtain the specific position (x',y') that corresponds to it in the original image beforehand. Therefore, in the embodiments of this disclosure, we adopt the approach of finding the inverse matrix of the first matrix, namely Mat-1*Dst(x,y)=(x',y'), and by swapping the left and right sides, we get (x',y')=Mat-1*Dst(x,y).

[0047] Using the method described above, the GPU can quickly determine the corresponding position of each pixel in the corrected image in the original image, and can directly process all pixels in parallel on the GPU.

[0048] Step C: Perform correction processing on the original image based on the second matrix. If the second matrix (also called the inverse matrix) is known, it can be understood that the above method allows the GPU to quickly determine the pixel position in the original image and the color of each pixel in the corrected image, thereby performing perspective correction processing on the original image and obtaining the corrected image.

[0049] Furthermore, it should be explained that due to the hardware characteristics of the CPU itself, serial calculations can only be performed on the entire pixel, meaning the CPU must calculate the position of each pixel individually, and cannot process floating-point numbers. For example, it cannot process half of the pixels, and can only process the complete pixels, resulting in relatively poor processing accuracy and relatively low processing efficiency for serial processing methods. Due to the hardware characteristics of the GPU itself, the GPU can normalize the image to the range of 0 to 1, meaning it can process floating-point numbers better and perform parallel processing on all pixels, thereby improving processing speed. Compared to perspective correction processing using the CPU directly, the embodiment of this disclosure employs perspective correction processing using the GPU, which not only has higher processing speed but also higher processing accuracy, and can reduce to some extent phenomena such as jaggedness and artifacts caused by the poor processing accuracy of the CPU.

[0050] In the embodiments of this disclosure, we have taken into full consideration that performing perspective correction in any direction in three-dimensional space would require extremely complex calculations. Therefore, by determining the transformation correspondence between pixels of a planar image from the viewing angle in two-dimensional space, we can easily and efficiently realize perspective correction processing in a custom direction, and also support processing using a GPU to effectively improve processing efficiency and accuracy.

[0051] The embodiments of this disclosure further include obtaining a corrected image, then performing an optimization process on the corrected image, including edge optimization and / or non-content area filling, to further ensure the image correction effect, and obtaining a target image based on the optimized image. Because the corrected image obtained by the above method may have problems such as jagged edges, artifacts, and blank areas (non-content areas), further optimization of the corrected image in the embodiments of this disclosure can improve the effect of the target image displayed to the user. For ease of understanding, this will be explained in detail below.

[0052] For illustrative purposes, Figure 3 shows a schematic diagram of the perspective transformation process, illustrating the original image and the corrected image obtained by applying perspective correction to the original image. As an example, the original image shows a columnar object photographed from the front, with the background surrounding the columnar object simply shown in gray. The perspective correction process is performed on the original image using the method described above to obtain the corrected image. After the perspective processing, the blank area around the outer edge of the corrected image contains no content, i.e., it becomes a non-content area. In actual application, it is understood that the corrected image obtained after perspective processing may have jagged edges or blank areas (non-content areas). However, in order to provide the user with a good visual experience, the embodiments of this disclosure may optimize the edges of the content area (e.g., the trapezoidal edge in Figure 3) and further fill in the non-content areas.

[0053] In actual application, content and non-content areas in the corrected image may be identified based on a mask. For ease of understanding, a schematic diagram of mask generation shown in Figure 4 may be referenced. In Figure 4, the original coordinates are the pixel coordinates in the original image. Multiplying the original coordinates by a homography matrix yields the sampling coordinates, which are the pixel coordinates in the corrected image. It may then be determined whether the pixel coordinates in the corrected image are within the content area. If it is determined that they are, black is output without the need for filling; otherwise, filling is required, and white is output. This yields a result mask (black and white mask). For illustrative purposes, a schematic diagram of a black and white mask shown in Figure 5 may be referenced. The role of the mask is to clearly distinguish between content and non-content areas.

[0054] If the optimization process includes edge optimization, performing the optimization on the corrected image includes applying a transparency gradient to the content region of the corrected image, where the transparency of the content region gradually decreases along the direction from the center of the content region to the edge of the content region. The transparency of the edges of the content region is lowest, and the transparency of the central part of the content region is highest. For ease of understanding, you may refer to the schematic diagram of the gradual decrease in transparency shown in Figure 6. This method, in which the transparency gradually decreases from the center of the content region to the edge of the content region, can improve the jaggedness of the edges to some extent.

[0055] If the optimization process includes filling in non-content areas, performing the optimization process on the corrected image may refer to steps (1) and (2) below.

[0056] Step (1): An initial filling is performed on the non-content area based on the content area using a predetermined matching algorithm to obtain the initial filled area. Exemplarily, the matching algorithm may be a stereo matching algorithm, and specifically, based on the core idea that there is similarity between adjacent pixels in an image, the continuity of the image can be used to fill the non-content area with the edges of the content area and adjacent content, that is, the matching algorithm is used to find areas that merge with the edges and perform the initial filling. For ease of understanding, a feasible method of the stereo matching algorithm will be briefly explained. First, in the initialization stage, one offset amount is randomly assigned to each pixel in the original image, and one corresponding pixel is found in the result image. Then, in the propagation stage, each pixel checks whether the offset from the adjacent block provides a better match, and if so, the patch offset of the adjacent block is adopted. Subsequently, in the search stage, each pixel point finds a better matching offset in place of the current offset within a concentric circle centered on the current offset. The above method can be used to fill pixels in the non-content area. In this, the content area corresponds to the original image, and the non-content area corresponds to the result image.

[0057] Step (2): Pixel optimization processing is performed on the initial filled region using a predetermined deep learning algorithm.

[0058] In the embodiments of this disclosure, we have taken into full consideration the possibility that the effect of directly filling non-content areas using a matching algorithm may not be natural. Therefore, we may further optimize the initial filled areas using a deep learning algorithm. The embodiments of this disclosure do not limit the deep learning algorithm, and for example, a neural network model may be pre-trained using a training set. Exemplarily, the training set includes a first training image and a second training image. The first training image is an image in which blank areas have been filled by a matching algorithm. For example, this may be obtained by first acquiring a sample image that does not contain blank areas, then converting a specified area in the sample image into a blank area to obtain an image containing blank areas, and then filling the blank areas using a matching algorithm. The second training image may be the above sample image that does not contain blank areas, or it may be the most natural original image. The second training image allows the neural network model to oversee the optimization effect on the filled areas of the first training image, enabling the neural network model to perform pixel optimization processing more accurately and realistically, resulting in a more natural blank area filling result.

[0059] For the sake of understanding, the embodiments of this disclosure further provide the image processing flowchart shown in Figure 7, which schematically illustrates that the user first performs horizontal or vertical perspective correction on the original image, then performs custom direction perspective correction, and finally optimizes the image obtained by the free correction process. In other words, Figure 7 shows three important stages: horizontal / vertical correction in three-dimensional space, free correction in two-dimensional space, and processing of blank areas by an intelligent algorithm. In the horizontal / vertical correction process in 3D space, image scaling information and image rotation information corresponding to the original image are obtained, and a transformation matrix is ​​calculated by scaling the image and moving the camera angle (corresponding to image rotation). Thus, horizontal / vertical correction may be achieved by applying the transformation matrix. After that, the image after horizontal / vertical correction may be directly used as input for free correction (i.e., custom direction perspective correction processing). In the free correction process in 2D space, target point position information corresponding to the original image is obtained, and then a homography matrix is ​​calculated, the inverse matrix of the homography matrix is ​​found, and parallel processing is performed on all pixels by combining the GPU and the inverse matrix, and an edge-optimized image is obtained by performing an optimization process on the GPU edges. Finally, the region to be filled in the edge-optimized image (the non-content region mentioned above) is identified using a mask, and for example, a matching algorithm may be used to find similar regions in the image and fill them. Finally, the filling effect is post-optimized using a deep learning algorithm to obtain the result picture, which is the target image mentioned above. Figure 7 is merely one example of applying the image processing method described in the embodiments of this disclosure, and is not intended to limit it. In actual application, depending on the user's needs, only horizontal / vertical correction may be performed on the original image, only free correction may be performed on the original image, free correction may be performed first and then horizontal / vertical correction may be performed, and some of the steps in Figure 7 may be reduced or adjusted. For example, edge optimization may not be performed due to processing resource constraints, and other matching algorithms may be employed.

[0060] Based on the image processing method provided in the embodiments of this disclosure, the user can be provided with a perspective correction function for a preset direction (horizontal / vertical) and a perspective correction function for a custom direction. The user may flexibly select either one or combine them according to their needs and perform multiple adjustments on the image. Specifically, assuming that the user first triggers the horizontal correction function to perform horizontal correction on image A, image A may be treated as the source image to be processed, and image A may be processed by applying the target processing measures corresponding to the preset direction provided in the embodiments of this disclosure to obtain image A'. If the user then triggers the custom correction function, image A' can be treated as the source image to be processed again, and processed using the target processing measures corresponding to the custom direction. In other words, the target image obtained by each processing can be used as the source image for the correction processing function separately selected by the user, and correction processing can be continued. With this method, the user can repeatedly perform adjustments until the desired perspective effect is obtained, fully satisfying the user's needs.

[0061] From the above, the image processing method provided in the embodiments of this disclosure can not only support perspective correction processing between fixed directions such as horizontal / vertical and arbitrary directions customized by the user, but also achieve a better balance between processing efficiency and display effect after image correction by using different correction methods for fixed directions and custom directions. Furthermore, when performing custom perspective correction, parallel calculations based on the inverse matrix may be performed using a GPU, which not only increases the image processing speed but also further improves processing accuracy because the GPU can perform floating-point calculations. In addition, by performing transparency asymptotic processing on edges, phenomena such as jaggedness and artifacts can be effectively reduced. Moreover, by combining conventional algorithms with deep learning algorithms, the effect of filling correction blank areas can be effectively ensured.

[0062] Corresponding to the image processing method described above, Figure 8 is a schematic diagram of the configuration of an image processing apparatus provided by an embodiment of the present disclosure, which can be implemented by software and / or hardware and can usually be integrated into an electronic device, as shown in Figure 8,

[0063] A direction acquisition module 802 for obtaining the source image to be processed, and the perspective correction direction of the source image, including a pre-set direction and / or a custom direction,

[0064] An information acquisition module 804 for acquiring image transformation information corresponding to the original image, which includes one or more of image scaling information, image rotation information, and target point position information, depending on the perspective correction direction,

[0065] The system includes a correction module 806 for performing correction processing on the original image using image conversion information to obtain a corrected image.

[0066] The above-mentioned technical solution provided in the embodiments of this disclosure can determine image transformation information corresponding to the perspective correction direction of the original image, and since the image transformation information corresponding to different perspective correction directions may be different, the perspective correction processing can be more focused on the original image, thereby effectively improving the perspective correction effect.

[0067] In some embodiments, if the perspective correction direction includes the preset direction, the image conversion information includes image scaling information and / or image rotation information, and if the perspective correction direction includes the custom direction, the image conversion information includes target point position information.

[0068] In some embodiments, the correction module 806 is specifically used to determine camera movement information according to the image rotation information, to determine pixel point position transformation information of the original image according to the image scaling information and the camera movement information, and to perform perspective correction processing on the original image according to the pixel point position transformation information of the original image.

[0069] In some embodiments, the image scaling information and the image rotation information are set by the user through designated controls on the interface.

[0070] In some embodiments, the correction module 806 is specifically used to determine a first matrix for representing the pixel transformation correspondence between the original image and the corrected image based on the target point position information, which includes planar position information of target points corresponding to the original image and the corrected image, respectively; to obtain a second matrix by finding the inverse matrix of the first matrix; and to perform a correction process on the original image based on the second matrix.

[0071] In some embodiments, the target point position information is determined based on the user dragging a contact point on the original image.

[0072] In some embodiments, the apparatus further includes an optimization module for performing optimization processing on the corrected image, including edge optimization processing and / or non-content area filling processing, and obtaining a target image based on the optimized image.

[0073] In some embodiments, the optimization process includes edge optimization, and the optimization module is specifically used to perform transparency gradient processing on the content region of the corrected image, wherein the transparency of the content region gradually decreases along the direction from the center of the content region to the edge of the content region.

[0074] In some embodiments, the optimization process includes a non-content area filling process, and the optimization module is specifically used to obtain an initial filled area by performing an initial filling on the non-content area based on the content area using a predetermined matching algorithm, and to perform a pixel optimization process on the initial filled area using a predetermined deep learning algorithm.

[0075] The image processing apparatus provided in the embodiments of this disclosure can perform the image processing method provided in any embodiment of this disclosure and has functional modules and beneficial effects corresponding to the performance of the method.

[0076] For the sake of convenience and conciseness of explanation, the specific working procedures for the embodiments of the apparatus described above can be referenced from the corresponding procedures in the embodiments of the method, and will not be described again here, as will be obvious to those skilled in the art.

[0077] Figure 9 is a schematic diagram of the configuration of an electronic device provided by an embodiment of the present disclosure. As shown in Figure 9, the electronic device 900 comprises one or more processors 901 and memory 902.

[0078] The processor 901 may be a central processing unit (CPU), or another form of processing unit having data processing capability and / or instruction execution capability, and may control other components in the electronic device 900 to perform a desired function.

[0079] The memory 902 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored in the computer-readable storage media, and the processor 901 may execute the program instructions to implement the image processing method and / or other desired functions of the embodiments of this disclosure described above. The computer-readable storage media may further store various contents, such as input signals, signal components, noise components, etc.

[0080] In one example, the electronic device 900 may further include an input device 903 and an output device 904, and these components are interconnected by a bus system and / or other forms of connection mechanisms (not shown).

[0081] Furthermore, this input device 903 may further include, for example, a keyboard, a mouse, etc.

[0082] This output device 904 can output various types of information, including determined distance information and direction information, to an external source. This output device 904 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0083] Of course, for the sake of simplification, Figure 9 shows only some of the components of the electronic device 900 relating to this disclosure, and components such as buses and input / output interfaces are omitted. Furthermore, the electronic device 900 may include any other appropriate components depending on the specific application.

[0084] The embodiments of this disclosure may also include, in addition to the above-described methods and apparatus, a computer program product that, when executed by a processor, includes computer program instructions causing the processor to execute the image processing method provided by the embodiments of this disclosure.

[0085] The computer program product may be written in any combination of one or more programming languages, the programming languages ​​including object-oriented programming languages ​​such as Java and C++, and also including conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, run as a standalone software package, partially on the user's computing device and partially on a remote computing device, or fully on a remote computing device or server.

[0086] Furthermore, an embodiment of the present disclosure may be a computer-readable storage medium containing computer program instructions that cause the processor to execute the image processing method provided by the embodiment of the present disclosure when executed by the processor.

[0087] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any combination thereof. More specific examples (not an exhaustive list) of readable storage media include electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0088] The embodiments of the present disclosure further provide a computer program product that, when executed by a processor, includes a computer program / instructions that implements the image processing method in the embodiments of the present disclosure.

[0089] In this text, relational terms such as “first” and “second” are used solely to distinguish one entity or operation from another, and do not necessarily require or suggest that such an actual relationship or order exists between these entities or operations. Furthermore, the terms “includes,” “contains,” or any other variations thereof are intended to include non-exclusive inclusion such that a process, method, article, or apparatus containing a set of elements includes not only these elements but also other elements not explicitly listed, or elements specific to such a process, method, article, or apparatus. Unless otherwise specified, an element limited by the phrase “contains one…” does not preclude the existence of another identical element in a process, method, article, or apparatus containing the said element.

[0090] The above are merely specific embodiments of the Disclosure, intended to enable those skilled in the art to understand or implement the Disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the Disclosure. Accordingly, the Disclosure is not limited to these embodiments described herein, but rather conforms to the broadest scope of the principles and novel features disclosed herein.

Claims

1. The process involves obtaining the source image to be processed, and the perspective correction direction of the source image, including a pre-set direction and / or a custom direction. Depending on the perspective correction direction, image transformation information corresponding to the original image is obtained, which includes one or more of the following: image scaling information, image rotation information, and target point position information. The method is characterized by including performing perspective correction processing on the original image based on the image conversion information to obtain a corrected image. Image processing methods.

2. The image processing method according to claim 1, characterized in that the aforementioned preset direction includes the horizontal direction and / or the vertical direction.

3. When the perspective correction direction includes the preset direction, the image conversion information includes image scaling information and / or image rotation information. The image conversion information includes the target point position information when the perspective correction direction includes the custom direction, characterized in that The image processing method according to claim 2.

4. If the perspective correction direction includes the predetermined direction, performing perspective correction processing on the original image based on the image conversion information is: Determining camera movement information according to the aforementioned image rotation information, The pixel point position transformation information of the original image is determined according to the image scaling information and the camera movement information, The method is characterized by including performing perspective correction processing on the original image according to the pixel point position conversion information of the original image. The image processing method according to claim 3.

5. The image processing method according to claim 4, characterized in that the image rotation information is set by the user through a control specified on the interface.

6. If the perspective correction direction includes the custom direction, performing perspective correction processing on the original image based on the image conversion information is: Based on target point position information including planar position information of target points corresponding to the original image and the corrected image, a first matrix is ​​determined to represent the pixel conversion correspondence between the original image and the corrected image. The inverse matrix of the first matrix is ​​obtained to obtain the second matrix, The method is characterized by including performing perspective correction processing on the original image based on the second matrix, The image processing method according to claim 3.

7. The image processing method according to claim 6, characterized in that the target point position information is determined by the user dragging a contact point on the original image.

8. The image processing method further includes: The process includes performing an optimization process on the corrected image, including edge optimization and / or non-content area filling, and obtaining a target image based on the optimized image. The image processing method according to any one of claims 1 to 7.

9. The optimization process includes edge optimization, and performing the optimization process on the corrected image is This includes performing a transparency gradient process on the content area of ​​the corrected image, The transparency of the content area is characterized by gradually decreasing along the direction from the center of the content area to the edge of the content area. The image processing method according to claim 8.

10. The optimization process includes filling in non-content areas, and performing the optimization process on the corrected image is Using a predetermined matching algorithm, the initial filling of the non-content area is performed based on the content area of ​​the corrected image to obtain the initial filled area. The method is characterized by including performing pixel optimization processing on the initial filled region using a predetermined deep learning algorithm, The image processing method according to claim 8.

11. A direction acquisition module for obtaining the source image to be processed and the perspective correction direction of the source image, including a preset direction and / or a custom direction, An information acquisition module for acquiring image transformation information corresponding to the original image, which includes one or more of image scaling information, image rotation information, and target point position information, depending on the perspective correction direction. The system is characterized by comprising a correction module for performing perspective correction processing on the original image based on the image conversion information to obtain a corrected image, Image processing device.

12. Processor and The processor comprises a memory for storing executable instructions, The processor is characterized by reading the executable instruction from the memory and executing the instruction to realize the image processing method described in any one of claims 1 to 7. electronic equipment.

13. A computer-readable storage medium characterized by storing a computer program for performing the image processing method described in any one of claims 1 to 7.

14. A computer program characterized in that, when executed by a computing device, it causes the computing device to execute the image processing method described in any one of claims 1 to 7.

Images