Image processing method and device, electronic device, storage medium, and computer program

Abstract

The embodiments of the present disclosure relate to an image processing method, apparatus, device, and medium, which include: obtaining an original image to be processed and a perspective correction direction of the original image, including a preset direction and / or a custom direction; obtaining image transformation information corresponding to the original image according to the perspective correction direction, the image transformation information including one or more of image scaling information, image rotation information, and target point position information; and performing a correction process on the original image based on the image transformation information to obtain a corrected image. The embodiments of the present disclosure can effectively improve the perspective correction effect.

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims priority to a Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on December 13, 2022, bearing application number 202211602553.5 and entitled "Image processing method, device, equipment and medium," the entire contents of which are incorporated herein by reference.

[0002] (Technical field) The present disclosure relates to the field of image processing technology, and more particularly to an image processing method, device, apparatus, and medium. [Background technology]

[0003] Increasingly, users are performing post-processing such as optimization on captured images, and there are various optimization needs, such as beautifying, changing filters, adding stickers, etc. One of these optimization needs is changing the perspective effect of an image. For example, a user may dislike the image obtained by photographing object A from the side, and would like to obtain an image obtained by photographing object A from the front. In other words, by changing the photographing angle of the image using post-image processing, an image with a perspective effect that meets their needs can be obtained. However, the conventional perspective correction effect is not excellent, and it is difficult to meet users' needs. Summary of the Invention

[0004] SUMMARY OF THE INVENTION The present disclosure provides an image processing method, apparatus, device, and medium to solve or at least partially solve the above technical problems.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including: obtaining an original image to be processed and a perspective correction direction of the original image, including a preset direction and / or a custom direction; obtaining image transformation information corresponding to the original image according to the perspective correction direction, the image transformation information including one or more of image scaling information, image rotation information, and target point position information; and performing a correction process on the original image based on the image transformation information to obtain a corrected image.

[0006] In a second aspect, an embodiment of the present disclosure provides an image processing device, further comprising: an orientation acquisition module for acquiring an original image to be processed and a perspective correction orientation of the original image, including a preset orientation and / or a custom orientation; an information acquisition module for acquiring image transformation information corresponding to the original image, including one or more of image scaling information, image rotation information, and target point position information according to the perspective correction orientation; and a correction module for performing a correction process on the original image based on the image transformation information to obtain a corrected image.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device further comprising a processor and a memory for storing instructions executable by the processor, wherein the processor reads the executable instructions from the memory and executes the instructions to realize the image processing method provided by the embodiment of the present disclosure.

[0008] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium having stored thereon a computer program for executing the image processing method provided by the embodiments of the present disclosure.

[0009] The above technical solutions provided by the embodiments of the present disclosure can determine image transformation information corresponding to the perspective correction direction of the original image. Since the image transformation information corresponding to different perspective correction directions may be different, the perspective correction process can be more narrowly performed on the original image, and the perspective correction effect can be effectively improved.

[0010] It should be understood that the material described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following description. [Brief explanation of the drawings]

[0011] The drawings herein, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the disclosure.

[0012] In the following, in order to more clearly explain the technical solutions in the embodiments of the present disclosure or the prior art, the drawings necessary for explaining the embodiments or the prior art will be briefly described, and it is obvious to those skilled in the art that other drawings can be obtained from these drawings without any creative work.

[0013] [Figure 1] FIG. 1 is a schematic diagram of the flow of an image processing method provided by an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of a planar mapping relationship provided by an embodiment of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram of a perspective transformation process provided by an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of mask generation provided by an embodiment of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of a black and white mask provided by an embodiment of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram of the gradual decrease in transparency provided by an embodiment of the present disclosure. [Figure 7] 1 is a flowchart of image processing provided by an embodiment of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram illustrating a configuration of an image processing device provided by an embodiment of the present disclosure. [Figure 9] 1 is a schematic diagram illustrating the configuration of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] In order to make the above objectives, features and advantages of the present disclosure more clearly understood, the technical solution of the present disclosure will be further described below. It should be noted that, unless contradictory, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0015] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure; however, the present disclosure may be embodied in other forms different from those described herein, and the embodiments in the specification are merely some of the embodiments of the present disclosure, but not all of the embodiments.

[0016] 1 is a flow diagram of an image processing method provided by an embodiment of the present disclosure, which may be performed by an image processing device, which may be realized in software and / or hardware, or may be integrated into a general electronic device. As shown in FIG. 1, the method mainly includes the following steps S102 to S106:

[0017] Step S102: Obtain an original image to be processed and a perspective-corrected direction of the original image, including a preset direction and / or a custom direction.

[0018] In practical applications, the original image may be an image uploaded or specified by a user. There is no limitation on the source of the original image, and the embodiments of the present disclosure do not limit the content of the original image. The perspective correction direction can directly affect the perspective effect. Typically, the perspective effect of an object can be changed directly by using the shooting angle / orientation of a camera. The intuitive sense of perspective effect is that close objects are larger and far objects are smaller, and it can be understood that the effect of an object's closeness and distance is different depending on the shooting angle. For example, the perspective effect obtained by a camera photographing an object from the front is different from the perspective effect obtained by a camera photographing an object from above and below. In many cases, due to various factors such as space constraints and shooting level, the perspective effect of an image photographed by a user differs from the perspective effect actually desired by the user. Therefore, users may wish to change the perspective effect of an image by post-processing the image. Specifically, users can set the perspective correction direction they want to adjust for the original image. The perspective correction direction may be a preset direction provided by an image editing application or a user-customized direction, thereby meeting various user needs.

[0019] In some specific embodiments, the preset direction includes a horizontal direction and / or a vertical direction, and a displayable perspective effect in a horizontal direction can be understood as a camera photographing an object along the horizontal direction of the object, for example, a camera located on the left side of the object and photographing the object toward the right side of the object at a certain deflection angle, and a displayable perspective effect in a vertical direction can be understood as a camera photographing an object along the vertical direction of the object, for example, a camera photographing an object from above downward at a certain pitch angle or from below upward. Furthermore, the above horizontal and vertical directions should not be understood as completely horizontal or completely vertical, and may actually be deviated within a certain angle range, and are not limited herein.

[0020] Step S104: According to the perspective correction direction, image transformation information corresponding to the original image is obtained, including one or more of image scaling information, image rotation information, and target point position information.

[0021] In order to perform better image correction, in the embodiments of the present disclosure, the characteristics of the preset direction and the custom direction are fully taken into consideration. The image transformation information for performing perspective correction for different directions may be different. That is, the image transformation information required for perspective correction is first narrowed down and obtained according to the perspective correction direction. Therefore, different processing measures may be adopted. The processing measures corresponding to the preset direction and the custom direction are different. The processing measures are used to indicate the method of performing perspective correction processing on the original image, thereby achieving both processing accuracy and processing efficiency.

[0022] In some specific embodiments, the image transformation information corresponding to the preset direction and the custom direction is different, and the spatial dimension of the processing measures adopted is different accordingly. This spatial dimension is the visual angle of the spatial dimension when processing the image, such as a three-dimensional spatial visual angle or a two-dimensional spatial visual angle, and the image processing method is also different accordingly. For example, if the preset direction includes a horizontal direction and / or a vertical direction and the perspective correction direction includes the preset direction, it is determined that the image transformation information corresponding to the perspective correction direction includes image scaling information and / or image rotation information, and further determined that the target processing measures corresponding to the perspective correction direction include performing perspective correction on the original image based on the three-dimensional space. If the perspective correction direction includes a custom direction, it is determined that the image transformation information corresponding to the perspective correction direction includes image scaling information and / or image rotation information, the image transformation information includes target point position information, and further determined that the target processing measures corresponding to the perspective correction direction include performing perspective correction on the original image based on the two-dimensional space. In the embodiments of the present disclosure, it is fully taken into consideration that the perspective correction in the horizontal / vertical directions is relatively similar to the simulation of translating a camera left / right / up / down in three-dimensional space. Therefore, it is more convenient to process image correction in three-dimensional space. Therefore, image scaling information and / or image rotation information may be obtained and used for correction processing. However, processing perspective correction in any direction in three-dimensional space requires extremely complex calculations. Therefore, in the embodiments of the present disclosure, a two-dimensional space viewing angle is adopted, and target point position information including planar position information of target points corresponding to each of the original image and the corrected image may be obtained and processed. By determining the position transformation correspondence relationship between pixels of the planar image, perspective correction processing in a custom direction can be easily and efficiently realized.

[0023] According to the above method, the image transformation information required for different directions can be determined and further acquired. In practical application, the image transformation information can be determined based on user operation, for example, the image transformation information can be acquired by the user operating a specified control on the interface, or the image transformation information can be acquired according to the user's operation on the image, but this is not limited thereto.

[0024] Step S106: Correction processing is performed on the original image based on the image conversion information to obtain a corrected image.

[0025] When the perspective correction direction corresponding to the original image is determined, the desired image transformation information is determined accordingly, that is, the target processing measure to be adopted is further determined, and then the perspective correction process is performed on the original image using the target processing measure to obtain a corrected image.

[0026] The above technical solutions provided by the embodiments of the present disclosure can determine image transformation information corresponding to the perspective correction direction of the original image, and the image transformation information corresponding to different perspective correction directions can be different, so that the perspective correction process on the original image can be more targeted and the perspective correction effect can be improved.

[0027] For ease of understanding, the following describes image conversion information corresponding to each of the preset directions and custom directions, and a method (abbreviated as processing measure) for correcting the original image based on the image conversion information.

[0028] (1) Pre-defined direction processing measures:

[0029] If the perspective correction direction includes a preset direction, and the preset direction includes a horizontal direction and / or a vertical direction, the image transformation information includes image scaling information and / or image rotation information.

[0030] In a specific implementation, image scaling information and / or image rotation information corresponding to a preset direction may be first obtained. In practical application, both the image scaling information and the image rotation information may be set by a user through a designated control on the interface. That is, the image scaling information and / or the image rotation information may be determined based on a user's operation. For example, a horizontal trigger control and a vertical trigger control may be set on the interface, and an adjustment control, i.e., the designated control, may be set simultaneously. For example, when the horizontal trigger control is in an active state, if the user triggers the adjustment control, it is deemed that the user has set image scaling information and / or image rotation information corresponding to the horizontal direction. The embodiments of the present disclosure are not limited to the form of the designated control. For example, the designated control may be a slide bar control, where the slide bar is similar to the displayable scale of a staff. A user can change the perspective effect by moving the position of the index point of the slide bar on the slide bar. Each scale has a mapping relationship with information such as image scaling information and image rotation information. In some specific embodiments, the position of the index point of the slide bar may also have a mapping relationship with image scaling information and image rotation information. In other words, a user only needs to change the position of the index point to change the image scaling information and image rotation information at the same time, which is very simple and quick.

[0031] After obtaining the image transformation information, correction processing can be performed on the original image based on the image transformation information, which may be realized in some embodiments by referring to the following steps a to c.

[0032] Step a: Determine camera movement information according to image rotation information. Because object movement is relative, that is, if you want the object to move left in the field of view, it is equivalent to the camera moving right, so camera movement information can be determined according to image rotation information.

[0033] Step b: Pixel point position conversion information of the original image is determined according to the image scaling information and camera movement information.

[0034] In some specific embodiments, image scaling information may be represented by a model transformation matrix (model_matrix) to represent the scaling of the image itself, and camera movement information may be represented by a camera movement view matrix (simply referred to as view matrix, or view_matrix) to represent the movement of the camera. Furthermore, an image transformation matrix (mvp_matrix) capable of representing pixel point position transformation information of the original image may be determined by further combining a predetermined perspective projection matrix (projection_matrix). Specifically, the image transformation matrix may be obtained based on the product of the model transformation matrix, the camera movement view matrix, and the perspective projection matrix. Furthermore, in the embodiments of the present disclosure, fixed parameters of the perspective projection matrix may be set in advance, and the above matrix may be determined by determining image scaling information and image rotation information set by the user for the original image mainly in response to user operation, without knowing information such as camera parameters at the time of capturing the original image. Furthermore, image scaling may change the aspect ratio and is not limited to proportional scaling.

[0035] Taking the user performing positive horizontal correction on the original image as an example, the embodiments of the present disclosure may perform the following operations on the original image: keep the left edge of the original image stationary, slightly increase the width of the original image (i.e., modify the model_matrix matrix), rotate the left edge of the original image as the Y axis (i.e., modify the view_matrix matrix), and then further combine the perspective projection matrix. Multiplying the three matrices results in an image transformation matrix corresponding to the pixel point position transformation information. The above width adjustment method and rotation angle can both be determined based on the user's operation results on controls such as sliders.

[0036] Step c: A correction process is performed on the original image according to the pixel point position conversion information of the original image, which is to say, a perspective correction is performed on the original image.

[0037] Specifically, the pixel point position transformation information represented by the image transformation matrix may be sent to the GPU as a parameter to perform the rendering process, thereby realizing correction in the horizontal or vertical direction.

[0038] In the embodiments of the present disclosure, perspective correction in a preset direction such as horizontal / vertical is relatively similar to a simulation of translating a camera left / right / up / down in three-dimensional space, and the above method can easily and reliably realize perspective correction processing in a preset direction.

[0039] (2) Custom direction processing measures:

[0040] When the perspective correction direction includes a custom direction, the image transformation information includes target point position information. Specifically, in this implementation, target point position information corresponding to the custom direction may be first obtained. Exemplarily, the target point position information includes planar position information of target points corresponding to the original image and the corrected image, respectively, where the corrected image is an image obtained after perspective correction processing of the original image according to the custom direction. The planar position information of target points corresponding to the original image and the corrected image is determined based on the user dragging contact points on the original image. The contact points are points touched by the user on the original image. In some examples, the target points are determined based on the contact points, and the contact points may be directly used as the target points, or points having a predetermined association relationship with the contact points may be used as the target points, but this is not limited thereto. Exemplarily, the target points may correspond to four vertices of the original image. The user can set a custom perspective correction direction by touching one or more vertices on the original image and dragging the vertices in any direction as needed to change the vertex positions. Based on user operation, the positions of target points in each of the original image and the corrected image can be determined, i.e., the correspondence between the same points in the original image and the corrected image can be determined, thereby making it easier to further establish the correspondence between the original image and the corrected image.

[0041] In some embodiments, the step of acquiring the image transformation information and then performing a correction process on the original image based on the image transformation information may be realized by referring to the following steps A to C.

[0042] Step A: Determine a first matrix based on the plane position information, where the first matrix is ​​for representing the pixel transformation correspondence between the original image and the corrected image, and the first matrix may be understood as a transformation matrix, and in some embodiments, the first matrix may be realized using a homography matrix.

[0043] For ease of understanding, the schematic diagram of the planar mapping relationship shown in Figure 2 may be referred to. This diagram simply illustrates the mapping relationship between the original image plane and the new image plane when the perspective effect is changed. For specific methods of determining the correspondence relationship, refer to the related art. Note that Figure 2 merely illustrates the principle and shows that the mapping relationship between two planes can be processed based on two-dimensional space. The specific principle will not be repeated here. In some specific embodiments, after knowing the planar position information of the target point in the original image and the planar position information of the target point in the corrected image, the homography matrix can be obtained by determining eight unknown quantities in the homography matrix according to four target points. The homography matrix is ​​a transformation matrix that can be used for image correction. The correspondence relationship between two images can be determined using four corresponding point pairs. The size of the homography matrix in the two-dimensional plane is 3*3. Specifically, according to the positional relationship of the target points in the two images, the parameters of the homography matrix can be determined as follows, where a 33 is mainly a 11 ~a 32 is a fixed value for determining

number

[0044] However, x', y' and u, v may be determined based on the planar position information of the target point in the original image and the planar position information of the target point in the corrected image, and w' and w may be regarded as predetermined parameters. According to the above method, the homography matrix Mat can be obtained.

[0045] Step B: Invert the first matrix to obtain a second matrix, where the second matrix may be an inverse matrix.

[0046] It can be seen that the first matrix describes the pixel offset method, i.e., for each pixel in the original image, the position of that pixel in the corrected image can be obtained by applying the first matrix. The formula is as follows: Dst(x,y)=Mat*(x',y'), where (x',y') corresponds to the pixel coordinate in the original image, and Dst(x,y) corresponds to the pixel coordinate in the corrected image. However, the above formula is more suitable for serial calculations and is inconsistent with the operating method of a GPU. The characteristics of a GPU indicate that it is suitable for parallel calculations. In actual applications, it can only obtain Dst(x,y) of the corrected image, without first obtaining the corresponding specific position (x',y') in the original image. Therefore, in the embodiment of the present disclosure, we adopt to find the inverse matrix of the first matrix, i.e., Mat-1*Dst(x,y)=(x',y'), and after left-right swapping, we get (x',y')=Mat-1*Dst(x,y).

[0047] The above scheme allows the GPU to quickly determine the location in the original image where each pixel in the corrected image corresponds, and allows all pixels to be processed directly in parallel on the GPU.

[0048] Step C: Perform correction processing on the original image based on the second matrix. If the second matrix (also called the inverse matrix) is known, it can be seen that the above method allows the GPU to quickly determine the pixel position in the original image and the pixel color for each pixel in the corrected image, thereby achieving perspective correction processing on the original image and obtaining the corrected image.

[0049] It should also be noted that due to the hardware characteristics of the CPU itself, it can only perform serial calculations for all pixels. That is, the CPU must calculate the position of each pixel individually and cannot process floating-point numbers. For example, it cannot process half pixels, but only full pixels. This results in relatively poor processing accuracy and relatively low processing efficiency for serial processing. The hardware characteristics of the GPU itself allow the GPU to normalize images to the range of 0 to 1, that is, it can better process floating-point numbers and perform parallel processing for all pixels, thereby improving processing speed. Compared to perspective correction processing directly using a CPU, the embodiments of the present disclosure employ perspective correction processing using a GPU, which not only achieves higher processing speed but also higher processing accuracy, thereby reducing to some extent phenomena such as jagged edges and artifacts caused by poor CPU processing accuracy.

[0050] In the embodiments of the present disclosure, it is fully taken into consideration that performing perspective correction in any direction in three-dimensional space requires extremely complex calculations. Therefore, by determining the conversion correspondence between pixels of a planar image from a two-dimensional spatial visual angle, perspective correction processing in a custom direction can be easily and efficiently realized, and processing using a GPU is supported, thereby effectively improving processing efficiency and processing accuracy.

[0051] In some embodiments of the present disclosure, the correction image is further optimized, including an edge optimization process and / or a non-content area filling process, to ensure a more effective image correction. The target image is then obtained based on the optimized image. Correction using the above method may result in problems such as jagged edges, artifacts, and blank areas (non-content areas). Therefore, in some embodiments of the present disclosure, the correction image is further optimized to improve the target image displayed to the user. For ease of understanding, the following is a detailed description.

[0052] For example, refer to the schematic diagram of perspective transformation processing shown in Figure 3, which schematically illustrates an original image and a corrected image obtained by performing perspective correction processing on the original image. Take the original image as an example, where the subject is a columnar object photographed from the front, and the background around the columnar object is simply shown in gray in the figure. The original image is subjected to perspective correction processing using the above method to obtain a corrected image. After perspective processing, the blank area around the periphery of the corrected image has no content, i.e., it becomes a non-content area. In practical applications, it is understood that the corrected image obtained after perspective processing may have jagged edges or blank areas (non-content areas). However, in order to provide a good visual experience to users, in embodiments of the present disclosure, the edges of the content area (e.g., the trapezoidal edges in Figure 3) may be optimized and the non-content area may be further filled in.

[0053] In practical application, content areas and non-content areas in the corrected image may be identified based on a mask. For ease of understanding, reference may be made to the schematic diagram of mask generation shown in Figure 4, in which the original coordinates in Figure 4 are pixel coordinates in the original image. The original coordinates are multiplied by a homography matrix to obtain sampled coordinates, which are pixel coordinates in the corrected image. It may be determined whether the pixel coordinates in the corrected image are within a content area. If it is determined that there is, no filling is required and black is output; otherwise, filling is required and white is output. This results in a result mask (black-and-white mask). For example, reference may be made to the schematic diagram of a black-and-white mask shown in Figure 5. The role of the mask is to clearly distinguish content areas from non-content areas.

[0054] If the optimization process includes edge optimization, performing the optimization process on the corrected image includes performing a transparency gradation process on the content area of ​​the corrected image, in which the transparency of the content area gradually decreases from the center of the content area to the edge of the content area. The transparency of the edge of the content area is the lowest, and the transparency of the central part of the content area is the highest. For ease of understanding, refer to the schematic diagram of gradual transparency decrease shown in Figure 6. By gradually decreasing the transparency from the center of the content area to the edge of the content area, jagged edges can be improved to a certain extent.

[0055] When the optimization process includes filling the non-content region, performing the optimization process on the corrected image may refer to the following steps (1) and (2).

[0056] Step (1): Using a predetermined matching algorithm, a non-content area is initially filled based on the content area to obtain an initial filled area. For example, the matching algorithm may be a stereo matching algorithm. Specifically, based on the core idea of ​​similarity between adjacent pixels in an image, image continuity can be utilized to fill the non-content area with the edge of the content area and the adjacent content. That is, a matching algorithm is used to find an area that merges with the edge to perform the initial filling. For ease of understanding, a possible implementation of the stereo matching algorithm will be briefly described. First, in the initialization phase, a random offset is assigned to each pixel in the original image, and a corresponding pixel is found in the resulting image. Then, in the propagation phase, each pixel checks whether the offset from the neighboring block provides a better match. If so, the patch offset of the neighboring block is adopted. Then, in the search phase, each pixel point finds a better match within a concentric circle centered on the current offset, replacing the current offset. Using the above method, pixel filling of the non-content area can be achieved. The content area corresponds to the original image, and the non-content area corresponds to the resulting image.

[0057] Step (2): Pixel optimization processing is performed on the initial filling area using a predetermined deep learning algorithm.

[0058] In the embodiments of the present disclosure, the possibility that the effect of directly filling a non-content area using a matching algorithm may not be natural is fully taken into consideration. Therefore, a deep learning algorithm may be further used to optimize the initial filling area. The embodiments of the present disclosure do not limit the deep learning algorithm. For example, a training set may be used to pre-train a neural network model. For example, the training set may include a first training image and a second training image. The first training image is an image whose blank area has been filled using a matching algorithm. For example, the first training image may be obtained by first obtaining a sample image that does not contain a blank area, then converting a specified area in the sample image into a blank area to obtain an image that contains a blank area, and then filling the blank area using a matching algorithm. The second training image may be the sample image that does not contain a blank area and is also the most natural original image. The second training image allows the neural network model to supervise the optimization effect of the filling area of ​​the first training image, allowing the neural network model to perform a more accurate and realistic pixel optimization process, resulting in a more natural blank area filling result.

[0059] For ease of understanding, the embodiment of the present disclosure further provides an image processing flowchart shown in Fig. 7, which schematically illustrates that a user first performs horizontal or vertical perspective correction on the original image, then performs custom perspective correction, and finally optimizes the resulting image through free correction. That is, Fig. 7 illustrates three important stages: horizontal / vertical correction in three-dimensional space, free correction in two-dimensional space, and blank area processing using an intelligent algorithm. In the process of horizontal / vertical correction in three-dimensional space, image scaling information and image rotation information corresponding to the original image may be obtained, and then a transformation matrix may be calculated by scaling the image and moving the camera angle (corresponding to the image rotation), thereby realizing horizontal / vertical correction by applying the transformation matrix. After that, the image after horizontal / vertical correction may be directly used as the input for free correction (i.e., custom direction perspective correction processing). In the process of free correction in two-dimensional space, target point position information corresponding to the original image may be obtained, and then a homography matrix may be calculated, and the inverse matrix of the homography matrix may be obtained. The GPU may be combined with the inverse matrix to perform parallel processing on all pixels, and an optimization process may be performed on the edges of the GPU to obtain an edge-optimized image. Finally, a mask may be used to identify the areas to be filled in the edge-optimized image (the above-mentioned non-content areas), and for example, a matching algorithm may be used to find and fill similar areas in the image. Finally, a deep learning algorithm may be used to post-optimize the filling effect, and a result picture, which is the above-mentioned target image, may be obtained. It should be noted that FIG. 7 is merely one example of applying the above-mentioned image processing method provided by the embodiments of the present disclosure, and is not intended to be limiting. In actual application, depending on the needs of the user, only horizontal / vertical correction may be performed on the original image, only free correction may be performed on the original image, or free correction may be performed first and then horizontal / vertical correction may be performed. In addition, some steps in FIG. 7 may be reduced or adjusted. For example, edge optimization may not be performed due to processing resource constraints, and other matching algorithms may be adopted.

[0060] The image processing method provided by the embodiments of the present disclosure can provide a user with a perspective correction function in a preset direction (horizontal / vertical) and a perspective correction function in a custom direction. The user can flexibly select either one or a combination of them according to their needs, and can perform multiple adjustments on an image. Specifically, suppose a user first triggers the horizontal correction function to perform horizontal correction on image A. At this time, image A can be treated as the source image to be processed, and a target processing measure corresponding to the preset direction provided by the embodiments of the present disclosure can be applied to process image A to obtain image A'. When the user next triggers the custom correction function, image A' can be treated as the source image to be processed and processed using a target processing measure corresponding to the custom direction. That is, the target image obtained by each processing can be used as the source image for the correction processing function separately selected by the user for subsequent correction processing. This method allows the user to repeatedly perform adjustments until the desired perspective effect is achieved, fully satisfying the user's needs.

[0061] From the above, the image processing method provided by the embodiments of the present disclosure not only supports perspective correction processing between fixed directions such as horizontal and vertical and any direction customized by the user, but also uses different correction processing methods for fixed and custom directions, thereby achieving both better processing efficiency and improved display results after image correction. Furthermore, when performing custom perspective correction, a GPU may be used to perform parallel calculations based on an inverse matrix, which not only increases image processing speed but also improves processing accuracy because the GPU can perform floating-point calculations. Furthermore, transparency asymptotic processing of edges can effectively reduce phenomena such as jagged edges and artifacts. Furthermore, combining traditional algorithms with deep learning algorithms can effectively ensure the filling of correction gaps.

[0062] Corresponding to the above image processing method, FIG. 8 is a schematic diagram of the configuration of an image processing device provided by an embodiment of the present disclosure, which can be realized by software and / or hardware, and can usually be integrated into electronic equipment. As shown in FIG. 8,

[0063] an orientation acquisition module 802 for acquiring a source image to be processed and a perspective-corrected orientation of the source image, including pre-defined and / or custom orientations;

[0064] an information obtaining module 804 for obtaining image transformation information corresponding to the original image according to the perspective correction direction, the image transformation information including one or more of image scaling information, image rotation information, and target point position information;

[0065] and a correction module 806 for performing correction processing on the original image using the image conversion information to obtain a corrected image.

[0066] The above technical solution provided by the embodiments of the present disclosure can determine image transformation information corresponding to the perspective correction direction of the original image. Since the image transformation information corresponding to different perspective correction directions may be different, the perspective correction process can be more narrowly performed on the original image, and the perspective correction effect can be effectively improved.

[0067] In some embodiments, if the perspective correction direction comprises the preset direction, the image transformation information comprises image scaling information and / or image rotation information, and if the perspective correction direction comprises the custom direction, the image transformation information comprises target point position information.

[0068] In some embodiments, the correction module 806 is specifically used for determining camera movement information according to the image rotation information, determining pixel point position transformation information of the original image according to the image scaling information and the camera movement information, and performing perspective correction processing on the original image according to the pixel point position transformation information of the original image.

[0069] In some embodiments, the image scaling information and the image rotation information are set by a user through designated controls on an interface.

[0070] In some embodiments, the correction module 806 is specifically used for determining a first matrix for representing a pixel transformation correspondence relationship between the original image and the corrected image based on the target point position information including planar position information of target points corresponding to each of the original image and the corrected image; calculating an inverse matrix of the first matrix to obtain a second matrix; and performing a correction process on the original image based on the second matrix.

[0071] In some embodiments, the target point position information is determined based on a user dragging a contact point on the original image.

[0072] In some embodiments, the device further includes an optimization module for performing an optimization process on the corrected image, including an edge optimization process and / or a non-content area filling process, and obtaining a target image based on the image after the optimization process.

[0073] In some embodiments, the optimization process includes an edge optimization process, and an optimization module is specifically used to perform a transparency gradient process on a content region of the corrected image, where the transparency of the content region gradually decreases along a direction from the center of the content region to the edge of the content region.

[0074] In some embodiments, the optimization process includes a filling process for a non-content area, and the optimization module is specifically used for: performing an initial filling of the non-content area based on the content area using a predetermined matching algorithm to obtain an initial filled area; and performing a pixel optimization process for the initial filled area using a predetermined deep learning algorithm.

[0075] The image processing device provided by the embodiments of the present disclosure can execute the image processing method provided by any embodiment of the present disclosure, and includes functional modules and beneficial effects according to the execution of the method.

[0076] For convenience and brevity of explanation, those skilled in the art can clearly understand that the specific operating procedures of the above-described apparatus embodiments can refer to the corresponding procedures in the method embodiments and will not be repeated here.

[0077] 9 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 9, an electronic device 900 includes one or more processors 901 and a memory 902.

[0078] The processor 901 may be a central processing unit (CPU) or other form of processing device having data processing and / or instruction execution capabilities and may control other components in the electronic device 900 to perform desired functions.

[0079] The memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored in the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the image processing methods of the embodiments of the present disclosure described above and / or other desired functions. The computer-readable storage medium may also store various contents, such as an input signal, signal components, and noise components.

[0080] In one example, the electronic device 900 may further include input devices 903 and output devices 904, with these components interconnected by a bus system and / or other form of connection (not shown).

[0081] The input device 903 may further include, for example, a keyboard, a mouse, and the like.

[0082] This output device 904 can output various information including determined distance information, direction information, etc. This output device 904 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0083] Of course, for simplicity, Figure 9 shows only some of the components of the electronic device 900 according to the present disclosure, and omits components such as buses and input / output interfaces, etc. Furthermore, the electronic device 900 may include any other appropriate components depending on the specific application.

[0084] In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the image processing methods provided by embodiments of the present disclosure.

[0085] The computer program product may have program code for carrying out operations of embodiments of the present disclosure written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., as well as traditional procedural programming languages ​​such as "C" or similar programming languages. The program code may execute entirely on the user's computing device, partially on the user's computing device, as a separate software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0086] Furthermore, an embodiment of the present disclosure may be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform an image processing method provided by an embodiment of the present disclosure.

[0087] The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (not an exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0088] An embodiment of the present disclosure further provides a computer program product including a computer program / instruction that, when executed by a processor, implements the image processing method in an embodiment of the present disclosure.

[0089] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another and do not necessarily require or imply that such an actual relationship or order exists between those entities or operations. Furthermore, the terms "comprise," "include," and any other variations thereof are intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that includes a set of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or device. Unless further limited, an element qualified by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes that element.

[0090] The foregoing are merely specific embodiments of the present disclosure, intended to enable those skilled in the art to understand or realize the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. obtaining an original image to be processed and a perspective-corrected orientation of the original image, including a pre-defined orientation and / or a custom orientation; acquiring image transformation information corresponding to the original image, the image transformation information including one or more of image scaling information, image rotation information, and target point position information according to the perspective correction direction; and performing a correction process on the original image based on the image conversion information to obtain a corrected image. An image processing method comprising:

2. The preset direction includes a horizontal direction and / or a vertical direction.

2. The method of claim 1 .

3. when the perspective correction direction includes the predetermined direction, the image transformation information includes image scaling information and / or image rotation information; When the perspective correction direction includes the custom direction, the image conversion information includes the target point position information.

3. The method of claim 2.

4. When the perspective correction direction includes the predetermined direction, performing correction processing on the original image based on the image conversion information includes: determining camera movement information in response to the image rotation information; determining pixel point position conversion information of the original image according to the image scaling information and the camera movement information; performing a correction process on the original image in accordance with pixel point position conversion information of the original image; 4. The method of claim 3.

5. The image rotation information is set by a user through a designated control on an interface.

5. The method of claim 4.

6. When the perspective correction direction includes the custom direction, the step of performing correction processing on the original image based on the image conversion information includes: determining a first matrix for representing a pixel transformation correspondence relationship between the original image and the corrected image based on the target point position information including planar position information of target points corresponding to each of the original image and the corrected image; Inverting the first matrix to obtain a second matrix; performing a correction process on the original image based on the second matrix.

4. The method of claim 3.

7. The target point position information is determined by a user dragging a contact point on the original image.

7. The method of claim 6.

8. The method further comprises: performing an optimization process on the corrected image, the optimization process including an edge optimization process and / or a non-content region filling process, and obtaining a target image based on the optimized image; 8. The method according to any one of claims 1 to 7.

9. The optimization process includes edge optimization process, and performing the optimization process on the corrected image includes: performing transparency gradation processing on a content region of the corrected image; the transparency of the content area tapers along a direction from the center of the content area to the edge of the content area; 9. The method of claim 8.

10. The optimization process includes a filling process of a non-content area, and performing the optimization process on the corrected image includes: performing an initial filling of the non-content region based on the content region using a predetermined matching algorithm to obtain an initial filled region; and performing pixel optimization processing on the initial filling region using a predetermined deep learning algorithm.

9. The method of claim 8.

11. an orientation acquisition module for acquiring a source image to be processed and a perspective-corrected orientation of the source image, including a pre-defined orientation and / or a custom orientation; an information acquisition module for acquiring image transformation information corresponding to the original image according to the perspective correction direction, the image transformation information including one or more of image scaling information, image rotation information, and target point position information; a correction module for performing a correction process on the original image based on the image conversion information to obtain a corrected image.

1. An image processing device comprising:

12. a processor; a memory for storing instructions executable by the processor; The processor reads the executable instructions from the memory and executes the instructions to implement the image processing method according to any one of claims 1 to 10. An electronic device characterized by:

13. A computer program for executing the image processing method according to any one of claims 1 to 10 is stored. A computer-readable storage medium comprising:

14. tangibly stored on a non-transitory computer-readable medium and including machine-executable instructions; The machine-executable instructions, when executed, cause a machine to perform the image processing method of any one of claims 1 to 10. Computer program products.

Images