Image enhancement processing method, device, computer storage medium and terminal
By parallel processing of the local correction matrix and secondary rendering of the stitched panoramic images on the GPU, the real-time and computational cost of image enhancement processing in the panoramic video picture is solved, and efficient image enhancement effect is achieved.
Patent Information
- Application Number
- CN202411814448.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-10
- Publication Date
- 2025-04-01
AI Technical Summary
The prior art is difficult to achieve efficient image enhancement processing in panoramic video images, especially at ultra-high resolution, the static picture image correction calculation is large and cannot meet the real-time processing requirements.
By inputting the stitched panoramic image to the graphics processing unit (GPU), the local correction matrix of the original layer is acquired in parallel and the shading layer is secondary rendered, and image enhancement is achieved based on the original layer after curve correction and the shading layer after secondary rendering.
It realizes the rapid processing of static image correction in panoramic video images, meets real-time processing requirements, and maintains image enhancement effect at ultra-high resolution.
Smart Images
Figure CN119741237A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to multimedia information processing technology, and in particular to a method, device, computer storage medium and terminal for image enhancement processing. Background Art
[0002] Static image correction and image enhancement are two common multimedia information processing technologies. Among them, the common static image correction methods in related technologies include some image correction based on feature transformation (Fourier transform), and some image deformation including affine, projection and polar coordinate transformation. These methods can only perform calculations on static small-scale images through the central processing unit (CPU), and the typical application size is an image with a resolution of 1920*1080p. On small-scale images, processing such as histogram equalization sharpening, grayscale and contour detection can also be achieved, but these processing is difficult to achieve on images with a single-side size of more than 8192 pixels, and there are real-time problems. Image enhancement aims to improve the visual quality of video images to make them more suitable for human eye observation or computer analysis. It adjusts the image's brightness, contrast, clarity, color and other features to highlight specific information or improve overall readability.
[0003] Panoramic real-time video stitching is a technology that merges several video streams collected by cameras in real time into a seamless, continuous panoramic video image. It can provide an ultra-wide viewing angle of 360° or higher that far exceeds that of a single camera, providing users with an all-round (without blind spots) immersive monitoring perspective. While achieving an ultra-wide viewing angle, panoramic real-time video stitching introduces more real-time camera video streams, which leads to problems such as image distortion, color aberration and poor image output imaging at ultra-high video resolution after video image fusion. When panoramic real-time video is stitched and fused, each frame of the panoramic video screen is a seamless stitched panoramic image; under test conditions, after a frame of 8k resolution stitched panoramic image is converted into an RGB image, it takes more than 20 milliseconds to process the stitched panoramic image through static picture image correction; in order to ensure the stability of the panoramic video screen output, the frame processing interval of the stitched panoramic image contained in the panoramic video screen must be less than 25 milliseconds or even smaller; therefore, static picture image correction of the stitched panoramic image requires a very high operating rate, but the ultra-high resolution of the stitched panoramic image makes the static picture image correction calculation very large, and the typical graphics card processing size is limited to below 8192 resolution, therefore, the static picture image correction in the related technology cannot be used to process the stitched panoramic image in a timely manner, and cannot be used to achieve image enhancement of the panoramic video screen.
[0004] In summary, how to achieve image enhancement processing of panoramic video images has become a problem to be solved. Summary of the invention
[0005] The embodiment of the present application provides a method for image enhancement processing, comprising: inputting a stitched panoramic image into a graphics processing unit (GPU), wherein the stitched panoramic image includes an original layer and a cover layer; and performing the following processing on the stitched panoramic image by the GPU: Parallel acquisition of the local correction matrix of the original layer and secondary rendering of the cover layer; Performing curve correction of the original layer according to the obtained local correction matrix of the original layer and the predetermined curve correction parameters; The stitched panoramic image input to the GPU includes an image in a fused panoramic video screen; the image-enhanced stitched panoramic image of the panoramic video screen is obtained based on the original layer after curve correction and the cover layer after secondary rendering.
[0006] On the other hand, an embodiment of the present application further provides a computer storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned image enhancement processing method is implemented.
[0007] On the other hand, an embodiment of the present application further provides a terminal, comprising: a memory and a processor, wherein the memory stores a computer program; wherein: The processor is configured to execute the computer program in the memory; When the computer program is executed by the processor, the above-mentioned image enhancement processing method is implemented.
[0008] In another aspect, the embodiment of the present application further provides an apparatus for image enhancement processing, comprising: an input unit and a GPU unit; wherein: The input unit is configured to: input the stitched panoramic image to a graphics processing unit (GPU), wherein the stitched panoramic image includes an original layer and a cover layer; The GPU unit includes a parallel processing module and a correction module; wherein, The parallel processing module is set to: obtain the local correction matrix of the original layer in parallel and perform secondary rendering on the cover layer; The correction module is configured to: perform curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters; The stitched panoramic image input to the GPU includes an image in a fused panoramic video screen; the image-enhanced stitched panoramic image of the panoramic video screen is obtained based on the original layer after curve correction and the cover layer after secondary rendering.
[0009] The disclosed embodiment replaces the static picture image correction based on feature transformation performed on the central processing unit with an image correction process performed by the GPU including two processing steps of curve correction of the original layer and secondary rendering of the cover layer. A stitched panoramic image with image enhancement of the panoramic video picture is obtained based on the curve-corrected original layer and the secondary rendering of the cover layer. The efficient image processing capability of the GPU is utilized to parallelly process the acquisition of the local correction matrix of the original layer and the secondary rendering of the cover layer, thereby realizing fast processing of the static picture image correction, which can be completed in the frame processing interval of the panoramic video picture. In addition, the static picture image correction of the stitched panoramic image implemented based on the GPU realizes image enhancement of the panoramic video picture.
[0010] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.
[0012] Figure 1 A flowchart of the method for image enhancement processing according to an embodiment of the present disclosure; Figure 2 A schematic diagram of a stitched panoramic image inputted by an embodiment of the present disclosure; Figure 3 A schematic diagram of a stitched panoramic image rendered twice according to an embodiment of the present disclosure; Figure 4 A schematic diagram of an image mapping process performed in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of an image before fusion processing according to the first embodiment of the present disclosure; Figure 6 This is another schematic diagram of an image before fusion processing according to an embodiment of the present disclosure; Figure 7 A schematic diagram of viewing angle correction according to an embodiment of the present disclosure; Figure 8 The figure is a structural block diagram of the image enhancement processing device according to the embodiment of the present disclosure. DETAILED DESCRIPTION
[0013] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.
[0014] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.
[0015] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.
[0016] The inventors of the present application have analyzed and found that in order to achieve image enhancement processing of panoramic video pictures by performing static picture image correction on the stitched panoramic images, the static picture image correction processing needs to have a very high operating rate, but the ultra-high resolution of the stitched panoramic images increases the dimension of the local correction matrix involved in the static picture image correction process by more than ten times. The expansion of the dimension of the local correction matrix increases the calculation amount and complexity of the CPU by more than ten times, and its calculation time is much longer than the frame processing interval of the fusion to obtain the panoramic video picture. Therefore, static picture image correction cannot be applied to the stitched panoramic image processing in the process of fusing panoramic video pictures, and cannot be used to achieve image enhancement of panoramic video pictures.
[0017] Figure 1 Flow chart of the method for image enhancement processing according to an embodiment of the present disclosure. Figure 1 As shown, it includes: inputting the stitched panoramic image into a graphics processing unit (GPU), wherein the stitched panoramic image includes an original layer and a cover layer; and performing the following processing on the stitched panoramic image by the GPU: Step 101, obtaining the local correction matrix of the original layer and performing secondary rendering on the cover layer in parallel; Step 102: performing curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters; The stitched panoramic image input to the GPU includes images in the fused panoramic video screen; the image-enhanced stitched panoramic image of the panoramic video screen is obtained based on the original layer after curve correction and the cover layer after secondary rendering.
[0018] The disclosed embodiment replaces the static picture image correction based on feature transformation performed on the central processing unit with an image correction process performed by the GPU including two processing steps of curve correction of the original layer and secondary rendering of the cover layer. A stitched panoramic image with image enhancement of the panoramic video picture is obtained based on the curve-corrected original layer and the secondary rendering of the cover layer. The efficient image processing capability of the GPU is utilized to parallelly process the acquisition of the local correction matrix of the original layer and the secondary rendering of the cover layer, thereby realizing fast processing of the static picture image correction, which can be completed in the frame processing interval of the panoramic video picture. In addition, the static picture image correction of the stitched panoramic image implemented based on the GPU realizes image enhancement of the panoramic video picture.
[0019] The embodiments of the present disclosure may be applicable to stitched panoramic images including fusion or ultra-large resolution such as 13802x2956; in view of the problems of picture distortion and brightness mottle of stitched panoramic images at a panoramic scale, image correction of stitched panoramic images is achieved based on the method of the embodiments of the present disclosure.
[0020] In an exemplary embodiment, the curve correction parameters in the embodiment of the present disclosure can be set by a technician based on experience, and can be adjusted by the image quality of the stitched panoramic image after image correction.
[0021] In an exemplary embodiment, the disclosed embodiment can set the interval for performing the above-mentioned image enhancement processing according to the usage scenario. For example, for places including airports with high real-time and picture requirements, image enhancement processing is performed on each frame of the stitched panoramic image; for places including docks with low real-time and picture requirements, the above-mentioned image enhancement processing is performed every time interval of a preset number of frames. The image enhancement processing of the disclosed embodiment can adapt to image changes at various resolution scales, especially at resolutions exceeding the hardware limit of 8192 pixels, it still has a reliable image imaging effect.
[0022] In an exemplary embodiment, the embodiment of the present disclosure inputs the stitched panoramic image into the GPU through CUDA or shared memory.
[0023] In an exemplary embodiment, the embodiment of the present disclosure obtains a local correction matrix of an original layer, including: The local correction matrix is obtained by applying a local image distortion algorithm to the original layer.
[0024] In an exemplary embodiment, the embodiment of the present disclosure performs secondary rendering on the cover layer, including: Perform secondary rendering on the cover layer according to the pre-obtained 3A parameters; Among them, 3A parameters include: Auto Focus (AF), Auto Exposure (AE) and Auto White Balance (AWB).
[0025] In an exemplary embodiment, the embodiment of the present disclosure performs secondary rendering on the cover layer according to the pre-obtained 3A parameters, including: The brightness of the cover layer is smoothed and unified based on the exposure value of the automatic exposure in the 3A parameters to achieve secondary rendering.
[0026] Figure 2 A schematic diagram of a stitched panoramic image inputted by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown in the figure, the imbalance of automatic exposure will directly lead to the appearance of light spots with varying brightness in the picture, which has a great impact on the imaging effect of the full-frame picture. Figure 3 Schematic diagram of a stitched panoramic image rendered twice according to an embodiment of the present disclosure, such as Figure 3 As shown, the imaging quality of the stitched panoramic image is improved through the secondary rendering of the embodiment of the present disclosure.
[0027] In an exemplary embodiment, the method of the embodiment of the present disclosure further includes performing one or any combination of the following processing in parallel through a GPU: Affine, projection, polar coordinate transformation of image deformation, histogram equalization sharpening, grayscale and contour detection and other processing.
[0028] In an exemplary embodiment, the embodiment of the present disclosure performs curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters, including: According to the obtained local correction matrix of the original layer and the predetermined curve correction parameters, the curve correction of the original layer is performed using the Thin Plate Spline (TPS) method.
[0029] Due to the distortion of the image caused by stitching, in order to ensure that the geometric distortion of the image can be correctly corrected, the disclosed embodiment uses curve correction technology including the TPS method. The following is a brief description of the TPS method: using the TPS (Thine Plate Spline) method, the local nonlinear transformation parameters of the stitched panoramic image are determined by an array of control points; the TPS transformation process includes calculating theta (transformation parameter), and then solving the offset (offset) of the image plane in the x and y directions through the tps_grid function; in this process, the uniform_grid function creates a uniform grid whose value is between 0 and 1 for linear interpolation algorithm; the input of the tps_grid function includes the change parameter, the local capture control points on the target template, and the resolution of the local reference image; the output is a grid grid, which contains the new position of each pixel after the transformation. See Figure 4 , the embodiment of the present disclosure calculates the mapped coordinates of the four upper, lower, left and right boundaries of the image, and then determines the coordinates of the upper left and upper right corners of the transformed image; the graphic is back-projected onto the transformed image using back projection, and the pixel calculation uses two-dimensional linear interpolation. The back projection process involves the intrinsic and extrinsic parameters of the camera, as well as the coordinate transformation of the image. The curve correction process of the embodiment of the present disclosure segments the ultra-high resolution image into appropriate proportions according to size, and then performs local algorithm restoration in the domain of interest, leaving the correction matrix parameters. It should be noted that the local algorithm restoration adjustment of the embodiment of the present disclosure can be interactive, and the technician can perform reasonable image segmentation by inputting interactive instructions, calculate one by one, and then leave all the calculation parameters. This parameter is used in real-time processing to perform real-time image enhancement.
[0030] In an exemplary embodiment, the method of the embodiment of the present disclosure further includes: before fusing the images in the panoramic video screen, performing position calibration on the video images input by different cameras.
[0031] The disclosed embodiments improve the consistency of multi-camera input images by performing position calibration preprocessing on multi-input stream images through searching and confirming the image differences between different camera input streams, including but not limited to different zoom sizes, tilts and azimuth angles between video images caused by differences in installation design and physical placement of cameras, and differences in automatic exposure values and 3A parameters caused by the amount of incoming light.
[0032] In an exemplary embodiment, the method of the present disclosure further includes: The local correction matrix of the original layer is obtained in parallel and the perspective correction is performed on the stitched panoramic image before the secondary rendering of the cover layer.
[0033] In an exemplary embodiment, the perspective correction performed by the embodiment of the present disclosure includes: performing perspective transformation and affine transformation on the stitched panoramic image, typically using the getRotationMatrix2D function and the warpAffine function to perform perspective rotation, Figure 5 and Figure 6 is the image before fusion processing, Figure 7 Schematic diagram of viewing angle correction according to an embodiment of the present disclosure, Figure 7 As shown, the embodiment of the present disclosure provides smoother picture transition and better transition continuity for image fusion through perspective correction.
[0034] The disclosed embodiment performs an image enhancement processing test on an Nvidia GPU of model RTXA4000. When the resolution of the stitched panoramic image is 13802 pixels * 2956 pixels, the average processing speed of each frame of the stitched panoramic image is less than 20 milliseconds, and real-time adaptive adjustment can be achieved during the automatic white balance adjustment of the camera. While ensuring the smoothness of the output panoramic video stream, it can adapt to various lighting environments, thereby improving the client's user experience.
[0035] The embodiment of the present disclosure also provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image enhancement processing method is implemented.
[0036] The embodiment of the present disclosure further provides a terminal, comprising: a memory and a processor, wherein a computer program is stored in the memory; The processor is configured to execute the computer program in the memory; When the computer program is executed by a processor, the above-mentioned image enhancement processing method is implemented.
[0037] Figure 8 is a structural block diagram of the image enhancement processing device according to an embodiment of the present disclosure, such as Figure 8 As shown, it includes: an input unit and a GPU unit; wherein, The input unit is configured to: input the stitched panoramic image to a graphics processing unit (GPU), wherein the stitched panoramic image includes an original layer and a cover layer; The GPU unit includes a parallel processing module and a correction module; wherein, The parallel processing module is set to: obtain the local correction matrix of the original layer in parallel and perform secondary rendering on the cover layer; The correction module is configured to: perform curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters; The stitched panoramic image input to the GPU includes images in the fused panoramic video screen; the image-enhanced stitched panoramic image of the panoramic video screen is obtained based on the original layer after curve correction and the cover layer after secondary rendering.
[0038] In an exemplary embodiment, the input unit of the embodiment of the present disclosure is configured as follows: The stitched panoramic image is input into the GPU via the unified computing architecture CUDA or shared memory.
[0039] In an exemplary embodiment, the parallel processing module of the embodiment of the present disclosure is configured to obtain a local correction matrix of the original layer, including: The local correction matrix is obtained by applying a local image distortion algorithm to the original layer.
[0040] In an exemplary embodiment, the parallel processing module of the embodiment of the present disclosure is configured to perform secondary rendering on the cover layer, including: Perform secondary rendering on the cover layer according to the pre-obtained 3A parameters; Among them, the 3A parameters include: auto focus AF, automatic exposure AE and automatic white balance AWB.
[0041] In an exemplary embodiment, the parallel processing module of the embodiment of the present disclosure is further configured to perform one or any combination of the following processing on the stitched panoramic image in parallel through the GPU: Image deformation using affine, projection, and polar coordinate transformations, histogram equalization, sharpening, grayscaling, and contour detection.
[0042] In an exemplary embodiment, the correction module of the embodiment of the present disclosure is configured to perform curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters, including: According to the obtained local correction matrix of the original layer and the predetermined curve correction parameters, the curve correction of the original layer is performed using the TPS method.
[0043] In an exemplary embodiment, the apparatus of the embodiment of the present disclosure further includes a preprocessing unit configured to: Before fusing the images in the panoramic video screen, position calibration is performed on the video images input by different cameras; and / or, The local correction matrix of the original layer is obtained in parallel and the perspective correction is performed on the stitched panoramic image before the secondary rendering of the cover layer.
[0044] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
Claims
1. A method for image enhancement processing, characterized in that: include: The stitched panoramic image is input to a graphics processing unit (GPU), wherein the stitched panoramic image includes an original layer and a cover layer. The GPU performs the following processing on the stitched panoramic image: Parallel acquisition of the local correction matrix of the original layer and secondary rendering of the cover layer; Performing curve correction of the original layer according to the obtained local correction matrix of the original layer and the predetermined curve correction parameters; The stitched panoramic image input to the GPU includes an image in a fused panoramic video screen; the image-enhanced stitched panoramic image of the panoramic video screen is obtained based on the original layer after curve correction and the cover layer after secondary rendering.
2. The method according to claim 1, characterized in that The stitched panoramic image is input into the GPU via the unified computing architecture CUDA or shared memory.
3. The method according to claim 1, characterized in that The obtaining of the local correction matrix of the original layer includes: The local correction matrix is obtained by applying a local image distortion algorithm to the original layer.
4. The method according to claim 1, characterized in that: The second rendering of the cover layer includes: Performing secondary rendering on the cover layer according to the pre-obtained 3A parameters; The 3A parameters include: auto focus AF, automatic exposure AE and automatic white balance AWB.
5. The method according to any one of claims 1 to 4, characterized in that: The method further includes performing one or any combination of the following processing on the stitched panoramic image in parallel by a GPU: Image deformation using affine, projection, and polar coordinate transformations, histogram equalization, sharpening, grayscaling, and contour detection.
6. The method according to any one of claims 1 to 4, characterized in that: The method of performing curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters includes: According to the obtained local correction matrix of the original layer and the predetermined curve correction parameters, the curve correction of the original layer is performed using the TPS method.
7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Before fusing the images in the panoramic video screen, position calibration is performed on the video images input by different cameras; and / or, Before obtaining the local correction matrix of the original layer and re-rendering the cover layer in parallel, the stitched panoramic image is corrected for perspective.
8. A computer storage medium, wherein a computer program is stored in the computer storage medium, and when the computer program is executed by a processor, the image enhancement processing method according to any one of claims 1 to 7 is implemented.
9. A terminal, comprising: A memory and a processor, wherein the memory stores a computer program; wherein, The processor is configured to execute the computer program in the memory; When the computer program is executed by the processor, the image enhancement processing method according to any one of claims 1 to 7 is implemented.
10. An image enhancement processing device, comprising: Input unit and GPU unit; wherein, The input unit is configured to: input the stitched panoramic image to a graphics processing unit (GPU), wherein the stitched panoramic image includes an original layer and a cover layer; The GPU unit includes a parallel processing module and a correction module; wherein, The parallel processing module is set to: obtain the local correction matrix of the original layer in parallel and perform secondary rendering on the cover layer; The correction module is configured to: perform curve correction of the original layer according to the acquired local correction matrix of the original layer and the predetermined curve correction parameters; The stitched panoramic image input to the GPU includes an image in a fused panoramic video screen; the image-enhanced stitched panoramic image of the panoramic video screen is obtained based on the original layer after curve correction and the cover layer after secondary rendering.