Panoramic imaging method and system based on distributed cameras

Abstract

The invention discloses a panoramic imaging method and system based on distributed cameras, and the method comprises the steps: carrying out the shooting of a target region, and obtaining all local videos; processing each local video to obtain each local grayscale image; calculating a neighborhood texture fuzzy index according to the difference characteristic between the neighborhood contrast and the overall contrast of each pixel point in each local grayscale image and the neighborhood texture similarity characteristic of each pixel point, and performing fuzzy mode judgment based on the distribution characteristic of all the neighborhood texture fuzzy indexes of each local grayscale image; a fuzzy mode discrimination result is obtained; performing adaptive motion blur restoration processing on each local image according to the blur mode discrimination result to obtain each local restoration image; and carrying out image splicing on the local restoration images at the same moment to obtain a panoramic image. According to the panoramic imaging method and system based on the distributed camera provided by the embodiment of the invention, the imaging effect of panoramic imaging in a dynamic scene is improved.

Description

Technical Field

[0001] This invention relates to the field of panoramic imaging technology, and in particular to a panoramic imaging method and system based on distributed cameras. Background Technology

[0002] The basic principle of panoramic imaging is to generate a complete, all-around view by matching the corner points of multiple images and stitching the images together, avoiding information loss. This technology can capture a wider field of view than traditional cameras, thus providing richer environmental information, allowing users to immerse themselves in the overall space through panoramic images.

[0003] However, since panoramic imaging technology relies on corner matching between multiple images, and image stitching in dynamic scenes is affected by object movement and camera movement, resulting in blurring, corner matching errors and stitching misalignments, the imaging effect of panoramic imaging is poor.

[0004] Therefore, improving the imaging effect of panoramic imaging in dynamic scenes has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides a panoramic imaging method and system based on distributed cameras to solve the technical problem that panoramic imaging technology relies on corner matching between multiple images, and that image stitching in dynamic scenes is affected by object movement and camera movement, resulting in blurring, corner matching errors, and stitching misalignment, thus leading to poor panoramic imaging results.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a panoramic imaging method based on distributed cameras, comprising:

[0007] Each camera in the distributed camera system captures images of the target area to obtain local videos of the target area; image frames are extracted from each local video to obtain local images; and each local image is converted to grayscale to obtain local grayscale images.

[0008] The neighborhood texture blur index of each pixel is calculated based on the difference between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity of each pixel in each local grayscale image. The neighborhood texture blur index reflects the degree of texture blur in the neighborhood of the pixel.

[0009] Based on the distribution characteristics of the neighborhood texture blur index of each local grayscale image, the blur mode of each local image is determined to obtain the blur mode discrimination result of each local image; the blur mode discrimination result of each local image is used to perform adaptive motion blur restoration processing on each local image to obtain each restored local image.

[0010] By stitching together the locally restored images of adjacent cameras in the distributed camera system at the same time, a panoramic image of the target area is obtained.

[0011] As a preferred embodiment, the step of calculating the neighborhood texture blur index of each pixel based on the difference between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity feature of each pixel in each local grayscale image, includes:

[0012] With each pixel in each local grayscale image as the center and a preset length as the side length, a neighborhood window is constructed for each pixel, and the remaining pixels in the neighborhood window of each pixel are taken as the neighboring pixels of each pixel.

[0013] A first comparison analysis is performed on the neighborhood contrast of each pixel in the neighborhood window within each local grayscale image and the overall contrast of each local grayscale image to calculate the neighborhood contrast weakening factor of each pixel.

[0014] A second comparison analysis is performed on the texture features of each pixel in each local grayscale image and the texture features of the neighboring pixels of each pixel to calculate the neighborhood texture similarity index of each pixel.

[0015] Based on the linear relationship between the neighborhood contrast weakening factor of each pixel and the neighborhood texture similarity index of each pixel, the neighborhood texture blur index of each pixel is calculated.

[0016] As one preferred embodiment, the fuzzy pattern determination is designed as follows:

[0017] Based on the neighborhood texture blur index of each pixel in each local grayscale image, all pixels in each local grayscale image are filtered to obtain each weak texture pixel in each local grayscale image;

[0018] The blur mode discrimination result of each local image is determined based on the proportion of weak texture pixels in each local grayscale image, wherein the blur mode discrimination result includes no motion blur, local motion blur and global motion blur.

[0019] As a preferred embodiment, the step of filtering all pixels in each local grayscale image based on the neighborhood texture blur index of each pixel in each local grayscale image to obtain each weak texture pixel in each local grayscale image includes:

[0020] The gray value of each pixel in each local grayscale image is replaced with the neighborhood texture blur index of each pixel to obtain a texture blur saliency image corresponding to each local grayscale image;

[0021] The Otsu's method is used to perform image segmentation on each of the texture-blurred and significant images to obtain local binary images.

[0022] Pixels with a value of 1 in each of the local binary images are designated as weak texture pixels.

[0023] As one preferred embodiment, determining the blur mode discrimination result of each local image based on the proportion of weak texture pixels in each local grayscale image includes:

[0024] Calculate the ratio of the number of weak texture pixels in the local grayscale image corresponding to the current local image to the total number of pixels in the local grayscale image;

[0025] If the ratio is less than or equal to a preset first threshold, the blur pattern determination result of the current local image is no motion blur; if the ratio is greater than the preset first threshold and less than or equal to a preset second threshold, the blur pattern determination result of the current local image is local motion blur; if the ratio is greater than the preset second threshold, the blur pattern determination result of the current local image is global motion blur, wherein the preset first threshold is less than the preset second threshold.

[0026] As one preferred embodiment, the step of performing adaptive motion blur restoration processing on each local image based on the blur pattern discrimination result of each local image to obtain each restored local image includes:

[0027] If the blur pattern determination result of the current local image is no motion blur, then no motion blur restoration is performed on the current local image;

[0028] If the blur mode determination result of the current local image is the global motion blur, then the global motion blur restoration process is performed on the current local image, wherein the global motion blur restoration process is designed to use a motion blur image restoration model to perform overall motion blur restoration on the current local image;

[0029] If the blur mode determination result of the current local image is local motion blur, then a region-based motion blur restoration process is performed on the current local image. The region-based motion blur restoration process is designed to extract each motion blur region in the current local image and perform motion blur restoration on each motion blur region separately.

[0030] As one preferred embodiment, the regional motion blur restoration process is designed as follows:

[0031] Based on the difference features between adjacent local grayscale images of each extracted local video, each motion region within a local grayscale image is determined;

[0032] The motion blur index of each motion region is calculated based on the normal gradient feature of the edge line of each motion region and the neighborhood texture blur index of all pixels within the region, wherein the motion blur index reflects the degree of motion blur of the motion region;

[0033] The motion blur index of each motion region in each local image is normalized to obtain the motion blur normalization index of each motion region. Based on the comparison and analysis results of the motion blur normalization index of all motion regions and the preset blur threshold, each motion blur region in each local image is determined.

[0034] A motion blur image restoration model is used to restore the motion blur of each motion blur region in each local image, thereby obtaining a local restored image corresponding to each local image.

[0035] As one preferred embodiment, the step of calculating the motion blur index of each motion region based on the normal gradient features of the edge lines of each motion region and the neighborhood texture blur index of all pixels within the region includes:

[0036] For all edge pixels on the edge line of each motion region, a radial gray-level gradient sequence for each edge pixel is constructed based on the distribution characteristics of gray-level values ​​on the outer normal of each edge pixel.

[0037] The radial gray-level gradient sequence of each edge pixel is subjected to first-order difference to obtain the radial gray-level difference sequence of each edge pixel. The Hurst exponent of the radial gray-level difference sequence of each edge pixel is used as the gradient gradient persistence factor of each edge pixel. The gradient gradient persistence factor reflects the persistence characteristic of the outward gradient gradient of each edge pixel.

[0038] The gradient degradation blur index of each edge pixel is calculated based on the linear relationship between the gradient gradient duration factor and the gradient change rate of the gradient asymptote of each edge pixel; the gradient degradation blur indexes of all edge pixels in each motion region are sorted, and each blurred edge pixel in each motion region is determined based on the sorting result;

[0039] The motion blur index of each motion region is calculated based on the linear relationship between the gradient degradation blur index of all blurred edge pixels in each motion region and the neighborhood texture blur index of all pixels within each motion region.

[0040] As one preferred embodiment, the step of constructing a radial grayscale gradient sequence for each edge pixel based on the distribution characteristics of the grayscale values ​​along the outer normal of each edge pixel includes:

[0041] All pixels on the outer normal of each edge pixel are used as gradient analysis pixels for each edge pixel; the absolute value of the difference between the gray value of each gradient analysis pixel on the outer normal and the gray value of the next adjacent gradient analysis pixel is used as the gradient descent exponent for each gradient analysis pixel.

[0042] The first gradient descent exponent on the outer normal of each edge pixel is less than a preset threshold, and the zero gradient pixel of each edge pixel is taken as the gradient asymptote of each edge pixel.

[0043] Arrange the gray values ​​of all pixels on the gradient asymptote of each edge pixel in ascending order according to their distance from each edge pixel to obtain the radial gray-level gradient sequence of each edge pixel.

[0044] Another embodiment of the present invention provides a panoramic imaging system based on distributed cameras, implementing the panoramic imaging method based on distributed cameras as described above, including:

[0045] A distributed image acquisition module is used to capture images of a target area using each camera in a distributed camera system, thereby obtaining local videos of the target area; to extract image frames from each local video to obtain local images; and to perform grayscale processing on each local image to obtain local grayscale images.

[0046] The single-point blur estimation module is used to calculate the neighborhood texture blur index of each pixel based on the difference features between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity features of each pixel in each local grayscale image, wherein the neighborhood texture blur index reflects the degree of texture blur within the neighborhood of the pixel.

[0047] An adaptive image restoration module is used to determine the blur mode of each local image based on the distribution characteristics of the texture blur index of all the neighborhoods of each local grayscale image, and obtain the blur mode discrimination result of each local image; and to perform adaptive motion blur restoration processing on each local image based on the blur mode discrimination result of each local image, to obtain each restored local image.

[0048] The panoramic image stitching module is used to stitch together the local restored images of adjacent cameras in the distributed camera system at the same time to obtain a panoramic image of the target area.

[0049] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:

[0050] First, by analyzing the differences between the neighborhood contrast and overall contrast of each pixel in each local grayscale image, as well as the neighborhood texture similarity features of each pixel, the neighborhood texture blur index of each pixel is calculated to quantify the degree of texture blur within the neighborhood of each pixel. Then, the grayscale value of each pixel in each local grayscale image is replaced with the neighborhood texture blur index of each pixel to obtain a texture blur significant image corresponding to each local grayscale image. The less clear motion blur areas in the local grayscale images are highlighted, and the pixels in the motion blur areas, i.e., weak texture pixels, are initially screened out. Finally, the proportion of weak texture pixels in each local grayscale image is used to determine the local image. The system identifies the blur pattern and performs adaptive motion blur restoration on each local image to obtain various restored local images. For local images with only local motion blur, it first extracts each motion blur region in each local image and then performs adaptive motion blur restoration on each motion blur region separately. This solves the problem that the globally unified deblurring algorithm is difficult to handle mixed clear and blur regions in the image, which easily leads to overfitting of the clear part or under-restoration of the blur part, thus improving the motion blur restoration effect of local images. Finally, the system stitches together the restored local images of adjacent cameras in the distributed camera system at the same time to obtain a panoramic image of the target area, improving the imaging effect of panoramic imaging in dynamic scenes. Attached Figure Description

[0051] Figure 1 This is a flowchart of a panoramic imaging method based on a distributed camera in one embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of a zero-gradient pixel and a gradient asymptote in one embodiment of the present invention;

[0053] Figure 3 This is a structural block diagram of a panoramic imaging system based on a distributed camera in one embodiment of the present invention;

[0054] Figure label:

[0055] A. Edge pixels; B. Zero-gradient pixels; L. Gradient asymptote; 11. Distributed image acquisition module; 12. Single-point blur estimation module; 13. Adaptive image restoration module; 14. Panoramic image stitching module. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0057] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0058] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0059] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0060] One embodiment of the present invention provides a panoramic imaging method based on distributed cameras. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 The flowchart shown is a panoramic imaging method based on a distributed camera according to one embodiment of the present invention, including steps S1 to S4, as follows:

[0061] Step S1: Each camera in the distributed camera system captures images of the target area to obtain local videos of the target area; each local video is used to extract image frames to obtain local images; each local image is then converted to grayscale to obtain local grayscale images.

[0062] Specifically, time synchronization technology is first used to control all cameras in the distributed camera system to simultaneously capture images of the target area, obtaining various local videos of the target area. Then, FFmpeg is used to extract image frames from each local video to obtain various local images. Finally, each local image is processed into grayscale to obtain various local grayscale images.

[0063] It should be noted that FFmpeg is a powerful open-source multimedia processing tool and development kit, mainly used for tasks such as audio and video encoding and decoding, format conversion, and streaming media processing. It can decompose video into a sequence of images.

[0064] In one embodiment, the time synchronization technology includes hardware-triggered synchronization and software-controlled synchronization. Hardware-triggered synchronization sends a unified electrical signal to all cameras via external hardware to trigger exposure, while software-controlled synchronization sends synchronization acquisition commands to all cameras in the distributed camera system via host software.

[0065] Step S2: Calculate the neighborhood texture blur index of each pixel based on the difference between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity feature of each pixel in each local grayscale image. The neighborhood texture blur index reflects the degree of texture blurring within the neighborhood of the pixel.

[0066] Considering that in short exposure times and small areas, the motion of objects can usually be approximated as uniform linear motion, resulting in low contrast of each pixel within the motion area and similar texture features in the neighborhood, by analyzing the difference between the neighborhood contrast of each pixel and the overall image contrast, as well as the similarity of textures in the neighborhood of each pixel, we can initially screen pixels located in motion-blurred areas.

[0067] Specifically, a neighborhood window is constructed with each pixel as the center and a preset length as the side length. The remaining pixels within each pixel's neighborhood window are then used as the neighboring pixels of that pixel.

[0068] First, a comparison analysis is performed on the neighborhood contrast of each pixel in each local grayscale image within the neighborhood window and the overall contrast of each local grayscale image to calculate the neighborhood contrast weakening factor of each pixel.

[0069] In one embodiment, the first comparison analysis includes: taking the range of gray values ​​of all neighboring pixels of each pixel as the neighborhood contrast of each pixel, and taking the range of gray values ​​of all pixels in each local grayscale image as the overall contrast; inputting the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image into a weak contrast discrimination expression to calculate the neighborhood contrast weakening factor of each pixel in each local grayscale image.

[0070] It should be noted that the preset length is an odd number preset by the user, and the center pixel of the neighborhood window is the edge pixel. In this embodiment, the preset length is 9. For other implementation methods, the implementer can choose the desired value for the preset length. The weak contrast discrimination expression is as follows:

[0071] ConLower(P,x)=max{AllCon(P)-LocalCon(P,x),0}

[0072] Where ConLower(P,x) is the neighborhood contrast weakening factor of pixel x in local grayscale image P, AllCon(P) is the overall contrast of local grayscale image P, LocalCon(P,x) is the neighborhood contrast of pixel x in local grayscale image P, and max{} is the maximum value function.

[0073] It should be noted that when the neighborhood contrast of a pixel is less than the overall contrast of the corresponding local grayscale image, the larger the difference, the more likely the contrast in the neighborhood of the pixel is to be weakened due to motion blur, and the larger the output value of the weak contrast discrimination expression. When the neighborhood contrast of pixel x is greater than or equal to the overall contrast of the corresponding local grayscale image P, the value of AllCon(P)-LocalCon(P,x) is less than or equal to 0, indicating that the contrast in the neighborhood of pixel x is significant, and the possibility of the contrast being weakened due to motion blur is small, and the output value of the weak contrast discrimination expression ConLower(P,x) is 0.

[0074] Then, a second comparison analysis is performed on the texture features of each pixel in each local grayscale image and the texture features of the corresponding neighboring pixels to calculate the neighborhood texture similarity index of each pixel.

[0075] In one embodiment, the second comparison analysis includes: calculating the LBP operator for each pixel in each local grayscale image; using the Hamming distance between the first LBP operator of the current pixel and the second LBP operator of each neighboring pixel as the texture difference value between the current pixel and each neighboring pixel; using the average of the texture difference values ​​between the current pixel and all neighboring pixels as the neighborhood texture difference feature value of the current pixel; and performing inverse proportional normalization on the neighborhood texture difference feature values ​​of all pixels in each local grayscale image to obtain the neighborhood texture similarity index of each pixel.

[0076] It should be noted that the LBP operator uses the center pixel of the window as the threshold and compares the gray values ​​of the 8 adjacent pixels with it. If the gray value of the surrounding pixels is greater than the gray value of the center pixel, the pixel is marked as 1; otherwise, it is marked as 0. The first LBP operator and the second LBP operator are used to distinguish the texture features of the current pixel and the neighboring pixels.

[0077] In one embodiment, the neighborhood texture difference feature values ​​of all pixels in each local grayscale image are inversely normalized to obtain the neighborhood texture similarity index of each pixel. Specifically, this includes: taking the maximum value of the neighborhood texture difference feature values ​​of all pixels in each local grayscale image as the texture difference reference value of each local grayscale image; taking the ratio of the neighborhood texture difference feature value of each pixel in each local grayscale image to the corresponding texture difference reference value of the local grayscale image as the texture difference ratio of each pixel in each local grayscale image; and taking the difference between the value 1 and the texture difference ratio of each pixel as the neighborhood texture similarity index of each pixel.

[0078] It should be noted that inverse proportional normalization processing includes both normalization and inverse proportional processing. Normalization is achieved by calculating the ratio of the neighborhood texture difference feature value of each pixel in each local grayscale image to the corresponding texture difference reference value of the local grayscale image. Inverse proportional processing is achieved by calculating the difference between the value 1 and the texture difference ratio of each pixel. The closer the ratio between the neighborhood texture difference feature value of a pixel and the maximum value of the neighborhood texture difference feature values ​​of all pixels in the overall local grayscale image is to 1, the more significant the neighborhood texture difference of the pixel, the less similar the textures, and the closer the neighborhood texture similarity index is to 0, the lower the probability of motion blur.

[0079] Furthermore, based on the linear relationship between the neighborhood contrast weakening factor of each pixel and the neighborhood texture similarity index of each pixel, the neighborhood texture blur index of each pixel is calculated.

[0080] It should be noted that when motion blur occurs, the edges of objects extend along the direction of motion into linear or arc-shaped blurred trajectories, similar to the "ghosting" effect. If the texture features of the current pixel are more similar to the texture features of the other pixels in the neighborhood window, and the neighborhood contrast weakening factor of the current pixel is larger, it means that the textures in the neighborhood range of the current pixel are more similar and the contrast is more likely to be weakened, and motion blur is more likely to occur.

[0081] In one embodiment, the sum of the neighborhood contrast weakening factor and the corresponding neighborhood texture similarity index of each pixel is used as the neighborhood texture blur index of each pixel.

[0082] It should be noted that by comprehensively considering the contrast difference features between the neighborhood and the whole, as well as the texture similarity features within the neighborhood, the problem of missed detection or false detection caused by a single factor is avoided. For example, an image with similar overall texture and no blur is misjudged as a blurred overall image. It can accurately detect both local blur caused by object movement and overall image blur caused by camera movement.

[0083] Step S3: Based on the distribution characteristics of the texture blur index of all neighborhoods of each local grayscale image, perform blur mode judgment on each local image to obtain the blur mode discrimination result of each local image; perform adaptive motion blur restoration processing on each local image based on the blur mode discrimination result of each local image to obtain each restored local image.

[0084] It should be noted that in dynamic scenes, whether it is global blur caused by camera movement or local blur caused by the movement of external objects, adjacent pixels in the direction of movement will tend to be similar, resulting in blurry trailing and ghosting phenomena in the acquired local images. In order to avoid interference with the image stitching process of panoramic imaging, it is necessary to first determine the blur mode discrimination result of each local image and perform adaptive motion blur restoration processing.

[0085] It should be further explained that the blur mode judgment is designed to filter all pixels in each local grayscale image based on the neighborhood texture blur index of each pixel in each local grayscale image to obtain each weak texture pixel in each local grayscale image; and determine the blur mode judgment result of each local image based on the proportion of weak texture pixels in each local grayscale image. The blur mode judgment result includes no motion blur, local motion blur and global motion blur.

[0086] Specifically, the gray value of each pixel in each local grayscale image is replaced with the neighborhood texture blur index of each pixel to obtain a texture blur saliency image corresponding to each local grayscale image; the Otsu's method is used to perform image segmentation processing on each texture blur saliency image to obtain each local binary image; and the pixels with a pixel value of 1 in each local binary image are taken as weak texture pixels.

[0087] It should be noted that Otsu's method is an algorithm for determining the image binarization segmentation threshold. It divides the image into background and foreground parts based on the principle of maximizing the inter-class variance between the foreground and background images. Otsu's method is a well-known technique and will not be elaborated upon in this embodiment. First, by replacing the gray value of each pixel in each local grayscale image with the neighborhood texture blur index, the less clear motion blur areas in the local grayscale image can be highlighted. Then, by using Otsu's method to process the image with significant texture blur, pixels in the motion blur area, i.e., weak texture pixels, can be initially filtered out.

[0088] Furthermore, the ratio of the number of weak texture pixels in each local grayscale image to the total number of pixels in each local grayscale image is calculated. If the ratio is less than or equal to a preset first threshold, the blur mode of the corresponding local image is determined to be without motion blur. If the ratio is greater than the preset first threshold and less than or equal to a preset second threshold, the blur mode of the corresponding local image is determined to be local motion blur. If the ratio is greater than the preset second threshold, the blur mode of the corresponding local image is determined to be global motion blur.

[0089] It should be noted that both the preset first threshold and the preset second threshold are manually preset values. The preset first threshold is less than the preset second threshold. The value range of the preset first threshold is 0.1 to 0.2, and the value range of the preset second threshold is 0.8 to 0.9. In this embodiment, the preset first threshold is set to 0.1, and the preset second threshold is set to 0.9.

[0090] Furthermore, adaptive motion blur restoration processing is performed on each local image based on the blur pattern discrimination result of each local image to obtain each local restored image.

[0091] Specifically, adaptive motion blur restoration processing includes:

[0092] (1) If the blur mode discrimination result is no motion blur, then no motion blur restoration is performed on the local image.

[0093] (2) If the blur mode discrimination result is global motion blur, then the global motion blur restoration process is performed on the local image. The global motion blur restoration process is designed to use the motion blur image restoration model PSF to perform overall motion blur restoration on each local image.

[0094] It should be noted that the point spread function describes the energy distribution of an ideal point light source after passing through an optical system. In motion-blurred scenes, it is represented by the brightness distribution of the motion trajectory. The motion-blurred image restoration model PSF estimates the PSF parameters of the point spread function and uses a filtering algorithm based on the estimated PSF parameters to restore the motion-blurred image. The PSF parameters include the motion blur direction and the motion blur scale.

[0095] In one embodiment, the Radon transform is used to estimate the motion ambiguity direction of the point spread function.

[0096] (3) If the blur pattern discrimination result is local motion blur, then the local image is subjected to a regional motion blur restoration process. The regional motion blur restoration process is designed to extract each motion blur region in each local image and perform adaptive motion blur restoration on each motion blur region.

[0097] Specifically, the regional motion blur restoration process includes steps S301 to S304:

[0098] Step S301: Determine the motion region of each local grayscale image based on the difference features between adjacent local grayscale images of each extracted local video.

[0099] It should be noted that in local motion blur scenes, the target area may contain multiple objects with different directions of motion and speeds. Existing motion blur restoration techniques perform motion blur restoration in a single direction on the entire image, which will aggravate noise in static areas, cause distortion of the edges of some moving objects, and thus reduce the effect of subsequent panoramic imaging.

[0100] To improve the panoramic imaging effect in dynamic scenes, each motion-blurred region in each local image is extracted, and adaptive motion blur restoration is performed on each motion-blurred region.

[0101] Specifically, each local grayscale image of each local video is differentially processed with the adjacent previous local grayscale image, and the image obtained by differential processing is used as the differential image of each local grayscale image; an edge detection algorithm is used to perform image segmentation processing on the differential image of each local grayscale image to obtain each motion region in each local grayscale image.

[0102] Step S302: Calculate the motion blur index of each motion region based on the normal gradient features of the edge line of each motion region and the neighborhood texture blur index of all pixels within the region. The motion blur index reflects the degree of motion blur of the corresponding motion region.

[0103] It should be noted that motion blur is essentially the pixel integral of an image along the direction of motion. Neighboring pixels tend to become similar due to blurring, and the gradient along the direction of motion is smoothed, while the gradient along other directions remains unchanged or undergoes minor changes. First, the gradient change rate and persistence characteristics of each edge pixel in each motion region along the outer normal are compared and analyzed to calculate the gradient degradation blur index of each edge pixel, reflecting the probability of motion blur at each edge pixel. Then, the motion blur index of each motion region is determined based on the gradient degradation blur indices of all edge pixels in each motion region and the neighborhood texture blur indices of all pixels within each motion region.

[0104] Specifically, all pixels on the outer normal of each edge pixel in each motion region are taken as gradient analysis pixels of each edge pixel. The absolute value of the difference between the gray value of each gradient analysis pixel and the gray value of the next adjacent gradient analysis pixel is taken as the gradient descent exponent of each gradient analysis pixel. The first gradient analysis pixel on the outer normal of each edge pixel whose gradient descent exponent is less than a preset threshold is taken as the zero gradient pixel of each edge pixel. The line connecting each edge pixel and the corresponding zero gradient pixel is taken as the gradient asymptote of each edge pixel.

[0105] It should be noted that the outer normal is the curve's normal at a point P that is perpendicular to the tangent and extends outward from point P. The preset threshold is a manually set value, close to 0; in this embodiment, the preset threshold is set to 1. For details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 The diagram shown illustrates a zero-gradient pixel and a gradient asymptote in one embodiment of the present invention. Figure 2 In the diagram, point B is the zero-gradient pixel of edge pixel A, and line segment L is the gradient asymptote of edge pixel A.

[0106] It should be noted that since camera shake can also cause pixel gradients to oscillate and recover, in order to avoid the impact of camera shake on the evaluation of motion blur, the gray values ​​of all pixels on the gradient asymptote line of the current edge pixel are further arranged in ascending order according to their distance from the current edge pixel, to obtain the radial gray-level gradient sequence of each edge pixel.

[0107] For example, the gradient asymptotes L from edge pixel A to the corresponding zero gradient pixel B are A, C, D, E, F, B in sequence, and f(A) = 248, f(C) = 245, f(D) = 240, f(E) = 235, f(F) = 230, f(B) = 225. Then the radial gray-level gradient sequence V(A) of edge pixel A is [248, 245, 240, 235, 230, 225].

[0108] The radial gray-level gradient sequence of each edge pixel is first-order differencing to obtain the radial gray-level difference sequence of each edge pixel. The Hearst exponent of the radial gray-level difference sequence of each edge pixel is used as the gradient gradient persistence factor of each edge pixel. The gradient gradient persistence factor reflects the persistence characteristic of the outward gradient gradient of each edge pixel.

[0109] It should be noted that the Hearst exponent can be used to analyze whether time series data has long-term correlation, and thus measure the persistence of fluctuations in time series data. A high Hearst exponent indicates that the grayscale values ​​change continuously in one direction; a low Hearst exponent indicates that the grayscale values ​​oscillate frequently. The calculation of the Hearst exponent is a well-known technique, and will not be elaborated upon in this embodiment.

[0110] Furthermore, the gradient degradation blur index of each edge pixel is calculated based on the linear relationship between the gradient gradient duration factor and the gradient change rate of the gradient asymptote.

[0111] In one embodiment, the formula for calculating the gradient degradation blur index of each edge pixel is as follows:

[0112]

[0113] Where DeGrad(x) is the gradient degradation blur index of edge pixel x, and f(x) is the gray value of edge pixel x. Let be the gray value of the zero gradient pixel of edge pixel x, d(x) be the length of the gradient asymptote of edge pixel x, ()-1 indicates taking the reciprocal, and GradStab(x) be the gradient gradient duration factor of edge pixel x.

[0114] It should be noted that the gradient degradation blur index reflects the probability of motion blur at each edge pixel. The slower the gray value changes from the edge pixel to the corresponding zero gradient pixel, the higher the degree of blur at that edge pixel, and the more likely blurring is to occur. At the same time, the higher the gradient gradient persistence factor of the edge pixel, the more likely the blur at that edge pixel is caused by motion blur.

[0115] Furthermore, the gradient degradation blur index of all edge pixels in each motion region is sorted, and the blurry edge pixels of each motion region are determined based on the sorting results.

[0116] In one embodiment, the gradient degradation blur exponents of all edge pixels in each motion region are sorted in descending order to obtain the gradient degradation sequence of each motion region. The top n values ​​of the gradient degradation sequence of motion region z are then selected. z The edge pixels corresponding to each element are used as the blurred edge pixels of the motion region z, where n z =1 + round(per × N) z ), n z N represents the number of blurred edge pixels in the motion region z, where round() is the rounding function. z is the total number of edge pixels in the motion region z, and per is a preset ratio, which is 10% in this embodiment.

[0117] It should be noted that the rounding function is used to ensure that the number of blurred edge pixels in each motion region is an integer, and adding 1 avoids the number of blurred edge pixels being 0.

[0118] Furthermore, based on the linear relationship between the gradient degradation blur index of all blurred edge pixels in each motion region and the neighborhood texture blur index of all pixels within each motion region, the motion blur index of each motion region is calculated.

[0119] It should be noted that when the gradient degradation blur index of each blurred edge pixel in a certain motion region is larger, and the neighborhood texture blur index of each pixel inside the motion region is also larger, it indicates that the motion region is more likely to be a motion blur region with weakened internal texture and edge ghosting. By comprehensively considering the edge features and internal region features of the motion blur region, the accuracy of subsequent judgment of motion blur region is improved.

[0120] In one embodiment, the average gradient degradation blur index of all blurred edge pixels in each motion region is used as the edge blur index of each motion region, the average neighborhood texture blur index of all pixels within each motion region is used as the internal blur index of each motion region, and the sum of the edge blur index and the internal blur index of each motion region is used as the motion blur index of each motion region.

[0121] In another embodiment, the maximum value of the gradient degradation blur index of all blurred edge pixels in each motion region is taken as the edge blur index of each motion region, the median of the neighborhood texture blur index of all pixels within each motion region is taken as the internal blur index of each motion region, and the product of the edge blur index and the internal blur index of each motion region is taken as the motion blur index of each motion region.

[0122] Step S303: Normalize the motion blur index of each motion region in each local image to obtain the motion blur normalization index of each motion region. Determine each motion blur region in each local image based on the comparison and analysis results of the motion blur normalization index of all motion regions and the preset blur threshold.

[0123] In one embodiment, the Sigmoid normalization function is used to normalize the motion blur index of each moving region in each local image to obtain the motion blur normalization index of each moving region.

[0124] In another embodiment, the maximum value of the motion blur index of all moving regions in each local image is used as the motion blur reference value of each local image, and the ratio of the motion blur index of each moving region in each local image to the motion blur reference value of each local image is used as the motion blur normalization index of each moving region.

[0125] It should be noted that the preset fuzziness threshold is a value preset by the user, and the value range is 0 to 1.

[0126] In one embodiment, the preset fuzziness threshold is set to 0.7.

[0127] Step S304: The motion blur image restoration model PSF is used to restore the motion blur of each motion blur region in each local image, so as to obtain the local restored image corresponding to each local image.

[0128] Step S4: Stitch together the locally restored images of adjacent cameras in the distributed camera system at the same time to obtain a panoramic image of the target area.

[0129] It should be noted that the principle of stitching together multiple images at the same time to obtain a panoramic image is as follows: First, corner points are detected in each image. Then, the corner points in different images are associated to find the best matching point pairs, thereby calculating the transformation matrix between the images. Based on the transformation matrix, adjacent images are chained together to align overlapping areas. Finally, the images obtained from the initial stitching are fused to eliminate seams and ensure a natural transition, resulting in the final panoramic image.

[0130] By performing motion blur restoration on each image before corner detection, the accuracy of corner detection is improved. This solves the problem of blurring caused by object movement and camera movement in dynamic scenes, which leads to corner matching errors and misalignment. This ensures the accuracy of image stitching and improves the imaging effect of panoramic imaging.

[0131] The panoramic imaging method based on a distributed camera provided in this invention first analyzes the difference features between the neighborhood contrast and the overall contrast of each pixel in each local grayscale image, as well as the neighborhood texture similarity features of each pixel, to calculate the neighborhood texture blur index of each pixel, quantifying the degree of texture blur within the neighborhood of each pixel. Then, the grayscale value of each pixel in each local grayscale image is replaced with the neighborhood texture blur index of each pixel to obtain a texture blur significant image corresponding to each local grayscale image. This highlights the less clear motion blur areas in the local grayscale images, initially filtering out pixels in motion blur areas, i.e., weak texture pixels. Then, based on the weak texture pixels in each local grayscale image... The proportion of points determines the blur pattern discrimination result of each local image, and adaptive motion blur restoration processing is performed on each local image to obtain each local restored image. For local images with only local motion blur, each motion blur region in each local image is extracted first, and then adaptive motion blur restoration is performed on each motion blur region separately. This solves the problem that the globally unified deblurring algorithm is difficult to handle the mixed clear and blur regions in the image, which easily leads to overfitting of the clear part or under-restoration of the blur part, thus improving the motion blur restoration effect of local images. Finally, the local restored images of adjacent cameras in the distributed camera system at the same time are stitched together to obtain the panoramic image of the target area, which improves the imaging effect of panoramic imaging in dynamic scenes.

[0132] Another embodiment of the present invention provides a panoramic imaging system based on distributed cameras, implementing the panoramic imaging method based on distributed cameras as described above. For details, please refer to [link to relevant documentation]. Figure 3 , Figure 3 The diagram shown is a structural block diagram of a panoramic imaging system based on a distributed camera according to one embodiment of the present invention, comprising:

[0133] The distributed image acquisition module 11 is used to capture images of the target area using each camera in the distributed camera system to obtain local videos of the target area; to extract image frames from each local video to obtain local images; and to perform grayscale processing on each local image to obtain local grayscale images.

[0134] The single-point blur estimation module 12 is used to calculate the neighborhood texture blur index of each pixel based on the difference features between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity features of each pixel in each local grayscale image. The neighborhood texture blur index reflects the degree of texture blur within the neighborhood of the pixel.

[0135] The adaptive image restoration module 13 is used to determine the blur mode of each local image based on the distribution characteristics of the texture blur index of all neighborhoods of each local grayscale image, and obtain the blur mode discrimination result of each local image; and to perform adaptive motion blur restoration processing on each local image based on the blur mode discrimination result of each local image, so as to obtain each local restored image.

[0136] The panoramic image stitching module 14 is used to stitch together the local restored images of adjacent cameras in the distributed camera system at the same time to obtain a panoramic image of the target area.

[0137] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A panoramic imaging method based on distributed cameras, characterized in that, The method includes: Each camera in the distributed camera system captures images of the target area to obtain local videos of the target area; image frames are extracted from each local video to obtain local images; and each local image is converted to grayscale to obtain local grayscale images. The neighborhood texture blur index of each pixel is calculated based on the difference between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity of each pixel in each local grayscale image. The neighborhood texture blur index reflects the degree of texture blur in the neighborhood of the pixel. Based on the distribution characteristics of the neighborhood texture blur index of each local grayscale image, the blur mode of each local image is determined to obtain the blur mode discrimination result of each local image; the blur mode discrimination result of each local image is used to perform adaptive motion blur restoration processing on each local image to obtain each restored local image. By stitching together the locally restored images of adjacent cameras in the distributed camera system at the same time, a panoramic image of the target area is obtained.

2. The panoramic imaging method based on distributed cameras according to claim 1, characterized in that, The step of calculating the neighborhood texture blur index of each pixel based on the difference between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity feature of each pixel in each local grayscale image, includes: With each pixel in each local grayscale image as the center and a preset length as the side length, a neighborhood window is constructed for each pixel, and the remaining pixels in the neighborhood window of each pixel are taken as the neighboring pixels of each pixel. A first comparison analysis is performed on the neighborhood contrast of each pixel in the neighborhood window within each local grayscale image and the overall contrast of each local grayscale image to calculate the neighborhood contrast weakening factor of each pixel. A second comparison analysis is performed on the texture features of each pixel in each local grayscale image and the texture features of the neighboring pixels of each pixel to calculate the neighborhood texture similarity index of each pixel. Based on the linear relationship between the neighborhood contrast weakening factor of each pixel and the neighborhood texture similarity index of each pixel, the neighborhood texture blur index of each pixel is calculated.

3. The panoramic imaging method based on distributed cameras according to claim 1, characterized in that, The fuzzy pattern determination is designed as follows: Based on the neighborhood texture blur index of each pixel in each local grayscale image, all pixels in each local grayscale image are filtered to obtain each weak texture pixel in each local grayscale image; The blur mode discrimination result of each local image is determined based on the proportion of weak texture pixels in each local grayscale image, wherein the blur mode discrimination result includes no motion blur, local motion blur and global motion blur.

4. The panoramic imaging method based on distributed cameras according to claim 3, characterized in that, The step of filtering all pixels in each local grayscale image based on the neighborhood texture blur index of each pixel in each local grayscale image to obtain each weak texture pixel in each local grayscale image includes: The gray value of each pixel in each local grayscale image is replaced with the neighborhood texture blur index of each pixel to obtain a texture blur saliency image corresponding to each local grayscale image; The Otsu's method is used to perform image segmentation on each of the texture-blurred and significant images to obtain local binary images. Pixels with a value of 1 in each of the local binary images are designated as weak texture pixels.

5. The panoramic imaging method based on distributed cameras according to claim 3, characterized in that, The step of determining the blur mode discrimination result of each local image based on the proportion of weak texture pixels in each local grayscale image includes: Calculate the ratio of the number of weak texture pixels in the local grayscale image corresponding to the current local image to the total number of pixels in the local grayscale image; If the ratio is less than or equal to a preset first threshold, the blur pattern determination result of the current local image is no motion blur; if the ratio is greater than the preset first threshold and less than or equal to a preset second threshold, the blur pattern determination result of the current local image is local motion blur; if the ratio is greater than the preset second threshold, the blur pattern determination result of the current local image is global motion blur, wherein the preset first threshold is less than the preset second threshold.

6. The panoramic imaging method based on distributed cameras according to claim 3, characterized in that, The step of performing adaptive motion blur restoration processing on each local image based on the blur pattern discrimination result of each local image to obtain each restored local image includes: If the blur pattern determination result of the current local image is no motion blur, then no motion blur restoration is performed on the current local image; If the blur mode determination result of the current local image is the global motion blur, then the global motion blur restoration process is performed on the current local image, wherein the global motion blur restoration process is designed to use a motion blur image restoration model to perform overall motion blur restoration on the current local image; If the blur mode determination result of the current local image is local motion blur, then a region-based motion blur restoration process is performed on the current local image. The region-based motion blur restoration process is designed to extract each motion blur region in the current local image and perform motion blur restoration on each motion blur region separately.

7. The panoramic imaging method based on distributed cameras according to claim 6, characterized in that, The region-specific motion blur restoration process is designed as follows: Based on the difference features between adjacent local grayscale images of each extracted local video, each motion region within a local grayscale image is determined; The motion blur index of each motion region is calculated based on the normal gradient feature of the edge line of each motion region and the neighborhood texture blur index of all pixels within the region, wherein the motion blur index reflects the degree of motion blur of the motion region; The motion blur index of each motion region in each local image is normalized to obtain the motion blur normalization index of each motion region. Based on the comparison and analysis results of the motion blur normalization index of all motion regions and the preset blur threshold, each motion blur region in each local image is determined. A motion blur image restoration model is used to restore the motion blur of each motion blur region in each local image, thereby obtaining a local restored image corresponding to each local image.

8. The panoramic imaging method based on distributed cameras according to claim 7, characterized in that, The step of calculating the motion blur index of each motion region based on the normal gradient features of the edge lines of each motion region and the neighborhood texture blur index of all pixels within the region includes: For all edge pixels on the edge line of each motion region, a radial gray-level gradient sequence for each edge pixel is constructed based on the distribution characteristics of gray-level values ​​on the outer normal of each edge pixel. The radial gray-level gradient sequence of each edge pixel is subjected to first-order difference to obtain the radial gray-level difference sequence of each edge pixel. The Hurst exponent of the radial gray-level difference sequence of each edge pixel is used as the gradient gradient persistence factor of each edge pixel. The gradient gradient persistence factor reflects the persistence characteristic of the outward gradient gradient of each edge pixel. The gradient degradation blur index of each edge pixel is calculated based on the linear relationship between the gradient gradient duration factor and the gradient change rate of the gradient asymptote of each edge pixel; the gradient degradation blur indexes of all edge pixels in each motion region are sorted, and each blurred edge pixel in each motion region is determined based on the sorting result; The motion blur index of each motion region is calculated based on the linear relationship between the gradient degradation blur index of all blurred edge pixels in each motion region and the neighborhood texture blur index of all pixels within each motion region.

9. The panoramic imaging method based on distributed cameras according to claim 8, characterized in that, The step of constructing a radial grayscale gradient sequence for each edge pixel based on the distribution characteristics of the grayscale values ​​along the outer normal of each edge pixel includes: All pixels on the outer normal of each edge pixel are used as gradient analysis pixels for each edge pixel; the absolute value of the difference between the gray value of each gradient analysis pixel on the outer normal and the gray value of the next adjacent gradient analysis pixel is used as the gradient descent exponent for each gradient analysis pixel. The first gradient descent exponent on the outer normal of each edge pixel is less than a preset threshold, and the zero gradient pixel of each edge pixel is taken as the gradient asymptote of each edge pixel. Arrange the gray values ​​of all pixels on the gradient asymptote of each edge pixel in ascending order according to their distance from each edge pixel to obtain the radial gray-level gradient sequence of each edge pixel.

10. A panoramic imaging system based on distributed cameras, characterized in that, Implementing the panoramic imaging method based on distributed cameras as described in any one of claims 1 to 9, comprising: A distributed image acquisition module is used to capture images of a target area using each camera in a distributed camera system, thereby obtaining local videos of the target area; to extract image frames from each local video to obtain local images; and to perform grayscale processing on each local image to obtain local grayscale images. The single-point blur estimation module is used to calculate the neighborhood texture blur index of each pixel based on the difference features between the neighborhood contrast of each pixel in each local grayscale image and the overall contrast of each local grayscale image, and the neighborhood texture similarity features of each pixel in each local grayscale image, wherein the neighborhood texture blur index reflects the degree of texture blur within the neighborhood of the pixel. An adaptive image restoration module is used to determine the blur mode of each local image based on the distribution characteristics of the texture blur index of all the neighborhoods of each local grayscale image, and obtain the blur mode discrimination result of each local image; and to perform adaptive motion blur restoration processing on each local image based on the blur mode discrimination result of each local image, to obtain each restored local image. The panoramic image stitching module is used to stitch together the local restored images of adjacent cameras in the distributed camera system at the same time to obtain a panoramic image of the target area.

Images