Multi-view fusion vision monitoring method and system for long and narrow area

Abstract

The invention provides a multi-view fusion vision monitoring method and system for a long and narrow area, and relates to the technical field of image processing. According to the invention, the front-end acquisition device is arranged above the long and narrow area scene to obtain different focal length pictures of the long and narrow area scene; the image processing unit carries out formula stretching transformation processing / matrix stretching transformation processing on each focal section image, fusion splicing of multi-focus images is realized, and a multi-view fusion visual monitoring image with larger depth of field and clearer details is generated; according to the invention, the image of the spliced target area can be ensured to keep high definition from near to far, and the image can be presented in the image at similar magnification in the far and near directions; the problem that a multi-focus fusion image is large in view field but unclear in distant view or clear in distant view but small in view field is solved; therefore, the problem that when a traditional image processing technology is combined with follow-up technology fusion, a machine cannot obtain enough image details, and consequently the follow-up recognition result is affected is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-view fusion visual monitoring method and system for narrow and long areas. Background Art

[0002] With the advancement of science and technology and the development of the times, image monitoring technology is used in more and more scenarios, and combined with artificial intelligence, big data processing and other technologies, the technical effect of "1+1>2" is achieved; for example: through road monitoring images + vehicle behavior recognition technology, the automatic identification of road violations is realized; through playground monitoring images + AI face recognition technology, the automatic identification of student attendance is realized; through the safety passage scene + object recognition model, the automatic identification of illegal occupation of the safety passage is realized.

[0003] From the above examples, it is not difficult to see that the current image monitoring technology has not only remained at the stage of being understandable to "people", but has also developed to a new stage where it needs to be understood by "machines", which has brought new challenges to our image monitoring technology; however, existing image monitoring technology is often designed to meet the sensory experience of "people". This difference in the starting point of the underlying design has caused a series of contradictory things to occur when we use the existing image monitoring technology for subsequent technical integration (especially AI fusion empowerment).

[0004] Among them, the most typical is the field of large-scene monitoring. For the monitoring of narrow and long area scenes (such as highways, safety passages, etc.), the existing image monitoring technology will have the situation that the short-focus lens takes clear close-ups but blurred distant shots, while the long-focus lens takes clear distant shots but missing close-ups. The reason is that: in order to conform to the sensory intuition of "people", the existing technology uses multi-focus fusion of short-focus images and long-focus images into a panoramic image.

[0005] Although this method is consistent with human sensory intuition (content is coherent and distortion is small) and improves the clarity of distant objects compared to short-focus images, it is still very difficult to recognize and identify distant objects due to the limitations of image size and resolution. If the recognition ability of distant objects is improved by enlarging the image, the ability to observe close-up images is lost, which results in the "machine" being unable to obtain enough image details when combined with subsequent technologies, thus affecting subsequent recognition results.

[0006] Therefore, it is necessary to provide a multi-eye fusion visual monitoring method and system for narrow and long areas to solve the technical problems of existing multi-focus fusion images, such as a large field of view but unclear distant view, or a clear distant view but a small field of view. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-eye fusion visual monitoring method and system for narrow and long areas that is easy to operate, thereby solving the technical problems that the existing multi-focus fusion images have a large field of view but unclear distant view, or a clear distant view but a small field of view.

[0008] To achieve the above purpose, the present application proposes a multi-eye fusion visual monitoring system for narrow and long areas, which is used to monitor narrow and long area scenes, including a front-end acquisition device and an image processing unit; wherein,

[0009] Front-end acquisition equipment: arranged above the narrow and long area scene, used to obtain different focal length images of the narrow and long area scene; wherein, the front-end acquisition equipment is composed of a plurality of image acquisition units, which are all arranged on the same shooting position, and can independently adjust the shooting pitch angle and shooting focal length, and adjust the shooting pitch angle of each image acquisition unit so that the image contents of adjacent image acquisition units have overlapping areas;

[0010] Image processing unit: communicated with the front-end acquisition device, used for performing image stitching processing on each focal length picture, including a stretching transformation processing module and an image stitching module; wherein the stretching transformation processing module is used for performing stretching transformation processing on each focal length picture, and obtaining the target area image corresponding to each focal length picture through formula stretching transformation processing / matrix stretching transformation processing, and retaining the original image details of the target area image for output; the image stitching module sequentially stitches the target area images according to the setting order of the image acquisition unit to obtain a multi-eye fusion visual monitoring image.

[0011] On the other hand, the present invention provides a multi-eye fusion visual monitoring method for a narrow and long area, which is applied to the multi-eye fusion visual monitoring system for a narrow and long area as described above, and monitors the narrow and long area scene through the following steps:

[0012] Step 1: Obtain images of different focal lengths of a narrow and long area scene through the front-end acquisition device;

[0013] Step 2: The stretching transformation processing module is used to perform stretching transformation processing on each focal length picture, obtain the target area image corresponding to each focal length picture through formula stretching transformation processing / matrix stretching transformation processing, and retain the original image details of the target area image for output;

[0014] Step 3: The image stitching module stitches the target area images sequentially according to the setting order of the image acquisition unit to obtain a multi-eye fusion visual monitoring image.

[0015] As a further solution, the stretch transformation processing module performs formula stretch transformation processing through the following steps:

[0016] Step A1: determining a target area in each focal length image; wherein the target area is an isosceles trapezoidal area;

[0017] Step A2: Calculate the horizontal and vertical stretching ratios λ corresponding to the target area for each focal length image;

[0018] Step A3: stretching each row of pixels in the focal length image by a horizontal and vertical stretching ratio λ to keep the distance of the target area consistent;

[0019] Step A4: Execute steps A1 to A3 one by one for each focal length picture, obtain the target area image corresponding to each focal length picture, and complete the formula stretching transformation process;

[0020] The calculation formulas for the horizontal and vertical stretching ratios λ are:

[0021]

[0022] Where h is the height of the isosceles trapezoid, y is the row coordinate of the pixel, d1 is the length of the upper half of the base of the isosceles trapezoid, and d2 is the length of the lower half of the base of the isosceles trapezoid.

[0023] As a further solution, the stretch transformation processing module performs matrix stretch transformation processing through the following steps:

[0024] Step B1: determining a target area in each focal length image, and selecting four transformation feature points of the target area; wherein the target area is a trapezoidal area, including an upper left point of the trapezoidal area, an upper right point of the trapezoidal area, a lower point of the trapezoidal area, and a lower right point of the trapezoidal area;

[0025] Step B2: Calculate the homography transformation matrix corresponding to the four transformation feature points; wherein the homography transformation matrix is ​​used to transform the trapezoidal area transformed into the target area into a rectangular area;

[0026] Step B3: transforming the trapezoidal area of ​​the target area in the focal segment image by using a homography transformation matrix to obtain a transformed target area image;

[0027] Step B4: Execute steps B1 to B3 for each focal length picture one by one to obtain the target area image corresponding to each focal length picture, and complete the matrix stretching transformation processing.

[0028] As a further solution, the image stitching module sequentially stitches the target area images through the following steps:

[0029] Using the scale-invariant feature transformation algorithm, the feature points of all target area images in different scale spaces are calculated;

[0030] Use the FLANN algorithm to filter out high-quality matching points, and save the matching results of the feature points corresponding to the high-quality matching points;

[0031] Set the transformation matrix and perform affine transformation on the target area image;

[0032] Use a transformation matrix to map all target area images to the same coordinate system;

[0033] Use dynamic programming algorithm to find the best stitching line between the images of each target area, stitch the images according to the best stitching line, and remove the ghost images caused by the movement of moving objects in the stitching;

[0034] The optical flow method is used to realize multi-plane alignment of each target area image, so that the spliced ​​target area image has consistent pixel positions within the spliced ​​area;

[0035] Adjust the image brightness and contrast to achieve consistent exposure of the target area after stitching;

[0036] Pyramid fusion is used to fuse the foreground object into the background, achieving a smooth transition effect between the spliced ​​target area images and obtaining a multi-view fusion visual monitoring image.

[0037] As a further solution, the feature points of all target area images at different scales are calculated through the following steps:

[0038] Constructing a Gaussian pyramid of the target area image; wherein the Gaussian pyramid is implemented by Gaussian blurring and downsampling the target area image at different scales;

[0039] Based on the Gaussian pyramid, the difference between two adjacent layers of Gaussian images is calculated to construct the DOG pyramid; the DOG pyramid is used to capture edge information of different scales in the image;

[0040] Detect feature points in the DOG pyramid; wherein the feature points appear as local maximum or minimum values ​​in the scale space, and the feature points are detected within each scale layer and between adjacent scale layers to ensure scale invariance;

[0041] Accurately locate the feature points detected initially to determine the exact position and scale of the feature points;

[0042] Assign one or more directions to each extreme point; wherein the assigned directions are based on the gradient directions of the pixels around the feature point, so as to make the feature point descriptor invariant to image rotation;

[0043] The feature point descriptor is constructed using the pixel gradient information around the feature point to obtain the feature point feature vector;

[0044] Image matching is performed by comparing the feature vectors of feature points in two images.

[0045] As a further solution, high-quality matching points are screened out through the following steps:

[0046] Initialize the FLANN matcher and specify the index parameter index_params and the search parameter search_params;

[0047] Use the knnMatch method to perform K nearest neighbor matching and obtain several matching points;

[0048] Set the ratio and perform ratio test to select high-quality matching points;

[0049] The matching results are drawn through high-quality matching points, and the matching results of the feature points corresponding to the high-quality matching points are saved.

[0050] As a further solution, a transformation matrix is ​​set and an affine transformation is performed on the target area images, so that all target area images are mapped to the same coordinate system; wherein the transformation matrix includes elements corresponding to each feature point of the target area image.

[0051] As a further solution, the image stitching module also calibrates the area length and area width of the target area image corresponding to each focal length picture, normalizes and stretches each target area image according to the shortest area length, and stitches each adjacent target area image according to the intersection area of ​​the actual scene to obtain a multi-eye fusion visual monitoring image; wherein, in the multi-eye fusion visual monitoring image, the image length ratio of each target area image is the same as the area length ratio.

[0052] As a further solution, target speed measurement V is also performed through multi-eye fusion visual monitoring images;

[0053]

[0054] Wherein, Vx = target image x-axis pixel point change rate * region width / image width; Vy = target image y-axis pixel point change rate * region length / image length.

[0055] Compared with related technologies, the present invention can realize the fusion and stitching of multi-focus images to generate a multi-eye fusion visual monitoring image with greater depth of field and clearer details; it can ensure that the stitched image of the target area maintains a high clarity from near to far, and can also achieve that both near and far objects can be presented in the image with similar magnifications; it solves the problem that the multi-focus fusion image has a large field of view but unclear distant view or clear distant view but small field of view; thereby solving the problem that when traditional image processing technology is combined with subsequent technology fusion, the "machine" cannot obtain enough image details, thereby affecting subsequent recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 A schematic diagram of a multi-view fusion visual monitoring system for narrow and long areas provided by the present invention;

[0059] Figure 2 A schematic diagram of a formula stretching transformation processing scenario provided by the present invention;

[0060] Figure 3 A schematic diagram of a matrix stretch transformation processing scenario provided by the present invention;

[0061] Figure 4 A schematic diagram of a telephoto scene provided by the present invention;

[0062] Figure 5 A schematic diagram of a mid-focus scene provided by the present invention;

[0063] Figure 6 A schematic diagram of a short-focus image scene provided by the present invention;

[0064] Figure 7 A schematic diagram of a multi-view fusion visual monitoring image provided by the present invention;

[0065] Figure 8 A schematic diagram of a multi-view fusion visual monitoring method for narrow and long areas provided by the present invention;

[0066] Fig. 9 A schematic diagram of the process of sequentially stitching target area images provided by the present invention.

[0067] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0069] See also Figure 1 The embodiment of the present application provides a multi-eye fusion visual monitoring system for a narrow and long area, which is used to monitor the narrow and long area scene, including a front-end acquisition device and an image processing unit; wherein,

[0070] Front-end acquisition equipment: arranged above the narrow and long area scene, used to obtain different focal length images of the narrow and long area scene; wherein, the front-end acquisition equipment is composed of a plurality of image acquisition units, which are all arranged on the same shooting position, and can independently adjust the shooting pitch angle and shooting focal length, and adjust the shooting pitch angle of each image acquisition unit so that the image contents of adjacent image acquisition units have overlapping areas;

[0071] Image processing unit: communicated with the front-end acquisition device, used for performing image stitching processing on each focal length picture, including a stretching transformation processing module and an image stitching module; wherein the stretching transformation processing module is used for performing stretching transformation processing on each focal length picture, and obtaining the target area image corresponding to each focal length picture through formula stretching transformation processing / matrix stretching transformation processing, and retaining the original image details of the target area image for output; the image stitching module sequentially stitches the target area images according to the setting order of the image acquisition unit to obtain a multi-eye fusion visual monitoring image.

[0072] It should be noted that the front-end acquisition device uses multiple cameras or array cameras with different focal lengths to form a front-end acquisition device device. Figure 1 In the example arrangement diagram shown, cameras of different focal lengths (telephoto, medium focal length and short focal length) belong to the same position in space, and the focal length images are divided into telephoto images, medium focal length images and short focal length images; among them, the pitch angle and focal length need to be calibrated according to the actual shooting scene, and are set in the way of "expected preset value + later actual adjustment". The collected telephoto images, medium focal length images and short focal length images are shown as follows: Figure 4 , Figure 5 and Figure 6 shown.

[0073] There is some overlap in the captured images. The captured images are preprocessed and spliced ​​to output a multi-view fusion visual monitoring image (such as Figure 7 The number and position of multi-focal length lenses can be determined according to the focus area of ​​the shooting scene: for example, when shooting an airport runway and focusing on the aircraft on the runway, when arranging multi-focal length cameras, the focal plane should be located on the runway, and the number of focal lengths should be determined according to the length of the runway.

[0074] pass Figure 7 It is not difficult to see that the visual structure of the image does not conform to the observation habits of the human eye, because there will be distortion of objects and incoherence of the picture. However, for the machine, the image is clear enough and has enough details. It can be used without unnecessary processing at different focal lengths. For the "machine", Figure 7 It is very user-friendly. If you need an image that is more in line with human observation habits, you only need to perform some processing during the subsequent stitching (see Fig. 9 content).

[0075] See also Figure 2 The present invention provides a multi-eye fusion visual monitoring method for a narrow and long area, which is applied to the multi-eye fusion visual monitoring system for a narrow and long area as described above, and monitors the narrow and long area scene through the following steps:

[0076] Step 1: Obtain images of different focal lengths of a narrow and long area scene through the front-end acquisition device;

[0077] Step 2: The stretching transformation processing module is used to perform stretching transformation processing on each focal length picture, obtain the target area image corresponding to each focal length picture through formula stretching transformation processing / matrix stretching transformation processing, and retain the original image details of the target area image for output;

[0078] Step 3: The image stitching module stitches the target area images sequentially according to the setting order of the image acquisition unit to obtain a multi-eye fusion visual monitoring image.

[0079] The core of the solution of this embodiment is: through formula stretching transformation processing / matrix stretching transformation processing, the target area image is obtained and sequentially stitched, so as to ensure that the stitched target area image maintains a high clarity from near to far, and can also achieve that both near and far can be presented in the picture with similar magnification; it solves the problem of multi-focus fusion image with a large field of view but unclear distant view or clear distant view but small field of view, meets the "machine" demand for image details during subsequent technical integration, and solves the problem of clarity of different focal lengths that affects subsequent recognition results from the "bottom layer", which has great practical value at the current new technical stage.

[0080] It should be noted that the stretching transformation processing module adaptively selects the stretching transformation mode to be used, including formula stretching transformation processing and matrix stretching transformation processing. If clear formula parameters cannot be obtained, it switches to matrix stretching transformation processing, otherwise it uses formula stretching transformation processing.

[0081] When processing the formula stretch transformation: according to Figure 2 And formula (1) and formula (2),

[0082]

[0083] The horizontal and vertical stretching ratios λ of each row of images are calculated for each focal length. Formula (1) is obtained by the similar triangles. The stretching ratio λ in formula (2) refers to the stretching ratio of a row in the image (when the height from bottom to top is hy) compared to the bottom row. The principle is also through similar triangles.

[0084] In the matrix stretch transformation process: according to Figure 3 The four feature points of (a) correspond to Figure 3 (b) The four feature points are used to calculate the homography transformation matrix

[0085] Then Figure 3 (a) The real oblique image is transformed one-to-one according to the homography transformation matrix H, and the result is Figure 3 (b).

[0086] Crop out the target area of ​​each focal length and then fuse and stitch the entire image.

[0087] When performing matrix stretching transformation processing: it is necessary to determine the target area in each focal length picture and select four transformation feature points of the target area; wherein the target area is a trapezoidal area, including the upper left point of the trapezoidal area, the upper right point of the trapezoidal area, the lower point of the trapezoidal area and the lower right point of the trapezoidal area; and calculate the homography transformation matrix corresponding to the four transformation feature points; wherein the homography transformation matrix is ​​used to transform the trapezoidal area transformed into the target area into a rectangular area; and then transform the trapezoidal area of ​​the target area in the focal length picture through the homography transformation matrix to obtain the transformed target area image.

[0088] When performing image stitching, we can use a variety of methods, either directly stitching the images or using a certain stitching method to make the image visual effect better, thereby taking into account the dual visual needs of "people" and "machines", such as Fig. 9 The steps shown are used to sequentially stitch the target area images:

[0089] Using the scale-invariant feature transformation algorithm, the feature points of all target area images in different scale spaces are calculated;

[0090] Use the FLANN algorithm to filter out high-quality matching points, and save the matching results of the feature points corresponding to the high-quality matching points;

[0091] Set the transformation matrix and perform affine transformation on the target area image;

[0092] Use a transformation matrix to map all target area images to the same coordinate system;

[0093] Use dynamic programming algorithm to find the best stitching line between the images of each target area, stitch the images according to the best stitching line, and remove the ghost images caused by the movement of moving objects in the stitching;

[0094] The optical flow method is used to realize multi-plane alignment of each target area image, so that the spliced ​​target area image has consistent pixel positions within the spliced ​​area;

[0095] Adjust the image brightness and contrast to achieve consistent exposure of the target area after stitching;

[0096] Pyramid fusion is used to fuse the foreground object into the background, achieving a smooth transition effect between the spliced ​​target area images and obtaining a multi-view fusion visual monitoring image.

[0097] In the above image stitching scheme, the scale-invariant feature transform (SIFT) algorithm is used to calculate the feature points of all images in different scale spaces. The following steps can be followed:

[0098] Constructing Gaussian Pyramid: The SIFT algorithm first needs to construct a Gaussian pyramid of the image, which is achieved by Gaussian blurring and downsampling the image at different scales. Each layer of the Gaussian pyramid represents the representation of the image at different scales, providing a scale space for subsequent feature point detection.

[0099] Difference of Gaussian (DOG) pyramid: Based on the Gaussian pyramid, the difference between two adjacent layers of Gaussian images is calculated to construct the DOG pyramid. The DOG pyramid is used to capture edge information of different scales in the image, which is a key step in detecting scale-invariant feature points.

[0100] Spatial extreme point detection: Detect extreme points in the DOG pyramid, which appear as local maximum or minimum values ​​in the scale space. Extreme point detection needs to be performed within each scale layer and between adjacent scale layers to ensure scale invariance.

[0101] Precise localization of stable key points: Precisely localize the initially detected extreme points to determine the exact location and scale of the key points. This step involves refining the extreme points to ensure their stability in scale and direction.

[0102] Keypoint orientation assignment: Each keypoint is assigned one or more orientations, usually based on the gradient orientation of the pixels surrounding the keypoint. This makes the keypoint descriptor invariant to image rotation.

[0103] Key point description: The key point descriptor is constructed using the pixel gradient information around the key point. The SIFT algorithm usually uses a 128-dimensional feature vector to describe the local features of each key point. This descriptor is invariant to rotation, scale, and brightness changes.

[0104] Feature point matching: Finally, image matching is performed by comparing the key point feature vectors in the two images. Euclidean distance or cosine similarity is usually used to evaluate the similarity of feature vectors to achieve feature matching and target recognition.

[0105] Through these steps, the SIFT algorithm can calculate the feature points of the image in different scale spaces and construct descriptors for these feature points. These descriptors can be used for subsequent image matching and recognition tasks.

[0106] Using the FLANN algorithm to perform feature point matching follows these steps:

[0107] Import necessary libraries: Import NumPy, OpenCV, Matplotlib and other libraries for image processing and feature point matching.

[0108] Load images: Use cv2.imread() method to load two images and convert them into grayscale images.

[0109] Feature point detection and descriptor extraction: Detect key points and extract descriptors using feature detectors such as SIFT, SURF, or ORB.

[0110] Initialize the FLANN matcher: To create a FLANN matcher object, you need to specify the index parameter index_params and the search parameter search_params.

[0111] Perform KNN matching: Use the knnMatch method to perform K nearest neighbor matching, usually k=2.

[0112] Apply ratio test: In order to screen out good matching points, apply a ratio test, usually using a distance ratio of 0.75 or 0.5.

[0113] Draw the matching results: Use cv2.drawMatchesKnn() to draw the matching results.

[0114] To use the transformation matrix to perform an affine transformation on each frame, map all frames to the same coordinate system, and complete the image transformation, you can follow these steps:

[0115] Determine the transformation matrix: The transformation matrix describes the mapping relationship from one coordinate system to another. This matrix can be obtained by matching and solving feature points, such as using the SIFT algorithm to find matching point pairs between two images, and then using these point pairs to solve the homography matrix.

[0116] Read the image: Use OpenCV's cv2.imread() function to read the image that needs to be transformed.

[0117] Apply affine transformation: Use OpenCV's cv2.warpPerspective() function to perform an affine transformation on the image. This function requires the original image, the transformation matrix, and the size of the output image.

[0118] Display or save the transformed image: Use OpenCV's cv2.imshow() function to display the transformed image, or use cv2.imwrite() function to save it to a file.

[0119] To find the best stitching line for image stitching through dynamic programming algorithm, stitch the images according to the best stitching line, and remove the ghosting caused by the moving objects in the stitching, you can follow the following steps:

[0120] Calculate the energy value of each pixel: First, the energy value of each pixel in the overlapping area needs to be calculated. The energy value is usually composed of color difference and structural difference. The color difference can be obtained by calculating the difference in the color values ​​of overlapping pixels, while the structural difference can be calculated by edge detection operators (such as Sobel operator). The specific formula is as follows: E(x, y) = E c (x, y) 2 +E g (x, y); where Ec is the difference in color value, and Eg is the difference in structure value. The structure value can be calculated using the Sobel operator, and the calculation formula is as follows: Eg=Diff(I1x, I2x)·Diff(I1y, I2y); here, I1x, I1yI1x, I1y and I2x, I2yI2x, I2y are the Sobel edge detection results of the two images in the x and yy directions respectively.

[0121] Dynamic programming to find the best stitching: Use dynamic programming to calculate the minimum energy value and path for each pixel starting from the second row. For each column, consider the energy values ​​in the left, middle, and right directions, select the minimum energy value as the energy value of the current pixel, and record the path with the minimum energy value. This process continues until the last row of the image, and the minimum energy value of the last row is the total energy value of the best stitching path.

[0122] Image stitching based on the best stitching path: stitch the images based on the calculated best stitching path. Specifically, for each pixel on the best path, select the pixel value of the image on the left side of the path, and for the pixel on the right side of the path, select the pixel value of the image on the right side of the path.

[0123] Ghost removal: Since moving objects may have ghost images during the stitching process, they can be removed by the following methods: Improved weighted average fusion: In the stitching area, the pixel values ​​of the two images are weighted averaged, and the weight can be determined based on the position and motion state of the pixel in the overlapping area. Region growing method: Starting from the best stitching line, grow to both sides, and decide whether to include the pixel in the stitching result based on the energy value and color similarity of the pixel.

[0124] To use the optical flow method to achieve multi-plane alignment of images, ensure that the stitched images have consistent pixel positions within the stitching area, and ensure that the stitched images are continuous, seamless, or misaligned, you can follow the following steps:

[0125] Calculate optical flow: The purpose of the optical flow method is to estimate the movement of pixels in an image between consecutive frames. First, a Gaussian pyramid needs to be established for each frame. The optical flow is calculated starting from the top layer (the lowest resolution). The optical flow of each point in the top layer image is obtained by minimizing the sum of the matching errors within the neighborhood of each point. This process is carried out from the top layer downward until the bottom layer of the pyramid (the original image) is reached.

[0126] Feature point matching: Use optical flow to find matching point pairs in the overlapping areas of the images. These point pairs will be used for subsequent image alignment and stitching. Initial motion estimation can be performed through template matching, and then further matching can be performed using optical flow.

[0127] Eliminate mismatched points: Due to the high complexity and similarity of microscopic images, the optical flow method may have many mismatches. Therefore, a mismatched point elimination method based on feature point similarity and position relationship can be introduced to improve the registration accuracy.

[0128] Global optimization: After obtaining the matching point pairs, a global optimization method (such as RANSAC) is used to calculate the displacement between images to further optimize the matching results.

[0129] Image warping and alignment: Based on the results of the optical flow calculation, the images are warped and aligned. This step requires careful operation to avoid introducing other types of visual artifacts. For overlapping areas, the images are first warped and then aligned, which is a challenging process because warping is prone to under-constraint problems.

[0130] Image Fusion: After image alignment, the images need to be seamlessly fused together. This can be achieved using image fusion techniques such as Poisson fusion or multi-band fusion to eliminate ghosting and ensure continuity in the stitching area.

[0131] Processing moving objects: For ghost images caused by moving objects during stitching, you can use the alt optical flow vector to perform a remap operation after optical flow calculation, then perform a crop operation, and finally input the image into the network for deep learning fusion to solve the ghost image problem.

[0132] To use pyramid fusion technology to blend foreground objects into the background and achieve a smooth transition effect, follow these steps:

[0133] Construct Gaussian pyramid: Construct Gaussian pyramids for the foreground and background images respectively. Gaussian pyramids are a collection of images obtained by recursively performing Gaussian blur and downsampling on the image. This helps to process images at different resolutions, thereby achieving a smooth fusion effect.

[0134] Constructing Laplacian Pyramid: Starting from Gaussian Pyramid, construct Laplacian Pyramid. Laplacian Pyramid is obtained by subtracting adjacent layers from Gaussian Pyramid, which can capture the detailed information of the image.

[0135] Pyramid level fusion: At each level of the Laplacian pyramid, a weighted fusion of the foreground and background images is performed based on the area that needs to be fused. This can be achieved by creating a gradient mask that is gradient in the overlapping area of ​​the foreground and background, thereby achieving a smooth transition.

[0136] Reconstruct the image: Start from the lowest level of the Laplacian pyramid and reconstruct the image step by step upwards. At each level, add the upsampled image to the Laplacian layer to finally get the fused image.

[0137] Optimize stitching: In order to further optimize the transition effect of the stitching, a dynamic programming algorithm can be used to find the best stitching line, and then perform smooth transition processing on the image based on the best stitching line.

[0138] Apply masking and weighted fusion: In overlapping areas, the foreground and background images are combined using a weighted fusion method. A smooth transition can be achieved by calculating a mask for the foreground image and then blending it with the background image.

[0139] Processing transparent areas: For transparent areas in the foreground image, special processing is required to ensure a smooth transition. The final smooth transition effect can be achieved by weighted fusion of transparent areas and non-transparent areas, and then merging the results.

[0140] Through the above steps, a smooth transition fusion of the foreground object and the background can be achieved, and a multi-view fusion visual monitoring image can be output. This method is particularly suitable for scenes that require fine control of the fusion effect, such as image editing, video production and other fields.

[0141] In addition, we also calibrate the area length and area width of the target area image corresponding to each focal length, normalize and stretch each target area image according to the shortest area length, and stitch the adjacent target area images according to the intersection area of ​​the actual scene to obtain a multi-eye fusion visual monitoring image; in this way, the multi-eye fusion visual monitoring image we obtained can produce a corresponding effect with the actual physical scene, and keep the image pixel correspondence equal to the actual correspondence, and then realize the subsequent target speed measurement.

[0142] When measuring the speed of a target, the traditional image processing method will cause the speed measurement algorithm to be complex and have large errors due to factors such as the small number of pixels at a long distance. In this embodiment, the pixels of the distant scene are retained during speed measurement, and each pixel can correspond to the actual physical scene. Therefore, when measuring the speed of a target, it only needs to calculate the x-axis speed Vx according to the x-axis pixel change rate of the target image*region width / image width, and calculate the y-axis speed Vy according to the y-axis pixel change rate of the target image*region length / image length, and the target speed measurement can be directly integrated.

[0143] After completing the precise speed measurement, we can also combine target recognition and tracking. By connecting the type, speed, appearance time, and disappearance time between the upper and lower images, we can achieve target matching between the upper and lower images, making target tracking continuous. There is no need for other complex algorithm processing processes, which further proves that this solution is more suitable for machine learning image processing than traditional solutions.

[0144] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multi-eye fusion visual monitoring system for narrow and long areas, used to monitor narrow and long area scenes, characterized in that: It includes front-end acquisition equipment and image processing unit; wherein, Front-end acquisition equipment: arranged above the narrow and long area scene, used to obtain different focal length images of the narrow and long area scene; wherein, the front-end acquisition equipment is composed of a plurality of image acquisition units, which are all arranged on the same shooting position, and can independently adjust the shooting pitch angle and shooting focal length, and adjust the shooting pitch angle of each image acquisition unit so that the image contents of adjacent image acquisition units have overlapping areas; Image processing unit: communicated with the front-end acquisition device, used for performing image stitching processing on each focal length picture, including a stretching transformation processing module and an image stitching module; wherein the stretching transformation processing module is used for performing stretching transformation processing on each focal length picture, and obtaining the target area image corresponding to each focal length picture through formula stretching transformation processing / matrix stretching transformation processing, and retaining the original image details of the target area image for output; the image stitching module sequentially stitches the target area images according to the setting order of the image acquisition unit to obtain a multi-eye fusion visual monitoring image.

2. A multi-eye fusion visual monitoring method for a narrow and long area, applied to a multi-eye fusion visual monitoring system for a narrow and long area as claimed in claim 1, characterized in that: To monitor a narrow and long area, perform the following steps: Step 1: Obtain images of different focal lengths of a narrow and long area scene through the front-end acquisition device; Step 2: The stretching transformation processing module is used to perform stretching transformation processing on each focal length picture, obtain the target area image corresponding to each focal length picture through formula stretching transformation processing / matrix stretching transformation processing, and retain the original image details of the target area image for output; Step 3: The image stitching module stitches the target area images sequentially according to the setting order of the image acquisition unit to obtain a multi-eye fusion visual monitoring image.

3. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 2 is characterized in that: The stretching transformation processing module performs formula stretching transformation processing through the following steps: Step A1: determining a target area in each focal length image; wherein the target area is an isosceles trapezoidal area; Step A2: Calculate the horizontal and vertical stretching ratios λ corresponding to the target area for each focal length image; Step A3: stretching each row of pixels in the focal length image by a horizontal and vertical stretching ratio λ to keep the distance of the target area consistent; Step A4: Execute steps A1 to A3 one by one for each focal length picture, obtain the target area image corresponding to each focal length picture, and complete the formula stretching transformation process; Among them, the calculation formulas of the horizontal and vertical stretching ratios λ are: Where h is the height of the isosceles trapezoid, y is the row coordinate of the pixel, d1 is the length of the upper half of the base of the isosceles trapezoid, and d2 is the length of the lower half of the base of the isosceles trapezoid.

4. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 2 is characterized in that: The stretch transformation processing module performs matrix stretch transformation processing through the following steps: Step B1: determining a target area in each focal length image, and selecting four transformation feature points of the target area; wherein the target area is a trapezoidal area, including an upper left point of the trapezoidal area, an upper right point of the trapezoidal area, a lower point of the trapezoidal area, and a lower right point of the trapezoidal area; Step B2: Calculate the homography transformation matrix corresponding to the four transformation feature points; wherein the homography transformation matrix is ​​used to transform the trapezoidal area transformed into the target area into a rectangular area; Step B3: transforming the trapezoidal area of ​​the target area in the focal segment image by using a homography transformation matrix to obtain a transformed target area image; Step B4: Execute steps B1 to B3 for each focal length picture one by one to obtain the target area image corresponding to each focal length picture, and complete the matrix stretching transformation processing.

5. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 2 is characterized in that: The image stitching module sequentially stitches the target area images through the following steps: Using the scale-invariant feature transformation algorithm, the feature points of all target area images in different scale spaces are calculated; Use the FLANN algorithm to filter out high-quality matching points, and save the matching results of the feature points corresponding to the high-quality matching points; Set the transformation matrix and perform affine transformation on the target area image; Use a transformation matrix to map all target area images to the same coordinate system; Use dynamic programming algorithm to find the best stitching line between the images of each target area, stitch the images according to the best stitching line, and remove the ghost images caused by the movement of moving objects in the stitching; The optical flow method is used to realize multi-plane alignment of each target area image, so that the spliced ​​target area image has consistent pixel positions within the spliced ​​area; Adjust the image brightness and contrast to achieve consistent exposure of the target area after stitching; Pyramid fusion is used to fuse the foreground object into the background, achieving a smooth transition effect between the spliced ​​target area images and obtaining a multi-view fusion visual monitoring image.

6. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 5, characterized in that: The feature points of all target area images in different scale spaces are calculated through the following steps: Constructing a Gaussian pyramid of the target area image; wherein the Gaussian pyramid is implemented by Gaussian blurring and downsampling the target area image at different scales; Based on the Gaussian pyramid, the difference between two adjacent layers of Gaussian images is calculated to construct the DOG pyramid; the DOG pyramid is used to capture edge information of different scales in the image; Detect feature points in the DOG pyramid; wherein the feature points appear as local maximum or minimum values ​​in the scale space, and the feature points are detected within each scale layer and between adjacent scale layers to ensure scale invariance; Accurately locate the initially detected feature points to determine their exact location and scale; Assign one or more directions to each extreme point; wherein the assigned directions are based on the gradient directions of the pixels around the feature point, so as to make the feature point descriptor invariant to image rotation; The feature point descriptor is constructed using the pixel gradient information around the feature point to obtain the feature point feature vector; Image matching is performed by comparing the feature vectors of feature points in two images.

7. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 6, characterized in that: Filter out high-quality matching points through the following steps: Initialize the FLANN matcher and specify the index parameter index_params and the search parameter search_params; Use the knnMatch method to perform K nearest neighbor matching and obtain several matching points; Set the ratio and perform ratio test to select high-quality matching points; The matching results are drawn through high-quality matching points, and the matching results of the feature points corresponding to the high-quality matching points are saved.

8. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 7, characterized in that: By setting a transformation matrix and performing affine transformation on the target area images, all the target area images are mapped to the same coordinate system; wherein the transformation matrix includes elements corresponding to each feature point of the target area image.

9. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 2, characterized in that: The image stitching module also calibrates the area length and area width of the target area image corresponding to each focal length picture, normalizes and stretches each target area image according to the shortest area length, and stitches each adjacent target area image according to the intersection area of ​​the actual scene to obtain a multi-eye fusion visual monitoring image; wherein, in the multi-eye fusion visual monitoring image, the image length ratio of each target area image is the same as the area length ratio.

10. The multi-eye fusion visual monitoring system for narrow and long areas according to claim 9, characterized in that: The target speed V is also measured by multi-eye fusion visual monitoring images; Wherein, Vx = target image x-axis pixel point change rate * region width / image width; Vy = target image y-axis pixel point change rate * region length / image length.

Images