A working face panoramic splicing method and device fusing an improved optical flow algorithm

Abstract

This application relates to the field of video surveillance and image processing technology in coal mines, and discloses a method and apparatus for panoramic stitching of working faces using an improved optical flow algorithm. The method includes the following steps: First, multi-scale adaptive preprocessing is performed on the original image; an anti-interference weight model based on texture entropy and gradient is constructed, and a sparse optical flow field is generated using Retinex illumination component separation technology to suppress dust and illumination interference. Second, feature matching is guided by the sparse optical flow field, and a dynamic region mask is generated by combining target detection. The optical flow transformation and feature transformation matrices are then adaptively weighted and fused. Finally, an improved Poisson fusion algorithm is used to repair dynamically occluded areas, and the optimal seam path is planned by calculating the optical flow variance. A dynamic window is then used for smooth fusion. This invention effectively solves the problems of stitching misalignment and artifacts caused by high dust levels, illumination fluctuations, and rapid equipment movement in coal mines, achieving high-quality seamless panoramic stitching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal mine underground video monitoring and image processing, in particular to a working face panoramic splicing method and device fusing an improved optical flow algorithm. BACKGROUND

[0002] With the advancement of intelligent construction of coal mines, remote visual monitoring of fully mechanized working faces has become a key link to ensure safety in production and realize unmanned operation. Due to the narrow space and numerous equipment in the underground working face, the field of view of a single camera is extremely limited and cannot cover the entire operation scene. Therefore, using multi-camera image splicing technology to construct a panoramic video stream of the working face has become the main means to obtain global visual perception.

[0003] Existing image splicing technologies mainly include feature-based methods and region-based methods. Feature-based methods (such as SIFT and SURF algorithms) usually rely on matching of features such as corner points or spots extracted from images. However, the underground environment of coal mines is extremely harsh, with high concentrations of dispersed dust in the air, and the surfaces of coal walls and hydraulic supports often have single or repetitive textures. In such an environment, dust noise is easily misjudged as feature points, and low-texture areas are difficult to extract enough effective features, resulting in frequent false matching or insufficient matching of feature matching algorithms, which cannot calculate accurate homography matrices.

[0004] Region-based methods do not need to extract artificially designed features, but directly use image pixel domain information to construct registration optimization targets. Methods based on optical flow are a typical subclass. On the other hand, although methods based on optical flow can better handle scenes with parallax by estimating the motion field of pixels, the calculation of traditional optical flow algorithms heavily relies on the assumption of constant brightness. There are a large number of non-uniform artificial light sources (such as mine lamps and support lamps) in fully mechanized working faces, and with the movement of the coal mining machine, the lighting conditions and shadow distribution fluctuate dramatically in real time, which seriously undermines the premise of constant brightness, resulting in large deviations in optical flow calculation. In addition, the rapid operation of the coal mining machine and the scraper conveyor often produces large displacement motion, which easily exceeds the small displacement constraint range of traditional optical flow algorithms, causing the algorithm to fall into local extreme values and fail.

[0005] In addition, the fully mechanized working face is a typical high-dynamic scene, and there is a complex relative motion and occlusion relationship between the foreground equipment and the background coal wall. Existing splicing schemes often lack effective perception and processing mechanisms for dynamic regions. When dealing with dynamic occlusions, it is easy to produce field-of-view loss (i.e., black hole effect) or ghosting phenomenon due to misplacement of objects at the splicing site, which seriously affects the visual consistency of the panoramic video and the effectiveness of the monitoring. Therefore, there is an urgent need for a working face panoramic splicing technology that can resist dust interference, adapt to changes in lighting, and effectively handle dynamic occlusions. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a working face panoramic stitching method and device fusing an improved optical flow algorithm, which solves the problems of stitching misplacement, dynamic occlusion loss and visual artifacts caused by unstable feature matching and large optical flow calculation deviation of the existing image stitching technology under the complex working conditions of high dust, severe light fluctuation and rapid equipment movement in the coal mine.

[0007] To achieve the above object, the present application is implemented by the following technical solutions: In a first aspect, the present application provides a working face panoramic stitching method fusing an improved optical flow algorithm.

[0008] S1, obtain the original image sequence of the working face, and perform multi-scale adaptive preprocessing on the original image sequence; construct an anti-interference weight model based on texture entropy and gradient, and combine with the Retinex light component separation technology to generate a sparse optical flow field capable of representing the equipment motion state through pyramid iteration calculation; S2, use the sparse optical flow field to guide feature matching to obtain a feature matching transformation matrix; at the same time, generate a dynamic region mask combining the target detection result and the optical flow amplitude, and perform adaptive weighted fusion on the transformation matrix based on the optical flow and the feature matching transformation matrix based on the dynamic region mask to obtain a final fusion transformation matrix; S3, predict the overlapping area based on the sparse optical flow field, repair the dynamic occlusion area in the overlapping area using an improved Poisson fusion algorithm, plan the best seam path by calculating the optical flow variance, and use a dynamic window to smooth and fuse the images to generate a panoramic stitched image.

[0009] Further, in the multi-scale adaptive preprocessing stage, the method compares the global average optical flow amplitude of the previous frame with a preset speed threshold to determine the processing resolution of the current frame. When the global average optical flow amplitude is detected to be greater than the speed threshold, the image resolution is reduced, thereby expanding the optical flow search range and reducing the calculation load when the equipment moves rapidly; otherwise, a higher resolution is maintained to retain texture details.

[0010] Further, in constructing the anti-interference weight model, the method calculates the normalized gradient amplitude of the image using the Sobel operator, and calculates the normalized texture entropy based on the gray histogram probability distribution within the local window. The local texture entropy is obtained by statistically calculating the negative sum of the product of the gray level probability and the natural logarithm of the probability. Then, according to a preset fusion coefficient, the gradient amplitude and the texture entropy are weighted and combined to calculate the anti-interference weight factor. The weight factor is introduced as a penalty term into the optical flow constraint equation to suppress the dust interference area with high entropy value and enhance the equipment edge area with high gradient.

[0011] Furthermore, the illumination component separation technique employs the Retinex algorithm to decompose the image into illuminance and reflectance components, retaining only the reflectance component. The method further calculates the Hessian matrix of the reflectance component and uses the eigenvalues ​​of this matrix to perform edge enhancement on the reflectance component. The enhanced image is then used as input for optical flow calculation to eliminate the influence of uneven illumination from mine lamps.

[0012] Furthermore, when generating the sparse optical flow field, the method divides the image into different regions based on the anti-interference weighting factor and the optical flow amplitude, and adopts a non-uniform sampling strategy. A larger pixel sampling interval is used for low-weight and low-speed static regions, while a smaller pixel sampling interval is used for high-weight or high-speed dynamic regions, thereby achieving sparsity pruning of the optical flow field.

[0013] Furthermore, in the registration stage, the method uses the motion vectors provided by the sparse optical flow field to predict the target position of feature points from the previous frame in the current frame, and restricts the search range of feature matching to a local window around the target position, using Hamming distance to complete the matching of feature descriptors. This step combines the continuous tracking capability of the optical flow method with the geometric constraint capability of the feature matching method.

[0014] Furthermore, in the transformation matrix fusion stage, the method utilizes an object detection model to obtain the device bounding box and combines it with the mean of local optical flow amplitude to generate a pixel-level state mask. This mask divides the image into defined static regions, defined dynamic regions, and transition regions. The method assigns weights to the optical flow-based transformation matrix and the feature matching transformation matrix respectively: the weight of the optical flow transformation matrix is ​​emphasized in the defined static regions, the weight of the feature matching transformation matrix is ​​emphasized in the defined dynamic regions, and linear interpolation of the weights is performed in the transition regions. Finally, the fused transformation matrix is ​​obtained through weighted summation and Gaussian filtering smoothing.

[0015] Furthermore, in the dynamic stitching stage, the method uses the mean of the sparse optical flow field to predict the overlapping region and calculates the normalized cross-correlation coefficient to verify the effectiveness of the overlap. For dynamically occluded regions within the overlapping region where the difference in optical flow vector exceeds a threshold, the method selects the same location region of adjacent, unoccluded frames on the time axis as a reference template. Guided by the gradient field of the reference template and constrained by the boundary pixel values ​​of the region to be repaired in the current frame, the method generates filling pixels by iteratively solving the Poisson equation.

[0016] Furthermore, in the image fusion stage, the method selects stable regions with small optical flow neighborhood variance within the overlapping area to plan the optimal seam path. The size of the fusion window is adaptively adjusted according to the dynamic attributes of the region: a fixed-size large window is used for static regions, while an adaptive small window that decreases as the optical flow amplitude increases is used for dynamic regions. Within the window, distance-based gradient weighting is performed on the pixels on both sides of the seam to achieve a smooth transition.

[0017] Secondly, the present invention provides a working surface panoramic stitching device that integrates an improved optical flow algorithm.

[0018] The device includes: an adaptive preprocessing module configured to acquire the original image sequence of the working surface and perform multi-scale adaptive preprocessing on the original image sequence; a robust optical flow calculation module configured to construct an anti-interference weight model based on texture entropy and gradient, and combine Retinex illumination component separation technology to generate a sparse optical flow field that can characterize the motion state of the device through pyramid iterative calculation; a feature fusion and registration module configured to guide feature matching using the sparse optical flow field and obtain the feature matching transformation matrix; simultaneously, it generates a dynamic region mask by combining the target detection results and optical flow amplitude, and performs adaptive weighted fusion of the optical flow-based transformation matrix and the feature matching transformation matrix based on the dynamic region mask to obtain the final fusion transformation matrix; and a dynamic stitching and repair module configured to predict the overlapping region based on the sparse optical flow field, repair the dynamic occlusion region within the overlapping region using an improved Poisson fusion algorithm, plan the optimal seam path by calculating the optical flow variance, and use a dynamic window to smoothly fuse the images to generate a panoramic stitched image.

[0019] This invention provides a method and apparatus for panoramic stitching of working surfaces by integrating an improved optical flow algorithm. It has the following beneficial effects: 1. This invention solves the motion estimation problem in downhole environments with high dust and fluctuating lighting by constructing an anti-interference weight model based on texture entropy and gradient, and combining it with Retinex illumination component separation technology. It uses texture entropy to distinguish between diffuse dust and structural equipment, uses illumination separation to eliminate interference from brightness changes, and introduces anti-interference weights into the optical flow constraint equation to ensure that the optical flow calculation focuses on effective textures such as equipment edges, thereby obtaining accurate equipment motion vectors even under harsh working conditions.

[0020] 2. This invention achieves high-precision image registration in dynamic scenes through restricted feature matching guided by sparse optical flow and matrix fusion strategy based on dynamic mask. It uses optical flow prior to narrow the feature search range to improve matching efficiency. At the same time, it combines target detection and optical flow amplitude to divide the image into dynamic and static regions and adaptively allocates the fusion weights of optical flow transformation and feature transformation, thus overcoming the cumulative drift problem of the single optical flow method and the matching failure problem of the single feature method in weak texture regions.

[0021] 3. This invention improves the visual consistency of panoramic video by using optical flow variance-guided optimal seam planning and dynamic window fusion algorithm, combined with improved Poisson fusion repair technology. It can automatically avoid planning seams in areas of sudden motion changes and dynamically adjust the fusion window size according to the speed of object movement. At the same time, it uses temporal reference frames to repair field occlusion caused by device movement, eliminating stitching marks and dynamic ghosting, and achieving seamless panoramic stitching. Attached Figure Description

[0022] Figure 1 This is a functional module architecture diagram of the device of the present invention; Figure 2 This is a flowchart illustrating the overall process of the method of the present invention. Figure 3 This is a detailed flowchart of the robust optical flow calculation of the present invention. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See attached document Figure 1 This invention provides a panoramic stitching device for working faces that integrates an improved optical flow algorithm. The device is applied to the fully mechanized mining working face environment in underground coal mines and includes: an image acquisition module 100, a data transmission module 200, a central processing module 300, and a visualization interaction module 400.

[0025] The image acquisition module 100 is used to acquire real-time video data of the fully mechanized mining face. The image acquisition module 100 includes several high-resolution industrial cameras arranged linearly. The industrial cameras are mounted below the top beam of the hydraulic support, and the fields of view of adjacent industrial cameras have a preset overlap range. In this embodiment, the image sensor resolution of the industrial cameras is configured as follows: The pixel count is set at 30 frames per second. The industrial camera is equipped with a global shutter to synchronize exposure during equipment movement, eliminating motion blur and rolling shutter effects.

[0026] One end of the data transmission module 200 is electrically connected to the image acquisition module 100, and the other end is electrically connected to the central processing module 300. The data transmission module 200 uses a gigabit Ethernet or industrial fiber optic communication network to transmit the raw wide dynamic range (WDR) video stream output by the image acquisition module 100 to the central processing module 300.

[0027] The central processing module 300 is the core computing unit of the device, including a processor, memory, and a graphics acceleration unit (GPU). The memory stores computer-executable instructions. When the instructions are executed by the processor, the central processing module 300 is configured to perform logical operations for the panoramic stitching method that integrates the improved optical flow algorithm. Specifically, the central processing module 300 is functionally divided into: an image preprocessing unit 301, a robust optical flow calculation unit 302, a feature fusion and registration unit 303, and a dynamic stitching and repair unit 304.

[0028] The image preprocessing unit 301 is configured to receive the original image and perform adaptive multi-scale scaling on the original image based on the global optical flow amplitude of the previous frame. The image preprocessing unit 301 uses a bilinear interpolation algorithm to perform image scale transformation operations, downsampling the original image to the target resolution.

[0029] The robust optical flow calculation unit 302 is configured to calculate the anti-interference weighting factor based on texture entropy and gradient features, and generate a sparse optical flow field. The robust optical flow calculation unit 302 first uses the Sobel operator to calculate the image in... gradient of direction and gradient of direction The normalized gradient magnitude is obtained. Meanwhile, the robust optical flow calculation unit 302 divides the image into... A local window of pixels, statistically analyzing the grayscale histogram probability within the window. ,in Indicates grayscale level. The robust optical flow calculation unit 302 calculates the texture entropy according to the following formula.

[0030] ; in, Set as This is used to prevent singularities in logarithmic operations. The robust optical flow calculation unit 302 further calculates the weighting factor according to the following formula.

[0031] ; in, This is the normalized texture entropy. The preset fusion coefficients are used. The robust optical flow calculation unit 302 utilizes this weighting factor. As a penalty, the optical flow vector is calculated iteratively using the Gaussian pyramid.

[0032] The feature fusion registration unit 303 is configured to calculate the image transformation matrix by combining the sparse optical flow field and the feature matching algorithm. The feature fusion registration unit 303 uses the optical flow vector output by the robust optical flow calculation unit 302 to predict the target location of feature points, and performs feature point search and matching within a preset pixel range centered on the target location. The feature fusion registration unit 303 is also configured with a target detection network (e.g., a YOLO model) to identify device regions and generate dynamic region masks, thereby performing weighted fusion of the optical flow transformation matrix and the feature matching transformation matrix.

[0033] The dynamic stitching repair unit 304 is configured to perform projection transformation on the image based on the fused transformation matrix and repair occlusion defects in overlapping areas. The dynamic stitching repair unit 304 monitors the difference in optical flow vectors between adjacent pixels. When the difference exceeds a preset threshold, it is determined to be a dynamic occlusion area, and the Poisson fusion algorithm is used to fill pixels with the same location area in temporally adjacent frames as a reference template.

[0034] The visualization interaction module 400 is connected to the central processing module 300 to display the spliced ​​panoramic video stream in real time and provide a human-machine interface for operators to monitor the equipment status.

[0035] See attached document Figure 2 This invention provides a panoramic stitching method for working surfaces that integrates an improved optical flow algorithm. The method includes the following steps: a dust and illumination robust optical flow field calculation step S100, an optical flow and feature matching fusion registration step S200, and an optical flow-guided dynamic stitching step S300.

[0036] In step S100, which involves robust optical flow field calculations for dust and illumination, the system first acquires the original image sequence captured by an industrial camera, with a single frame resolution of [resolution missing]. Pixels. The system performs multi-scale adaptive preprocessing on the input raw image: the system calculates the global average optical flow amplitude of the previous frame image. And compare the amplitude with a preset speed threshold. (Set to 15 pixels / frame) for comparison. When At that time, the system uses bilinear interpolation to scale the current frame image to... Pixel; when At that time, the system will scale the current frame image to... Pixels. This dynamic scaling strategy is used to expand the optical flow search range and reduce computational load when the device is moving rapidly.

[0037] Subsequently, in step S100, the system constructs an anti-interference weight model based on texture entropy and gradient. The system uses the Sobel operator to calculate the normalized gradient magnitude of the image. and based on 8 grayscale histogram within a local window Calculate normalized texture entropy Texture entropy The calculation follows the formula: ,in The system uses the formula Calculate weighting factors ,in The value is set to 0.6. The system incorporates this weighting factor into the optical flow constraint equation and uses a 4-layer Gaussian pyramid structure to perform iterative calculations from coarse to fine, outputting a sparse optical flow field.

[0038] In the optical flow and feature matching fusion registration step S200, the system calculates the transformation matrix by combining the sparse optical flow field and the feature matching algorithm. The system then scales the original image to... The system calculates pixels and extracts simplified 64-dimensional SIFT feature descriptors. Using the sparse optical flow field output in step S100 as prior information, the system predicts the estimated positions of feature points from the previous frame in the current frame and limits the search radius for feature matching to the area surrounding the estimated positions. Within the pixel range, similarity is measured using Hamming distance.

[0039] Simultaneously, in step S200, the system performs dynamic region recognition. The system runs the YOLOv8-mini object detection model to obtain the target bounding box of the device, and generates a pixel-level state mask by combining the mean of optical flow amplitude. The state mask divides the image into defined static regions, defined dynamic regions, and transition regions. For different regions, the system applies an optical flow-based transformation matrix. and feature-matching-based transformation matrix Assign different weights and The system is based on the formula. Calculate the final fusion transformation matrix and perform a field transformation on this matrix. Gaussian filtering smoothing process.

[0040] In the optical flow-guided dynamic stitching step S300, the system uses the mean direction of the sparse optical flow field to predict the overlapping region of adjacent images and calculates the normalized cross-correlation coefficient (NCC) within this region. When the overlap is valid, the system determines that the overlap is valid. The system traverses the overlapping area and calculates the difference in optical flow vectors between adjacent pixels. When the difference is greater than 5 pixels / frame, it is determined to be a dynamic occlusion area. The system performs improved Poisson fusion repair on the dynamic occlusion area: it selects the same location area of ​​temporally adjacent and unoccluded frames as a reference template, generates filling pixels by solving the Poisson equation, and sets the number of iterations to 5 or until the pixel grayscale error is less than 2.

[0041] Finally, in step S300, the system plans the optimal seam path and performs image fusion. The system calculates the optical flow vector of the overlapping area. The system uses neighborhood variance to filter out optical flow smoothing regions with variance less than 0.5 and determines the seam path within these regions. A dynamic window fusion strategy is employed: for static regions, a dynamic window fusion strategy is used. Fixed window; for dynamic areas, use a size that is [not specified]. to An adaptive window is used between the images. The system performs distance-weighted calculations on the image pixels on both sides of the seam to achieve gradient connection of pixel grayscale, ultimately generating a panoramic stitched image.

[0042] See attached document Figure 3 This embodiment details the technical process of acquiring a sparse optical flow field that characterizes the actual motion state of equipment in the high dust and drastically fluctuating light conditions of underground coal mines. This is achieved by acquiring the image output from the image acquisition module. After resolving the original image sequence, the system first performs multi-scale adaptive preprocessing. The system reads the global average optical flow amplitude calculated from the previous frame image. This serves as a priori indicator of the motion state in the current frame. The system has a preset velocity threshold. 15 pixels per frame.

[0043] When detected When the current working face equipment is in a slow advancing or stationary state, the system uses a bilinear interpolation algorithm to interpolate the current frame image. Scaling to uniform The standard processing size of a pixel is used to preserve sufficient texture detail for subsequent calculations. When detected... This indicates that the coal mining machine or scraper conveyor is in a high-speed operation state. The large displacement movement causes the optical flow calculation to get stuck in a local extremum. The system automatically triggers the secondary scaling logic to further downsample the image. Pixels. This dynamic resolution adjustment strategy ensures that the motion capture range covers the device's maximum moving speed while reducing the computational load of subsequent gradient calculations and optical flow iterations.

[0044] To address the issue of diffuse dust interference in downhole environments, the system constructs an anti-interference weight model based on texture entropy and gradient to distinguish between dust noise and equipment edges. The system then applies this model to the pre-processed images. The Sobel operator of the convolution kernel calculates the gradient in the horizontal direction. and vertical gradient Based on this, a normalized gradient magnitude is synthesized. Since dust appears as irregular, diffuse textures in images, while mechanical equipment exhibits directional structural textures, the system introduces local texture entropy as a distinguishing feature. The system divides the image into non-overlapping segments. Pixel local window, statistical window for grayscale histogram probability distribution The system calculates the local texture entropy of each window based on the texture entropy calculation formula disclosed in Embodiment 1 or Embodiment 2. During this calculation process, the system calculates the probability. Logarithmic operations are performed and weighted summation is performed, where high entropy values ​​represent dust areas with disordered gray-scale distribution, and low entropy values ​​represent coal wall or equipment surface with uniform texture.

[0045] The system further calculates the anti-interference weighting factor. The system normalizes the calculated texture entropy to obtain... The weighted fusion is then performed according to the weighting factor calculation formula disclosed in the foregoing embodiments. In this step, the fusion coefficient... It is precisely configured to 0.6. This parameter configuration makes the weighting factor... It is not only directly proportional to the gradient magnitude, but also inversely proportional to the texture entropy. During the solution of the optical flow constraint equations, the system uses this weighting factor. As a penalty coefficient for data items, for high-entropy dust regions with lower weights, optical flow calculation will rely more on regularization terms (smoothing constraints) to suppress spurious optical flow vectors generated by dust flow; for device edge regions with higher weights, optical flow calculation will strictly follow the grayscale conservation assumption to ensure the accuracy of motion estimation.

[0046] To address the issues of uneven and dynamically changing illumination in mining lamps, the system incorporates illumination component separation and enhancement processing before optical flow calculation. Utilizing the Retinex algorithm, the system decomposes the image into an illuminance component reflecting light intensity and a reflectance component reflecting the material properties of the object. The system discards the illuminance component, which is susceptible to the position and angle of the light source, retaining only the reflectance component as the input source for optical flow calculation. To further enhance the device contours under low illumination, the system constructs a Hessian matrix for the reflectance component image. The system calculates the optical flow for each pixel. The system calculates the second-order partial derivative matrix within the neighborhood and solves for its eigenvalues. It then uses the eigenvector directions corresponding to the large eigenvalues ​​to enhance ridge and corner features in the image, generating an edge-enhanced feature map for subsequent optical flow field iterations.

[0047] The system employs a pyramid optical flow calculation and iterative optimization strategy to solve for the optical flow field. The system constructs a Gaussian pyramid with four resolution levels. During the optical flow solution process, the system adopts a coarse-to-fine layer-by-layer propagation strategy. At the top of the pyramid (the lowest resolution layer), the system is configured to perform a fixed number of iterations of 3 to quickly converge and capture the overall motion trend of the device, preventing initial estimation errors under large displacements. At the bottom of the pyramid (the highest resolution layer), the system is configured to perform a fixed number of iterations of 5 to refine the optical flow field and capture the local motion of small components such as hydraulic support push rods. During the iteration of each layer, the system calculates the Euclidean distance between the optical flow vectors of two adjacent iterations in real time. When this distance is less than 0.1 pixels / frame, the system determines that the algorithm has converged and terminates the iteration of the current layer early.

[0048] To meet the real-time requirements of downhole video processing, the system finally performs a sparsity pruning operation on the generated dense optical flow field. The system then uses calculated weighting factors... and optical current amplitude The image pixels are divided into three regions and a non-uniform sampling strategy is adopted. For and The system determines the pixel / frame area to be either a background coal wall or a dust interference area (static area), and adopts... Sparse sampling is performed at large pixel intervals; for or The system identifies areas of pixels / frames as critical equipment areas (dynamic areas) such as the coal mining machine cutting section or moving supports, and adopts... The system performs dense sampling at small pixel intervals; for transition regions that do not meet the above conditions, the system uses... Pixel-spaced sampling. The sparse optical flow field output by this step includes position coordinates. and its corresponding motion vector This serves as the input data for subsequent feature registration and stitching.

[0049] See attached document Figure 1 - Appendix Figure 3 This embodiment details how to achieve high-precision image registration in dynamic and complex scenes by utilizing the prior motion information of sparse optical flow fields and the geometric constraints of feature matching. The registration mechanism of this embodiment aims to address the technical challenges of cumulative drift in fast-moving objects using single optical flow methods and the loss of feature points in dust-occluded areas using single feature matching methods. The system constructs a robust transformation model through optical flow-guided feature matching and a region-based weighted fusion strategy.

[0050] Optical flow-guided constrained feature matching: The system first downsamples the input image, then... Scaling the original resolution to To reduce the computational time required for feature extraction, a simplified version of the SIFT (Scale Invariant Feature Transform) algorithm is used at this resolution. To improve real-time performance, the system fixes the number of Gaussian difference pyramid layers in the standard SIFT algorithm to four and compresses the dimension of the feature descriptors from the standard 128 dimensions to 64 dimensions. This dimensionality reduction is achieved through principal component analysis (PCA) or dimensionality reduction projection, preserving key gradient direction information while reducing memory usage and matching computation.

[0051] In the feature matching stage, the system abandons the traditional full-image brute-force search strategy and instead adopts an optical flow-guided local search strategy. The system calls the sparse optical flow field data generated in Example 3 as the motion prior. For any feature point in the previous frame image... The system is based on its corresponding motion vector in the sparse optical flow field. Calculate the predicted coordinates of the feature point in the current frame. The system predicts coordinates. A local search window is constructed centered on a specific element. Traditional algorithms typically set the search radius to over 100 pixels or perform a full-image search, while this embodiment forcibly compresses the matching search radius to a smaller value. Pixels. Within a preset local search window, the system calculates the Hamming distance between candidate feature points and target feature descriptors. The system uses Hamming distance instead of Euclidean distance as the similarity metric and leverages binary XOR operations to accelerate the comparison process of 64-dimensional descriptors, thereby completing coarse matching of feature point pairs in milliseconds.

[0052] Dynamic region mask generation based on YOLO and optical flow: To accurately distinguish between moving equipment areas and static background areas during image registration, the system executes dual-verification region recognition logic. The system loads and runs the lightweight object detection model YOLOv8-mini. This model, pre-trained on an underground equipment dataset, can output the bounding box coordinates of key equipment such as coal mining machines and hydraulic supports in real time.

[0053] Simultaneously, the system performs local statistical analysis on the sparse optical flow field output in Example 3. The system calculates the optical flow vector point's... The mean optical flow amplitude within the neighborhood is denoted as The system sets dynamic judgment thresholds. .

[0054] Based on the target detection results and optical flow statistics, the system generates a pixel-level state mask, dividing the image into three regions: Define the dynamic region: This region contains pixels that are both inside the device bounding box output by the YOLO model and whose corresponding average optical flow magnitude is [value missing]. Greater than the threshold This represents the part of the equipment that is undergoing actual displacement.

[0055] Define the static region: the pixels within this region are located outside the device bounding box, and their average optical flow amplitude is [value missing]. Less than or equal to the threshold This typically corresponds to a relatively static background such as the coal face or the roof of the roadway.

[0056] Transition region: The remaining pixel region that does not meet the above two conditions, usually located at the junction of the device edge and the background.

[0057] Adaptive weight fusion of transformation matrix: The system calculates the transformation matrix based on the sparse optical flow field. Transformation matrix based on feature matching point pairs . It mainly reflects the continuous motion trend at the pixel level, while It mainly reflects the geometric rigidity transformation based on feature points.

[0058] The system uses the generated pixel-level state mask to define the state of each pixel in the image. Dynamically allocate fusion weights. Let... The weights are the values ​​of the optical flow transformation matrix. Weights for the feature matching transformation matrix: In a defined static region, feature point extraction is unstable due to dust interference. Therefore, the system prioritizes the continuity of the optical flow field and configures... for , for .

[0059] In defining the dynamic region, due to optical flow tracking slippage caused by rapid device movement, the system focuses on geometric constraints for feature matching and configuration. for , for .

[0060] In the transition region, the system uses a linear interpolation algorithm to calculate... and This allows the values ​​to transition smoothly between static and dynamic weights, avoiding image tearing caused by sudden weight changes.

[0061] The system, based on the fusion formula disclosed in Embodiment 2 above, utilizes the aforementioned weights to... and Perform a weighted summation to generate the final fusion transformation matrix field. To eliminate high-frequency noise generated by pixel-level fusion, the system finally applies a matrix field to the fused matrix. A Gaussian low-pass filter is used for spatial smoothing to ensure that the transformation parameters of the final output are continuous in spatial distribution, providing an accurate geometric reference for subsequent image stitching.

[0062] This embodiment describes in detail how to eliminate field-of-view occlusion (i.e., black hole effect) and visual discontinuities at seams in dynamic scenes after image registration is completed, thereby generating a complete and seamless panoramic image.

[0063] The splicing scheme in this embodiment is divided into three logical stages: adaptive overlapping area positioning, dynamic occlusion area repair, and optimal smooth seam fusion.

[0064] Adaptive overlapping region localization: The system first uses the sparse optical flow field generated in Example 3 to quickly predict the overlapping region. The system calculates the mean direction and magnitude of all effective optical flow vectors in the sparse optical flow field to infer the overall translation trend of the current frame relative to the previous frame. Based on this translation trend, the system delineates a rectangular prediction region (ROl) on the current frame image, which represents the theoretical image overlap range.

[0065] Within the defined prediction area, the system performs normalized cross-correlation coefficient (NCC) calculation to accurately verify the effectiveness of the overlap. The system calculates the ratio of the product of the covariance and standard deviation of the pixel grayscale values ​​of the two images to be stitched within the prediction area. The system sets the overlap determination threshold to 0.7. When the calculated NCC value is greater than 0.7, the system determines that the area is a valid overlap region and initiates the subsequent stitching process; if the NCC value is less than or equal to 0.7, the system determines that there is no effective visual correlation between the images (e.g., a large and rapid rotation of the device causing a complete switch in the field of view). In this case, the system directly outputs the original resolution image without performing forced stitching to avoid erroneous image overlay.

[0066] Dynamic occlusion area (black hole) repair: In underground fully mechanized mining faces, the rapid movement of the coal mining machine or supports can cause field-of-view occlusion between consecutive frames. This occlusion often manifests as black pixel gaps in traditional stitching. This embodiment identifies and repairs these gaps by monitoring the continuity of the optical flow field.

[0067] The system iterates through each pixel within the effective overlapping area and calculates the difference between its optical flow vector and the optical flow vector of its adjacent pixels (e.g., the pixel on the right). When the system detects that the difference in optical flow vectors between adjacent pixels exceeds a preset threshold (specifically set to 5 pixels / frame), it determines that there is motion tearing at that location, which is a dynamic occlusion edge caused by device movement.

[0068] For the identified dynamically occluded regions, the system employs an improved Poisson fusion algorithm for texture reconstruction. This algorithm is based on gradient domain operations, and its core logic does not directly copy pixel values. Instead, it clones the gradient field of the source image region into the target region by solving the Poisson equation.

[0069] Specifically, the system selects the same spatial region from adjacent frames on the timeline that are determined to be unoccluded (usually the previous or next frame) as a reference template. The system extracts the texture features and grayscale distribution patterns of the reference template to construct a guiding gradient field. Using the boundary pixel values ​​of the region to be repaired in the current frame as constraints (Dirichlet boundary conditions), the system solves the Poisson equation to generate filling pixels by minimizing the difference between the gradient of the region to be reconstructed and the guiding gradient field.

[0070] To meet the real-time requirements of industrial environments, the system has optimized the solution process for the Poisson equation by accelerating it: the maximum number of iterations for the solver is limited to 5, or the iteration is terminated immediately when the pixel grayscale reconstruction error is less than 2 gray levels. This strategy reduces computation time while ensuring a natural visual transition.

[0071] Optimal seam planning and smooth integration: To eliminate seam artifacts at image stitching points, the system abandons the traditional fixed straight seam approach and instead employs an optimal seam search strategy based on optical flow variance. The system calculates the optical flow vector within the overlapping region. Neighborhood variance. The variance value reflects the intensity and disorder of the movement in that area.

[0072] The system uses threshold filtering (variance less than 0.5) to identify areas with gentle changes in optical flow (typically background coal walls or smooth equipment surfaces), and strictly limits the optimal seam path to these areas. This strategy effectively avoids equipment edges, abrupt changes in motion, or complex dusty areas, preventing ghosting or misalignment caused by seams passing through objects.

[0073] After determining the joint path, the system employs a dynamic window fusion algorithm to smoothly transition the images on both sides of the joint. The size of the fusion window is adaptively adjusted based on the dynamic attributes of the region: for regions determined to be static (such as the coal wall background), the system uses a size of [missing information]. A fixed large window for each pixel. Large window blending enables long-distance grayscale gradation, effectively smoothing out lighting differences caused by different angles of the miner's lamp.

[0074] For areas deemed dynamic (such as moving support panels), the system uses dimensions within... to Adaptive small windows between pixels. The system adjusts the window size inversely based on the magnitude of the optical flow: the larger the optical flow magnitude (the faster the movement), the smaller the fusion window. This strategy aims to reduce the fusion range and prevent blurring or ghosting on fast-moving objects.

[0075] Finally, the system performs gradient weighting calculations within the fusion window. For pixels in the image to the left of the seam, the system assigns them a weight that decreases as they get closer to the seam; for pixels in the image to the right of the seam, the system assigns them a weight that increases as they get closer to the seam. Through this distance weighting, the system achieves a smooth linear or non-linear transition of pixel grayscale at the seam between the two images, ultimately generating a panoramic stitched video stream with high visual consistency and no obvious discontinuities.

[0076] See attached document Figure 1This embodiment describes in detail the apparatus structure for performing the above-described method steps from a functional logic perspective. The apparatus is specifically implemented as a software system, hardware circuitry, or a combination of both running on a computer device.

[0077] The present invention provides a working surface panoramic stitching device that integrates an improved optical flow algorithm. The device includes: an adaptive preprocessing module 601, a robust optical flow calculation module 602, a feature fusion and registration module 603, and a dynamic stitching and repair module 604.

[0078] The adaptive preprocessing module 601 is configured to receive raw high-resolution image data input from the image acquisition hardware. (pixels). This module integrates a velocity monitoring unit and a multi-scale scaling unit. The velocity monitoring unit continuously reads the global average optical flow amplitude of the previous frame and logically compares this amplitude with a preset velocity threshold (15 pixels / frame). The multi-scale scaling unit responds to the comparison result of the interference monitoring unit: when the amplitude is small or equal to the threshold, it performs a scaling operation using bilinear interpolation to downsample the image to the first resolution. When the amplitude exceeds the threshold, a scaling operation is performed based on the bilinear interpolation algorithm to downsample the image to the second resolution. The output of this module is the scaled image data, which serves as the input stream for subsequent modules.

[0079] The robust optical flow calculation module 602 is the core computing unit of the device, used to extract reliable motion information in complex interference environments. This module specifically includes the following sub-units: Anti-interference weight generation unit: This unit is equipped with Sobel operator convolution kernels and local histogram statistical logic. It calculates the gradient magnitude and local texture entropy of the image in parallel and synthesizes anti-interference weight factors according to the weighting formula disclosed in Embodiment 1. This unit marks dust interference by identifying high-entropy regions and device edges by identifying high-gradient regions.

[0080] Illumination Processing Unit: This unit runs the Retinex decomposition algorithm to separate the input image into reflectance and illuminance components, discarding the illuminance component. Subsequently, this unit applies the Hessian matrix to the reflectance component, using second-order derivative features to enhance the geometric edges in the image.

[0081] Pyramid Iteration Unit: This unit constructs a 4-layer Gaussian pyramid data structure in memory. It solves the optical flow constraint equations in a coarse-to-fine order. At the top of the pyramid, this unit performs a fixed 3 iterations; at the bottom of the pyramid, it performs a fixed 5 iterations. This unit also includes convergence detection logic, which interrupts the computation when the Euclidean distance between adjacent iteration results is less than 0.1 pixels.

[0082] Sparse sampling unit: This unit receives the anti-interference weighting factor and the calculated dense optical flow field. Based on the weighting factor and the magnitude of the optical flow, it divides the image into static background areas, dynamic device areas, and transition areas, and applies the corresponding methods to each area. , and The non-uniform spacing of pixels downsamples the optical flow field, outputting sparse optical flow field data.

[0083] The feature fusion and registration module 603 is configured to generate image transformation parameters by combining optical flow priors and feature geometric constraints. This module includes: Constrained Feature Extraction Unit: This unit performs image extraction... Resolution transformation is performed, and a simplified version of the SIFT algorithm is run to extract 64-dimensional feature descriptors. This unit utilizes the sparse optical flow field output by the robust optical flow calculation module 602 to predict feature point locations and, in the vicinity of the predicted locations... Perform Hamming distance-based feature matching within the pixel radius.

[0084] Dynamic Mask Generation Unit: This unit integrates a pre-trained YOLOv8-mini object detection network to output device bounding boxes. Simultaneously, this unit calculates the local mean of optical flow amplitude. This unit combines bounding box information and optical flow amplitude information through a logical AND operation to generate pixel-level masks that identify dynamic regions, static regions, and transition regions.

[0085] Weighted Matrix Fusion Unit: This unit receives both optical flow-based and feature matching-based transformation matrices. Based on the region identifiers output by the dynamic mask generation unit, this unit assigns different fusion weights to different regions (static regions emphasize optical flow weights, dynamic regions emphasize feature weights), and generates the final smooth transformation matrix field through weighted summation and Gaussian filtering.

[0086] The dynamic stitching and repair module 604 is configured to perform the final compositing and repair of the image. This module includes: Overlap Detection Unit: This unit predicts the overlap range using the optical flow mean and calculates the normalized cross-correlation coefficient (NCC). This unit triggers the stitching logic only when the NCC value is greater than 0.7; otherwise, it directly outputs the original image.

[0087] Occlusion Repair Unit: This unit monitors the optical flow difference between adjacent pixels within the overlapping area. When the difference value exceeds 5 pixels / frame, this unit locks the corresponding dynamically occluded region and calls the Poisson fusion algorithm solver. The solver uses temporally adjacent unoccluded frames as templates and iteratively solves the Poisson equation (maximum 5 iterations) to fill the pixels in the occluded region.

[0088] Smooth Blending Unit: This unit calculates the optical flow variance to plan the optimal seam path. This unit is configured with dynamic window generation logic for static region generation. Fixed window for dynamic region generation An adaptive window for pixels. This unit performs distance-based gradient weighting within the window to complete the grayscale transition between the two sides of the seam.

[0089] Furthermore, one embodiment of the present invention provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs are configured to perform the working surface panoramic stitching method of the fusion improved optical flow algorithm described in the above embodiments.

[0090] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the panoramic stitching method for the working surface of the fusion improved optical flow algorithm described in the above embodiments. The storage medium includes, but is not limited to, random access memory, read-only memory, flash memory, hard disk drive, or solid-state drive.

Claims

1. A panoramic stitching method for working surfaces integrating an improved optical flow algorithm, characterized in that, Includes the following steps: S1. Obtain the original image sequence of the working surface, perform multi-scale adaptive preprocessing on the original image sequence, construct an anti-interference weight model based on texture entropy and gradient, and combine Retinex illumination component separation technology to generate a sparse optical flow field that can characterize the motion state of the device through pyramid iterative calculation. S2. Using the sparse optical flow field to guide feature matching, obtain the feature matching transformation matrix. At the same time, combine the target detection result and the optical flow amplitude to generate a dynamic region mask. Based on the dynamic region mask, perform adaptive weighted fusion of the optical flow-based transformation matrix and the feature matching transformation matrix to obtain the final fused transformation matrix. S3. Based on the sparse optical flow field, predict the overlapping area, use the improved Poisson fusion algorithm to repair the dynamic occlusion area in the overlapping area, calculate the optimal seam path by calculating the optical flow variance, and use a dynamic window to smoothly fuse the image to generate a panoramic stitched image.

2. The panoramic stitching method for the working surface fused with the improved optical flow algorithm according to claim 1, characterized in that, The multi-scale adaptive preprocessing of the original image sequence specifically includes: Calculate the global average optical flow amplitude of the previous frame image; The global average optical flow amplitude is compared with a preset velocity threshold. If the global average optical flow amplitude is less than or equal to the velocity threshold, the current frame image is scaled to the first resolution using bilinear interpolation. If the global average optical flow amplitude is greater than the velocity threshold, the current frame image is scaled to a second resolution, wherein the second resolution is smaller than the first resolution.

3. The panoramic stitching method for the working surface fused with the improved optical flow algorithm according to claim 1, characterized in that, The construction of the anti-interference weight model based on texture entropy and gradient specifically includes: The Sobel operator is used to calculate the gradients of the image in the horizontal and vertical directions, respectively, to obtain the normalized gradient magnitudes. The image is divided into non-overlapping local windows, and the probability distribution of the gray-level histogram within each local window is calculated. The local texture entropy is calculated based on the probability distribution of the gray-level histogram, where the local texture entropy is the negative sum of the product of the gray-level probability and the natural logarithm of the gray-level probability; The local texture entropy is normalized to obtain the normalized texture entropy; Calculate the anti-interference weighting factor based on the preset fusion coefficient; The anti-interference weight factor is calculated as follows: the product of the fusion coefficient and the normalized gradient magnitude, plus the product of the difference after subtracting the fusion coefficient and the difference after subtracting the normalized texture entropy; the anti-interference weight factor is introduced as a penalty term into the optical flow constraint equation.

4. The panoramic stitching method for the working surface fused with the improved optical flow algorithm according to claim 1, characterized in that, The specific components of the Retinex illumination component separation technology include: The Retinex algorithm is used to decompose the image into illuminance and reflectance components. The illuminance component is discarded, and the reflectance component is retained. The Hessian matrix of the reflectance component is then calculated. The eigenvalues ​​of the Hessian matrix are solved, and the reflection components are enhanced using the eigenvalues. The enhanced image is then used as the input for optical flow calculation.

5. A panoramic stitching method for working surfaces based on an improved optical flow algorithm according to claim 3, characterized in that, The generation of a sparse optical flow field capable of characterizing the motion state of the device also includes sparsification pruning of the dense optical flow field: Determine the anti-interference weighting factor and optical flow amplitude of each pixel; If the anti-interference weight factor is less than the first weight threshold and the optical flow amplitude is less than the first amplitude threshold, it is determined to be a static region, and sampling is performed using the first pixel interval; If the anti-interference weight factor is greater than the second weight threshold or the optical flow amplitude is greater than the second amplitude threshold, it is determined to be a dynamic region, and sampling is performed using the second pixel interval; if the above conditions are not met, sampling is performed using the third pixel interval. Wherein, the first pixel interval is greater than the third pixel interval, and the third pixel interval is greater than the second pixel interval.

6. The panoramic stitching method for the working surface fused with the improved optical flow algorithm according to claim 1, characterized in that, The specific features of using the sparse optical flow field to guide feature matching include: Extract feature descriptors from the image and compress the dimensions of the feature descriptors to a preset dimension; Based on the motion vector corresponding to the feature point in the previous frame in the sparse optical flow field, predict the target coordinate position of the feature point in the previous frame in the current frame; A local search window is constructed within a preset pixel radius, centered on the target coordinates. Within the local search window, the similarity between candidate feature points and target feature points is calculated using Hamming distance to complete feature matching.

7. The panoramic stitching method for the working surface fused with the improved optical flow algorithm according to claim 1, characterized in that, The adaptive weighted fusion of the optical flow-based transformation matrix and the feature matching transformation matrix based on the dynamic region mask specifically includes: The bounding box of the device is obtained using a target detection model, and the mean local optical flow amplitude of the sparse optical flow field is calculated. Generate pixel-level state masks: if a pixel is located within the bounding box and the average local optical flow amplitude is greater than a preset dynamic threshold, it is marked as a defined dynamic region; if a pixel is located outside the bounding box and the average local optical flow amplitude is less than or equal to the preset dynamic threshold, it is marked as a defined static region; otherwise, it is marked as a transition region. A first weight is assigned to the optical flow-based transformation matrix, and a second weight is assigned to the feature matching transformation matrix; in the defined static region, the first weight is set to a value greater than the second weight; in the defined dynamic region, the first weight is set to a value less than the second weight; in the transition region, the first weight and the second weight are calculated by linear interpolation. The two transformation matrices are weighted and summed according to the first weight and the second weight, and the summation result is smoothed by Gaussian filtering to obtain the fused transformation matrix.

8. The panoramic stitching method for the working surface fused with the improved optical flow algorithm according to claim 1, characterized in that, The method of using an improved Poisson fusion algorithm to repair dynamically occluded areas within overlapping regions specifically includes: The mean direction of the sparse optical flow field is calculated to predict the extent of the overlapping region, and the normalized cross-correlation coefficient is calculated within the overlapping region. When the normalized cross-correlation coefficient is greater than a preset effective threshold, the pixels within the overlapping region are traversed. Calculate the optical flow vector difference between adjacent pixels. If the optical flow vector difference is greater than a preset difference threshold, the location is determined to be a dynamic occlusion area. Select the same position region of adjacent, unobstructed frames on the timeline as the reference template; A Poisson equation is constructed, guided by the gradient field of the reference template, and constrained by the boundary pixel values ​​of the region to be repaired in the current frame. The Poisson equation is then iteratively solved to generate fill pixels.

9. A panoramic stitching method for working surfaces based on an improved optical flow algorithm according to claim 1, characterized in that, The method of using dynamic windows to smoothly blend images specifically includes: Calculate the neighborhood variance of the optical flow vector within the overlapping region, filter out regions where the neighborhood variance is less than a preset variance threshold, and plan the optimal seam path within the region. The fusion window size is determined based on the dynamic properties of the region: a large window of fixed size is used for static regions, and an adaptive small window that decreases as the optical flow amplitude increases is used for dynamic regions. Within the fusion window, image pixels on one side of the seam are assigned a weight that decreases as they get closer to the seam, while image pixels on the other side of the seam are assigned a weight that increases as they get closer to the seam, and the weighted pixel values ​​are calculated.

10. A panoramic stitching device for a working surface integrating an improved optical flow algorithm, characterized in that, A panoramic stitching method for working surfaces using an improved optical flow algorithm as described in any one of claims 1-9, comprising: An adaptive preprocessing module is configured to acquire the original image sequence of the working surface and perform multi-scale adaptive preprocessing on the original image sequence. The robust optical flow calculation module is configured to build an anti-interference weight model based on texture entropy and gradient, and combined with Retinex illumination component separation technology, it generates a sparse optical flow field that can characterize the motion state of the device through pyramid iterative calculation. The feature fusion and registration module is configured to guide feature matching using the sparse optical flow field to obtain the feature matching transformation matrix; simultaneously, it generates a dynamic region mask by combining the target detection result and the optical flow amplitude, and performs adaptive weighted fusion of the optical flow-based transformation matrix and the feature matching transformation matrix based on the dynamic region mask to obtain the final fused transformation matrix. The dynamic stitching and repair module is configured to predict the overlapping area based on the sparse optical flow field, repair the dynamically occluded area within the overlapping area using an improved Poisson fusion algorithm, plan the optimal seam path by calculating the optical flow variance, and use a dynamic window to smoothly fuse the image to generate a panoramic stitched image.

Images