A machine vision-based vertical shaft panorama image splicing method

Abstract

The present application relates to the technical field of image stitching, and particularly relates to a vertical shaft panoramic image stitching method based on machine vision. The present application firstly performs optical flow analysis on continuous frame shaft wall images to obtain a camera displacement vector, then obtains an imaging ratio between a pixel point and a shaft wall, and obtains a transverse drift allowance of the pixel point; in each shaft wall image, a parallax ratio coefficient between a guide rail and the shaft wall at an angle of each pixel point is obtained, a transverse drift displacement component of the pixel point is obtained, and then a de-shock displacement vector of each pixel point is obtained; then a distortion displacement vector of each pixel point is obtained, and the shaft wall image is corrected in combination with the imaging ratio; the corrected shaft wall image is sequentially stitched to obtain a panoramic image. The present application introduces a physical mechanical constraint of a hoisting system to solve the technical problem that a geometrically correct and physically credible panoramic image cannot be obtained in a dynamic swing and repeated texture environment, and improves the vertical shaft panoramic image stitching quality.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, and more specifically to a method for stitching panoramic images of a vertical shaft based on machine vision. Background Technology

[0002] Vertical shafts are the core passage of a mine hoisting system, and their shaft wall safety monitoring is crucial for production safety. Traditional maintenance methods rely on slow, manual inspections, which are inefficient and pose safety hazards. With the development of machine vision technology, stitching together continuous images collected during the hoisting process into a panoramic image is a highly efficient means of achieving digital monitoring of the shaft wall and quantitative analysis of cracks.

[0003] Current panoramic image stitching for vertical shafts primarily relies on feature point matching or global optical flow algorithms to extract overlapping features between adjacent frames for alignment. However, the mechanical movement gap between the hoisting container and the guide rail causes nonlinear lateral swaying or vibration of the hoisting container under the constraint of the rigid guide rail, resulting in significant depth parallax. Conventional global motion compensation cannot decouple this parallax, leading to misalignment or ghosting in the stitched images. Furthermore, the repetitive texture of the shaft wall makes feature matching prone to longitudinal spurious slip (false matching), resulting in geometric distortion of the panoramic image and making it impossible to support quantitative measurement of crack width. Summary of the Invention

[0004] To address the poor quality of existing vertical shaft panoramic image stitching methods, this invention provides a machine vision-based method for stitching vertical shaft panoramic images, comprising: Acquire continuous frames of well wall images captured by the camera inside the vertical shaft during its vertical lifting process on the track; perform optical flow analysis on the well wall images to obtain the displacement vector of each pixel in the well wall images, and obtain the camera displacement vector; Divide the continuous frame well wall images to obtain all well wall image sequences, with different sequences corresponding to well walls of the same length; in each sequence, based on the longitudinal component of the displacement vector of all pixels in the well wall image, the well wall length corresponding to the sequence, and the camera acquisition parameters, obtain the imaging ratio between the pixel and the well wall, and combine the preset movement gap of the track to obtain the lateral drift tolerance of the pixel. In each well wall image, based on the longitudinal component of the camera displacement vector and the longitudinal component of the displacement vector of each pixel, the parallax ratio coefficient between the guide rail and the well wall at each pixel angle is obtained, and the lateral drift displacement component of the pixel is obtained by combining the lateral component of the camera displacement vector, and the vibration reduction displacement vector of each pixel is obtained based on the lateral drift displacement component. In each wellbore image, the distortion displacement vector of each pixel is obtained based on the lateral drift tolerance, the deformation consistency characteristics of adjacent pixels, and the seismic displacement vector of each pixel. The wellbore image is then corrected by combining the imaging ratio. The corrected wellbore images are then stitched together sequentially to obtain a panoramic image.

[0005] Furthermore, the method for obtaining the camera displacement vector includes: Based on the distribution characteristics of the lateral component of the displacement vector of pixels at the same position in any consecutive preset number of well wall images, representative parameters of pixels at each position in the non-guide rail region are obtained. Based on the representative parameters, the guide rail region in each wellbore image is determined, and the camera displacement vector is obtained according to the distribution characteristics of the displacement vectors of the pixels in the guide rail region in all wellbore images.

[0006] Furthermore, the method for acquiring the wellbore image sequence includes: The images collected also include fixed-distance crossbeams in the vertical shaft; in each image of the well wall, the response parameters of the crossbeams are obtained based on the longitudinal gradient of all pixels in the preset central region. In continuous frame wellbore images, based on the temporal changes of the beam response parameters corresponding to the preset central region of the wellbore image, the beam positioning frame is determined, and the continuous frame wellbore images are divided based on the beam positioning frame to obtain all wellbore image sequences.

[0007] Furthermore, the method for obtaining the imaging ratio includes: For the first image sequence of the well wall in the acquisition time sequence, the imaging ratio between the pixel and the well wall is obtained based on the pinhole imaging model and the camera acquisition parameters; the camera acquisition parameters include focal length, sensor pixel size and shooting distance. For a sequence of images other than the first wellbore image, the total span of the pixels is determined based on the longitudinal component of the displacement vector of all pixels in all wellbore images. The ratio of the total span of the pixels to the length of the corresponding wellbore in the sequence is used as the imaging ratio between the pixels and the wellbore.

[0008] Furthermore, the method for obtaining the lateral drift tolerance includes: In each sequence, the product of the imaging ratio and the preset floating gap is used as the lateral drift tolerance of the pixel.

[0009] Furthermore, the method for obtaining the disparity scaling factor includes: When the magnitude of the longitudinal component of the camera displacement vector is less than a preset displacement threshold, the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle is set to a preset value; when the magnitude of the longitudinal component of the camera displacement vector is not less than the preset displacement threshold, the ratio of the longitudinal component of the pixel displacement vector to the longitudinal component of the camera displacement vector is used as the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle.

[0010] Furthermore, the method for obtaining the lateral drift displacement component includes: The product of the parallax ratio coefficient between the guide rail and the well wall at each pixel angle and the lateral component of the camera displacement vector is used as the lateral drift displacement component of each pixel.

[0011] Furthermore, the method for obtaining the vibration reduction displacement vector includes: Subtract the lateral drift displacement component from the displacement vector of each pixel to obtain the de-vibration displacement vector.

[0012] Furthermore, the method for obtaining the distorted displacement vector includes: In each wellbore image, the distortion displacement vector of each pixel is used as the optimization variable. Based on the difference between the distortion displacement vector of each pixel and the vibration reduction displacement vector, the difference between the distortion displacement vector of each pixel and the corresponding distortion displacement vector of other pixels within a preset range, and the difference between the distortion displacement vector of each pixel and the allowable lateral drift, a function model is constructed. Based on the optimization variable corresponding to the minimum value of the function model, the distortion displacement vector of each pixel is obtained.

[0013] Furthermore, the correction of the wellbore image includes: In each wellbore image, a scaling factor is determined based on the deviation of the imaging ratio from the preset resolution to scale the wellbore image vertically, and the pixels are horizontally translated by combining the horizontal component of the distortion displacement vector of each pixel to obtain the corrected wellbore image.

[0014] This invention has the following beneficial effects: First, it acquires continuous frame images of the well wall captured by a camera inside the shaft during vertical lifting on a track, providing a foundation for subsequent analysis, correction, and stitching; it performs optical flow analysis on the well wall images to obtain the displacement vector of each pixel in the well wall images, and also obtains the camera displacement vector, preparing for depth parallax in subsequent stages; it divides the continuous frame images of the well wall, acquiring all well wall image sequences to avoid accumulated errors; then, in each sequence, it obtains the imaging ratio between the pixel and the well wall, performing a mapping between the real world and the imaging, and then, combined with the preset movement gap of the track, obtains the pixel's position on the well wall. The invention addresses the following: First, it decouples the depth parallax in each wellbore image, obtaining the parallax ratio coefficient between the guide rail and the wellbore at each pixel angle. Then, it combines the lateral component of the camera displacement vector to obtain the lateral drift displacement component of the pixel, thereby eliminating lateral drift caused by camera vibration and obtaining the vibration-reducing displacement vector for each pixel. Second, based on the lateral drift tolerance, the deformation consistency characteristics of adjacent pixels, and the vibration-reducing displacement vector of each pixel, it obtains the compliant distortion displacement vector for each pixel and corrects the wellbore image by combining it with the imaging ratio. Finally, it stitches the corrected wellbore images sequentially to obtain a panoramic image. This invention solves the technical problem of obtaining geometrically correct and physically reliable panoramic images in dynamic swing and repetitive texture environments by introducing physical and mechanical constraints of the hoisting system, thus improving the quality of panoramic image stitching for vertical shafts. Attached Figure Description

[0015] Figure 1 The flowchart illustrates a machine vision-based panoramic image stitching method for vertical shafts provided by this invention. Detailed Implementation

[0016] like Figure 1 The flowchart shown is a machine vision-based panoramic image stitching method for vertical shafts provided in an embodiment of the present invention, specifically including: Step S1: Acquire continuous frame images of the shaft wall captured by the camera inside the shaft during the vertical lifting process on the track; perform optical flow analysis on the shaft wall images to obtain the displacement vector of each pixel in the shaft wall images, and obtain the camera displacement vector.

[0017] It should be noted that the specific implementation scenario targeted by the embodiments of the present invention is as follows: the camera is installed on the hoisting system of the vertical shaft, the camera is vertically lifted along the guide rail and takes a series of images of the shaft wall during the lifting process to analyze and stitch them into a panoramic image.

[0018] The well wall is usually annular, and the camera's field of view may not be able to fully cover it. Each series of well wall images may only cover one vertical column of the well wall, so it may be necessary to adjust the camera angle multiple times to capture all vertical columns. The acquisition and stitching methods for all vertical columns are the same. In this embodiment of the invention, only one vertical column is used as an example for analysis and description, and will not be repeated.

[0019] Specifically, a gigabit Ethernet industrial camera with a resolution of, for example, 2448×2048, and a global shutter is selected to prevent motion blur. The industrial camera is installed on top of the hoisting container, such as a cage, in the vertical shaft hoisting system, with the optical axis perpendicular to the shaft wall to capture images vertically towards the wall. A wide-angle lens, such as an 8mm focal length, is selected for the camera to ensure that the field of view covers the guide rail and most (e.g., at least 3m) of the shaft wall. High-intensity explosion-proof LED line light sources are symmetrically installed on both sides of the camera, using a low-angle side lighting method to enhance the shadow contrast of cracks and surface irregularities in the shaft wall. The hoisting container moves vertically along the rigid guide rail at a speed of 2-4m / s; in this embodiment, a uniform speed of 4m / s is adopted. The camera captures continuous frames of shaft wall images during the vertical hoisting process on the track. The frame rate range is 30-60fps, ensuring that the overlap rate between adjacent frames is greater than 60%; in this embodiment, 60fps is adopted.

[0020] In other embodiments, implementers may also adjust the data acquisition process parameters according to specific scenarios and actual applications.

[0021] It should be noted that during the vertical lifting process, the lifting container may be affected by airflow or mechanical movement gaps, resulting in random lateral flapping or twisting, which in turn generates parallax. Therefore, in addition to the well wall, the well wall image must also include rigid guide rails. The guide rails provide a basis for subsequent analysis of the camera's motion attitude, helping to decouple and compensate for parallax, thereby improving the stitching effect. In the continuous frame well wall images, horizontally traversing fixed-distance beams will also appear periodically. The fixed-distance beams can be used as longitudinal anchor points to discretize the segmented well wall, so as to individually correct the continuous frame well wall image corresponding to each segment of the well wall, avoiding cumulative error interference.

[0022] Since the motion of pixels at different locations in the well wall image is not consistent due to depth difference or lens distortion, optical flow analysis can obtain the displacement vector of each pixel, providing a basis for subsequent decoupling of parallax in conjunction with camera motion attitude for compensation; therefore, this embodiment of the invention further performs optical flow analysis on the well wall image to obtain the displacement vector of each pixel in the well wall image and obtain the camera displacement vector.

[0023] As an example, using the first frame of the well wall image as a reference, dense optical flow algorithms such as the Farneback algorithm or the TV-L1 algorithm are used to calculate the displacement vector of each pixel in each well wall image except the first frame. The displacement vector is an initial displacement estimate that includes interference such as camera vibration and noise. Implementers may also use other optical flow analysis algorithms, which are well known technologies along with dense optical flow algorithms and will not be described in detail here.

[0024] Considering that the guide rail is the rigid foreground closest to the camera, its motion information in the well wall image is less affected by depth parallax and most directly reflects the changes in the camera's physical attitude, making it the best reference benchmark for solving camera motion. At the same time, in the initial lifting phase of the lifting system, the lifting environment is relatively simple, and the position distribution of the guide rail in the well wall image is relatively fixed, and the lateral displacement changes of the corresponding pixels are relatively stable and consistent. By comparing the changes in the displacement vectors of pixels at the same position in the well wall image, we can help capture the difference in stability between the displacement changes of the guide rail and the well wall, thereby determining the guide rail region in the well wall image. Then, based on the motion changes of the pixels in the guide rail region, we can solve the camera motion and determine the camera displacement vector. Based on this, in a preferred embodiment of the present invention, the method for obtaining the camera displacement vector includes: Based on the distribution characteristics of the lateral component of the displacement vector of pixels at the same position in any consecutive preset number of wellbore images, representative parameters of pixels at each position that are in the non-guide rail region are obtained; based on the representative parameters, the guide rail region in each wellbore image is determined, and the camera displacement vector is obtained according to the distribution characteristics of the displacement vector of pixels in the guide rail region in all wellbore images.

[0025] As an example, the preset number is no less than 100, and 100 is used in this example. First, construct the image coordinate system for each wellbore image (e.g., with the image center as the origin). In the first 100 consecutive wellbore images, the variance of the lateral component of the displacement vector of all pixels with the same coordinate position is used as the representative parameter of the corresponding non-guide rail region pixels. Further, set a judgment threshold and select positions in each wellbore image that are less than the judgment threshold as guide rail regions. After determining the guide rail regions in each wellbore image, the mean value of the displacement vector of the pixels in the guide rail regions of all wellbore images can be used as the camera displacement vector.

[0026] In other examples, implementers may also arbitrarily select 100 consecutive frames of wellbore images for analysis to determine representative parameters.

[0027] The variance calculation method is as follows: For each horizontal component, its magnitude L is taken. The sign of the magnitude of the horizontal component to the left is set to -, and the sign of the magnitude of the horizontal component to the right is set to +. Then, the variance is calculated on the dimension with signed values. The larger the variance, the greater the lateral change fluctuation of the displacement vector, the lower the probability of it being a pixel in the guide rail area, and the larger the representative parameter. The judgment threshold is obtained by collecting the camera noise benchmark in a static state. For example, based on the 3σ principle, three times the variance of the camera static noise is taken as the judgment threshold. The implementer can also adjust it himself.

[0028] Step S2: Divide the continuous frame well wall images and obtain all well wall image sequences. Different sequences correspond to well walls of the same length. In each sequence, based on the longitudinal component of the displacement vector of all pixels in the well wall image, the well wall length corresponding to the sequence, and the camera acquisition parameters, obtain the imaging ratio between the pixel and the well wall, and combine the preset movement gap of the track to obtain the allowable lateral drift of the pixel.

[0029] Further segmenting the continuous frame wellbore images prepares for subsequent segmental calibration of the wellbore images in the corresponding wellbore image sequence for each segment in the vertical direction, in order to avoid cumulative errors.

[0030] Preferably, in one embodiment of the present invention, considering that the fixed-distance crossbeams on the well wall of the vertical shaft can provide physical anchor points for segmentation, and that the crossbeams are the most obvious horizontal markers in the well shaft, appearing as significant horizontal edges in the well wall image; by analyzing the longitudinal gradient of pixels in a preset central region of the well wall image, it is possible to detect horizontal edges, i.e., determine the probability of the crossbeams appearing. Furthermore, setting a preset central region instead of performing gradient analysis on the entire well wall image reduces computational load, thereby quickly and accurately locating the exact moment of the crossbeam's appearance in continuous well wall images, thus dividing and acquiring a sequence of well wall images, and ensuring that different sequences correspond to well walls of the same length (because the crossbeams are fixed-distance, the spacing between adjacent crossbeams is the same, and the corresponding well wall lengths are the same); the method for acquiring the well wall image sequence includes: The collected images also include a fixed-distance crossbeam in the vertical shaft. In each well wall image, the crossbeam response parameters are obtained based on the longitudinal gradient of all pixels in the preset central region. In continuous frame well wall images, the crossbeam positioning frame is determined based on the temporal change of the crossbeam response parameters corresponding to the preset central region of the well wall image. Based on the crossbeam positioning frame, continuous frame well wall images are divided to obtain all well wall image sequences.

[0031] As an example, the preset center region is set to 1 / 3 of the well wall image, and the center of the preset center region coincides with the center of the well wall image. The implementer can also adjust the region size, such as setting it to 20%-70% of the well wall image. Then, in the preset center region, the vertical gradient of each pixel is calculated using the vertical Sobel operator, and the absolute values ​​of the magnitudes of the vertical gradients of all pixels are summed as the beam response parameter. The beam response parameter reflects the probability of a beam appearing in the preset center region, preparing for the subsequent determination of the beam positioning frame. The beam response parameters are further sorted according to the acquisition sequence of the wellbore images to construct a time-series variation curve. Peak detection is performed on the time-series variation curve, and the wellbore image corresponding to each peak is used as the beam positioning frame. Starting from the first wellbore image, the beam positioning frame is used as the last frame of each wellbore image sequence, thereby obtaining all wellbore image sequences.

[0032] It should be noted that the application of the longitudinal Sobel operator, the fitting of time-series variation curves, and peak detection are all well-known techniques and will not be elaborated further.

[0033] Since the complete operation process of the hoisting container includes three stages: start-up and acceleration, constant speed operation, and deceleration and stopping, the speed changes linearly or in an S-shape during the acceleration and deceleration stages, and is not a constant value; in addition, there may be resistance effects such as airflow or uneven distribution of friction in the shaft, which will cause the hoisting container to produce instantaneous speed fluctuations. Images acquired under non-uniform speed conditions may have certain distortions. For example, when the instantaneous speed is high, the shaft wall is compressed, and when the speed is slow, the shaft wall is stretched. Directly stitching panoramic images of the shaft wall in chronological order will not produce good results. Based on this, in each sequence, the embodiments of the present invention obtain the imaging ratio between the pixel and the well wall according to the longitudinal component of the displacement vector of all pixels in the well wall image, the well wall length corresponding to the sequence, and the camera acquisition parameters. The imaging ratio can help to build a rigid mapping between imaging and reality, quantify the number of pixels corresponding to a unit well wall length in the well wall image, and prepare for the subsequent conversion of the preset sliding gap of the guide rail in the real world into the lateral drift tolerance of the pixels in the image.

[0034] Preferably, in one embodiment of the present invention, considering that the longitudinal component of the displacement vector of a pixel in each wellbore image sequence can help evaluate the pixel span, the pixel span characterizes the imaging length corresponding to the wellbore length, and its ratio to the wellbore length in the real physical world can help evaluate the imaging ratio between the pixel and the wellbore; and considering that the displacement vector of the pixel in the first wellbore image sequence cannot be evaluated, and therefore the pixel span cannot be evaluated, the imaging ratio between the pixel and the wellbore can be evaluated based on the imaging principle; the method for obtaining the imaging ratio includes: For the first wellbore image sequence in the acquisition timeline, the imaging ratio between the pixel and the wellbore is obtained based on the pinhole imaging model and the camera acquisition parameters; the camera acquisition parameters include focal length, sensor pixel size and shooting distance. For a sequence of images other than the first wellbore image, the total span of the pixels is determined based on the longitudinal component of the displacement vector of all pixels in all wellbore images. The ratio of the total span of the pixels to the length of the corresponding wellbore in the sequence is used as the imaging ratio between the pixels and the wellbore.

[0035] As an example, for the first wellbore image sequence in the acquisition timeline, based on the pinhole imaging principle, we know that: The imaging size h of the well wall is equal to the number of pixels k multiplied by the sensor pixel size p, from which the imaging ratio can be deduced. The unit is pixels / mm. Wherein, the shooting distance D is the design calibration distance when the camera is installed.

[0036] For a sequence of wellbore images that is not the first in the acquisition time sequence, the mean of the longitudinal components of the displacement vectors of all pixels in each wellbore image is taken as the pixel displacement span in the wellbore image; the sum of the pixel displacement spans in all wellbore images in the sequence is taken as the total pixel span, which represents the cumulative number of pixel rows in the longitudinal direction and is used to represent the imaging length of the corresponding wellbore in the sequence; then, the total pixel span in the image is taken as the numerator, the corresponding wellbore length in the sequence is taken as the denominator, and the ratio of the fractions is taken as the imaging ratio between the pixel and the wellbore.

[0037] After obtaining the imaging ratio between the pixel and the well wall, the lateral drift tolerance of the pixel can be obtained by combining the preset movement gap of the track. The lateral drift tolerance quantifies the pixel offset tolerance caused by the lateral drift deviation caused by the preset movement gap in the image, which prepares for subsequent constraint of pixel drift to accurately evaluate the distortion displacement vector of the pixel.

[0038] Preferably, in one embodiment of the present invention, since the imaging scale is based on the longitudinal imaging of the well wall, the lateral drift deviation of the camera caused by the preset movement gap in the real world can be simultaneously mapped to the image. Therefore, the method for obtaining the lateral drift tolerance includes: In each sequence, the product of the imaging ratio and the preset movement gap is used as the lateral drift tolerance of the pixel. The preset movement gap refers to the design gap for the canister lug movement, which needs to be determined based on the relevant design data of the shaft hoisting system. In this embodiment, the preset movement gap is 10mm. In other embodiments, a relaxation factor such as 1-1.5 can be introduced to appropriately increase the lateral drift tolerance.

[0039] Step S3: In each well wall image, based on the longitudinal component of the camera displacement vector and the longitudinal component of the displacement vector of each pixel, obtain the parallax ratio coefficient between the guide rail and the well wall at the angle of each pixel, and combine it with the lateral component of the camera displacement vector to obtain the lateral drift displacement component of the pixel, and obtain the vibration reduction displacement vector of each pixel based on the lateral drift displacement component.

[0040] In vertical shaft imaging, the guide rail is closer to the camera, while the shaft wall is farther away. According to the perspective projection law, the imaging motion displacement of objects closer to the camera is greater than that of objects farther away. By utilizing this difference in longitudinal displacement, the relative depth of each pixel in the shaft wall image can be deduced, thereby accurately calculating the projection of the lateral vibration of the hoisting container caused by the movement gap onto the shaft wall depth plane.

[0041] Based on this, the embodiments of the present invention first obtain the parallax ratio coefficient between the guide rail and the well wall at each pixel angle in each well wall image, according to the longitudinal component of the camera displacement vector and the longitudinal component of the displacement vector of each pixel. Since the well wall is farther from the camera than the guide rail, the lateral drift projection on the well wall is relatively smaller for the same camera shaking. The parallax ratio coefficient decouples the depth parallax between the guide rail and the well wall, thus preparing for the subsequent evaluation of how much error the camera's lateral vibration has caused on the well wall, i.e., calculating the lateral drift displacement component of each pixel, by combining the lateral component of the camera displacement vector. The lateral drift displacement component characterizes the lateral error caused by the movement gap, providing a benchmark for the subsequent acquisition of the vibration reduction displacement vector of each pixel.

[0042] Preferably, in one embodiment of the present invention, considering the perspective imaging law of "faster near, slower far", the depth ratio between the guide rail and the well wall of the acquisition scene can be deduced; and when the longitudinal displacement of the pixel is small, it can be regarded as the lifting system being in a stationary or extremely low-speed state, and the parallax caused by the depth difference can be ignored, so the parallax ratio coefficient between the guide rail and the well wall can be set to a preset value; based on this, the method for obtaining the parallax ratio coefficient includes: When the magnitude of the longitudinal component of the camera displacement vector is less than the preset displacement threshold, the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle is set to the preset value; when the magnitude of the longitudinal component of the camera displacement vector is not less than the preset displacement threshold, the ratio of the longitudinal component of the pixel displacement vector to the longitudinal component of the camera displacement vector is used as the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle.

[0043] As an example, the preset displacement threshold is set to 0.5 pixels / frame, which can be adjusted by the implementer. The preset value ranges from 0.8 to 1. In this example, it is set to 1, which means that the parallax effect is considered to be small when the camera running speed is low. When the magnitude of the longitudinal component of the camera displacement vector is greater than or equal to 0.5, the magnitude of the longitudinal component of the pixel displacement vector is used as the numerator, and the magnitude of the longitudinal component of the camera displacement vector is used as the denominator. The ratio of the fraction is used as the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle. The parallax ratio coefficient is inversely proportional to the depth ratio (the depth of the camera from the well wall / the depth of the camera from the guide rail).

[0044] Preferably, in one embodiment of the present invention, the lateral drift of the camera can be converted into the lateral drift of pixels on the well wall by using a parallax scaling factor. The method for obtaining the lateral drift displacement component includes: The product of the parallax ratio coefficient between the guide rail and the well wall at each pixel angle and the lateral component of the camera displacement vector is used as the lateral drift displacement component of each pixel.

[0045] After obtaining the lateral drift displacement component of each pixel in each wellbore image, the lateral deviation can be further eliminated to obtain the vibration-reduced displacement vector of each pixel. The vibration-reduced displacement vector removes the error caused by the lateral jitter of the camera due to the floating gap, and initially corrects the pixels, preparing for the subsequent improvement of the panoramic image stitching effect.

[0046] Preferably, in one embodiment of the present invention, the method for obtaining the vibration reduction displacement vector includes: Subtract the lateral drift displacement component from the displacement vector of each pixel to obtain the vibration-reduced displacement vector.

[0047] It should be noted that the above vector decomposition operations include addition, subtraction, averaging, and decomposition of horizontal and vertical components, all of which are well-known techniques and will not be elaborated further.

[0048] Step S4: In each wellbore image, based on the lateral drift tolerance, the deformation consistency features of adjacent pixels, and the seismic displacement vector of each pixel, the distortion displacement vector of each pixel is obtained, and the wellbore image is corrected in combination with the imaging ratio; the corrected wellbore images are stitched together sequentially to obtain a panoramic image.

[0049] Considering that repeated textures on the well wall may cause the optical flow algorithm to fail, resulting in the calculation of large slips that violate physical common sense, this embodiment of the invention further needs to combine the lateral drift tolerance with the deformation consistency feature of adjacent pixels to analyze and eliminate erroneous slips in the de-vibration displacement vector of each pixel, thereby obtaining the distortion displacement vector of each pixel in each well wall image. The distortion displacement vector is the true residual parallax observation value (compliant deviation) after removing camera vibration and erroneous slips, and then compensates and corrects the well wall image.

[0050] Preferably, in one embodiment of the present invention, the method for obtaining the distorted displacement vector includes: In each wellbore image, the distortion displacement vector of each pixel is used as the optimization variable. Based on the difference between the distortion displacement vector of each pixel and the de-vibration displacement vector, the difference between the distortion displacement vector of each pixel and the corresponding distortion displacement vector of other pixels within a preset range, and the difference between the distortion displacement vector of each pixel and the allowable lateral drift, a function model is constructed. Based on the optimization variable corresponding to the minimum value of the function model, the distortion displacement vector of each pixel is obtained.

[0051] As an example, taking any wellbore image as an example, we construct a function model, the expression of which is: ; It is a function model; These are the pixel coordinates; The coordinate system for the wellbore image; For pixels The distorted displacement vector is also an optimization variable; For pixels The de-vibration displacement vector; For pixels The difference between the distorted displacement vector of the current pixel and the distorted displacement vector of each other pixel within the preset range; is the Euclidean norm notation, used to measure vector differences; This is the fidelity term in the function model, used to constrain the distorted displacement vector of a pixel to be as close as possible to the de-shaking displacement vector, thus preventing texture loss. This is a smoothing term in the function model, used to ensure the texture consistency or similarity of the local well wall and avoid generating isolated noise points; This is a constraint term in the function model, used for drift constraints, which forces the distortion displacement vector of the pixel to be limited within the influence range of the drift gap.

[0052] Among them, the preset local range is such as pixels. The local image region is centered on the image. In this example, the size of the local image region is 3×3. The implementer can also select a preset number of nearest pixels for calculation. ; , This is the allowable lateral drift. For pixels The lateral component of the distorted displacement vector; when the lateral component of the distorted displacement vector is greater than the allowable lateral drift, a larger penalty will be imposed on the constraint term.

[0053] Then, the function model is solved using the successive over-relaxation iterative method (SOR) or the primal-dual algorithm. The minimum value; the above model solution algorithm is a well-known technology and will not be described in detail; the optimization variables corresponding to the minimum value include the distortion displacement vector of each pixel in the well wall image.

[0054] After obtaining the distortion displacement vector of each pixel, the well wall image can be further corrected by combining the imaging ratio; finally, the corrected well wall images are stitched together to obtain a panoramic image.

[0055] Preferably, in one embodiment of the present invention, correcting the wellbore image includes: In each wellbore image, a scaling factor is determined based on the deviation of the imaging scale from the preset resolution to scale the wellbore image vertically, and the pixels are translated laterally by combining the lateral component of the distortion displacement vector of each pixel to obtain the corrected wellbore image.

[0056] As an example, taking any well wall image as an example, first determine the preset resolution of the panoramic image, such as 1 pixel / mm; then divide the preset resolution by the imaging ratio to obtain the scaling factor; perform bilinear interpolation scaling on each well wall image along the longitudinal Y-axis to eliminate image distortion caused by camera running speed, and obtain the scaled initial corrected well wall image. Then, in the initial corrected wellbore image, the sum of the lateral drift displacement component and the distortion displacement vector of each pixel is calculated. The magnitude of the lateral component corresponding to the sum is used as the lateral translation amount, and a reverse translation is performed along the direction of the lateral component corresponding to the sum to offset vibration error and compliance error. A geometric remapping table can also be constructed based on the lateral translation amount to perform geometric correction on the wellbore image, and an interpolation algorithm (such as bilinear interpolation) is used to fill pixel holes to obtain the corrected wellbore image.

[0057] Finally, all the corrected well wall images are stitched together to obtain a panoramic image corresponding to a vertical column in the well wall. Then, all the panoramic images of the vertical columns are stitched together horizontally to obtain a panoramic image of the entire well wall.

[0058] In one embodiment of the present invention, a center strip method can be used for seamless stitching. For example, a strip of a preset height is selected at the center of each corrected well wall image, and all strips are stitched together sequentially. The preset height is set as the pixel displacement span in the well wall image (obtained in step S2, and will not be repeated here). In another embodiment, the corrected well wall images can also be overlapped and stitched based on feature matching, wherein the overlapping areas between adjacent corrected well wall images are weighted and averaged for fusion. The above image stitching methods are all well-known techniques and will not be described in detail here.

[0059] In summary, this invention first acquires continuous frame images of the shaft wall captured by a camera during vertical lifting on a track within the shaft; performs optical flow analysis on the shaft wall images to obtain the displacement vector of each pixel in the shaft wall images, and also obtains the camera displacement vector; divides the continuous frame shaft wall images to obtain all shaft wall image sequences; in each sequence, based on the longitudinal component of the displacement vector of all pixels in the shaft wall images, the corresponding shaft wall length, and the camera acquisition parameters, obtains the imaging ratio between the pixel and the shaft wall, and combines this with the preset movement gap of the track to obtain the allowable lateral drift of the pixel; in each shaft wall image, based on... The longitudinal component of the camera displacement vector and the longitudinal component of the displacement vector of each pixel are used to obtain the parallax ratio coefficient between the guide rail and the well wall at each pixel angle. The lateral drift displacement component of the pixel is then obtained by combining this with the lateral component of the camera displacement vector. Based on this lateral drift displacement component, the vibration reduction displacement vector of each pixel is obtained. In each well wall image, the distortion displacement vector of each pixel is obtained based on the lateral drift tolerance, the deformation consistency characteristics of adjacent pixels, and the vibration reduction displacement vector of each pixel. This distortion displacement vector is then used to correct the well wall image by combining it with the imaging ratio. The corrected well wall images are then stitched together sequentially to obtain a panoramic image. This invention solves the technical problem of being unable to obtain geometrically correct and physically reliable panoramic images in dynamic swing and repetitive texture environments by introducing physical and mechanical constraints of the hoisting system, thus improving the quality of panoramic image stitching for vertical shafts.

Claims

1. A method for stitching panoramic images of a vertical shaft based on machine vision, characterized in that, The method includes: Acquire continuous frames of well wall images captured by the camera inside the vertical shaft during its vertical lifting process on the track; perform optical flow analysis on the well wall images to obtain the displacement vector of each pixel in the well wall images, and obtain the camera displacement vector; Divide the continuous frame well wall images to obtain all well wall image sequences, with different sequences corresponding to well walls of the same length; in each sequence, based on the longitudinal component of the displacement vector of all pixels in the well wall image, the well wall length corresponding to the sequence, and the camera acquisition parameters, obtain the imaging ratio between the pixel and the well wall, and combine the preset movement gap of the track to obtain the lateral drift tolerance of the pixel. In each well wall image, based on the longitudinal component of the camera displacement vector and the longitudinal component of the displacement vector of each pixel, the parallax ratio coefficient between the guide rail and the well wall at each pixel angle is obtained, and the lateral drift displacement component of the pixel is obtained by combining the lateral component of the camera displacement vector, and the vibration reduction displacement vector of each pixel is obtained based on the lateral drift displacement component. In each wellbore image, the distortion displacement vector of each pixel is obtained based on the lateral drift tolerance, the deformation consistency characteristics of adjacent pixels, and the seismic displacement vector of each pixel. The wellbore image is then corrected by combining the imaging ratio. The corrected wellbore images are then stitched together sequentially to obtain a panoramic image.

2. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the camera displacement vector includes: Based on the distribution characteristics of the lateral component of the displacement vector of pixels at the same position in any consecutive preset number of well wall images, representative parameters of pixels at each position in the non-guide rail region are obtained. Based on the representative parameters, the guide rail region in each wellbore image is determined, and the camera displacement vector is obtained according to the distribution characteristics of the displacement vectors of the pixels in the guide rail region in all wellbore images.

3. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for acquiring the wellbore image sequence includes: The images collected also include fixed-distance crossbeams in the vertical shaft; in each image of the well wall, the response parameters of the crossbeams are obtained based on the longitudinal gradient of all pixels in the preset central region. In continuous frame wellbore images, based on the temporal changes of the beam response parameters corresponding to the preset central region of the wellbore image, the beam positioning frame is determined, and the continuous frame wellbore images are divided based on the beam positioning frame to obtain all wellbore image sequences.

4. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the imaging ratio includes: For the first image sequence of the well wall in the acquisition time sequence, the imaging ratio between the pixel and the well wall is obtained based on the pinhole imaging model and the camera acquisition parameters; the camera acquisition parameters include focal length, sensor pixel size and shooting distance. For a sequence of images other than the first wellbore image, the total span of the pixels is determined based on the longitudinal component of the displacement vector of all pixels in all wellbore images. The ratio of the total span of the pixels to the length of the corresponding wellbore in the sequence is used as the imaging ratio between the pixels and the wellbore.

5. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the lateral drift tolerance includes: In each sequence, the product of the imaging ratio and the preset floating gap is used as the lateral drift tolerance of the pixel.

6. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the disparity scaling factor includes: When the magnitude of the longitudinal component of the camera displacement vector is less than a preset displacement threshold, the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle is set to a preset value; when the magnitude of the longitudinal component of the camera displacement vector is not less than the preset displacement threshold, the ratio of the longitudinal component of the pixel displacement vector to the longitudinal component of the camera displacement vector is used as the parallax ratio coefficient between the guide rail and the well wall at the corresponding pixel angle.

7. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the lateral drift displacement component includes: The product of the parallax ratio coefficient between the guide rail and the well wall at each pixel angle and the lateral component of the camera displacement vector is used as the lateral drift displacement component of each pixel.

8. The method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the vibration reduction displacement vector includes: Subtract the lateral drift displacement component from the displacement vector of each pixel to obtain the de-vibration displacement vector.

9. A method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, The method for obtaining the distorted displacement vector includes: In each wellbore image, the distortion displacement vector of each pixel is used as the optimization variable. Based on the difference between the distortion displacement vector of each pixel and the vibration reduction displacement vector, the difference between the distortion displacement vector of each pixel and the corresponding distortion displacement vector of other pixels within a preset range, and the difference between the distortion displacement vector of each pixel and the allowable lateral drift, a function model is constructed. Based on the optimization variable corresponding to the minimum value of the function model, the distortion displacement vector of each pixel is obtained.

10. A method for stitching panoramic images of a vertical shaft based on machine vision according to claim 1, characterized in that, Corrected wellbore images include: In each wellbore image, a scaling factor is determined based on the deviation of the imaging ratio from the preset resolution to scale the wellbore image vertically, and the pixels are horizontally translated by combining the horizontal component of the distortion displacement vector of each pixel to obtain the corrected wellbore image.

Images