360-degree panoramic image splicing method for mining area environment

Abstract

The invention discloses a 360-degree panoramic image splicing method for a mining area environment. The splicing process comprises seven steps of offline calibration of cameras, online acquisition of images of four cameras, distortion correction, perspective transformation, image splicing, image enhancement processing and image display. According to the 360-degree panoramic image splicing method, through enhancement technologies such as dynamic brightness correction, histogram equalization and color balance, the brightness and the contrast ratio of the images collected under the low-light condition of the mining area are effectively improved, and the image quality is remarkably improved. And secondly, the weighted average method is adopted to carry out image stitching, so that the ghosting phenomenon in the traditional method is solved, and the stitching naturalness and precision are improved. And finally, the whole method has relatively high adaptability, can cope with dynamic changes in the working environment of the mining area, provides reliable technical support for safety monitoring of the mining area, and ensures that operators can obtain accurate visual information, so that the safety and efficiency of working in the mining area are improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a 360° panoramic image stitching method for a mining environment. Background Art

[0002] The mining operation environment is complex, usually facing dim lighting conditions and dynamically changing scenes. These factors bring significant challenges to the stitching of panoramic images. The existing 360° panoramic image stitching technology performs well in traditional environments, but faces some problems in the special environment of mining areas.

[0003] First, the low light conditions in the mining area result in insufficient brightness of the captured images, which affects the stitching effect and subsequent analysis. Traditional brightness enhancement technology is often difficult to effectively improve the brightness and contrast of the image when dealing with low-light environments. In addition, multi-camera systems are prone to ghosting when stitching images. Existing ghosting detection and elimination methods usually rely on feature point matching in a static environment, which is not effective in the dynamically changing mining environment, resulting in a decrease in the quality of the stitched image.

[0004] Therefore, there is an urgent need to develop a 360° panoramic image stitching method for the special environment of mining areas to effectively improve the brightness and quality of images and provide more reliable visual information for mine safety monitoring. Summary of the invention

[0005] In response to the above technical problems, the present invention proposes a 360° panoramic image stitching method for a mining environment. The 360° panoramic image stitching method can effectively improve the brightness and contrast of images collected under low light conditions in the mining area, significantly improve the image quality, and provide reliable technical support for mine safety monitoring, ensuring that operators can obtain accurate visual information, thereby improving the safety and efficiency of mining operations.

[0006] A 360° panoramic image stitching method for a mining environment, the stitching process is:

[0007] S1, offline camera calibration: obtain the internal and external parameters of the camera;

[0008] S2, online acquisition of four camera images: online acquisition of image information of four cameras initially installed on the front, rear, left, and right sides of the vehicle body;

[0009] S3, distortion correction: perform distortion correction on the four camera images acquired to generate distortion-free images;

[0010] S4, perspective transformation: perform perspective transformation on the undistorted image to generate a top-view image, and adjust and optimize the camera installation angle according to the top-view image;

[0011] S5, image stitching: stitching the four overhead images after perspective transformation, and fusing them using weighted average method to form a 360° panoramic image;

[0012] S6, Image Enhancement: Perform image enhancement processing on 360° panoramic images, including dynamic brightness correction, histogram equalization and color balance;

[0013] S7, image display: displaying the processed 360° panoramic image in real time.

[0014] As a preferred embodiment of the above technical solution, in step S1, the internal and external parameters of the offline calibration camera are calibrated using the Zhang Zhengyou calibration method, and the specific calibration process is:

[0015] Take multiple sets of images using a checkerboard pattern of known size to cover different angles and positions;

[0016] Process the captured images, extract feature points and match them to calculate the camera's internal parameters;

[0017] The position and posture of the camera in the world coordinate system are calculated using the corresponding external parameters.

[0018] As a preferred embodiment of the above technical solution, in step S4, the perspective transformation is implemented by calculating the projection matrix, converting the image in the camera coordinate system into the top view image coordinate system, and adjusting the installation angle of the camera according to the top view image. The specific implementation process is:

[0019] Determine the installation position and angle of the camera in order to establish the relationship between the camera coordinate system and the world coordinate system. The projection matrix P can be expressed as follows:

[0020] P=K·[R|t]

[0021] Among them, K is the intrinsic parameter matrix of the camera, R is the rotation matrix, and t is the translation vector;

[0022] By combining the internal and external parameters of the camera, a projection matrix is ​​constructed to achieve the conversion from the camera coordinate system to the world coordinate system;

[0023] When performing a perspective transformation, first perform a coordinate transformation on each pixel point (x, y) to calculate its position in the world coordinate system (X, Y, Z):

[0024]

[0025] By defining the projection plane of the bird's-eye view and setting the corresponding Z value, the image can be presented in a bird's-eye view. Specifically, by setting the camera's height and bird's-eye view angle, the three-dimensional coordinates are projected onto a two-dimensional plane to generate a bird's-eye view image, and the camera's installation angle is adjusted according to the bird's-eye view image.

[0026] As a preferred embodiment of the above technical solution, in step S5, the weighted average method performs linear operations based on the gradient between pixels in the overlapping area of ​​adjacent cameras, thereby generating a stitched image with a natural transition. The weighted average fusion formula is as follows:

[0027]

[0028] Among them, I(x,y) is the pixel value at (x,y) after fusion, I0(x,y) is the pixel value of image A at (x,y), I1(x,y) is the pixel value of image B at (x,y), ω is the weight corresponding to the pixel at (x,y), and the formula is:

[0029]

[0030] Among them, dI0(x,y) represents the distance from the (x,y) pixel to the contour of image A minus the overlapping area, and dI1(x,y) represents the distance from the contour of image B minus the overlapping area.

[0031] As a preferred embodiment of the above technical solution, in step S6, the 360° panoramic image enhancement processing flow is:

[0032] By analyzing image characteristics, calculating global and local brightness information, determining ghosting and other areas, and evaluating image contrast to identify key enhancement areas;

[0033] Use computer vision techniques to detect ghost areas in images and determine their location and extent;

[0034] Dynamic brightness correction is performed on the panoramic image. A brightness correction method based on the RGB three-channel proportional coefficient is applied to the detected ghosting area to dynamically adjust its brightness to achieve balance. For areas where ghosting is not detected, a brightness correction algorithm based on local gain compensation is used to enhance the contrast and clarity of the overall image.

[0035] Use histogram equalization to enhance the details of the dynamic corrected image to improve clarity and contrast;

[0036] Apply median filtering to smooth the image and remove noise while preserving edge information;

[0037] By analyzing the color histogram, adjusting the hue and saturation, a natural and realistic visual effect is achieved.

[0038] As a preferred embodiment of the above technical solution, in step S7, there are two 360° panoramic image display modes:

[0039] Display the stitched image separately. In the stitched image display mode, the generated 360° panoramic image is displayed as a whole image.

[0040] One screen displays four pictures. In the one-screen display four-picture mode, the system will display the images captured by the four cameras in the front, back, left and right sides side by side.

[0041] The beneficial effects of the present invention are:

[0042] The 360° panoramic image stitching method effectively improves the brightness and contrast of images collected under low-light conditions in the mining area through enhancement technologies such as dynamic brightness correction, histogram equalization and color balance, and significantly improves the image quality. Secondly, the weighted average method is used for image stitching to solve the occurrence of ghosting in traditional methods, thereby improving the naturalness and accuracy of stitching. Finally, the overall method has strong adaptability and can cope with dynamic changes in the mining operating environment, providing reliable technical support for mine safety monitoring, ensuring that operators can obtain accurate visual information, thereby improving the safety and efficiency of mining operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The figure is a flow chart of the 360° panoramic image stitching method of the present invention.

[0044] Figure 2 This is a schematic diagram of the calibration cloth placement and parameter settings.

[0045] Figure 3 A schematic diagram of obtaining a projection matrix by manually selecting landmark points.

[0046] Figure 4 for Figure 3 The corresponding top view is generated.

[0047] Figure 5 Flowchart of the image enhancement process.

[0048] Figure 6 This is a diagram of the 360° panoramic image stitching display interface.

[0049] Figure 7 The interface diagram is displayed separately for four cameras. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 2 A 360° panoramic image stitching method for a mining environment is shown, and the stitching process is as follows:

[0052] S1, offline camera calibration: obtain the internal and external parameters of the camera;

[0053] Specifically, the offline calibration camera uses the Zhang Zhengyou calibration method, which takes pictures of a checkerboard pattern of known size and obtains at least 10 sets of images at different angles and positions to ensure the accuracy of the calibration results. During the calibration process, computer vision libraries such as OpenCV are used to process the image, extract the corner points of the checkerboard, and use the SIFT algorithm for feature matching. The camera's intrinsic parameters (such as focal length, principal point position, distortion coefficient) and extrinsic parameters (i.e., the position and posture of the camera in the world coordinate system) are calculated through the matched feature points. Finally, the images that are not involved in the calibration are verified by reprojection to ensure the accuracy of the calibration results.

[0054] S2, online acquisition of four camera images: online acquisition of image information of four cameras initially installed on the front, rear, left, and right sides of the vehicle body;

[0055] Specifically, four cameras are arranged around the vehicle body to achieve a 360° all-round view. The cameras should be installed at an appropriate height to avoid being blocked by obstacles and maintain a suitable horizontal and vertical angle.

[0056] S3, distortion correction: perform distortion correction on the four camera images acquired to generate distortion-free images;

[0057] Specifically, after image acquisition, the internal parameters obtained by calibration are used to perform distortion correction to eliminate the influence of lens distortion on image quality and generate a distortion-free image. Figure 2 The calibration cloth shown is used for calibration and parameter setting to determine the projection area of ​​each camera.

[0058] S4, perspective transformation: perform perspective transformation on the undistorted image to generate a top-view image, and adjust and optimize the camera installation angle according to the top-view image;

[0059] Specifically, Figure 2 , Figure 3 As shown in the figure, according to the set parameters, calibration cloths are placed around the vehicle body to ensure that the overlapping area of ​​each calibration cloth covers the field of view of adjacent cameras, where: Figure 3 It is a manual calibration to obtain the projection matrix diagram. Figure 4 for Figure 3 The corresponding projection diagram.

[0060] Use the camera to capture the image of the calibration cloth in real time, calculate its field of view boundary according to the position and angle of the camera, and ensure that the calibration plate is completely within the visible range of the camera.

[0061] In the acquired image, the positions of the landmarks are manually selected. When selecting, it should be ensured that these landmarks have sufficient coverage in all overlapping areas for subsequent feature matching and projection matrix calculation.

[0062] According to the coordinates of the selected marker points, the two-dimensional coordinates of each selected marker point in the camera coordinate system are paired with its three-dimensional coordinates in the world coordinate system. The least squares method is used to solve the linear equations to estimate the projection matrix P. The projection matrix P can be expressed as follows:

[0063] P=K·[R|t]

[0064] Among them, K is the intrinsic parameter matrix of the camera, R is the rotation matrix, and t is the translation vector.

[0065] Reprojection validation is performed using images that were not used in the calibration to evaluate the accuracy of the projection matrix.

[0066] After obtaining the projection matrix, when performing perspective transformation, first perform coordinate transformation on each pixel point (x, y) and calculate its position (X, Y, Z) in the world coordinate system:

[0067]

[0068] By defining the projection plane of the top-down perspective and setting the corresponding Z value, the image is presented in a top-down manner. Specifically, by setting the height and top-down angle of the camera, the three-dimensional coordinates are projected onto a two-dimensional plane to generate a top-down image.

[0069] Adjust the installation angle of the camera according to the top view image, and repeat steps S2 to S4 until the requirements are met.

[0070] S5, image stitching: stitching the four overhead images after perspective transformation, and fusing them using weighted average method to form a 360° panoramic image;

[0071] Specifically, the image after perspective transformation will fall into the area set in advance in step S4. There are overlapping areas between adjacent cameras. In order to solve the ghosting problem, the overlapping areas of the four corners are extracted and smoothed.

[0072] The weighted average method is used to process the overlapping area to ensure the natural transition of the stitched image. By defining the weight ω, a linear operation is performed based on the gradient between pixels in the overlapping area of ​​adjacent cameras to calculate the fused pixel value. The formula is:

[0073]

[0074] Among them, I(x,y) is the pixel value at (x,y) after fusion, I0(x,y) is the pixel value of image A at (x,y), I1(x,y) is the pixel value of image B at (x,y), ω is the weight corresponding to the pixel at (x,y), and the formula is:

[0075]

[0076] Among them, dI0(x,y) represents the distance from the (x,y) pixel to the contour of image A minus the overlapping area, and dI1(x,y) represents the distance from the contour of image B minus the overlapping area.

[0077] The top-view images of the four cameras are fused in adjacent directions to obtain a 360° panoramic image, thereby completing the 360° panoramic image stitching.

[0078] S6, Image Enhancement: Perform image enhancement processing on 360° panoramic images, including dynamic brightness correction, histogram equalization and color balance;

[0079] Specifically, Figure 5 As shown, this is a flowchart of the image enhancement process:

[0080] Step S601, analyzing image features:

[0081] In image brightness analysis, the global brightness is calculated first, by counting the brightness value of each pixel in the entire stitched image and taking the average to obtain the global brightness. Then, the image is divided into multiple small blocks, and the local brightness is calculated block by block. By comparing the brightness of each small block with the global brightness, the area with brightness significantly lower than the average level is identified to determine the ghosting area and other parts that need to be adjusted.

[0082] In contrast analysis, the image's brightness histogram is used to evaluate contrast. Images with high contrast show a wide range in brightness distribution, while images with low contrast are concentrated at specific brightness values. By analyzing the distribution of the histogram, key areas that need to be enhanced can be effectively identified, especially those with low contrast and blurred details.

[0083] Step S602, overlapping area ghost detection:

[0084] Canny edge detection is used to process the image and extract the edge information in the image. Then, contour extraction technology is used to analyze the detected edges and identify the shapes and positions of different objects.

[0085] Step S603, dynamic brightness correction:

[0086] For the detected ghosting area, a brightness correction method based on the RGB three-channel proportional coefficient is used. This method first sums the R, G, and B channels of all pixels in the overlapping area between images, and then obtains the brightness difference based on the summation result to determine the proportional coefficient between the three channels. Finally, the three-channel values ​​of the pixels are corrected according to the coefficients obtained above.

[0087] For the area where ghosting is not detected, a brightness correction method based on local gain compensation is adopted. This method stipulates that the error function is the normalized sum of the gain intensity errors of all pixels in the image overlap area, expressed as:

[0088]

[0089] Among them, Γ is the overlapping area between the images to be stitched, g i , g j are the gain coefficients of the two pixel intensity values, H ij is the projection transformation relationship. To simplify the calculation and improve the alignment accuracy, the overlapping area point P i The intensity value is approximately equal to the average intensity value of all pixels in the overlapping area. It is expressed as:

[0090]

[0091] When the gain coefficient is 0, E is 0, and the optimal solution is obtained. To obtain the optimal solution, the gain coefficient g should be set i , g j tend to the same priority. Then the updated error E d It is expressed as:

[0092]

[0093] Among them, N ij is the number of pixels in the overlapping area, and the parameter σ N and σ g are the standard deviations of the normalized intensity error and gain value, respectively. The gain coefficient is obtained by d The derivative is obtained by taking the derivative and setting it to 0. Finally, when the gain coefficient is obtained, the pixel points in the image are multiplied by the gain coefficient to perform brightness correction.

[0094] Step S604: image detail enhancement:

[0095] The dynamically corrected image is subjected to histogram equalization processing to improve the visibility of details by expanding the brightness distribution of the image.

[0096] Step S605, image denoising:

[0097] The median filter denoising algorithm is used to smooth the enhanced image to remove random noise in the image while retaining the edge information.

[0098] Step S606, color balance:

[0099] When performing color histogram analysis, the brightness distribution of each color channel (red, green, and blue) is first counted and the corresponding histogram is generated. By comparing the histograms of different channels, it is possible to find out whether a color channel is excessive or insufficient relative to other channels, thereby determining the direction of adjustment. Next, adjust the hue and saturation of the image. Hue adjustment can be achieved by mapping specific colors to new values, making the image present a more balanced and natural color. Saturation adjustment makes the colors in the image more vivid or soft by increasing or decreasing the concentration of the color.

[0100] S7, image display: displaying the processed 360° panoramic image in real time.

[0101] In step S7, there are two 360° panoramic image display modes, which can meet the needs of different users and operation scenarios:

[0102] Display the stitched image separately. In the stitched image display mode, the generated 360° panoramic image is displayed as a whole image. Figure 7 As shown;

[0103] In the mode of displaying four images on one screen, the system will display the images captured by the four cameras side by side, such as Figure 7 As shown, it is convenient for operators to monitor the surrounding environment in real time.

[0104] The invention discloses a 360° panoramic image stitching method for a mining environment. The method comprises the following steps: first, calibrating the camera offline to obtain the camera internal and external parameters; then, acquiring the images of four cameras installed on the front, back, left and right sides of the vehicle body online, and performing distortion correction on the images; then, generating a top view image through perspective transformation, and adjusting the installation angle of the camera according to the image; then, stitching the images after perspective transformation, and using the weighted average method to fuse the images from different perspectives; next, performing image enhancement processing on the stitched images, including dynamic brightness correction, histogram equalization and color balance, to improve the overall clarity and visualization effect of the image; finally, displaying the processed images in real time. The method, first, effectively improves the brightness and contrast of the images collected under low light conditions in the mining area, and significantly improves the image quality. Secondly, the occurrence of ghosting phenomenon in the traditional method is solved, thereby improving the naturalness and accuracy of stitching. Finally, it can cope with the dynamic changes in the mining operation environment, provide reliable technical support for mining safety monitoring, ensure that the operators can obtain accurate visual information, and thus improve the safety and efficiency of mining operations.

[0105] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A 360° panoramic image stitching method for a mining environment, characterized in that: The splicing process is: S1, offline camera calibration: obtain the internal and external parameters of the camera; S2, online acquisition of four camera images: online acquisition of image information of four cameras initially installed on the front, rear, left, and right sides of the vehicle body; S3, distortion correction: perform distortion correction on the four camera images acquired to generate distortion-free images; S4, perspective transformation: perform perspective transformation on the undistorted image to generate a top-view image, and adjust and optimize the camera installation angle according to the top-view image; S5, image stitching: stitching the four overhead images after perspective transformation, and fusing them using weighted average method to form a 360° panoramic image; S6, Image Enhancement: Perform image enhancement processing on 360° panoramic images, including dynamic brightness correction, histogram equalization and color balance; S7, image display: displaying the processed 360° panoramic image in real time.

2. The 360° panoramic image stitching method according to claim 1, characterized in that: In step S1, the internal and external parameters of the offline calibration camera are calibrated using the Zhang Zhengyou calibration method. The specific calibration process is as follows: Take multiple sets of images using a checkerboard pattern of known size to cover different angles and positions; Process the captured images, extract feature points and match them to calculate the camera's internal parameters; The position and posture of the camera in the world coordinate system are calculated using the corresponding external parameters.

3. The 360° panoramic image stitching method according to claim 1, characterized in that: In step S4, the perspective transformation is implemented by calculating the projection matrix, converting the image in the camera coordinate system into the top view image coordinate system, and adjusting the installation angle of the camera according to the top view image. The specific implementation process is: Determine the installation position and angle of the camera in order to establish the relationship between the camera coordinate system and the world coordinate system. The projection matrix P can be expressed as follows: P=K·[R|t] Among them, K is the intrinsic parameter matrix of the camera, R is the rotation matrix, and t is the translation vector; By combining the internal and external parameters of the camera, a projection matrix is ​​constructed to achieve the conversion from the camera coordinate system to the world coordinate system; When performing a perspective transformation, first perform a coordinate transformation on each pixel point (x, y) to calculate its position in the world coordinate system (X, Y, Z): By defining the projection plane of the bird's-eye view and setting the corresponding Z value, the image can be presented in a bird's-eye view. Specifically, by setting the camera's height and bird's-eye view angle, the three-dimensional coordinates are projected onto a two-dimensional plane to generate a bird's-eye view image, and the camera's installation angle is adjusted according to the bird's-eye view image.

4. The 360° panoramic image stitching method according to claim 1, characterized in that: In step S5, the weighted average method performs linear operations based on the gradient between pixels in the overlapping area of ​​adjacent cameras to generate a stitched image with a natural transition. The weighted average fusion formula is as follows: Among them, I(x,y) is the pixel value at (x,y) after fusion, I0(x,y) is the pixel value of image A at (x,y), I1(x,y) is the pixel value of image B at (x,y), ω is the weight corresponding to the pixel at (x,y), and the formula is: Among them, dI0(x,y) represents the distance from the (x,y) pixel to the contour of image A minus the overlapping area, and dI1(x,y) represents the distance from the contour of image B minus the overlapping area.

5. The 360° panoramic image stitching method according to claim 1, characterized in that: In step S6, the 360° panoramic image enhancement process is as follows: By analyzing image characteristics, calculating global and local brightness information, determining ghosting and other areas, and evaluating image contrast to identify key enhancement areas; Use computer vision techniques to detect ghost areas in images and determine their location and extent; Dynamic brightness correction is performed on the panoramic image. A brightness correction method based on the RGB three-channel proportional coefficient is applied to the detected ghosting area to dynamically adjust its brightness to achieve balance. For areas where ghosting is not detected, a brightness correction algorithm based on local gain compensation is used to enhance the contrast and clarity of the overall image. Use histogram equalization to enhance the details of the dynamic corrected image to improve clarity and contrast; Apply median filtering to smooth the image and remove noise while preserving edge information; By analyzing the color histogram, adjusting the hue and saturation, a natural and realistic visual effect is achieved.

6. The 360° panoramic image stitching method according to claim 1, characterized in that: In step S7, there are two 360° panoramic image display modes: Display the stitched image separately. In the stitched image display mode, the generated 360° panoramic image is displayed as a whole image. One screen displays four pictures. In the one-screen display four-picture mode, the system will display the images captured by the four cameras in the front, back, left and right sides side by side.

Images