A video splicing method and device for a fully-mechanized coal mining face

Abstract

The application provides a video splicing method and device for a fully-mechanized coal mining face, and the method comprises the following steps: acquiring images collected by each fixed-focus camera; performing image segmentation on the collected images to obtain foreground images, background images and segmentation line information corresponding to each fixed-focus camera; obtaining a first offset and a second offset corresponding to each pair of adjacent fixed-focus cameras according to matching positions and matching ranges; performing foreground image splicing and image fusion based on the first offset to obtain foreground images of the fully-mechanized coal mining face; performing background image splicing and image fusion based on the second offset to obtain background images of the fully-mechanized coal mining face; performing image fusion on the foreground images and the background images of the fully-mechanized coal mining face to obtain a preliminary panoramic image; and performing post-processing on the preliminary panoramic image to obtain a final panoramic image of the fully-mechanized coal mining face. The application improves the splicing accuracy of the fully-mechanized coal mining face.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a video stitching method and apparatus for a fully mechanized coal mining face. Background Technology

[0002] The fully mechanized coal mining face is the core production area of ​​coal mining, and effective video surveillance can ensure safe production underground.

[0003] In fully mechanized mining faces, where the face width typically reaches hundreds of meters, the observation area of ​​a single camera is limited, necessitating the use of multiple cameras to form a monitoring network. However, this network often suffers from significant overlap and redundancy, requiring operators to switch between multiple split-screen views, which can impair observation efficiency, hinder overall situational awareness, and impede intelligent analysis and remote control. Stitching together multi-camera videos to generate a panoramic view of the fully mechanized mining face can effectively address these issues. However, stitching a panoramic view of a fully mechanized mining face presents challenges due to the complex underground coal mine environment, intricate geometric relationships between cameras, and high precision requirements, resulting in issues such as insufficient stitching accuracy. Summary of the Invention

[0004] To address the problems in the prior art, embodiments of the present invention provide a video stitching method and apparatus for fully mechanized coal mining faces, which can at least partially solve the problems existing in the prior art.

[0005] In a first aspect, the present invention proposes a video stitching method for a fully mechanized coal mining face, comprising: Images are captured from each fixed-focus camera in the video stream of the fully mechanized coal mining face; wherein the video stream of the fully mechanized coal mining face is based on the capture of each fixed-focus camera; the captured images contain at least one marker code, which is pre-configured on the coal mining machine and the side guard plate; Based on the segmentation location, the acquired image corresponding to each fixed-focus camera is segmented to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera; wherein, the foreground image and the background image have overlapping areas, and the segmentation location is obtained in advance; Based on the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras, feature matching is performed on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras to obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras are obtained in advance. Based on the arrangement order of the fixed-focus cameras, the foreground images corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera, the images are sequentially stitched and horizontally fused to obtain the foreground image of the fully mechanized coal mining face; and based on the arrangement order of the fixed-focus cameras, the background images corresponding to each adjacent fixed-focus camera, and the second offset corresponding to each adjacent fixed-focus camera, the images are sequentially stitched and horizontally fused to obtain the background image of the fully mechanized coal mining face. Based on the segmentation line information, the foreground image and the background image of the fully mechanized coal mining face, a vertical image fusion is performed to obtain a preliminary panoramic image of the fully mechanized coal mining face; The preliminary panoramic image of the fully mechanized coal mining face is post-processed to obtain the final panoramic image of the fully mechanized coal mining face.

[0006] Secondly, the present invention provides a video splicing device for a fully mechanized coal mining face, comprising: The acquisition module is used to acquire images captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face; wherein the video stream of the fully mechanized coal mining face is based on the acquisition by each fixed-focus camera; the acquired images contain at least one marker code, which is pre-configured on the coal mining machine and the side guard plate; The image segmentation module is used to segment the acquired images corresponding to each fixed-focus camera based on the segmentation position, and obtain the foreground image, background image and segmentation line information corresponding to each fixed-focus camera; wherein, the foreground image and the background image have overlapping areas, and the segmentation position is obtained in advance; The feature matching module is used to perform feature matching on the foreground and background images corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range of each pair of adjacent fixed-focus cameras, and obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range of each pair of adjacent fixed-focus cameras are obtained in advance. The horizontal stitching module is used to sequentially stitch and fuse horizontal images according to the arrangement order of each fixed-focus camera, the foreground image corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera to obtain the foreground image of the fully mechanized coal mining face; and to sequentially stitch and fuse horizontal images according to the arrangement order of each fixed-focus camera, the background image corresponding to each adjacent fixed-focus camera, and the second offset corresponding to each adjacent fixed-focus camera to obtain the background image of the fully mechanized coal mining face. The vertical stitching module is used to perform vertical image fusion based on the dividing line information, the foreground image and the background image of the fully mechanized coal mining face, to obtain a preliminary panoramic image of the fully mechanized coal mining face. The post-processing module is used to perform post-processing on the preliminary panoramic image of the fully mechanized coal mining face to obtain the final panoramic image of the fully mechanized coal mining face.

[0007] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the video stitching method for fully mechanized coal mining faces described in any of the above embodiments.

[0008] Fourthly, the present invention provides a computer-readable storage medium storing a computer program / instruction that, when executed by a processor, implements the video stitching method for fully mechanized coal mining faces described in any of the above embodiments.

[0009] Fifthly, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the video stitching method for fully mechanized coal mining faces described in any of the above embodiments.

[0010] The video stitching method and apparatus for a fully mechanized coal mining face provided in this invention can acquire images from each fixed-focus camera in the video stream of the fully mechanized coal mining face; perform image segmentation on the acquired image corresponding to each fixed-focus camera based on the segmentation position to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera; perform feature matching on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras to obtain the first offset and second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range corresponding to each adjacent fixed-focus camera are obtained in advance; and the foreground image and background image corresponding to each adjacent fixed-focus camera are obtained according to the arrangement order of each fixed-focus camera and the foreground image of each adjacent fixed-focus camera. The images and the first offsets corresponding to each adjacent fixed-focus camera are sequentially stitched together and horizontally fused to obtain the foreground image of the fully mechanized coal mining face. Then, based on the arrangement order of the fixed-focus cameras, the background images corresponding to each adjacent fixed-focus camera, and the second offsets corresponding to each adjacent fixed-focus camera, the images are sequentially stitched together and horizontally fused to obtain the background image of the fully mechanized coal mining face. Based on the segmentation line information, the foreground image, and the background image of the fully mechanized coal mining face, vertical image fusion is performed to obtain a preliminary panoramic image of the fully mechanized coal mining face. The preliminary panoramic image of the fully mechanized coal mining face is then post-processed to obtain the final panoramic image of the fully mechanized coal mining face, thus improving the stitching accuracy of the fully mechanized coal mining face. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of the video splicing system for a fully mechanized coal mining face provided in the first embodiment of the present invention.

[0013] Figure 2 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the second embodiment of the present invention.

[0014] Figure 3 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the third embodiment of the present invention.

[0015] Figure 4 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the fourth embodiment of the present invention.

[0016] Figure 5 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the fifth embodiment of the present invention.

[0017] Figure 6 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the sixth embodiment of the present invention.

[0018] Figure 7 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the seventh embodiment of the present invention.

[0019] Figure 8 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the eighth embodiment of the present invention.

[0020] Figure 9 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the ninth embodiment of the present invention.

[0021] Figure 10 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the tenth embodiment of the present invention.

[0022] Figure 11 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the eleventh embodiment of the present invention.

[0023] Figure 12 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the twelfth embodiment of the present invention.

[0024] Figure 13 This is a schematic diagram of the structure of the video splicing device for a fully mechanized coal mining face provided in the thirteenth embodiment of the present invention.

[0025] Figure 14 This is a schematic diagram of the physical structure of a computer device provided in the fourteenth embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of laws and regulations. The user information in the embodiments of this application is obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been agreed upon by the customer.

[0027] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution in this application will be explained below.

[0028] Video stitching technology can spatially match and align overlapping image sequences captured by multiple cameras, and then fuse them to form a wide-view panoramic image containing complete information about the work surface.

[0029] However, the underground environment of coal mines has special characteristics, which poses a severe challenge to image stitching technology: (1) Complex environment: poor lighting conditions, serious coal dust interference, and frequent equipment vibration; (2) Large data processing volume: real-time stitching processing of video streams from a large number of cameras (e.g., 103 cameras); (3) Complex geometric relationship: the cameras are installed on the outer cylinder of the hydraulic support column, and the geometric relationship between the cameras changes dynamically as the hydraulic support rises, falls and extends; (4) High stitching accuracy requirements: accurate stitching of key equipment such as coal mining machines and hydraulic supports is required, and the error cannot exceed 2 pixels.

[0030] Therefore, the following common technical challenges exist in video stitching of fully mechanized coal mining faces: (1) Mismatch between 2D and 3D scenes: Existing methods are mainly based on 2D image processing technology and do not make full use of the 3D geometric information of the scene, resulting in insufficient stitching accuracy in 3D scenes with significant depth changes; (2) Incompatibility between static and dynamic environments: Existing methods either rely on real-time feature matching (which has the defects of large computational load and poor robustness) or use fixed transformation matrices (which have the defects of not being able to adapt to dynamic changes), making it difficult to maintain high-precision stitching in dynamic industrial environments; (3) Global and local contradictions: Using a single transformation matrix for global processing cannot take into account the stitching accuracy of the foreground and background, and regional processing lacks effective segmentation basis; (4) Trade-off between real-time performance and accuracy: High-precision feature matching and transformation calculation are time-consuming and difficult to meet the requirements of real-time stitching of multiple cameras; simplifying the algorithm will sacrifice stitching quality; (5) Difficulty in calibration and maintenance: Traditional calibration methods have a large workload and long cycle, and cannot adapt to the dynamic changes of equipment, resulting in high maintenance costs.

[0031] Therefore, this application provides a video stitching method for fully mechanized coal mining faces, which utilizes 3D spatial information to guide image stitching, improving stitching accuracy in 3D scenes; it can estimate camera pose in real time and dynamically adapt to changes in equipment position; it provides intelligent layered processing based on scene understanding, taking into account the stitching quality of different depth areas; it improves the real-time performance of multi-camera video stream processing while ensuring stitching accuracy; and it simplifies calibration and maintenance processes, improving the system's practicality and reliability. This application solves the problem of high-precision real-time video stitching in complex 3D scenes such as fully mechanized coal mining faces.

[0032] Figure 1 This is a schematic diagram of the structure of the video splicing system for a fully mechanized coal mining face provided in the first embodiment of the present invention, as shown below. Figure 1 As shown, the video splicing system for a fully mechanized coal mining face provided in this embodiment of the invention includes multiple fixed-focus cameras 101, a video acquisition and synchronization unit 102, an image processing unit 103, a display and monitoring unit 104, and a storage unit 105, wherein: Each fixed-focus camera 101 is mounted on the outer cylinder of the column of an adjacent hydraulic support. Each fixed-focus camera 101 is fixed to the outer cylinder of the column via a detachable dustproof and explosion-proof structure. This structure design does not affect the lifting and telescopic movements of the hydraulic support. The field of view of each fixed-focus camera 101 covers the adjacent area, and the fields of view of adjacent fixed-focus cameras 101 overlap to form a continuous visual band. The overlap rate of the fields of view of adjacent fixed-focus cameras 101 can be no less than 20%. The number of fixed-focus cameras 101 is set according to actual needs; for example, 103 intrinsically safe fixed-focus cameras with a resolution of 1920×1080 pixels can be used. Using fixed-focus cameras avoids the distortion problems that may be caused by zoom lenses.

[0033] The video acquisition and synchronization unit 102 is connected to each fixed-focus camera 101 to acquire the video streams output by each fixed-focus camera 101 and perform time synchronization processing. The video acquisition and synchronization unit 102 uses a unified clock synchronization protocol (such as the PTP protocol) to ensure the time consistency of multiple video frames, avoiding splicing jitter problems caused by time asynchrony. The video acquisition and synchronization unit 102 can connect to all cameras via industrial Ethernet, receive video data in real time, and transmit the synchronized video frames to the image processing unit 103.

[0034] The image processing unit 103 executes the video stitching method for a fully mechanized coal mining face provided in this embodiment of the invention to obtain a final panoramic image of the fully mechanized coal mining face. The image processing unit 103 then sends the final panoramic image of the fully mechanized coal mining face to the display and monitoring unit 104.

[0035] The display and monitoring unit 104 is used to display the final panoramic image, realizing the visual monitoring of the fully mechanized coal mining face. The display and monitoring unit 104 can support high-resolution display, such as 6490×1385 pixels, to ensure the clarity of the displayed image.

[0036] Storage unit 105 is used to store the final panoramic image. Storage unit 105 can support playback of historical final panoramic images and can be used for accident tracing.

[0037] Multiple marker codes are deployed at the longwall mining face of the coal mine. These marker codes must meet the following requirements: they are installed on the coal mining machine body and side guards; at least one fixed-focus camera 101 can clearly capture the marker codes on the coal mining machine body and side guards; the coordinates of each marker code are recorded in the coal mining machine coordinate system; the marker codes are installed flat and horizontally, and the markings are clear and undamaged. ArUco marker codes can be used.

[0038] The following describes the specific implementation process of the video stitching method for fully mechanized coal mining faces provided in this embodiment of the invention, taking the image processing unit as the execution subject as an example.

[0039] Figure 2 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the second embodiment of the present invention, as shown below. Figure 2 As shown in the embodiment of the present invention, the video stitching method for a fully mechanized coal mining face includes: S201. Obtain images captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face; wherein the video stream of the fully mechanized coal mining face is based on the images captured by each fixed-focus camera; the captured images contain at least one marker code, which is pre-configured on the coal mining machine and the side guard plate; Specifically, multiple marker codes are configured on the coal mining machine and the side support plate. Fixed-focus cameras mounted on the outer cylinders of adjacent hydraulic supports capture images of the fully mechanized coal mining face, resulting in a video stream of the face. The image processing unit can obtain the images captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face. Each captured image contains at least one marker code. Adjacent fixed-focus cameras have overlapping fields of view, and their captured images share at least one common marker code. Because multiple fixed-focus cameras provide video streams, to avoid splicing jitter caused by time asynchrony, the video stream of the fully mechanized coal mining face provided to the image processing unit is a time-synchronized video stream.

[0040] Furthermore, for each image acquired by a fixed-focus camera, distortion parameters can be selected according to the fixed-focus camera type, and distortion correction can be performed using a remapping method to output the corrected image. Distortion correction supports both CPU and GPU processing methods.

[0041] S202. Based on the segmentation position, perform image segmentation on the acquired image corresponding to each fixed-focus camera to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera; wherein, the foreground image and the background image have overlapping areas, and the segmentation position is obtained in advance; Specifically, the image processing unit performs image segmentation on the acquired image corresponding to each fixed-focus camera based on the segmentation position, dividing the acquired image into two images. This allows the acquisition unit to obtain the foreground image, background image, and segmentation line information for each fixed-focus camera. The background image is the upper part of the acquired image, including the coal face. The foreground image is the lower part of the acquired image, including the front surface of the coal mining machine. During image segmentation, some overlapping areas are retained, meaning the foreground and background images partially overlap. The segmentation line information includes the segmentation position and the overlap position between the foreground and background images. The segmentation position is pre-determined and used to divide the acquired image into the background and foreground images.

[0042] For example, if the segmentation position is 50% of the height of the acquired image, moving the segmentation position upwards by 5% of the image height yields the upper segmentation line, and moving it downwards by 5% yields the lower segmentation line. The image between the upper and lower segmentation lines is the overlapping region. The upper segmentation line segments the acquired image, with the portion below it designated as the foreground image, and the lower segmentation line segments the image, with the portion above it designated as the background image.

[0043] For example, the segmentation location can be obtained through 3D modeling. A 3D surface model of the coal mining machine can be created based on its design drawings; the 3D coordinates of the segmentation location can be plotted in the world coordinate system of the 3D surface model; and the 3D coordinates of the segmentation location can be transformed to the pixel coordinate system corresponding to each fixed-focus camera to obtain the segmentation location.

[0044] S203. Based on the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras, feature matching is performed on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras to obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras are obtained in advance. Specifically, the image processing unit performs feature matching on the foreground image corresponding to each pair of adjacent fixed-focus cameras based on the matching position and matching range, thereby obtaining a first offset for each pair of adjacent fixed-focus cameras. The image processing unit also performs feature matching on the background image corresponding to each pair of adjacent fixed-focus cameras based on the matching position and matching range, thereby obtaining a second offset for each pair of adjacent fixed-focus cameras.

[0045] The matching position and matching range of each adjacent fixed-focus camera are obtained in advance. The specific acquisition process is described in detail below and will not be repeated here.

[0046] S204. Based on the arrangement order of each fixed-focus camera, the foreground image corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera, perform stitching and horizontal image fusion sequentially to obtain the foreground image of the fully mechanized coal mining face; and based on the arrangement order of each fixed-focus camera, the background image corresponding to each adjacent fixed-focus camera, and the second offset corresponding to each adjacent fixed-focus camera, perform stitching and horizontal image fusion sequentially to obtain the background image of the fully mechanized coal mining face. Specifically, the image processing unit stitches the foreground images corresponding to each fixed-focus camera in the order of their arrangement. Each time the foreground images corresponding to adjacent fixed-focus cameras are stitched together, horizontal image fusion is performed based on the foreground images corresponding to adjacent fixed-focus cameras and the first offset corresponding to adjacent fixed-focus cameras. After the foreground images corresponding to all fixed-focus cameras are stitched together, the foreground image of the fully mechanized coal mining face is obtained.

[0047] The image processing unit stitches the background images corresponding to each fixed-focus camera in the order of their arrangement. Each time the background images corresponding to adjacent fixed-focus cameras are stitched together, horizontal image fusion is performed based on the background images corresponding to adjacent fixed-focus cameras and the second offset corresponding to adjacent fixed-focus cameras. After the background images corresponding to all fixed-focus cameras are stitched together, the background image of the coal mine fully mechanized mining face is obtained.

[0048] This application divides the acquired images into background and foreground images, and stitches the foreground and background images of each fixed-focus camera separately, and optimizes the stitching parameters accordingly. This approach can take into account the image stitching of different depth areas of the acquired images, and the stitching quality is optimal in both the foreground and background areas, thus improving the overall visual effect.

[0049] S205. Based on the segmentation line information, the foreground image and the background image of the fully mechanized coal mining face, perform vertical image fusion to obtain a preliminary panoramic image of the fully mechanized coal mining face. Specifically, the image processing unit uses the segmentation line information to fuse the foreground and background images of the fully mechanized coal mining face in the vertical direction, thereby obtaining a preliminary panoramic image of the fully mechanized coal mining face.

[0050] S206. Post-process the preliminary panoramic image of the fully mechanized coal mining face to obtain the final panoramic image of the fully mechanized coal mining face.

[0051] Specifically, the image processing unit performs post-processing on the preliminary panoramic image of the fully mechanized coal mining face to obtain the final panoramic image of the fully mechanized coal mining face. The specific post-processing procedure is set according to actual needs, and this embodiment of the invention does not limit it.

[0052] For example, post-processing may include grayscale and binarization of the preliminary panoramic image of the fully mechanized coal mining face, followed by morphological operations, namely, dilation followed by erosion, which can effectively connect broken areas in the image and remove noise points. During binarization, the threshold can be set to 5.

[0053] For example, post-processing may also include effective region extraction and cropping of the preliminary panoramic image after morphological operations. The preliminary panoramic image after morphological operations is then subjected to contour detection, maximum area contour filtering, bounding rectangle calculation and boundary expansion, and image cropping in sequence to obtain the final panoramic image of the fully mechanized coal mining face.

[0054] A contour detection algorithm is performed on the binarized image of the initial panoramic image to obtain a set of all closed contours. All detected closed contours are traversed, and the area of ​​each contour (i.e., the number of pixels it encloses) is calculated. The areas of all closed contours are compared, and the closed contour with the largest area is selected as the valid image region. This largest closed contour corresponds to the non-black border portion in the stitched image.

[0055] After obtaining the largest closed contour, find the coordinates of all points within that largest closed contour: The left boundary x1 is the smallest x-coordinate among the contour points; the right boundary x2 is the largest x-coordinate among the contour points; the upper boundary y1 is the smallest y-coordinate among the contour points; and the lower boundary y2 is the largest y-coordinate among the contour points.

[0056] The circumscribed rectangle is a rectangle with a starting point (x1, y1), a width of W, and a height of H. The width W = right boundary x2 - left boundary x1, and the height H = bottom boundary y2 - top boundary y1.

[0057] To avoid over-cropping, you can extend the rectangle's boundary outward by a few pixels (e.g., 5 pixels), and then use this outer rectangle to crop the original image.

[0058] The video stitching method for a fully mechanized coal mining face provided in this invention can acquire images from each fixed-focus camera in the video stream of the fully mechanized coal mining face; perform image segmentation on the acquired image corresponding to each fixed-focus camera based on the segmentation position to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera; perform feature matching on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range, respectively, to obtain the first offset and second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range corresponding to each adjacent fixed-focus camera are obtained in advance; based on the arrangement order of each fixed-focus camera and the foreground image corresponding to each adjacent fixed-focus camera... The foreground image of the fully mechanized coal mining face is obtained by sequentially stitching and horizontally fusing images based on the first offset of each adjacent fixed-focus camera and the first offset of each adjacent fixed-focus camera. Then, the background image of the fully mechanized coal mining face is obtained by sequentially stitching and horizontally fusing images based on the arrangement order of the fixed-focus cameras, the background image of each adjacent fixed-focus camera, and the second offset of each adjacent fixed-focus camera. Finally, the preliminary panoramic image of the fully mechanized coal mining face is post-processed to obtain the final panoramic image of the fully mechanized coal mining face, thus improving the stitching accuracy of the fully mechanized coal mining face.

[0059] Figure 3This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the third embodiment of the present invention, as shown below. Figure 3 As shown, based on the above embodiments, the video stitching method for fully mechanized coal mining faces provided by the embodiments of the present invention further includes: S301. Obtain the first frame image captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face; Specifically, before stitching together the images captured by each fixed-focus camera, it is necessary to obtain the position and orientation of each fixed-focus camera, as well as the stitching parameters of each adjacent fixed-focus camera. The image processing unit can obtain the first frame image captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face.

[0060] The stitching parameters include matching position, matching range, and segmentation position. The matching position is the normalized location of the overlapping area between adjacent fixed-focus camera images within their respective images. The matching range is the search area extending left and right from the matching position, typically 20%-25% of the image width. The segmentation position is used to divide the acquired image into foreground and background images; the segmentation position can be a fixed proportion of the acquired image's height or obtained based on a 3D model of the coal mining machine.

[0061] S302. Perform marker detection based on the first frame image corresponding to each fixed-focus camera to obtain the marker points corresponding to each fixed-focus camera and the relationship point set of each fixed-focus camera. Each point in the relationship point set of each fixed-focus camera has three-dimensional coordinates and corresponding pixel coordinates. Specifically, the image processing unit detects the identifier of the marker point corresponding to each fixed-focus camera from the first frame image corresponding to each fixed-focus camera, and then extracts the pixel coordinates of the four corner points of each marker point. Based on the identifier of the marker point corresponding to each fixed-focus camera, the three-dimensional coordinates of the four corner points of each marker point can be obtained. The three-dimensional coordinates of the four corner points of the marker point are three-dimensional coordinates in the coal mining machine coordinate system. By mapping the three-dimensional coordinates of the four corner points of the marker point corresponding to the fixed-focus camera to the pixel coordinates of the four corner points of the marker point corresponding to the fixed-focus camera, a relation point set of the fixed-focus camera is formed. Each point in the relation point set of the fixed-focus camera has three-dimensional coordinates and corresponding pixel coordinates.

[0062] For example, the ArUco marker detection process is illustrated below. The first frame image corresponding to the fixed-focus camera is converted to grayscale to obtain a corresponding grayscale image. Then, the ArUco detector is used to detect markers in the grayscale image corresponding to the first frame image, obtaining the ID number of each marker in the first frame image. Next, the pixel coordinates of the four corner points of each marker in the first frame image are extracted. The three-dimensional coordinates of the four corner points of each marker are obtained by querying the ID number of each marker. A correspondence is established between the pixel coordinates of the four corner points of each marker and the three-dimensional coordinates of the four corner points of each marker. Each corner point of each marker is treated as a point in the relation point set of the fixed-focus camera, and the three-dimensional coordinates of each point correspond to the pixel coordinates.

[0063] The three-dimensional coordinates of the four corner points of each marker point are obtained in advance. During the installation of each marker point, the three-dimensional coordinates of the four corner points of each marker point in the coal mining machine coordinate system can be measured and recorded. The ID number of each marker point can be associated with the three-dimensional coordinates of its four corner points and stored in a database for easy retrieval. The three-dimensional coordinates of the four corner points of each marker point are coordinates in the coal mining machine coordinate system. The coal mining machine coordinate system is a three-dimensional rectangular coordinate system with the geometric center of the coal mining machine as the reference origin.

[0064] For example, AprilTag markers can be used instead of ArUco markers, and the PnP algorithm can still be performed for pose estimation. AprilTag has stronger detection robustness and performs better in low-resolution or blurry images.

[0065] S303. Solve the pose of each fixed-focus camera based on the relationship point set of each fixed-focus camera to obtain the position and attitude angle of each fixed-focus camera, as well as the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera. Specifically, for each fixed-focus camera, the image processing unit can solve for the camera's pose based on the camera's relation point set, obtaining the camera's position and attitude angles. The fixed-focus camera's position is its three-dimensional coordinates in the coal mining machine coordinate system, and its attitude angles are relative to the coal mining machine coordinate system. During the pose calculation process, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera's coordinate system can be obtained. The three-dimensional coordinate system refers to the coal mining machine coordinate system.

[0066] For example, the position and pose angles of a fixed-focus camera can be solved using the Perspective-n-Point (PnP) algorithm. Using the 3D coordinates and pixel coordinates of the corresponding marker points in the correspondence set of marker points, the camera intrinsic matrix of the fixed-focus camera, and distortion coefficients as input data, the PnP algorithm can calculate the rotation and translation vectors from the coal mining machine coordinate system to the fixed-focus camera coordinate system. The rotation vector can be converted into a rotation matrix. The Levenberg-Marquardt (LM) algorithm can be used to perform nonlinear optimization on the PnP solution, for example, by calling the cv2.solvePnPRefineLM function from the open-source computer vision library to optimize the pose parameters.

[0067] Let the rotation matrix be R and the translation vector be t. Then, the three-dimensional coordinates of the optical center of the fixed-focus camera in the coal mining machine coordinate system can be expressed as C = -R. T Let t and C represent the three-dimensional coordinates of the optical center of the fixed-focus camera in the coal mining machine coordinate system, R represent the rotation matrix from the coal mining machine coordinate system to the fixed-focus camera coordinate system, and t represent the translation vector from the coal mining machine coordinate system to the fixed-focus camera coordinate system. The three-dimensional coordinates of the optical center of the fixed-focus camera in the coal mining machine coordinate system represent the position of the fixed-focus camera.

[0068] By extracting Euler angles from the rotation matrix R, the yaw, pitch, and roll angles of the fixed-focus camera relative to the coal mining machine coordinate system can be obtained.

[0069] For a fixed-focus camera, the spatial distance between the fixed-focus camera and the marker point can be calculated using the three-dimensional coordinates of the camera's optical center and the center of the corresponding marker point. Specifically, the three-dimensional coordinates of the marker point's center can be obtained using the three-dimensional coordinates of its four corner points.

[0070] Based on the pixel size of the marker point in the image captured by the fixed-focus camera and the physical size of the marker point in 3D space, the conversion ratio from pixel size in the image captured by the fixed-focus camera to physical size in 3D space can be obtained. If the pixel size of the marker point in the image captured by the fixed-focus camera is 50 pixels, and the physical size of the marker point in 3D space is 0.1 meters, then the conversion ratio pixel_per_M = 50 pixels / 0.1 meters = 500 pixels / meter.

[0071] If a fixed-focus camera corresponds to multiple marker points, then by detecting each marker point in the first frame image corresponding to the fixed-focus camera, the position and orientation of the fixed-focus camera relative to each marker point, as well as the conversion ratio, can be obtained. A weighted average of the position and orientation of the fixed-focus camera relative to each marker point can be calculated, and the result of this weighted average can be used as the final conversion ratio. The weights of each marker point corresponding to the fixed-focus camera can be determined based on the size of each marker point.

[0072] S304. Based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, the intrinsic parameters and distortion coefficients of each fixed-focus camera, calculate the overlapping area of ​​adjacent fixed-focus cameras, obtain the matching position corresponding to each pair of adjacent fixed-focus cameras, and obtain the corresponding matching range.

[0073] Specifically, in the coordinate system of the coal mining machine, the surface of the acquisition machine is sampled, and the obtained sampling points constitute the surface sampling point set of the coal mining machine. Based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, and the intrinsic parameters and distortion coefficients of each fixed-focus camera, the pixel coordinates of the surface sampling point set projected onto the image plane of each fixed-focus camera are obtained. Based on the pixel coordinates corresponding to the surface sampling point set and the effective image range of each fixed-focus camera, the visible area range corresponding to each fixed-focus camera is determined. Based on the surface sampling points corresponding to the visible area range of each pair of adjacent fixed-focus cameras, the spatial overlap area of ​​each pair of adjacent fixed-focus cameras is obtained. Based on the spatial overlap area of ​​each pair of adjacent fixed-focus cameras and the segmentation rules, the spatial segmentation position of each pair of adjacent fixed-focus cameras is obtained; wherein, the segmentation rules are preset. The spatial segmentation position of each pair of adjacent fixed-focus cameras is transformed to the corresponding pixel coordinate system to obtain the matching position and corresponding matching range of each pair of adjacent fixed-focus cameras. The number of sampling points included in the surface sampling point set is set according to actual needs, and this embodiment of the invention does not limit this. For example, 200 points can be taken from each of the front, rear, left, and right surfaces of the coal mining machine, and 200 points can be taken evenly on each surface.

[0074] Figure 4 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the fourth embodiment of the present invention, as shown below. Figure 4 As shown, based on the above embodiments, further, the step of calculating the overlapping area of ​​adjacent fixed-focus cameras according to the surface sampling point set of the coal mining machine, the position and attitude angle of each fixed-focus camera, and obtaining the matching position and corresponding matching range for each pair of adjacent fixed-focus cameras includes: 401. Based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, and the intrinsic parameters and distortion coefficients of each fixed-focus camera, obtain the pixel coordinates of the image plane projected from the surface sampling point set to each fixed-focus camera. Specifically, for each fixed-focus camera, the three-dimensional coordinates of each point in the surface sampling point set of the coal mining machine can be transformed into the camera coordinate system of the fixed-focus camera using a rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of the fixed-focus camera. Then, the coordinates of the surface sampling point set of the coal mining machine in the camera coordinate system of the fixed-focus camera are projected onto the normalized image plane of the fixed-focus camera to obtain the normalized coordinates of each point in the surface sampling point set of the coal mining machine. Next, the distortion coefficients of the fixed-focus camera are used to correct the distortion of the normalized coordinates of each point in the surface sampling point set of the coal mining machine to obtain the corrected coordinates of each point in the surface sampling point set of the coal mining machine. Finally, the intrinsic parameters of the fixed-focus camera are used to convert the corrected coordinates of each point in the surface sampling point set of the coal mining machine into pixel coordinates to obtain the pixel coordinates of the surface sampling point set of the coal mining machine projected onto the image plane of the fixed-focus camera.

[0075] To facilitate the acquisition of the surface sampling point set of the coal mining machine, a 3D model of the coal mining machine can be created based on its 3D drawings. Marking codes are then configured on this 3D surface model, corresponding to the actual marking codes on the machine; that is, the 3D coordinates of the marking codes on the model are the same as those on the actual machine. The world coordinate system of the 3D surface model and the actual coordinate system of the coal mining machine can be considered equivalent. The surface sampling point set can be obtained by sampling 1000 discrete points from each face of the 3D surface model.

[0076] After projecting the surface sampling points of the coal mining machine onto the image plane of a fixed-focus camera to obtain pixel coordinates, the correspondence between the 3D coordinates and pixel coordinates of the surface sampling points can be verified by the conversion ratio from pixels to physical dimensions in 3D space in the image captured by the fixed-focus camera. For example, take any two surface sampling points, calculate the actual distance between the two surface sampling points in the world coordinate system and the pixel distance in the pixel coordinate system corresponding to the fixed-focus camera, calculate the ratio of the pixel distance to the physical distance, and compare the above ratio with the conversion ratio from pixels to physical dimensions in 3D space in the image captured by the fixed-focus camera. If the absolute value of the difference between the above ratio and the conversion ratio is less than a preset value, then the correspondence between the 3D coordinates and pixel coordinates of the surface sampling points is correct.

[0077] 402. Determine the visible area range corresponding to each fixed-focus camera based on the pixel coordinates corresponding to the surface sampling point set and the effective image range of each fixed-focus camera; Specifically, each fixed-focus camera has an effective image range, meaning the image captured by the fixed-focus camera has image boundaries. Due to the long length of the coal mining machine, the fixed-focus camera can only capture images of a portion of the fully mechanized coal mining face. The pixel coordinates of some points in the surface sampling point set fall within the image captured by the fixed-focus camera, while the pixel coordinates of others are outside the image. By comparing the pixel coordinates corresponding to the surface sampling point set with the effective image range of the fixed-focus camera, the pixels corresponding to the surface sampling points within the effective image range constitute the effective projection points of the surface sampling points corresponding to the fixed-focus camera. Boundary extraction of the distribution range of the effective projection points of the surface sampling points corresponding to the fixed-focus camera yields the visible area range corresponding to the fixed-focus camera. The effective image range of the fixed-focus camera is preset.

[0078] For example, the effective image range of a fixed-focus camera is represented as follows: image width greater than or equal to 0 and less than a first threshold, and image height greater than or equal to 0 and less than a second threshold. The pixel coordinates corresponding to points in the surface sampling point set are (u, v). If u is greater than or equal to 0 and less than the first threshold, and v is greater than or equal to 0 and less than the second threshold, then this point is within the effective image range of the fixed-focus camera, and is considered the effective projection point of the surface sampling point corresponding to the fixed-focus camera. The first and second thresholds are set according to actual conditions, and this embodiment of the invention does not impose limitations.

[0079] After obtaining all valid projection points of the surface sampling point set within the effective image range of the fixed-focus camera, the distribution range of the valid projection points of the surface sampling points corresponding to each fixed-focus camera is statistically analyzed. The left boundary of the visible area range corresponding to the fixed-focus camera is the minimum image width coordinate among the valid projection points, the right boundary is the maximum image width coordinate among the valid projection points, the upper boundary is the minimum image height coordinate among the valid projection points, and the lower boundary is the maximum image height coordinate among the valid projection points.

[0080] 403. Based on the surface sampling points corresponding to the visible area range of each pair of adjacent fixed-focus cameras, obtain the spatial overlap area of ​​each pair of adjacent fixed-focus cameras; Specifically, for a pair of adjacent fixed-focus cameras, by comparing the surface sampling points corresponding to the visible area range of the two fixed-focus cameras, we can obtain the surface sampling points with the same three-dimensional coordinates. The area formed by all the surface sampling points with the same three-dimensional coordinates corresponding to the adjacent fixed-focus cameras is the spatial overlap area of ​​the adjacent fixed-focus cameras.

[0081] Wherein, the surface sampling points corresponding to the visible area range of the fixed-focus camera refer to the surface sampling points whose pixel coordinates are within the visible area range of the fixed-focus camera. 404. Based on the spatial overlap area of ​​each pair of adjacent fixed-focus cameras and the segmentation rules, obtain the spatial segmentation position of each pair of adjacent fixed-focus cameras; wherein, the segmentation rules are preset. Specifically, for each pair of adjacent fixed-focus cameras, the spatial segmentation position of the image plane of the adjacent fixed-focus cameras is determined from the spatial overlap area of ​​the adjacent fixed-focus cameras according to the segmentation rules. That is, the position of the segmentation line of the image plane of the adjacent fixed-focus cameras in three-dimensional space. The segmentation rules are preset.

[0082] For example, if the segmentation rule is average segmentation, meaning that the spatial overlap area of ​​adjacent fixed-focus cameras is divided equally between each of the adjacent fixed-focus cameras, then the spatial segmentation position is the middle position of the spatial overlap area. For the surface sampling points of each row in the spatial overlap area, the left and right boundary points of each row are obtained. The midpoint of the left and right boundary points of each row in the spatial overlap area is calculated, and the midpoints of each row are connected to obtain the spatial segmentation position.

[0083] 405. Transform the spatial segmentation position of each pair of adjacent fixed-focus cameras to the corresponding pixel coordinate system to obtain the matching position and matching range of each pair of adjacent fixed-focus cameras.

[0084] Specifically, for a pair of adjacent fixed-focus cameras, after obtaining the spatial segmentation positions of the adjacent fixed-focus cameras, the spatial segmentation positions are transformed into the two pixel coordinate systems of the adjacent fixed-focus cameras respectively, to obtain the matching position corresponding to each fixed-focus camera in the pair. The matching range can be set to a% of the image width of the fixed-focus camera, centered on the matching position. a is set according to actual needs, and is not limited in this embodiment of the invention. Figure 5 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the fifth embodiment of the present invention, as shown below. Figure 5 As shown, based on the above embodiments, obtaining the segmentation position further includes: S501. Based on the design drawings of the coal mining machine, establish a three-dimensional surface model of the coal mining machine; Specifically, the three-dimensional dimensions of the coal mining machine can be obtained from its design drawings. Based on these dimensions, a three-dimensional surface model of the coal mining machine can be created using 3D modeling software.

[0085] S502. Draw the three-dimensional coordinates of the segmented position in the world coordinate system where the three-dimensional surface model of the coal mining machine is located; Specifically, in order to more accurately segment the foreground and background images of the images captured by the fixed-focus camera, this application introduces the three-dimensional coordinate information of the segmentation position into traditional two-dimensional image segmentation. By drawing the segmentation position within the world coordinate system of the three-dimensional surface model of the coal mining machine, the three-dimensional coordinates of the segmentation position can be obtained. The location of the segmentation position in the world coordinate system is set according to actual needs, and this embodiment of the invention does not impose any limitations.

[0086] S503. Transform the three-dimensional coordinates of the segmentation position to the pixel coordinate system corresponding to each fixed-focus camera to obtain the segmentation position.

[0087] Specifically, after obtaining the three-dimensional coordinates of the segmentation position, the three-dimensional coordinates of the segmentation position are transformed into the pixel coordinate system corresponding to each fixed-focus camera to obtain the segmentation position. The segmentation position includes the segmentation position corresponding to each fixed-focus camera, that is, the pixel coordinates of the segmentation position in the pixel coordinate system corresponding to each fixed-focus camera. The pixel coordinates of the segmentation position in the pixel coordinate system corresponding to each fixed-focus camera can form a segmentation line.

[0088] In order to transform the 3D coordinates of the segmented location to the pixel coordinate system corresponding to the fixed-focus camera, it is necessary to first establish the transformation relationship between the world coordinate system and the pixel coordinate system corresponding to the fixed-focus camera. The specific process is as follows: The first step is to place markers in the world coordinate system. Markers at the same positions can be placed in the world coordinate system according to the position coordinates of multiple markers already placed on the longwall mining face. Each marker has three-dimensional coordinates in the world coordinate system, and the three-dimensional coordinates of the four corner points of the marker can be obtained. ArUco markers can be used.

[0089] The world coordinate system and the coal mining machine coordinate system have the same origin, consistent coordinate axes, and identical coordinate measurement scales. Therefore, the world coordinate system and the coal mining machine coordinate system are equivalent. Any point in the world coordinate system has the same three-dimensional coordinates as its corresponding point in the coal mining machine coordinate system.

[0090] The second step is to obtain the pixel coordinates of the marker code in the pixel coordinate system corresponding to the fixed-focus camera. Images of the fully mechanized coal mining face are acquired, resulting in images containing at least one marker code. Marker point detection is performed on the acquired images to obtain the pixel coordinates of the four corner points of each marker.

[0091] The third step involves obtaining the transformation relationship between the world coordinate system and the corresponding pixel coordinate system of the fixed-focus camera based on the 3D coordinates and pixel coordinates of the marker code, as well as the intrinsic parameter matrix and distortion coefficients of the fixed-focus camera. Using the 3D coordinates and pixel coordinates of the marker code, the camera intrinsic parameter matrix, and distortion coefficients of the fixed-focus camera as input data, the rotation and translation vectors from the world coordinate system to the fixed-focus camera coordinate system can be calculated using the PnP algorithm. Based on these vectors, the transformation relationship between the world coordinate system and the fixed-focus camera coordinate system can be established. The PnP algorithm can be implemented using the OpenCV PnP solver function `cv2.solvePnP`.

[0092] Step 4: Based on the matrix of the fixed-focus camera, obtain the transformation relationship from the camera coordinate system of the fixed-focus camera to the corresponding pixel coordinate system of the fixed-focus camera.

[0093] Step 5: Based on the transformation relationship from the world coordinate system to the camera coordinate system of the fixed-focus camera, and the transformation relationship from the camera coordinate system of the fixed-focus camera to the image coordinate system corresponding to the fixed-focus camera, obtain the transformation relationship from the world coordinate system to the pixel coordinate system corresponding to the fixed-focus camera.

[0094] After obtaining the transformation relationship from the world coordinate system to the pixel coordinate system corresponding to the fixed-focus camera, and given the three-dimensional coordinates of the segmentation position, the three-dimensional coordinates of the segmentation position can be transformed to the pixel coordinate system corresponding to the fixed-focus camera using the transformation relationship, thus obtaining the pixel coordinates of the segmentation position.

[0095] Figure 6 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the sixth embodiment of the present invention, as shown below. Figure 6 As shown, based on the above embodiments, the further step of performing image segmentation on the acquired image corresponding to each fixed-focus camera based on the segmentation position to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera includes: S601. Draw a segmentation curve in the image captured by the fixed-focus camera based on the segmentation position; Specifically, the pre-set segmentation positions are obtained, and segmentation curves are drawn on the captured images corresponding to the segmentation positions of the fixed-focus camera.

[0096] For example, if the segmentation position is half the height of the image captured by the fixed-focus camera, then the segmentation curve is drawn in the image captured by the fixed-focus camera according to half the height of the captured image.

[0097] For example, the segmentation curve can be smoothed to avoid image jitter.

[0098] S602. Based on the segmentation curve, extend the overlap height vertically and horizontally in the acquired image corresponding to the fixed-focus camera to obtain the foreground segmentation line and the background segmentation line; wherein, the overlap height is preset. Specifically, to better perform image fusion, overlapping areas are left when the acquired images are segmented vertically. In the acquired image corresponding to the fixed-focus camera, the foreground segmentation line is obtained by extending the overlap height upward based on the segmentation curve, and the background segmentation line is obtained by extending the overlap height downward based on the segmentation curve. The overlap height is preset.

[0099] For example, if a 10% overlap is left when segmenting an image vertically, and the segmentation point is located at half the height of the captured image, then the overlap height is 5% of the captured image height. Moving the segmentation line downwards by 5% of the captured image height yields the background segmentation line. Moving the segmentation line upwards by 5% of the captured image height yields the foreground segmentation line.

[0100] S603. Obtain the image below the foreground segmentation line from the acquired image corresponding to the fixed-focus camera, and use it as the foreground image corresponding to the fixed-focus camera. Obtain the image above the background segmentation line from the acquired image corresponding to the fixed-focus camera, and use it as the background image corresponding to the fixed-focus camera. Also, obtain the foreground segmentation line and the background segmentation line as segmentation line information.

[0101] Specifically, after obtaining the foreground and background segmentation lines, the image captured by the fixed-focus camera is cropped according to the foreground segmentation line to obtain the image below the foreground segmentation line, which is used as the foreground image of the fixed-focus camera; the image is cropped according to the background segmentation line to obtain the image above the background segmentation line, which is used as the background image of the fixed-focus camera. The foreground and background segmentation lines are then obtained as segmentation line information.

[0102] Figure 7 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the seventh embodiment of the present invention, as shown below. Figure 7 As shown, based on the above embodiments, further, the step of performing feature matching on the foreground and background images corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range of each pair of adjacent fixed-focus cameras, and obtaining the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras, includes: S701. Determine the pixel range of the overlapping area between the foreground and background images of adjacent fixed-focus cameras based on the matching position and matching range of the adjacent fixed-focus cameras. Specifically, for adjacent fixed-focus cameras, the overlapping pixel range can be determined in the foreground and background images of each fixed-focus camera by using the matching positions and matching ranges corresponding to the adjacent fixed-focus cameras. This yields the overlapping pixel range of the foreground image and the overlapping pixel range of the background image for each fixed-focus camera. The overlapping pixel ranges of the foreground image and the background image of each of the two adjacent fixed-focus cameras are then used as the overlapping pixel ranges of the foreground and background images corresponding to the adjacent fixed-focus cameras.

[0103] S702. Based on the pixel range of the overlapping area of ​​the foreground images of adjacent fixed-focus cameras, crop the foreground and background images of the two fixed-focus cameras in the adjacent fixed-focus cameras respectively to obtain the overlapping area of ​​the foreground image and the overlapping area of ​​the background image of each of the two fixed-focus cameras in the adjacent fixed-focus cameras. Specifically, based on the pixel range of the overlapping area of ​​the foreground images of each of the adjacent fixed-focus cameras, the foreground image of each fixed-focus camera is cropped to obtain the overlapping area of ​​the foreground image of each fixed-focus camera; based on the pixel range of the overlapping area of ​​the background images of each of the adjacent fixed-focus cameras, the background image of each fixed-focus camera is cropped to obtain the overlapping area of ​​the background image of each fixed-focus camera.

[0104] S703. Extract feature points from the overlapping areas of the foreground and background images of the two adjacent fixed-focus cameras to obtain the feature points of the overlapping areas of the foreground and background images of the two adjacent fixed-focus cameras. Specifically, feature points are extracted from the overlapping regions of the foreground images of each of the adjacent fixed-focus cameras to obtain feature points of the overlapping regions of the foreground images of each fixed-focus camera; feature points are extracted from the overlapping regions of the foreground images of each of the adjacent fixed-focus cameras to obtain feature points of the overlapping regions of the foreground images of each fixed-focus camera.

[0105] Feature point extraction can employ methods such as ORB feature extraction, accelerated KAZE (AKAZE) feature extraction, or SuperPoint feature extraction, and the choice can be made according to actual needs. This embodiment of the invention does not impose any limitations.

[0106] S704. Perform feature matching on the feature points of the overlapping area of ​​the foreground images of two adjacent fixed-focus cameras to obtain the matching feature points of the foreground images of the adjacent fixed-focus cameras; and perform feature matching on the feature points of the overlapping area of ​​the background images of two adjacent fixed-focus cameras to obtain the matching feature points of the background images of the adjacent fixed-focus cameras. Specifically, for adjacent fixed-focus cameras, feature matching is performed on the feature points of the overlapping regions of the foreground images of the two fixed-focus cameras to obtain the corresponding matching feature points of the foreground images of the adjacent fixed-focus cameras. These matching feature points are the same points in the foreground images of the two fixed-focus cameras. Similarly, feature matching is performed on the feature points of the overlapping regions of the background images of the two fixed-focus cameras to obtain the corresponding matching feature points of the background images of the adjacent fixed-focus cameras. These matching feature points are the same points in the background images of the two fixed-focus cameras.

[0107] Specifically, the Fast Library for Approximate Nearest Neighbors (FLANN) matcher or the Brute-Force (BF) matcher can be used to perform feature matching on the foreground images of two adjacent fixed-focus cameras. Unreliable matches are filtered using the Lowe's ratio test, and outlier matches are removed using the Random Sample Consensus (RANSAC) algorithm. Similarly, the FLANN or BF matcher can be used to perform feature matching on the background images of two adjacent fixed-focus cameras. Unreliable matches are filtered using the Lowe's ratio test, and outlier matches are removed using the RANSAC algorithm.

[0108] S705. Based on the matching feature points of the foreground image corresponding to the adjacent fixed-focus camera, obtain the first offset corresponding to the adjacent fixed-focus camera; and based on the matching feature points of the background image corresponding to the adjacent fixed-focus camera, obtain the second offset corresponding to the adjacent fixed-focus camera.

[0109] Specifically, for the foreground image matching feature points corresponding to adjacent fixed-focus cameras, each of the two fixed-focus cameras has pixel coordinates in its foreground image. The difference between the two pixel coordinates of each foreground image matching feature point is calculated to obtain the translation amount of each foreground image matching feature point. The median of the translation amounts of each foreground image matching feature point is statistically analyzed, and the obtained median is used as the first offset corresponding to the adjacent fixed-focus camera.

[0110] For background image matching feature points corresponding to adjacent fixed-focus cameras, pixel coordinates are respectively found in the background images of the two adjacent fixed-focus cameras. The difference between the two pixel coordinates of each background image matching feature point is calculated to obtain the translation amount of each background image matching feature point. The median of the translation amounts of each background image matching feature point is statistically analyzed, and the obtained median is used as the second offset corresponding to the adjacent fixed-focus cameras.

[0111] Figure 8This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the eighth embodiment of the present invention, as shown below. Figure 8 As shown, based on the above embodiments, further, the step of sequentially stitching and horizontal image fusion according to the arrangement order of each fixed-focus camera, the foreground image corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera to obtain the foreground image of the coal mine fully mechanized mining face includes: S801. Stitch together the foreground images corresponding to adjacent fixed-focus cameras in the order of their arrangement. Specifically, the fixed-focus cameras are arranged sequentially along the hydraulic support, following a specific order along the width of the support. The foreground images corresponding to adjacent fixed-focus cameras are then stitched together according to their arrangement.

[0112] S802. Based on the first offset corresponding to each pair of adjacent fixed-focus cameras, perform horizontal image fusion on the foreground images corresponding to each pair of adjacent fixed-focus cameras.

[0113] Specifically, for adjacent fixed-focus cameras, the foreground images of the two fixed-focus cameras overlap. By using the first offset corresponding to the adjacent fixed-focus cameras, the foreground images of the two fixed-focus cameras are progressively fused in the horizontal direction to achieve a smooth transition in the overlapping area and improve the quality of the stitching of adjacent foreground images.

[0114] Figure 9 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the eighth embodiment of the present invention, as shown below. Figure 9 As shown, based on the above embodiments, the further step of performing horizontal image fusion on the foreground images corresponding to each pair of adjacent fixed-focus cameras based on the first offset corresponding to each pair of adjacent fixed-focus cameras includes: S901. Determine the size of the first splicing canvas based on the foreground image corresponding to the adjacent fixed-focus camera and the first offset; Specifically, the foreground image corresponding to adjacent fixed-focus cameras includes the foreground images of two fixed-focus cameras. One of these cameras is listed first in the sequence and is called the preceding fixed-focus camera, while the other is listed last and is called the following fixed-focus camera. The size of the first stitching canvas can be determined based on the foreground images of the preceding and following fixed-focus cameras and the first offset. Since the foreground images of the preceding and following fixed-focus cameras overlap, the width of the stitching canvas is less than the sum of the widths of the foreground images of the preceding and following fixed-focus cameras.

[0115] For example, the width of the foreground image of the preceding fixed-focus camera is w1, and the height is h1; the width of the foreground image of the following fixed-focus camera is w2, and the height is h2; the first offset includes an offset dw in the width direction and an offset dh in the height direction. Then the width of the stitched canvas is w1 + w2 - dw, and the height of the stitched canvas is h1 + h2 - dh.

[0116] S902. Based on the size of the first splicing canvas, create a first splicing canvas and place the foreground image of the preceding fixed-focus camera in the adjacent fixed-focus camera; Specifically, a first splicing canvas is created based on the size of the first splicing canvas, and the created first splicing canvas is a blank canvas. The foreground image of the preceding fixed-focus camera in the adjacent fixed-focus camera group is first placed in the blank canvas.

[0117] For example, the foreground image of the preceding fixed-focus camera in an adjacent fixed-focus camera can be placed on the far left of the stitching canvas, and the left edge of the foreground image of the preceding fixed-focus camera can be aligned with the left edge of the stitching canvas.

[0118] S903. Based on the foreground image of the preceding fixed-focus camera in the adjacent fixed-focus cameras and the first offset, obtain a first overlapping region image, and based on the foreground image of the following fixed-focus camera in the adjacent fixed-focus cameras and the first offset, obtain a second overlapping region image. Specifically, in the foreground image of the preceding fixed-focus camera among adjacent fixed-focus cameras, the foreground image of the preceding fixed-focus camera is cropped from right to left according to the first offset to obtain a first overlapping region image. In the foreground image of the following fixed-focus camera among adjacent fixed-focus cameras, the foreground image of the following fixed-focus camera is cropped from left to right according to the first offset to obtain a second overlapping region image. The first overlapping region image and the second overlapping region image are of equal size.

[0119] S904. The first overlapping region image and the second overlapping region image are weighted and fused according to the first horizontal gradient weight matrix to obtain the overlapping region image of the foreground images corresponding to adjacent fixed-focus cameras. Specifically, a first horizontal gradient weight matrix is ​​created, corresponding to the size of the first overlapping region image and the second overlapping region image. The element values ​​of the first horizontal gradient weight matrix change linearly in the horizontal direction, from 0 to 1. The first horizontal gradient weight matrix is ​​used to fuse the first overlapping region image and the second overlapping region image into a single image, obtaining the overlapping region image of the foreground images corresponding to adjacent fixed-focus cameras.

[0120] S905, Place the overlapping region image of the foreground image corresponding to the adjacent fixed-focus camera and the remaining image of the foreground image of the subsequent fixed-focus camera in the adjacent fixed-focus camera into the first splicing canvas; wherein, the remaining image of the foreground image of the subsequent fixed-focus camera in the adjacent fixed-focus camera is the image remaining after removing the second overlapping region image from the foreground image of the subsequent fixed-focus camera in the adjacent fixed-focus camera.

[0121] Specifically, after obtaining the overlapping region images of the foreground images corresponding to adjacent fixed-focus cameras, the overlapping region images of the foreground images corresponding to adjacent fixed-focus cameras are placed into the first stitching canvas and merged with the foreground image of the preceding fixed-focus camera among the adjacent fixed-focus cameras. Then, the remaining image of the foreground image of the subsequent fixed-focus camera among the adjacent fixed-focus cameras is placed into the first stitching canvas, adjacent to the overlapping region images of the foreground images corresponding to the adjacent fixed-focus cameras. From the foreground image of the subsequent fixed-focus camera, the second overlapping region image is removed to obtain the remaining image of the foreground image of the subsequent fixed-focus camera among the adjacent fixed-focus cameras.

[0122] Figure 10 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the tenth embodiment of the present invention, as shown below. Figure 10 As shown, based on the above embodiments, further, the step of sequentially stitching and horizontally fusing the images according to the arrangement order of the fixed-focus cameras, based on the background images corresponding to each adjacent fixed-focus camera and the second offset corresponding to each adjacent fixed-focus camera, to obtain the background image of the coal mine fully mechanized mining face includes: S1001. Stitch together the background images corresponding to adjacent fixed-focus cameras in the order of their arrangement. Specifically, the fixed-focus cameras are arranged sequentially along the hydraulic support, following a specific order along the width of the support. The background images corresponding to adjacent fixed-focus cameras are then stitched together according to their arrangement.

[0123] S1002. Based on the second offset corresponding to each pair of adjacent fixed-focus cameras, perform horizontal image fusion on the overlapping area of ​​the background images corresponding to each pair of adjacent fixed-focus cameras.

[0124] Specifically, for adjacent fixed-focus cameras, the background images of the two fixed-focus cameras overlap. By using the second offset corresponding to the adjacent fixed-focus cameras, the background images of the two fixed-focus cameras are gradually fused in the horizontal direction to achieve a smooth transition in the overlapping area and improve the quality of the stitching of adjacent background images.

[0125] Figure 11This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the eleventh embodiment of the present invention, as shown below. Figure 11 As shown, based on the above embodiments, the further step of performing horizontal image fusion on the overlapping region of the foreground images corresponding to each pair of adjacent fixed-focus cameras based on the second offset corresponding to each pair of adjacent fixed-focus cameras includes: S1101. Determine the size of the second splicing canvas based on the background image and the second offset corresponding to the adjacent fixed-focus camera; Specifically, the background image corresponding to an adjacent fixed-focus camera includes the background images of two fixed-focus cameras. One of these cameras is listed first in the sequence and is called the preceding fixed-focus camera, while the other is listed last and is called the following fixed-focus camera. The size of the second stitching canvas can be determined based on the background images of the preceding and following fixed-focus cameras, as well as the second offset. Since the background images of the preceding and following fixed-focus cameras overlap, the width of the stitching canvas is less than the sum of the widths of the background images of the preceding and following fixed-focus cameras.

[0126] S1102. Based on the size of the second splicing canvas, create a second splicing canvas and place the background image of the preceding fixed-focus camera in the adjacent fixed-focus camera; Specifically, a second splicing canvas is created based on the size of the second splicing canvas, and the created second splicing canvas is a blank canvas. The background image of the preceding fixed-focus camera from the adjacent fixed-focus cameras is first placed in the blank canvas.

[0127] S1103. Based on the background image of the preceding fixed-focus camera in the adjacent fixed-focus cameras and the second offset, obtain the third overlapping region image, and based on the background image of the following fixed-focus camera in the adjacent fixed-focus cameras and the second offset, obtain the fourth overlapping region image. Specifically, in the background image of the preceding fixed-focus camera among adjacent fixed-focus cameras, the background image of the preceding fixed-focus camera is cropped from right to left according to the second offset to obtain the third overlapping region image. In the background image of the following fixed-focus camera among adjacent fixed-focus cameras, the background image of the following fixed-focus camera is cropped from left to right according to the second offset to obtain the fourth overlapping region image. The third overlapping region image and the fourth overlapping region image are of equal size.

[0128] S1104. The third and fourth overlapping region images are weighted and fused according to the second level gradient weight matrix to obtain the overlapping region images of the background images corresponding to adjacent fixed-focus cameras. Specifically, a second horizontal gradient weight matrix is ​​created, corresponding to the size of the third and fourth overlapping region images. The element values ​​of the second horizontal gradient weight matrix change linearly in the horizontal direction, from 0 to 1. The third and fourth overlapping region images are then fused into a single image using the second horizontal gradient weight matrix, resulting in the overlapping region image of the background images corresponding to adjacent fixed-focus cameras.

[0129] S1105. Place the overlapping area image of the background image corresponding to the adjacent fixed-focus camera and the remaining image of the background image of the subsequent fixed-focus camera in the adjacent fixed-focus camera into the second splicing canvas; wherein, the remaining image of the background image of the subsequent fixed-focus camera in the adjacent fixed-focus camera is the image remaining after removing the fourth overlapping area image from the background image of the subsequent fixed-focus camera in the adjacent fixed-focus camera.

[0130] Specifically, after obtaining the overlapping region images of the background images corresponding to adjacent fixed-focus cameras, the overlapping region images of the background images corresponding to adjacent fixed-focus cameras are placed in the second stitching canvas, adjacent to the background image of the preceding fixed-focus camera among the adjacent fixed-focus cameras. Then, the remaining image of the background image of the subsequent fixed-focus camera among the adjacent fixed-focus cameras is placed in the second stitching canvas, adjacent to the overlapping region images of the background images corresponding to the adjacent fixed-focus cameras. By removing the fourth overlapping region image from the background image of the subsequent fixed-focus camera, the remaining image of the background image of the subsequent fixed-focus camera among the adjacent fixed-focus cameras can be obtained.

[0131] Figure 12 This is a flowchart illustrating the video stitching method for a fully mechanized coal mining face provided in the twelfth embodiment of the present invention, as shown below. Figure 12 As shown, based on the above embodiments, the step of performing vertical image fusion based on the segmentation line information, the foreground image and the background image of the fully mechanized coal mining face to obtain a preliminary panoramic image of the fully mechanized coal mining face includes: S1201. Based on the foreground image and background image of the fully mechanized coal mining face and the dividing line information, obtain the overlapping area of ​​the foreground image and the overlapping area of ​​the background image. Specifically, the segmentation information includes the segmentation position and the overlap position of the foreground image and the background image. By using the segmentation position and the overlap position of the foreground image and the background image in the foreground image of the fully mechanized coal mining face, the overlap area of ​​the foreground image can be obtained. By using the segmentation position and the overlap position of the foreground image and the background image in the background image of the fully mechanized coal mining face, the overlap area of ​​the background image can be obtained.

[0132] S1202. The overlapping regions of the foreground image and the background image are weighted and fused according to the vertical gradient weight matrix to obtain the overlapping region image of the coal mine fully mechanized mining face. Specifically, a vertical gradient weight matrix is ​​created that corresponds to the size of the overlapping regions of the foreground image and the background image. The element values ​​of the vertical gradient weight matrix change linearly in the vertical direction, from 0 to 1. The overlapping regions of the foreground image and the background image are then merged into a single image using the vertical gradient weight matrix to obtain the overlapping region image of the fully mechanized coal mining face.

[0133] S1203. Stitch together the foreground image of the fully mechanized coal mining face, the overlapping area image of the fully mechanized coal mining face, and the remaining image of the background image of the fully mechanized coal mining face to obtain a preliminary panoramic image of the fully mechanized coal mining face; wherein, the remaining image of the fully mechanized coal mining face is the image remaining after removing the overlapping area of ​​the background image.

[0134] Specifically, the foreground image of the fully mechanized coal mining face is placed on a blank canvas. Then, the overlapping area image of the fully mechanized coal mining face is placed on the canvas and merged with the foreground image. Next, the remaining background image of the fully mechanized coal mining face is placed on the canvas adjacent to the overlapping area image. The overlapping area of ​​the background image is then removed from the background image of the fully mechanized coal mining face to obtain the remaining background image of the fully mechanized coal mining face.

[0135] The following specific embodiment illustrates the video stitching method for fully mechanized coal mining faces provided by the present invention.

[0136] A coal mine's longwall mining face is approximately 300 meters wide and equipped with 103 intrinsically safe fixed-focus cameras. These cameras are sequentially mounted on the outer cylinders of the hydraulic support columns. Each camera has a resolution of 1920×1080 pixels and a frame rate of 25fps. Fifteen ArUco markers, each 0.1m × 0.1m in size, are spaced apart on the coal mining machine. The position of each ArUco marker is precisely recorded relative to the machine. Adjacent intrinsically safe fixed-focus cameras can jointly capture at least two ArUco markers.

[0137] Steps S301 to S304 are executed on the first frame image of each fixed-focus camera to obtain the pose parameters of 103 fixed-focus cameras and the stitching parameters of 102 pairs of adjacent fixed-focus cameras. During pose estimation, each fixed-focus camera detects 2 to 4 marker points. After calculating the pose using the PnP algorithm, the LM algorithm is used for optimization. A stable pose result is obtained after weighted fusion of multiple marker points. In the 3D modeling stage, the front surface of the coal mining machine is defined as a multi-dimensional plane 1 to 3 meters away from the camera, and the coal wall plane is defined as a plane 2 to 5 meters away from the camera. The segmentation line positions of the foreground and background images are calculated, with a typical segmentation range of 45% to 55% of the image height.

[0138] Steps S201 to S206 are executed for each frame of images from subsequent fixed-focus cameras. Distortion correction is accelerated using a GPU, with a single-frame processing time of approximately 5 milliseconds. After image segmentation, the foreground image height is approximately 55% of the original image, and the background image height is approximately 52% of the original image, with a 7% vertical overlap. Feature matching uses the AKAZE algorithm, extracting 500 to 1000 feature points from the images acquired by each pair of adjacent fixed-focus cameras. After Lowe ratio testing and RANSAC filtering, 50 to 200 reliable matching points are retained, achieving a matching success rate of over 98%. Feature matching and translation calculation for a single pair of images takes approximately 20 milliseconds.

[0139] In the image fusion stage, the overlapping area accounts for 20% to 30% of the width of a single image. A gradient weight matrix achieves a smooth transition, resulting in no obvious seams in the overlapping area after fusion. Foreground image stitching is performed pairwise, with 103 images stitched into a foreground image approximately 6500 pixels wide, taking about 1 second. Background image stitching also takes about 1 second. When merging foreground and background images, vertical weighted fusion is performed in the 7% overlap area to maintain the continuity of the segmentation line. In the black border removal stage, approximately 2% to 5% of the image edges are cropped to output the final panoramic image.

[0140] In a laboratory environment, using images with a resolution of 1920×1080, 20 marker points were evenly selected on the surface of the coal mining machine. After 100 repeated tests, the average positional deviation after stitching was 1.5 pixels, and the maximum deviation did not exceed 2 pixels, meeting the monitoring requirements. Under different lighting conditions, coal dust interference, and hydraulic support movement, the system could operate stably, with a stitching success rate remaining above 98%.

[0141] This application enables panoramic real-time monitoring of the fully mechanized mining face, allowing operators to observe a complete view of the working face from the ground monitoring center, significantly improving the safety monitoring effect and the level of intelligence.

[0142] The video stitching method for fully mechanized coal mining faces provided in this application can solve the following technical problems: (1) This invention solves the problem of insufficient stitching accuracy of 2D feature matching in 3D scenes: Existing technologies are based solely on 2D image feature matching. In 3D scenes with significant differences in foreground and background depth (such as between the front surface of a coal mining machine and the coal wall), a single homography transformation cannot accurately describe the spatial geometric relationship, leading to stitching misalignment and ghosting. This application achieves high-precision stitching by establishing a 3D spatial model and calculating the real geometric relationship between images in three-dimensional space.

[0143] (2) Solved the problem of stitching failure caused by dynamic changes in camera pose: During operation, the hydraulic support will perform actions such as lifting, moving and pushing, which will cause dynamic changes in the position and angle of the camera, which cannot be adapted by the existing fixed transformation matrix method. This application estimates the camera pose in real time through ArUco markers and dynamically calculates the transformation parameters to adapt to changes in equipment position.

[0144] (3) This application solves the problem of decreased stitching quality caused by uniform processing of foreground and background: Existing technologies use the same processing strategy for the entire image, which cannot take into account the stitching accuracy of different depth areas. This application uses intelligent layering based on 3D modeling results to divide the image into foreground (front surface of the coal mining machine) and background (coal wall plane) for separate processing, and optimizes the stitching parameters accordingly.

[0145] (4) Solved the computational efficiency problem of real-time stitching of multiple cameras: In a large-scale scene with 103 cameras, the traditional full-image feature matching computation is huge and difficult to process in real time. This application predicts the location of overlapping areas by 3D modeling, narrows the feature matching range, and adopts a hierarchical parallel processing strategy to achieve real-time stitching while ensuring accuracy (stitching time of two images is less than or equal to 50ms).

[0146] (5) Solved the problems of stitching gaps and fusion quality: Existing simple fusion methods are prone to producing obvious stitching gaps, brightness discontinuities and ghosting phenomena. This application adopts a progressive multi-band fusion algorithm, combined with local color matching and smooth weight allocation, to achieve seamless and natural image fusion.

[0147] (6) It solves the problems of large calibration workload and difficult maintenance: Traditional pre-calibration methods are labor-intensive, time-consuming, and require frequent recalibration. This application uses ArUco markers to achieve automated pose estimation, simplifying the calibration process and reducing maintenance costs.

[0148] This application enables real-time video stitching of 103 intrinsically safe fixed-focus cameras in the complex 3D environment of a fully mechanized coal mining face, generating high-quality panoramic images (stitching accuracy less than 2 pixels, stitching success rate ≥98%), providing a reliable visual perception foundation for intelligent coal mining.

[0149] Figure 13This is a schematic diagram of the structure of the video splicing device for a fully mechanized coal mining face provided in the thirteenth embodiment of the present invention, as shown below. Figure 13 As shown in the embodiment of the present invention, the video splicing device for a fully mechanized coal mining face includes an acquisition module 1301, an image segmentation module 1302, a feature matching module 1303, a horizontal splicing module 1304, a vertical splicing module 1305, and a post-processing module 1306, wherein: The acquisition module 1301 is used to acquire images captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face; wherein the video stream of the fully mechanized coal mining face is based on the acquisition by each fixed-focus camera; the acquired images contain at least one marker code, which is pre-configured on the coal mining machine and the side guard plate; the image segmentation module 1302 is used to perform image segmentation on the acquired images corresponding to each fixed-focus camera based on the segmentation position, to obtain the foreground image, background image and segmentation line information corresponding to each fixed-focus camera; wherein the foreground image and the background image have overlapping areas, and the segmentation position is pre-obtained; the feature matching module 1303 is used to perform feature matching on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras, to obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras; wherein each pair of adjacent fixed-focus cameras corresponds to The matching position and matching range are obtained in advance; the horizontal stitching module 1304 is used to perform stitching and horizontal image fusion sequentially according to the arrangement order of each fixed-focus camera, the foreground image corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera to obtain the foreground image of the coal mine fully mechanized mining face; and according to the arrangement order of each fixed-focus camera, the background image corresponding to each adjacent fixed-focus camera, and the second offset corresponding to each adjacent fixed-focus camera, to perform stitching and horizontal image fusion sequentially to obtain the background image of the coal mine fully mechanized mining face; the vertical stitching module 1305 is used to perform vertical image fusion according to the dividing line information, the foreground image and the background image of the coal mine fully mechanized mining face to obtain the preliminary panoramic image of the coal mine fully mechanized mining face; the post-processing module 1306 is used to post-process the preliminary panoramic image of the coal mine fully mechanized mining face to obtain the final panoramic image of the coal mine fully mechanized mining face.

[0150] Specifically, multiple marker codes are configured on the coal mining machine and the side protection plate. Fixed-focus cameras mounted on the outer cylinders of adjacent hydraulic supports capture images of the fully mechanized coal mining face, obtaining a video stream of the face. The acquisition module 1301 can acquire the images captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face. Each captured image contains at least one marker code. Adjacent fixed-focus cameras have overlapping fields of view, and their captured images share at least one common marker code. Since multiple fixed-focus cameras provide video streams, to avoid splicing jitter caused by time asynchrony, the video stream of the fully mechanized coal mining face provided to the image processing unit is a time-synchronized video stream.

[0151] The image segmentation module 1302 segments the acquired images corresponding to each fixed-focus camera according to the segmentation position, dividing the acquired image into two images. This provides the foreground image, background image, and segmentation line information for each fixed-focus camera. The background image is the upper part of the acquired image, including the coal face. The foreground image is the lower part of the acquired image, including the front surface of the coal mining machine. During image segmentation, some overlapping areas are retained, meaning the foreground and background images partially overlap. The segmentation line information includes the segmentation position and the overlap position between the foreground and background images. The segmentation position is pre-determined and used to divide the acquired image into the background and foreground images.

[0152] The feature matching module 1303 performs feature matching on the foreground image corresponding to each pair of adjacent fixed-focus cameras based on the matching position and matching range, thereby obtaining a first offset for each pair of adjacent fixed-focus cameras. The feature matching module 1303 also performs feature matching on the background image corresponding to each pair of adjacent fixed-focus cameras based on the matching position and matching range, thereby obtaining a second offset for each pair of adjacent fixed-focus cameras.

[0153] The horizontal stitching module 1304 stitches the foreground images corresponding to each fixed-focus camera sequentially according to their arrangement. Each time the foreground images corresponding to adjacent fixed-focus cameras are stitched together, horizontal image fusion is performed based on the foreground images of the adjacent fixed-focus cameras and a first offset. After all the foreground images corresponding to the fixed-focus cameras are stitched together, the foreground image of the fully mechanized coal mining face is obtained. The horizontal stitching module 1304 also stitches the background images corresponding to each fixed-focus camera sequentially according to their arrangement. Each time the background images corresponding to adjacent fixed-focus cameras are stitched together, horizontal image fusion is performed based on the background images of the adjacent fixed-focus cameras and a second offset. After all the background images corresponding to the fixed-focus cameras are stitched together, the background image of the fully mechanized coal mining face is obtained.

[0154] The vertical stitching module 1305 uses the dividing line information to fuse the foreground and background images of the fully mechanized coal mining face in the vertical direction, thereby obtaining a preliminary panoramic image of the fully mechanized coal mining face.

[0155] The post-processing module 1306 performs post-processing on the preliminary panoramic image of the fully mechanized coal mining face to obtain the final panoramic image of the fully mechanized coal mining face. The specific post-processing procedure is set according to actual needs, and this embodiment of the invention does not limit it.

[0156] The video stitching device for a fully mechanized coal mining face provided in this invention can acquire images from each fixed-focus camera in the video stream of the fully mechanized coal mining face; perform image segmentation on the acquired image corresponding to each fixed-focus camera based on the segmentation position to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera; perform feature matching on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range of each pair of adjacent fixed-focus cameras to obtain the first offset and second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range corresponding to each adjacent fixed-focus camera are obtained in advance; based on the arrangement order of each fixed-focus camera and the foreground image corresponding to each adjacent fixed-focus camera... The foreground image of the fully mechanized coal mining face is obtained by sequentially stitching and horizontally fusing images based on the first offset of each adjacent fixed-focus camera and the first offset of each adjacent fixed-focus camera. Then, the background image of the fully mechanized coal mining face is obtained by sequentially stitching and horizontally fusing images based on the arrangement order of the fixed-focus cameras, the background image of each adjacent fixed-focus camera, and the second offset of each adjacent fixed-focus camera. Finally, the preliminary panoramic image of the fully mechanized coal mining face is post-processed to obtain the final panoramic image of the fully mechanized coal mining face, thus improving the stitching accuracy of the fully mechanized coal mining face.

[0157] Based on the above embodiments, the video stitching device for a fully mechanized coal mining face provided in this embodiment of the invention further includes a first frame image acquisition module, a marker detection module, a pose solving module, and an acquisition module, wherein: The first-frame image acquisition module is used to acquire the first-frame image captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face; the marker detection module is used to perform marker detection based on the first-frame image corresponding to each fixed-focus camera, to obtain the marker points corresponding to each fixed-focus camera and the relationship point set of each fixed-focus camera, wherein each point in the relationship point set of each fixed-focus camera has three-dimensional coordinates and corresponding pixel coordinates; the pose solving module is used to solve the pose of each fixed-focus camera based on the relationship point set of each fixed-focus camera, to obtain the position and attitude angle of each fixed-focus camera, as well as the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera; the acquisition module is used to calculate the overlapping area of ​​adjacent fixed-focus cameras based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, the intrinsic parameters and distortion coefficients of each fixed-focus camera, to obtain the matching position corresponding to each pair of adjacent fixed-focus cameras and obtain the corresponding matching range.

[0158] Based on the above embodiments, the obtaining module further includes a first obtaining unit, a first determining unit, a second obtaining unit, a third obtaining unit, and a fourth obtaining unit, wherein: The first obtaining unit is used to obtain the pixel coordinates of the surface sampling point set projected onto the image plane of each fixed-focus camera based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, and the intrinsic parameters and distortion coefficients of each fixed-focus camera. The determining unit is used to determine the visible area range corresponding to each fixed-focus camera based on the pixel coordinates corresponding to the surface sampling point set and the effective image range of each fixed-focus camera. The second obtaining unit is used to obtain the spatial overlap area of ​​each pair of adjacent fixed-focus cameras based on the surface sampling points corresponding to the visible area range of each pair of adjacent fixed-focus cameras. The third obtaining unit is used to obtain the spatial segmentation position of each pair of adjacent fixed-focus cameras based on the spatial overlap area of ​​each pair of adjacent fixed-focus cameras and the segmentation rules, wherein the segmentation rules are preset. The fourth obtaining unit is used to transform the spatial segmentation position of each pair of adjacent fixed-focus cameras to the corresponding pixel coordinate system to obtain the matching position and the corresponding matching range of each pair of adjacent fixed-focus cameras.

[0159] Based on the above embodiments, obtaining the segmentation position further includes: Based on the design drawings of the coal mining machine, a three-dimensional surface model of the coal mining machine is established; the three-dimensional coordinates of the segmentation position are drawn in the world coordinate system of the three-dimensional surface model of the coal mining machine; the three-dimensional coordinates of the segmentation position are transformed into the pixel coordinate system corresponding to each fixed-focus camera to obtain the segmentation position.

[0160] Based on the above embodiments, the image segmentation module 1302 further includes a drawing unit, a fifth obtaining unit, and an acquisition unit, wherein: The drawing unit is used to draw a segmentation curve in the image captured by the fixed-focus camera according to the segmentation position; the fifth obtaining unit is used to extend the overlap height above and below the image captured by the fixed-focus camera based on the segmentation curve to obtain the foreground segmentation line and the background segmentation line; wherein, the overlap height is preset; the obtaining unit is used to obtain the image below the foreground segmentation line from the image captured by the fixed-focus camera as the foreground image of the fixed-focus camera, obtain the image above the background segmentation line from the image captured by the fixed-focus camera as the background image of the fixed-focus camera, and obtain the foreground segmentation line and the background segmentation line as segmentation line information.

[0161] Based on the above embodiments, the feature matching module 1303 further includes a second determining unit, a cropping unit, a feature point extraction unit, a feature matching unit, and a sixth obtaining unit, wherein: The second determining unit is used to determine the pixel range of the overlapping area of ​​the foreground and background images of adjacent fixed-focus cameras based on the matching position and matching range of the adjacent fixed-focus cameras; the cropping unit is used to crop the foreground and background images of the two adjacent fixed-focus cameras respectively based on the pixel range of the overlapping area of ​​the foreground images of the adjacent fixed-focus cameras, to obtain the overlapping area of ​​the foreground image and the overlapping area of ​​the background image of each of the two adjacent fixed-focus cameras; the feature point extraction unit is used to extract feature points from the overlapping area of ​​the foreground image and the overlapping area of ​​the background image of each of the two adjacent fixed-focus cameras, to obtain the overlapping area of ​​the foreground image and the overlapping area of ​​the background image of each of the two adjacent fixed-focus cameras. The system includes feature points in the overlapping regions of the foreground and background images; a feature matching unit performs feature matching on the feature points in the overlapping regions of the foreground images of two adjacent fixed-focus cameras to obtain matching feature points in the foreground images of the adjacent fixed-focus cameras; and performs feature matching on the feature points in the overlapping regions of the background images of two adjacent fixed-focus cameras to obtain matching feature points in the background images of the adjacent fixed-focus cameras; a sixth obtaining unit obtains a first offset corresponding to the adjacent fixed-focus cameras based on the matching feature points in the foreground images of the adjacent fixed-focus cameras; and obtains a second offset corresponding to the adjacent fixed-focus cameras based on the matching feature points in the background images of the adjacent fixed-focus cameras.

[0162] Based on the above embodiments, the horizontal stitching module 1304 further includes a first stitching unit and a first image fusion unit, wherein: The first stitching unit is used to stitch together the foreground images corresponding to adjacent fixed-focus cameras in the order of their arrangement; the first image fusion unit is used to perform horizontal image fusion on the foreground images corresponding to each pair of adjacent fixed-focus cameras based on the first offset corresponding to each pair of adjacent fixed-focus cameras.

[0163] Based on the above embodiments, the first splicing unit further includes a first determining subunit, a first creating subunit, a first obtaining subunit, a second obtaining subunit, and a first placing subunit, wherein: The first determining subunit is used to determine the size of the first stitching canvas based on the foreground images corresponding to adjacent fixed-focus cameras and the first offset; the first creating subunit is used to create the first stitching canvas based on the first stitching canvas size and place the foreground image of the preceding fixed-focus camera among the adjacent fixed-focus cameras; the first obtaining subunit is used to obtain a first overlapping region image based on the foreground image of the preceding fixed-focus camera among the adjacent fixed-focus cameras and the first offset, and to obtain a second overlapping region image based on the foreground image of the following fixed-focus camera among the adjacent fixed-focus cameras and the first offset; the second obtaining subunit is used to perform weighted fusion of the first overlapping region image and the second overlapping region image according to the first horizontal gradient weight matrix to obtain the overlapping region image of the foreground images corresponding to adjacent fixed-focus cameras; the first placing subunit is used to place the overlapping region image of the foreground images corresponding to adjacent fixed-focus cameras and the remaining image of the foreground image of the following fixed-focus camera among the adjacent fixed-focus cameras into the stitching canvas; wherein, the remaining image of the foreground image of the following fixed-focus camera among the adjacent fixed-focus cameras is the image remaining after removing the second overlapping region image from the foreground image of the following fixed-focus camera among the adjacent fixed-focus cameras.

[0164] Based on the above embodiments, the horizontal stitching module 1304 further includes a second stitching unit and a second image fusion unit, wherein: The second stitching unit is used to stitch the background images corresponding to adjacent fixed-focus cameras in the order of their arrangement; the second image fusion unit is used to perform horizontal image fusion on the overlapping area of ​​the background images corresponding to each pair of adjacent fixed-focus cameras based on the second offset corresponding to each pair of adjacent fixed-focus cameras.

[0165] Based on the above embodiments, the second splicing unit further includes a second determining subunit, a second creating subunit, a third obtaining subunit, a fourth obtaining subunit, and a second placing subunit, wherein: The second determining subunit is used to determine the size of the second stitching canvas based on the background image corresponding to the adjacent fixed-focus camera and the second offset; the second creating subunit is used to create the second stitching canvas based on the second stitching canvas size and place the background image of the preceding fixed-focus camera among the adjacent fixed-focus cameras; the third obtaining subunit is used to obtain the third overlapping region image based on the background image of the preceding fixed-focus camera among the adjacent fixed-focus cameras and the second offset, and to obtain the fourth overlapping region image based on the background image of the following fixed-focus camera among the adjacent fixed-focus cameras and the second offset; the fourth obtaining subunit is used to perform weighted fusion of the third overlapping region image and the fourth overlapping region image according to the second horizontal gradient weight matrix to obtain the overlapping region image of the background image corresponding to the adjacent fixed-focus camera; the second placing subunit is used to place the overlapping region image of the background image corresponding to the adjacent fixed-focus camera and the remaining image of the background image of the following fixed-focus camera among the adjacent fixed-focus cameras into the second stitching canvas; wherein, the remaining image of the background image of the following fixed-focus camera among the adjacent fixed-focus cameras is the image remaining after removing the fourth overlapping region image from the background image of the following fixed-focus camera among the adjacent fixed-focus cameras.

[0166] Based on the above embodiments, the vertical splicing module 1305 further includes a seventh obtaining unit, an eighth obtaining unit, and a third splicing unit, wherein: The seventh obtaining unit is used to obtain the overlapping region of the foreground image and the overlapping region of the background image based on the foreground image and the background image of the fully mechanized coal mining face and the segmentation line information; the eighth obtaining unit is used to perform weighted fusion of the overlapping region of the foreground image and the overlapping region of the background image based on the vertical gradient weight matrix to obtain the overlapping region image of the fully mechanized coal mining face; the third stitching unit is used to stitch together the foreground image of the fully mechanized coal mining face, the overlapping region image of the fully mechanized coal mining face, and the remaining image of the background image of the fully mechanized coal mining face to obtain a preliminary panoramic image of the fully mechanized coal mining face; wherein, the remaining image of the fully mechanized coal mining face is the image remaining after removing the overlapping region of the background image.

[0167] This application has the following significant advantages compared to the prior art: (1) The splicing accuracy is significantly improved; By using 3D spatial modeling to guide image stitching, the overlapping areas and stitching positions are accurately calculated in 3D space, solving the stitching misalignment problem caused by perspective distortion in 3D scenes using traditional 2D feature matching methods. Actual measurements show a stitching accuracy of less than 2 pixels, representing a 3- to 5-fold improvement in accuracy compared to traditional methods.

[0168] (2) Adapting to dynamic environmental changes; Real-time pose estimation based on ArUco markers eliminates the need for pre-calibration and dynamically adapts to camera position changes caused by hydraulic support lifting, moving, and other actions. Compared to fixed transformation matrix methods, it eliminates stitching failures caused by equipment movement, significantly improving system reliability.

[0169] (3) Simplify calibration and maintenance; ArUco marker-based automated pose estimation simplifies and speeds up the calibration process, requiring only the recording of marker positions. Compared to traditional pre-calibration methods, the workload is reduced by over 80%, and system deployment time is shortened from days to hours. Furthermore, the entire system does not need to be recalibrated after camera replacement or adjustment, significantly reducing maintenance costs.

[0170] (4) Achieve foreground and background layered optimization; Intelligent layering based on 3D modeling results allows for separate processing of the foreground and background, with targeted optimization of stitching parameters. This solves the problem that a uniform processing method cannot adequately address different depth areas. The stitching quality is optimized in both the foreground and background areas, resulting in a significant improvement in overall visual quality.

[0171] (5) Excellent fusion quality; A progressive weighted blending algorithm is employed to achieve a smooth transition in overlapping areas. Combined with a top-to-bottom blending method guided by segmentation lines, it completely eliminates stitching gaps and ghosting effects. Compared to simple linear blending or hard switching methods, the visual effect is more natural and continuous.

[0172] (6) Meets real-time processing requirements; By predicting overlapping areas using 3D modeling to narrow down the feature matching range, and combining GPU acceleration and multiple optimization measures, image stitching for a single pair takes only 50 milliseconds. Even in large-scale scenarios with 103 cameras, real-time video stitching can still be achieved with a processing latency of approximately 200 milliseconds, meeting the needs of surveillance applications.

[0173] (7) Highly robust; Multi-marker fusion, RANSAC outlier filtering, and timing smoothing are among the measures that ensure the system can still work stably in complex environments such as coal dust interference, light changes, and partial shading. The splicing success rate is over 98%, which is much higher than the 85% to 90% success rate of traditional methods.

[0174] (8) It is highly practical; Specifically optimized for the characteristics of the underground coal mine environment, the system features an intrinsically safe explosion-proof design, convenient installation and maintenance, and stable and reliable operation. In practical applications, it has enabled panoramic monitoring of the fully mechanized mining face, providing a reliable visual perception foundation for intelligent coal mining, and possesses significant application value and promising prospects for widespread adoption.

[0175] In summary, this application solves the problem of high-precision real-time video stitching in complex 3D scenarios such as coal mine fully mechanized mining faces. It is technologically advanced, highly practical, and has significant application value.

[0176] Figure 14 This is a schematic diagram of the physical structure of the computer device provided in the fourteenth embodiment of the present invention, as shown below. Figure 14 As shown, the computer device may include a processor 1401, a communications interface 1402, a memory 1403, and a communication bus 1404, wherein the processor 1401, the communications interface 1402, and the memory 1403 communicate with each other via the communication bus 1404. The processor 1401 can call logical instructions in the memory 1403 to execute the methods provided in the above-described method embodiments.

[0177] Furthermore, the logical instructions in the aforementioned memory 1403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments.

[0179] This embodiment provides a computer-readable storage medium storing a computer program that causes the computer to perform the methods provided in the above-described method embodiments.

[0180] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0184] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0185] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for video stitching of a fully mechanized coal mining face, characterized in that, include: Images are captured from each fixed-focus camera in the video stream of the fully mechanized coal mining face; wherein the video stream of the fully mechanized coal mining face is based on the capture of each fixed-focus camera; the captured images contain at least one marker code, which is pre-configured on the coal mining machine and the side guard plate; Based on the segmentation location, the acquired image corresponding to each fixed-focus camera is segmented to obtain the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera; wherein, the foreground image and the background image have overlapping areas, and the segmentation location is obtained in advance; Based on the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras, feature matching is performed on the foreground image and background image corresponding to each pair of adjacent fixed-focus cameras to obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range corresponding to each pair of adjacent fixed-focus cameras are obtained in advance. Based on the arrangement order of the fixed-focus cameras, the foreground images corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera, the images are sequentially stitched and horizontally fused to obtain the foreground image of the fully mechanized coal mining face; and based on the arrangement order of the fixed-focus cameras, the background images corresponding to each adjacent fixed-focus camera, and the second offset corresponding to each adjacent fixed-focus camera, the images are sequentially stitched and horizontally fused to obtain the background image of the fully mechanized coal mining face. Based on the segmentation line information, the foreground image and the background image of the fully mechanized coal mining face, a vertical image fusion is performed to obtain a preliminary panoramic image of the fully mechanized coal mining face; The preliminary panoramic image of the fully mechanized coal mining face is post-processed to obtain the final panoramic image of the fully mechanized coal mining face.

2. The method according to claim 1, characterized in that, Also includes: The first frame image captured by each fixed-focus camera is obtained from the video stream of the fully mechanized coal mining face; Marker detection is performed based on the first frame image corresponding to each fixed-focus camera to obtain the marker points corresponding to each fixed-focus camera and the relationship point set of each fixed-focus camera. Each point in the relationship point set of each fixed-focus camera has three-dimensional coordinates and corresponding pixel coordinates. The pose of each fixed-focus camera is solved based on the set of relation points of each fixed-focus camera, and the position and attitude angle of each fixed-focus camera, as well as the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera are obtained. Based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, the intrinsic parameters and distortion coefficients of each fixed-focus camera, the overlapping area of ​​adjacent fixed-focus cameras is calculated to obtain the matching position and corresponding matching range for each pair of adjacent fixed-focus cameras.

3. The method according to claim 2, characterized in that, The step of calculating the overlapping area of ​​adjacent fixed-focus cameras based on the surface sampling point set of the coal mining machine, the position and attitude angle of each fixed-focus camera, and obtaining the matching position and corresponding matching range for each pair of adjacent fixed-focus cameras includes: Based on the surface sampling point set of the coal mining machine, the rotation matrix and translation vector from the three-dimensional coordinate system to the camera coordinate system of each fixed-focus camera, and the intrinsic parameters and distortion coefficients of each fixed-focus camera, the pixel coordinates of the surface sampling point set projected onto the image plane of each fixed-focus camera are obtained. Based on the pixel coordinates corresponding to the surface sampling point set and the effective image range of each fixed-focus camera, the visible area range corresponding to each fixed-focus camera is determined. Based on the surface sampling points corresponding to the visible area range of each pair of adjacent fixed-focus cameras, the spatial overlap area of ​​each pair of adjacent fixed-focus cameras is obtained. Based on the spatial overlap area of ​​each pair of adjacent fixed-focus cameras and the segmentation rules, the spatial segmentation position of each pair of adjacent fixed-focus cameras is obtained; wherein, the segmentation rules are preset. Transform the spatial segmentation positions of each pair of adjacent fixed-focus cameras to the corresponding pixel coordinate system to obtain the matching positions and matching ranges for each pair of adjacent fixed-focus cameras.

4. The method according to claim 1, characterized in that, Obtaining the segmentation position includes: Based on the design drawings of the coal mining machine, a three-dimensional surface model of the coal mining machine is established; Draw the three-dimensional coordinates of the segmented location in the world coordinate system of the three-dimensional surface model of the coal mining machine; The three-dimensional coordinates of the segmentation position are transformed into the pixel coordinate system corresponding to each fixed-focus camera to obtain the segmentation position.

5. The method according to claim 1, characterized in that, Image segmentation is performed on the acquired images corresponding to each fixed-focus camera based on the segmentation location, obtaining the foreground image, background image, and segmentation line information corresponding to each fixed-focus camera, including: Draw a segmentation curve based on the segmentation location in the image captured by the fixed-focus camera; Based on the segmentation curve, the foreground segmentation line and the background segmentation line are obtained by extending the overlap height vertically and horizontally in the image captured by the fixed-focus camera; wherein, the overlap height is preset. The image below the foreground segmentation line is obtained from the image captured by the fixed-focus camera and used as the foreground image of the fixed-focus camera. The image above the background segmentation line is obtained from the image captured by the fixed-focus camera and used as the background image of the fixed-focus camera. The foreground segmentation line and the background segmentation line are also obtained as segmentation line information.

6. The method according to claim 1, characterized in that, The step of performing feature matching on the foreground and background images corresponding to each pair of adjacent fixed-focus cameras based on the matching position and matching range of each pair of adjacent fixed-focus cameras to obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras includes: Based on the matching position and matching range of adjacent fixed-focus cameras, determine the pixel range of the overlapping area between the foreground and background images of adjacent fixed-focus cameras; Based on the pixel range of the overlapping area of ​​the foreground images of adjacent fixed-focus cameras, the foreground and background images of the two fixed-focus cameras in the adjacent fixed-focus cameras are cropped respectively to obtain the overlapping area of ​​the foreground image and the overlapping area of ​​the background image of each of the two fixed-focus cameras in the adjacent fixed-focus cameras. Feature points are extracted from the overlapping regions of the foreground and background images of two adjacent fixed-focus cameras to obtain the feature points of the overlapping regions of the foreground and background images of two adjacent fixed-focus cameras. Feature matching is performed on the feature points of the overlapping area of ​​the foreground images of two adjacent fixed-focus cameras to obtain the matching feature points of the foreground images of the adjacent fixed-focus cameras; and feature matching is performed on the feature points of the overlapping area of ​​the background images of two adjacent fixed-focus cameras to obtain the matching feature points of the background images of the adjacent fixed-focus cameras. The first offset of the adjacent fixed-focus camera is obtained by matching feature points in the foreground image corresponding to the adjacent fixed-focus camera; and the second offset of the adjacent fixed-focus camera is obtained by matching feature points in the background image corresponding to the adjacent fixed-focus camera.

7. The method according to claim 1, characterized in that, The process of stitching and horizontal image fusion sequentially based on the arrangement order of the fixed-focus cameras, the foreground images corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera to obtain the foreground image of the fully mechanized coal mining face includes: The foreground images corresponding to adjacent fixed-focus cameras are stitched together in the order in which they are arranged. Based on the first offset corresponding to each pair of adjacent fixed-focus cameras, horizontal image fusion is performed on the foreground images corresponding to each pair of adjacent fixed-focus cameras.

8. The method according to claim 7, characterized in that, The horizontal image fusion of the foreground images corresponding to each pair of adjacent fixed-focus cameras based on the first offset corresponding to each pair of adjacent fixed-focus cameras includes: The size of the first stitching canvas is determined based on the foreground images corresponding to adjacent fixed-focus cameras and the first offset. Based on the first splicing canvas size, create a first splicing canvas and place the foreground image of the preceding fixed-focus camera in the adjacent fixed-focus camera; Based on the foreground image of the preceding fixed-focus camera in the adjacent fixed-focus cameras and the first offset, a first overlapping region image is obtained, and based on the foreground image of the following fixed-focus camera in the adjacent fixed-focus cameras and the first offset, a second overlapping region image is obtained. The first overlapping region image and the second overlapping region image are weighted and fused according to the first horizontal gradient weight matrix to obtain the overlapping region image of the foreground images corresponding to adjacent fixed-focus cameras. The overlapping region image of the foreground image corresponding to the adjacent fixed-focus camera and the remaining image of the foreground image of the subsequent fixed-focus camera in the adjacent fixed-focus camera are placed into the stitching canvas; wherein, the remaining image of the foreground image of the subsequent fixed-focus camera in the adjacent fixed-focus camera is the image remaining after removing the second overlapping region image from the foreground image of the subsequent fixed-focus camera in the adjacent fixed-focus camera.

9. The method according to claim 1, characterized in that, The process of stitching and horizontal image fusion sequentially according to the arrangement order of the fixed-focus cameras, based on the background images corresponding to each adjacent fixed-focus camera and the second offset corresponding to each adjacent fixed-focus camera, to obtain the background image of the fully mechanized coal mining face includes: The background images corresponding to adjacent fixed-focus cameras are stitched together sequentially according to the arrangement order of the fixed-focus cameras. Based on the second offset corresponding to each pair of adjacent fixed-focus cameras, horizontal image fusion is performed on the overlapping area of ​​the background images corresponding to each pair of adjacent fixed-focus cameras.

10. The method according to claim 9, characterized in that, The horizontal image fusion of the overlapping region of the foreground images corresponding to each pair of adjacent fixed-focus cameras based on the second offset corresponding to each pair of adjacent fixed-focus cameras includes: The size of the second stitching canvas is determined based on the background image and the second offset corresponding to the adjacent fixed-focus cameras. Based on the second splicing canvas size, create a second splicing canvas and place the background image of the preceding fixed-focus camera in the adjacent fixed-focus camera; Based on the background image of the preceding fixed-focus camera in the adjacent fixed-focus cameras and the second offset, a third overlapping region image is obtained, and based on the background image of the following fixed-focus camera in the adjacent fixed-focus cameras and the second offset, a fourth overlapping region image is obtained. The third and fourth overlapping region images are weighted and fused according to the second level gradient weight matrix to obtain the overlapping region images of the background images corresponding to adjacent fixed-focus cameras. The overlapping area image of the background image corresponding to the adjacent fixed-focus camera and the remaining image of the background image of the subsequent fixed-focus camera in the adjacent fixed-focus camera are placed into the second splicing canvas; wherein, the remaining image of the background image of the subsequent fixed-focus camera in the adjacent fixed-focus camera is the image remaining after removing the fourth overlapping area image from the background image of the subsequent fixed-focus camera in the adjacent fixed-focus camera.

11. The method according to claim 1, characterized in that, The step of performing vertical image fusion based on the segmentation line information, the foreground image, and the background image of the fully mechanized coal mining face to obtain a preliminary panoramic image of the fully mechanized coal mining face includes: Based on the foreground and background images of the fully mechanized coal mining face and the segmentation line information, the overlapping areas of the foreground image and the background image are obtained. The overlapping regions of the foreground image and the background image are weighted and fused according to the vertical gradient weight matrix to obtain the overlapping region image of the fully mechanized coal mining face. By stitching together the foreground image of the fully mechanized coal mining face, the overlapping area image of the fully mechanized coal mining face, and the remaining image of the background image of the fully mechanized coal mining face, a preliminary panoramic image of the fully mechanized coal mining face is obtained; wherein, the remaining image of the fully mechanized coal mining face is the image remaining after removing the overlapping area of ​​the background image.

12. A video splicing device for a fully mechanized coal mining face, characterized in that, include: The acquisition module is used to acquire images captured by each fixed-focus camera from the video stream of the fully mechanized coal mining face; wherein the video stream of the fully mechanized coal mining face is based on the acquisition by each fixed-focus camera; the acquired images contain at least one marker code, which is pre-configured on the coal mining machine and the side guard plate; The image segmentation module is used to segment the acquired images corresponding to each fixed-focus camera based on the segmentation position, and obtain the foreground image, background image and segmentation line information corresponding to each fixed-focus camera; wherein, the foreground image and the background image have overlapping areas, and the segmentation position is preset; The feature matching module is used to perform feature matching on the foreground and background images corresponding to each pair of adjacent fixed-focus cameras according to the matching position and matching range of each pair of adjacent fixed-focus cameras, and obtain the first offset and the second offset corresponding to each pair of adjacent fixed-focus cameras; wherein, the matching position and matching range of each pair of adjacent fixed-focus cameras are obtained in advance. The horizontal stitching module is used to sequentially stitch and fuse horizontal images according to the arrangement order of each fixed-focus camera, the foreground image corresponding to each adjacent fixed-focus camera, and the first offset corresponding to each adjacent fixed-focus camera to obtain the foreground image of the fully mechanized coal mining face; and to sequentially stitch and fuse horizontal images according to the arrangement order of each fixed-focus camera, the background image corresponding to each adjacent fixed-focus camera, and the second offset corresponding to each adjacent fixed-focus camera to obtain the background image of the fully mechanized coal mining face. The vertical stitching module is used to perform vertical image fusion based on the segmentation line information, the foreground image and the background image of the fully mechanized coal mining face, to obtain a preliminary panoramic image of the fully mechanized coal mining face. The post-processing module is used to perform post-processing on the preliminary panoramic image of the fully mechanized coal mining face to obtain the final panoramic image of the fully mechanized coal mining face.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program / instructions that, when executed by a processor, implement the method described in any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 11.

Images