Open-pit mine panoramic video splicing method and device and medium

Abstract

The present disclosure provides an open-pit mine panoramic video splicing method, device and medium, relating to the technical field of image processing, and is used to solve the problem of how to splice to obtain an open-pit mine panoramic video image. The method comprises: acquiring an original image of an open-pit mine monitoring video, and preprocessing the original image into a to-be-spliced image; acquiring multi-level multi-scale splicing features of the to-be-spliced image by using a convolutional neural network with multi-layer convolution; and splicing the to-be-spliced image according to the multi-level multi-scale splicing features to acquire a panoramic image of the open-pit mine monitoring video. The present disclosure is targeted at the particularity and complexity of the open-pit mine scene, and uses a convolutional neural network with multi-layer convolution to acquire multi-level multi-scale splicing features of the to-be-spliced image. The obtained multi-level multi-scale splicing features can meet the requirements of open-pit mine panoramic video splicing, so as to obtain an open-pit mine panoramic video image that meets the requirements and help effectively monitor the open-pit mine scene.

Description

Technical Field

[0001] This disclosure relates at least to the field of image processing technology, and in particular to a method, apparatus and medium for panoramic video stitching in open-pit mines. Background Technology

[0002] The unique and complex nature of open-pit mining environments, such as barren terrain and indistinct textures, presents significant challenges to image stitching technology. Traditional image stitching methods often fail to achieve satisfactory results in such environments, resulting in low stitching accuracy. Summary of the Invention

[0003] The technical problem to be solved by this disclosure is to provide a method, apparatus and medium for stitching panoramic videos of open-pit mines, in order to address the above-mentioned shortcomings and solve the problem of how to stitch together panoramic video images of open-pit mines.

[0004] In a first aspect, this disclosure provides a method for stitching panoramic videos of open-pit mines, the method comprising:

[0005] Acquire the original images from the open-pit mine monitoring video and preprocess the original images into images to be stitched together;

[0006] A convolutional neural network with multiple convolutions is used to obtain multi-level and multi-scale stitching features of the images to be stitched together;

[0007] Based on multi-level and multi-scale stitching features, images are stitched together to obtain panoramic images of open-pit mine monitoring videos.

[0008] Furthermore, the original images from the open-pit mine monitoring video are acquired, and the original images are preprocessed into images to be stitched together, specifically including:

[0009] Continuously acquire surveillance videos captured by different monitoring cameras in the open-pit mine, and extract original images from the surveillance videos that are sequentially adjacent to each other and have the same monitoring time.

[0010] The original image is preprocessed into an image to be stitched by using an array point calibration plate method to correct image distortion and histogram equalization to enhance features.

[0011] Furthermore, a convolutional neural network with multiple convolutions is employed to obtain multi-level, multi-scale stitching features of the images to be stitched, specifically including:

[0012] Two adjacent images to be stitched are input into a convolutional neural network. The convolutional neural network consists of multiple convolutional layers, each with a convolutional kernel of a different scale, and each convolutional layer is followed by a pooling layer.

[0013] Each convolutional layer extracts the features of the two images to be stitched together based on its own convolutional kernel, and outputs two sets of features to the pooling layer.

[0014] Each pooling layer compares the two received features of this layer and selects the key points in the two features of this layer that are identified as the same feature region by the max pooling strategy.

[0015] A convolutional neural network fuses multi-layer feature representations of key points, outputs the pairing results of key points of two images to be stitched based on the multi-layer feature representations, and uses the pairing results as multi-level, multi-scale stitching features.

[0016] Furthermore, each convolutional layer extracts the features of the two images to be stitched together based on its own convolutional kernel, specifically including:

[0017] The first convolutional layer has a convolutional kernel of the first scale. The first feature value is calculated based on the first scale convolutional kernel and the first scale region at different positions of the two images to be stitched. Low-level features are extracted based on the first feature value. The low-level features include image edge features and image texture features.

[0018] The second convolutional layer has a second-scale convolutional kernel. The second feature value is calculated based on the second-scale convolutional kernel and the second-scale regions at different positions in the two images to be stitched. The intermediate features are extracted based on the second feature value. The intermediate features include image shape features and image color features.

[0019] The third convolutional layer has a third-scale convolutional kernel. The third feature value is calculated based on the third-scale convolutional kernel and the third-scale regions at different positions in the two images to be stitched. High-level features are extracted based on the third feature value. The high-level features include image object features and image scene features.

[0020] Furthermore, each pooling layer compares the two received feature samples and selects key points from the two feature samples that are identified as having the same feature region using a max pooling strategy. Specifically, this includes:

[0021] The first pooling layer compares the two received low-level features with a pooling window of the fourth scale to identify the same low-level feature regions where the similarity of the two first feature values ​​is greater than the first threshold. The first key point with the largest first feature value in the two same low-level feature regions is selected by the max pooling strategy.

[0022] The second pooling layer compares the two received mid-layer features with a pooling window of the fifth scale to identify the same mid-layer feature regions where the similarity of the second feature value of the two mid-layer features is greater than the second threshold. The second key point with the largest second feature value in the two same mid-layer feature regions is selected by the max pooling strategy.

[0023] The third pooling layer compares the two received high-level features with a pooling window of the sixth scale to identify the same high-level feature regions where the similarity of the third feature value of the two high-level feature regions is greater than the third threshold. The third key point with the largest third feature value in the two same high-level feature regions is selected by the max pooling strategy.

[0024] Furthermore, the convolutional neural network fuses multi-layer feature representations of key points, and outputs the pairing results of key points in the two images to be stitched based on these multi-layer feature representations. Specifically, this includes:

[0025] The fully connected layers of the convolutional neural network fuse two sets of first feature values, second feature values, and third feature values ​​for each first keypoint, second keypoint, and third keypoint respectively to obtain two comprehensive feature representations for each keypoint;

[0026] The FLANN fast nearest neighbor search library is used to match two composite feature representations of each keypoint to obtain successfully matched keypoints.

[0027] The key points that are successfully matched are paired and labeled in the two images to be stitched together, and then output through the output layer of the convolutional neural network.

[0028] Furthermore, the fast nearest neighbor search library FLANN is used to match the two combined feature representations of each keypoint to obtain successfully matched keypoints, specifically including:

[0029] Mark each key point in the two images to be stitched together, and obtain the two feature positions of each key point in the two images to be stitched together;

[0030] The first algorithm in FLANN is used to compute two composite feature representations for each keypoint to obtain the first part of the keypoints whose approximation of the two composite feature representations is greater than the fourth threshold.

[0031] The second algorithm in FLANN is used to calculate two sets of feature locations for each keypoint, so that the two sets of feature locations are located at the second part of the keypoints on the side edges of the selected direction of the two images to be stitched together.

[0032] Combine the key points from Part 1 and Part 2 to obtain the key points for a successful match.

[0033] Furthermore, the stitching of the image is based on multi-level, multi-scale stitching features, specifically including:

[0034] Move the two images to be stitched in the selected direction until the key points of the two images are successfully matched and overlap at their respective selected side edges.

[0035] Secondly, this disclosure provides a panoramic video stitching device for open-pit mines, the device comprising:

[0036] The preprocessing module is used to acquire the original images from the open-pit mine monitoring video and preprocess the original images into images to be stitched together.

[0037] The convolution operation module, connected to the preprocessing module, is used to obtain multi-level and multi-scale stitching features of the image to be stitched by employing a convolutional neural network with multiple convolutions.

[0038] The panoramic stitching module, connected to the convolution operation module, is used to stitch together images based on multi-level and multi-scale stitching features to obtain panoramic images of open-pit mine monitoring videos.

[0039] Thirdly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for stitching panoramic videos of open-pit mines.

[0040] This disclosure provides a method, apparatus, and medium for panoramic video stitching in open-pit mines. Addressing the unique and complex nature of open-pit mine scenes, it employs a convolutional neural network with multiple convolutional layers to acquire multi-level, multi-scale stitching features of the image to be stitched. These multi-level, multi-scale stitching features meet the requirements for panoramic video stitching in open-pit mines, thereby obtaining a satisfactory panoramic video image and facilitating effective monitoring of open-pit mine scenes. Attached Figure Description

[0041] Figure 1 This is a flowchart of a panoramic video stitching method for an open-pit mine according to an embodiment of this disclosure;

[0042] Figure 2 This is a schematic diagram of the structure of an open-pit mine panoramic video stitching device according to an embodiment of the present disclosure;

[0043] Figure 3 This is a flowchart of another open-pit mine panoramic video stitching method according to an embodiment of this disclosure;

[0044] Figure 4 This is a flowchart of an image distortion correction method according to an embodiment of the present disclosure;

[0045] Figure 5 This is a schematic diagram of a panoramic video image convolution operation processing procedure in an open-pit mine according to an embodiment of this disclosure;

[0046] Figure 6 This is an image of an open-pit mine panoramic video image before convolution processing, according to an embodiment of this disclosure.

[0047] Figure 7 This is an image processed by convolution operation of a panoramic video image of an open-pit mine according to an embodiment of this disclosure;

[0048] Figure 8This is a schematic diagram of a convolutional layer calculation according to an embodiment of this disclosure;

[0049] Figure 9 This is a schematic diagram of a pooling layer calculation according to an embodiment of the present disclosure. Detailed Implementation

[0050] To enable those skilled in the art to better understand the technical solutions of this disclosure, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0051] It is understood that the specific embodiments and accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0052] It is understood that, without conflict, the various embodiments and features in the embodiments of this disclosure can be combined with each other.

[0053] It is understood that, for ease of description, only the parts relevant to this disclosure are shown in the accompanying drawings, while parts unrelated to this disclosure are not shown in the drawings.

[0054] It is understood that each module or unit involved in the embodiments of this disclosure may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple modules or units may be integrated into one entity structure.

[0055] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this disclosure may occur in a different order than that marked in the accompanying drawings.

[0056] It is understood that the flowcharts and block diagrams of this disclosure illustrate the architecture, functions, and operations of possible implementations of systems, apparatuses, devices, and methods according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a module, unit, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagrams and flowcharts may be implemented using hardware-based devices to implement the specified function, or using a combination of hardware and computer instructions.

[0057] It is understood that the modules and units involved in the embodiments of this disclosure can be implemented by software or by hardware, for example, the modules and units can be located in a processor.

[0058] Example 1:

[0059] like Figure 1 As shown, this disclosure provides a method for stitching panoramic videos of open-pit mines, the method comprising:

[0060] S1. Obtain the original images from the open-pit mine monitoring video and preprocess the original images into images to be stitched together;

[0061] S2. Use a convolutional neural network with multiple convolutions to obtain multi-level and multi-scale splicing features of the image to be spliced.

[0062] S3. Based on the multi-level and multi-scale splicing features, splice the images to obtain a panoramic image of the open-pit mine monitoring video.

[0063] In this embodiment, the method, considering the unique and complex nature of open-pit mine scenes, employs a convolutional neural network with multiple convolutional layers to acquire multi-level, multi-scale stitching features of the image to be stitched. These acquired multi-level, multi-scale stitching features meet the requirements for panoramic video stitching in open-pit mines, thereby obtaining a satisfactory panoramic video image of the open-pit mine and facilitating effective monitoring of open-pit mine scenes. Figure 1 The method shown is applicable to, for example, Figure 2 The apparatus shown.

[0064] Specifically, currently, various enterprises have developed video surveillance systems for safety monitoring and prevention in open-pit mines. Cameras are installed at key production sites within the mines, and the images and video signals captured by these cameras are transmitted to a monitoring center via transmission equipment. Management personnel use these videos to monitor key locations, but this fails to achieve comprehensive monitoring of the entire mining area. However, open-pit mines employ an open mining method, and their production processes typically cover vast geographical areas. Furthermore, production in open-pit mines is usually continuous, involving uninterrupted blasting, excavation, and transportation operations. Based on existing video surveillance systems, it is difficult for remote monitoring personnel to effectively monitor and track this continuous production activity in open-pit mines, increasing the difficulty of remotely monitoring operational status and remotely controlling equipment. In addition, open-pit mines are located on outdoor slopes in harsh environments, affected by various environmental noises such as sunlight, wind, rain, and fog. These environmental noises can lead to limited field of view, blurred images, and unclear features in the images captured by cameras, affecting feature extraction and directly impacting the final video stitching results. Furthermore, open-pit mines are characterized by poor feature diversity, uniformity, and indistinct textures. Enhancing the features of open-pit mine sites to make images clearer and more textured would undoubtedly improve video stitching results. Due to the unique and complex nature of open-pit mine scenes, there is currently no suitable panoramic video stitching technology or solution for open-pit mines. Existing panoramic stitching algorithms use the SURF (Speeded-Up Robust Features) algorithm to extract feature points and generate feature descriptions. However, in practical applications, this algorithm cannot achieve real-time processing of video frames, essentially ruling it out for engineering use. Additionally, the K-nearest neighbor feature matching classification algorithm used has a relatively large matching error. Therefore, how to achieve panoramic video stitching in open-pit mines is a pressing technical problem that needs to be solved in this field.

[0065] In view of this, this embodiment provides as follows: Figure 3 The technical solution shown is for realizing open-pit mine video stitching. This solution provides a method for panoramic video stitching of open-pit mines, which includes: acquiring the original open-pit mine image, image distortion correction, image feature enhancement, acquiring multi-level and multi-scale image features, key point selection, key point multi-scale image feature fusion, feature matching, image fusion, and panoramic video stitching. The key point is that a convolutional neural network (CNN) is used to extract multi-scale and multi-level features of the open-pit mine image to improve the accuracy of image stitching.

[0066] In one embodiment, S1, acquiring the original image from the open-pit mine monitoring video, and preprocessing the original image into an image to be stitched, specifically includes:

[0067] Continuously acquire surveillance videos captured by different monitoring cameras in the open-pit mine, and extract original images from the surveillance videos that are sequentially adjacent to each other and have the same monitoring time.

[0068] The original image is preprocessed into an image to be stitched by using an array point calibration plate method to correct image distortion and histogram equalization to enhance features.

[0069] In this embodiment, step S1 corresponds to... Figure 3 The diagram shows the acquisition of original open-pit mine images, image distortion correction based on array point calibration plate method, and image feature enhancement based on histogram equalization.

[0070] The process of acquiring the original open-pit mine image is used to obtain the image sequence to be stitched corresponding to the open-pit mine video. The specific steps are as follows: First, acquire open-pit mine videos with the same number of frames from monitoring videos from different cameras. Then, perform frame extraction processing on the open-pit mine videos to obtain the image sequence to be stitched corresponding to the open-pit mine videos. The obtained image sequence S is... i ={a i ,b i ,…,x i ,…t i}, where i represents the i-th image sequence to be stitched together, and a, b, x, t represent open-pit mine images obtained from monitoring by different cameras.

[0071] The image distortion correction is used to correct the image distortion of the sequence of images to be stitched corresponding to the acquired open-pit mine video, and obtain the distortion-free image sequence to be stitched. Distortion includes lens distortion and perspective distortion caused by the camera itself, which can cause the geometric shape of the image to shift, straight lines to become curved, and the proportion and shape of objects to be distorted. When multiple images are stitched together, the stitching area may be misaligned due to the inconsistent distortion of adjacent images, resulting in the images being geometrically misaligned and forming obvious gaps or overlapping parts.

[0072] The image distortion correction process is as follows: Figure 4 As shown, the specific steps are as follows: First, the camera used in the open-pit mine is calibrated to obtain the camera's intrinsic and extrinsic parameters and distortion coefficients. Then, image distortion is corrected based on the array point calibration board method (which is a conventional image distortion correction). The image is then processed in real time through hardware acceleration using the FPGA (Field Programmable Gate Array) platform.

[0073] The specific process for calculating the distortion coefficient is as follows:

[0074] The first step is to assume that P = (X,Y,Z) is a point in the scene and (X,Y,Z) is the world coordinate of point P. Transform P from the world coordinate system to the camera coordinate system through rigid body transformation (rotation and translation).

[0075] The second step is to transform the image point p = (x, y) on the camera's imaging plane from the camera coordinate system through perspective projection; as shown in formula (1), where

[0076]

[0077] t is the translation vector, representing the translation from the origin of the world coordinate system to the origin of the camera coordinate system.

[0078] The third step is to transform the image point p from the imaging coordinate system to the pixel coordinate system by scaling and translation, resulting in point p = (u, v).

[0079] x'=x / z(2)

[0080] y'=y / z(3)

[0081] x"=x'(1+k1r 2 +k2·r 4 )+2p1x'y'+p2(r 2 +2x'1 2 (4)

[0082] y"=y'(1+k1·r 2 +k2·r 4 )+2p1x'y'+p1(r 2 +2x' 2 (5)

[0083] r 2 =x' 2 +y' 2 (6)

[0084] u = f x ·x"+cx (7)

[0085] v = f y ·y"+cy (8)

[0086] Where k1 and k2 represent radial deformation parameters, and p1 and p2 represent tangential deformation parameters.

[0087] The image feature enhancement is used to enhance the image contrast of the distortion-free image sequence to be stitched together, thereby obtaining a clear image sequence to be stitched together; the image contrast is enhanced based on histogram equalization to improve image sharpness, while making the image present a more uniform distribution, more consistent color distribution, and more vivid colors, so as to obtain a clearer image.

[0088] The specific process is as follows:

[0089] The first step involves decomposing the pixel values ​​of the distortion-corrected image to be stitched into three channels and calculating the image's grayscale histogram. Each of the RGB (Red (R), Green (G), Blue (B)) channels is then converted to a grayscale image. The RGB image is converted to grayscale using a weighted average method, as follows (this is the common practice):

[0090] Y (grayscale value) = 0.299 × R (red) + 0.587 × G (green) + 0.114 × B (blue)

[0091] A grayscale histogram represents the frequency of each grayscale level in an image.

[0092] The second step is to calculate the value s of each channel's pixel value after mapping by the cumulative distribution function. k As shown in formula (9):

[0093]

[0094] Among them, s k This represents the value of the current gray level after being mapped by the cumulative distribution function, where k is the gray level and n is the sum of the pixels in the image. j This represents the number of pixels at the current gray level, and L represents the total number of gray levels in the image. Here, the total number of gray levels in each channel of the image to be stitched is 256.

[0095] The third step is to perform the second step operation on each of the three channels to obtain the values ​​of the three channels after being mapped by the cumulative distribution function;

[0096] The fourth step is to merge the three channels to obtain a clear image to be stitched after image feature enhancement.

[0097] In one embodiment, S2, a convolutional neural network with multiple convolutions is used to obtain multi-level, multi-scale stitching features of the images to be stitched, specifically including:

[0098] Two adjacent images to be stitched are input into a convolutional neural network. The convolutional neural network consists of multiple convolutional layers, each with a convolutional kernel of a different scale, and each convolutional layer is followed by a pooling layer.

[0099] Each convolutional layer extracts the features of the two images to be stitched together based on its own convolutional kernel, and outputs two sets of features to the pooling layer.

[0100] Each pooling layer compares the two received features of this layer and selects the key points in the two features of this layer that are identified as the same feature region by the max pooling strategy.

[0101] A convolutional neural network fuses multi-layer feature representations of key points, outputs the pairing results of key points of two images to be stitched based on the multi-layer feature representations, and uses the pairing results as multi-level, multi-scale stitching features.

[0102] In this embodiment, step S2 corresponds to... Figure 3 The diagram illustrates the acquisition of multi-level, multi-scale image features based on convolutional neural networks, keypoint selection based on max pooling, and keypoint multi-scale image feature fusion.

[0103] The process of acquiring multi-level, multi-scale image features is used to perform feature learning on the clear image sequence to be stitched together, thereby obtaining a multi-level, multi-scale feature representation of the clear image sequence to be stitched together.

[0104] The key point selection is used to obtain the multi-level feature representation of the key points of each image to be stitched in the clear image sequence from the multi-level feature representation of the clear image sequence to be stitched.

[0105] The key point multi-scale image feature fusion is used to fuse the feature representations of key points at different scales in the images to be stitched together, to obtain a comprehensive feature representation of key points in each image to be stitched together, and then to perform feature matching.

[0106] The feature matching is used to match the key points of each image to be stitched together to obtain key point feature matching pairs.

[0107] The actual calculation result of step S2 is as follows Figure 5-7 As shown, firstly, images taken at the same time that are spatially adjacent in content space are input into the convolutional neural network, such as... Figure 6 The two images shown are processed as follows: Figure 5 After the convolution operation shown, the final output of the convolutional neural network is as follows: Figure 7 The image shown illustrates the stitching keypoint pairing effect. (Example:) Figure 5 The convolution operation shown includes multi-level, multi-scale feature map acquisition based on multiple convolutional layers, key point selection based on max pooling and multi-size (scale) key point feature fusion, and finally key point pairing.

[0108] The core features include: Multi-scale feature extraction: Convolutional neural networks (CNNs) can simultaneously capture both local details and global structure of an image through convolutional kernels of different scales. Multiple convolutional layers are designed, each responsible for extracting features at different scales, thus obtaining feature representations from coarse to fine. Multi-level feature extraction: Features are extracted at different depths within the CNN. These features contain information from low-level edges and textures to high-level semantics, helping to more comprehensively describe the intrinsic structure of open-pit mine images.

[0109] In one implementation, each convolutional layer extracts the features of the two images to be stitched together based on its own convolutional kernel, specifically including:

[0110] The first convolutional layer has a convolutional kernel of the first scale. The first feature value is calculated based on the first scale convolutional kernel and the first scale region at different positions of the two images to be stitched. Low-level features are extracted based on the first feature value. The low-level features include image edge features and image texture features.

[0111] The second convolutional layer has a second-scale convolutional kernel. The second feature value is calculated based on the second-scale convolutional kernel and the second-scale regions at different positions in the two images to be stitched. The intermediate features are extracted based on the second feature value. The intermediate features include image shape features and image color features.

[0112] The third convolutional layer has a third-scale convolutional kernel. The third feature value is calculated based on the third-scale convolutional kernel and the third-scale regions at different positions in the two images to be stitched. High-level features are extracted based on the third feature value. The high-level features include image object features and image scene features.

[0113] In this embodiment, the scale refers to the size of the convolution kernel; convolution kernels of different sizes will extract features at different scales. The acquisition of multi-level, multi-scale image features is used to perform feature learning on the clear image sequence to be stitched together, obtaining a multi-level, multi-scale feature representation of the clear image sequence to be stitched together, such as... Figure 5-7 As shown, the process includes: simultaneously inputting two adjacent images from the image enhancement sequence to be stitched into multiple convolutional neural networks of different sizes to obtain image features, and fusing shallow and deep features to obtain new multi-level, multi-scale image features; here, the size refers to the size of the CNN convolution kernel. Assuming the original image size is H×W, the convolution kernel size can be H / 2×W / 2, H / 4×W / 4, H / 8×W / 8, etc.

[0114] Different levels contain features of different granularities, mainly including low, medium, and high-level features. Low-level features include basic image features such as edges and textures. Medium-level features represent local patterns in the image, such as larger-scale textures, the shape of regions, and color distribution. These features are more stable than low-level features and can represent local areas of the image, but still do not capture the overall structure. High-level features represent the global structure and semantic information of the image, involving object outlines, scene understanding, and more.

[0115] The specific process of obtaining multi-level, multi-scale image features using CNN is as follows:

[0116] Each CNN contains multiple convolutional kernels used to perform convolution operations on image feature maps. Convolutional kernels of the same size extract features at the same scale, and the values ​​in the kernels are learned through model training. The number of convolutional kernels in each CNN layer can be defined based on the specific requirements of each layer. The size of the convolutional kernel represents the feature at that scale; different sized kernels can extract features at different scales in the image. The size of the convolutional kernel directly affects the area it can perceive, that is, its receptive field. A larger receptive field results in more global features extracted by the convolutional layer; a smaller receptive field results in more local features extracted by the convolutional layer.

[0117] The operation process of convolutional layers is as follows Figure 8 As shown, this operation can be understood as performing element-wise multiplication of the convolution kernel with a small region in the feature map, followed by summation to obtain the feature value of this small region. Then, a window is slid to the next region for convolution calculation, finally obtaining the convolution result and achieving local feature extraction. The image information of the open-pit mine is first characterized into the form of a feature matrix, which is then input into the CNN layer for feature extraction, and keypoint extraction is performed based on these features.

[0118] In one embodiment, each pooling layer compares two received feature sets and selects key points from the two feature sets that are identified as having the same feature region using a max pooling strategy. Specifically, this includes:

[0119] The first pooling layer compares the two received low-level features with a pooling window of the fourth scale to identify the same low-level feature regions where the similarity of the two first feature values ​​is greater than the first threshold. The first key point with the largest first feature value in the two same low-level feature regions is selected by the max pooling strategy.

[0120] The second pooling layer compares the two received mid-layer features with a pooling window of the fifth scale to identify the same mid-layer feature regions where the similarity of the second feature value of the two mid-layer features is greater than the second threshold. The second key point with the largest second feature value in the two same mid-layer feature regions is selected by the max pooling strategy.

[0121] The third pooling layer compares the two received high-level features with a pooling window of the sixth scale to identify the same high-level feature regions where the similarity of the third feature value of the two high-level feature regions is greater than the third threshold. The third key point with the largest third feature value in the two same high-level feature regions is selected by the max pooling strategy.

[0122] In this embodiment, key point selection is used to obtain multi-level feature representations of key points for each image to be stitched from the multi-level feature representations of the clear image sequence to be stitched. For the multi-level feature representations of the clear image sequence to be stitched, key point features are obtained based on max pooling. Max pooling refers to selecting the largest feature value within a region as the representative value of that region, such as... Figure 9 In this example, if the window size is 2x2, the maximum value within the first 2x2 window is selected as the feature value for that region. The specific process of MAXPooling to acquire keypoint features in the image is as follows: Assuming the pooling window size is X, i.e. Figure 9 The side length of the red square is defined as the horizontal / vertical displacement between two adjacent pooling windows, and each pooling window is unique during the pooling process; therefore, X = stride. Keypoints here refer to points in an image with significant local features. These points have strong discriminative power and can be used for matching and alignment between different images. By detecting keypoints in an image, similar or identical regions can be found between different images. These keypoints typically represent local textures, corners, edges, and other information in the image, helping to identify correspondences between images.

[0123] It is understandable that this embodiment uses pooling layers to select key points for splicing, without affecting the original function of pooling layers in convolutional neural networks.

[0124] In one embodiment, the convolutional neural network fuses multi-layer feature representations of key points and outputs the pairing result of key points in two images to be stitched based on the multi-layer feature representations, specifically including:

[0125] The fully connected layers of the convolutional neural network fuse two sets of first feature values, second feature values, and third feature values ​​for each first keypoint, second keypoint, and third keypoint respectively to obtain two comprehensive feature representations for each keypoint;

[0126] The FLANN fast nearest neighbor search library is used to match two composite feature representations of each keypoint to obtain successfully matched keypoints.

[0127] The key points that are successfully matched are paired and labeled in the two images to be stitched together, and then output through the output layer of the convolutional neural network.

[0128] In this embodiment, the extracted multi-scale features are fused. The fusion process includes the following steps: Feature concatenation: Features at different scales are concatenated along the feature dimension to form a new feature matrix; Feature enhancement: The dimensionality-reduced features are enhanced using a non-linear activation function to improve their representational ability. Based on the original feature fusion of the convolutional neural network, the multi-scale image feature fusion of key points is implemented. This is used to fuse the feature representations of key points at different scales in the images to be stitched together, obtaining a comprehensive feature representation of the key points in each image to be stitched together. This includes concatenating shallow and deep features, fusing the shallow and deep feature representations as a multi-level image feature representation of the image sequence to be stitched together, and then fusing the multi-scale features of the key points to obtain new multi-level, multi-scale key point features.

[0129] In one implementation, the fast nearest neighbor search library FLANN is used to match the two combined feature representations of each keypoint to obtain successfully matched keypoints, specifically including:

[0130] Mark each key point in the two images to be stitched together, and obtain the two feature positions of each key point in the two images to be stitched together;

[0131] The first algorithm in FLANN is used to compute two composite feature representations for each keypoint to obtain the first part of the keypoints whose approximation of the two composite feature representations is greater than the fourth threshold.

[0132] The second algorithm in FLANN is used to calculate two sets of feature locations for each keypoint, so that the two sets of feature locations are located at the second part of the keypoints on the side edges of the selected direction of the two images to be stitched together.

[0133] Combine the key points from Part 1 and Part 2 to obtain the key points for a successful match.

[0134] In this embodiment, after the fusion of multi-scale features, the Fast Nearest Neighbors (FLANN) library is used to achieve efficient matching of keypoint feature points. The specific steps are as follows: Keypoint detection: Keypoints are detected on the fused feature map; these keypoints represent representative feature locations in the image. Feature description: Feature descriptors are generated for each keypoint; these descriptors uniquely identify the features of the keypoint. Fast matching: The efficient algorithm in the FLANN library is used to perform a fast nearest neighbor search on the feature descriptors to achieve matching between keypoints. In image stitching, FLANN can quickly and accurately find corresponding feature points between images, matching the keypoints of each image to be stitched to obtain keypoint feature matching pairs.

[0135] The training and learning process of convolutional neural networks (CNNs) has corresponding references in existing technologies. This embodiment can further train an existing CNN for the open-pit mine image stitching task. Specifically, it uses manually proofread and reviewed open-pit mine stitching images as training samples, which are input into the CNN designed in this embodiment. This CNN has a conventional input layer, convolutional layer, pooling layer, fully connected layer, and output layer. It can use convolutional layers that have already undergone basic training. To achieve image stitching, the CNN processes two adjacent images at a time. The convolutional layers extract features from each image separately. After each convolutional layer, at the corresponding level, the pooling layer obtains the same or similar regions in the two images. Due to different scales, it can obtain the matching of objects such as ground, roads, machinery, houses, people, and minerals. Then, feature fusion is performed in the fully connected layer. FLANN is used to match the key points of feature fusion, that is, according to the requirements of image stitching, the stitching edges of objects that appear in both images are found. The trained CNN can then be applied to open-pit mine video image stitching.

[0136] In one embodiment, S3 involves stitching the image to be stitched according to multi-level, multi-scale stitching features, specifically including:

[0137] Move the two images to be stitched in the selected direction until the key points of the two images are successfully matched and overlap at their respective selected side edges.

[0138] In this embodiment, the images are seamlessly stitched together using an image fusion algorithm based on the matched key points. For example... Figure 3 As shown, the image fusion obtains the panoramic image stitching result based on the key point feature matching pairs. The panoramic video stitching is used to generate the panoramic image stitching result in real time, resulting in the final real-time panoramic video of the open-pit mine. Figure 7 The two images shown are moved to the same area for overlap, and after appropriate cropping and other processing, a stitched image can be obtained. This method not only improves the accuracy of stitching but also ensures the visual quality of the stitched image, providing high-quality image data support for terrain analysis and resource exploration in open-pit mines.

[0139] Example 2:

[0140] like Figure 2 As shown, this disclosure provides a panoramic video stitching device for open-pit mines, the device comprising:

[0141] Preprocessing module 1 is used to acquire the original images of the open-pit mine monitoring video and preprocess the original images into images to be stitched together;

[0142] The convolution operation module 2, connected to the preprocessing module 1, is used to obtain multi-level and multi-scale splicing features of the image to be spliced ​​by using a convolutional neural network with multiple convolutions.

[0143] The panoramic stitching module 3, connected to the convolution operation module 2, is used to stitch together images based on multi-level and multi-scale stitching features to obtain panoramic images of open-pit mine monitoring videos.

[0144] In one embodiment, the preprocessing module 1 specifically includes:

[0145] The frame extraction unit is used to continuously acquire monitoring videos captured by different monitoring cameras in the open-pit mine, and extract original images from the monitoring videos that are adjacent in sequence and have the same monitoring time.

[0146] The correction and enhancement unit, connected to the frame extraction unit, is used to correct image distortion of the original image using the array point calibration plate method and to perform feature enhancement using histogram equalization, so as to preprocess the original image into an image to be stitched.

[0147] In one embodiment, the convolution operation module 2 specifically includes:

[0148] The input unit is used to input two adjacent images to be stitched into the convolutional neural network. The convolutional neural network includes multiple convolutional layers, each with a convolutional kernel of a different scale, and a pooling layer after each convolutional layer.

[0149] The convolutional unit, connected to the input unit, is used by each convolutional layer to extract the features of the two images to be stitched based on its own convolutional kernel, and output the two sets of features to the pooling layer.

[0150] The pooling unit, connected to the convolutional unit, is used to compare the two received features of the same layer in each pooling layer, and select the key points that are identified as the same feature region in the two features of the same layer through the max pooling strategy.

[0151] The fusion output unit, connected to the pooling unit, is used to fuse multi-layer feature representations of key points in the convolutional neural network. Based on the multi-layer feature representations, it outputs the pairing results of key points of the two images to be stitched, and uses the pairing results as multi-level, multi-scale stitching features.

[0152] In one embodiment, the convolutional unit specifically includes:

[0153] The first convolutional unit is used in the first convolutional layer to have a convolutional kernel of the first scale. It calculates the first feature value based on the first scale convolutional kernel and the first scale region at different positions of the two images to be stitched. It extracts low-level features based on the first feature value. The low-level features include image edge features and image texture features.

[0154] The second convolutional unit is used in the second convolutional layer to have a second-scale convolutional kernel. It calculates the second feature value based on the second-scale convolutional kernel and the second-scale regions at different positions in the two images to be stitched. It extracts intermediate features based on the second feature value. The intermediate features include image shape features and image color features.

[0155] The third convolutional unit is used in the third convolutional layer with a third-scale convolutional kernel. Based on the third-scale convolutional kernel, the third feature value is calculated with the third-scale regions at different positions in the two images to be stitched. Based on the third feature value, high-level features are extracted, including image object features and image scene features.

[0156] In one embodiment, the pooling unit specifically includes:

[0157] The first pooling unit is used by the first pooling layer to compare two received low-level features with a pooling window of the fourth scale, so as to identify the same low-level feature regions where the similarity of the two first feature values ​​is greater than the first threshold, and select the first key point with the largest first feature value in the two same low-level feature regions through the max pooling strategy.

[0158] The second pooling unit is used by the second pooling layer to compare the two received mid-layer features with a pooling window of the fifth scale, so as to identify the same mid-layer feature regions where the similarity of the two second feature values ​​is greater than the second threshold, and select the second key point with the largest second feature value in the two same mid-layer feature regions through the max pooling strategy.

[0159] The third pooling unit is used by the third pooling layer to compare the two received high-level features with a pooling window of the sixth scale, in order to identify the same high-level feature regions where the similarity of the third feature value of the two features is greater than the third threshold. The third key point with the largest third feature value in the two same high-level feature regions is selected by the max pooling strategy.

[0160] In one embodiment, the fusion output unit specifically includes:

[0161] The fusion unit is used in the fully connected layer of the convolutional neural network to fuse two sets of first feature values, second feature values, and third feature values ​​for each first key point, second key point, and third key point, respectively, to obtain two comprehensive feature representations for each key point;

[0162] The matching unit, connected to the fusion unit, is used to match the two comprehensive feature representations of each keypoint using the fast nearest neighbor search library FLANN to obtain successfully matched keypoints.

[0163] The output unit, connected to the matching unit, is used to pair and mark the key points of successful matching in the two images to be stitched together, and output them through the output layer of the convolutional neural network.

[0164] In one embodiment, the matching unit specifically includes:

[0165] The location unit is used to mark each key point in the two images to be stitched together, and to obtain two feature positions of each key point in the two images to be stitched together.

[0166] The first matching unit is used to compute two comprehensive feature representations of each keypoint using the first algorithm in FLANN, so as to obtain the first part of the keypoints whose approximation of the two comprehensive feature representations is greater than the fourth threshold.

[0167] The second matching unit, connected to the position unit, is used to calculate two feature positions for each key point using the second algorithm in FLANN, so as to obtain the second part of the key points located at the selected directional side edges of the two images to be stitched together;

[0168] The synthesis unit, connected to the first matching unit and the second matching unit, is used to synthesize the first part of key points and the second part of key points to obtain the key points of successful matching.

[0169] In one embodiment, the panoramic stitching module 3 specifically includes:

[0170] The moving unit, connected to the integrating unit, is used to move two images to be stitched according to a selected direction until the key points of the two images to be stitched successfully match overlap at their respective selected directional side edges.

[0171] Example 3:

[0172] Embodiment 3 of this disclosure provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it implements the open-pit mine panoramic video stitching method as described in Embodiment 1, or the open-pit mine panoramic video stitching device as described in Embodiment 2.

[0173] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program units, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0174] Additionally, this disclosure may also provide a computer device including a memory and a processor. The memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the open-pit mine panoramic video stitching method as described in Embodiment 1. This computer device may be the open-pit mine panoramic video stitching device as described in Embodiment 2.

[0175] The memory is connected to the processor. The memory can be flash memory, read-only memory or other types of memory. The processor can be a central processing unit or a microcontroller.

[0176] Embodiments 1-3 of this disclosure provide a method, apparatus, and medium for stitching panoramic videos of open-pit mines. Addressing the unique and complex nature of open-pit mine scenes, a convolutional neural network with multiple convolutional layers is employed to acquire multi-level, multi-scale stitching features of the images to be stitched. These multi-level, multi-scale stitching features meet the requirements for panoramic video stitching in open-pit mines, thereby obtaining satisfactory panoramic video images of open-pit mines and facilitating effective monitoring of open-pit mine scenes.

[0177] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. A method for stitching panoramic videos of open-pit mines, characterized in that, The method includes: Acquire the original images from the open-pit mine monitoring video and preprocess the original images into images to be stitched together; A convolutional neural network with multiple convolutions is used to obtain multi-level and multi-scale stitching features of the images to be stitched together; Based on multi-level and multi-scale stitching features, images are stitched together to obtain panoramic images of open-pit mine monitoring videos.

2. The method according to claim 1, characterized in that, Acquire the raw images from the open-pit mine monitoring video, and preprocess the raw images into images to be stitched together, specifically including: Continuously acquire surveillance videos captured by different monitoring cameras in the open-pit mine, and extract original images from the surveillance videos that are sequentially adjacent to each other and have the same monitoring time. The original image is preprocessed into an image to be stitched by using an array point calibration plate method to correct image distortion and histogram equalization to enhance features.

3. The method according to claim 1 or 2, characterized in that, A convolutional neural network with multiple convolutions is used to obtain multi-level, multi-scale stitching features of the images to be stitched, specifically including: Two adjacent images to be stitched are input into a convolutional neural network. The convolutional neural network consists of multiple convolutional layers, each with a convolutional kernel of a different scale, and each convolutional layer is followed by a pooling layer. Each convolutional layer extracts the features of the two images to be stitched together based on its own convolutional kernel, and outputs two sets of features to the pooling layer. Each pooling layer compares the two received features of this layer and selects the key points in the two features of this layer that are identified as the same feature region by the max pooling strategy. A convolutional neural network fuses multi-layer feature representations of key points, outputs the pairing results of key points of two images to be stitched based on the multi-layer feature representations, and uses the pairing results as multi-level, multi-scale stitching features.

4. The method according to claim 3, characterized in that, Each convolutional layer extracts the features of the two images to be stitched together based on its own convolutional kernel, specifically including: The first convolutional layer has a convolutional kernel of the first scale. The first feature value is calculated based on the first scale convolutional kernel and the first scale region at different positions of the two images to be stitched. Low-level features are extracted based on the first feature value. The low-level features include image edge features and image texture features. The second convolutional layer has a second-scale convolutional kernel. The second feature value is calculated based on the second-scale convolutional kernel and the second-scale regions at different positions in the two images to be stitched. The intermediate features are extracted based on the second feature value. The intermediate features include image shape features and image color features. The third convolutional layer has a third-scale convolutional kernel. The third feature value is calculated based on the third-scale convolutional kernel and the third-scale regions at different positions in the two images to be stitched. High-level features are extracted based on the third feature value. The high-level features include image object features and image scene features.

5. The method according to claim 4, characterized in that, Each pooling layer compares the two received feature sets of its own layer and selects key points from the two feature sets that are identified as having the same feature region using a max pooling strategy. Specifically, this includes: The first pooling layer compares the two received low-level features with a pooling window of the fourth scale to identify the same low-level feature regions where the similarity of the two first feature values ​​is greater than the first threshold. The first key point with the largest first feature value in the two same low-level feature regions is selected by the max pooling strategy. The second pooling layer compares the two received mid-layer features with a pooling window of the fifth scale to identify the same mid-layer feature regions where the similarity of the two second feature values ​​is greater than the second threshold. The second key point with the largest second feature value in the two same mid-layer feature regions is selected by the max pooling strategy. The third pooling layer compares the two received high-level features with a pooling window of the sixth scale to identify the same high-level feature regions where the similarity of the third feature value of the two high-level feature regions is greater than the third threshold. The third key point with the largest third feature value in the two same high-level feature regions is selected by the max pooling strategy.

6. The method according to claim 5, characterized in that, A convolutional neural network fuses multi-layer feature representations of key points, and outputs the pairing results of key points of two images to be stitched based on these multi-layer feature representations. Specifically, this includes: The fully connected layers of the convolutional neural network fuse two sets of first feature values, second feature values, and third feature values ​​for each first keypoint, second keypoint, and third keypoint respectively to obtain two comprehensive feature representations for each keypoint; The FLANN fast nearest neighbor search library is used to match two composite feature representations of each keypoint to obtain successfully matched keypoints. The key points that are successfully matched are paired and labeled in the two images to be stitched together, and then output through the output layer of the convolutional neural network.

7. The method according to claim 6, characterized in that, The FLANN fast nearest neighbor search library is used to match the two combined feature representations of each keypoint to obtain successfully matched keypoints, specifically including: Mark each key point in the two images to be stitched together, and obtain the two feature positions of each key point in the two images to be stitched together; The first algorithm in FLANN is used to compute two composite feature representations for each keypoint to obtain the first part of the keypoints whose approximation of the two composite feature representations is greater than the fourth threshold. The second algorithm in FLANN is used to calculate two sets of feature locations for each keypoint, so that the two sets of feature locations are located at the second part of the keypoints on the side edges of the selected direction of the two images to be stitched together. Combine the key points from Part 1 and Part 2 to obtain the key points for a successful match.

8. The method according to claim 7, characterized in that, Image stitching is performed based on multi-level, multi-scale stitching features, specifically including: Move the two images to be stitched in the selected direction until the key points of the two images are successfully matched and overlap at their respective selected side edges.

9. A panoramic video stitching device for open-pit mines, characterized in that, The device includes: The preprocessing module is used to acquire the original images from the open-pit mine monitoring video and preprocess the original images into images to be stitched together. The convolution operation module, connected to the preprocessing module, is used to obtain multi-level and multi-scale stitching features of the image to be stitched by employing a convolutional neural network with multiple convolutions. The panoramic stitching module, connected to the convolution operation module, is used to stitch together images based on multi-level and multi-scale stitching features to obtain panoramic images of open-pit mine monitoring videos.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the open-pit mine panoramic video stitching method as described in any one of claims 1-8.

Images