Fully mechanized coal mining face panoramic video stitching method, device, equipment, medium and product

Abstract

The invention discloses a fully mechanized coal mining face panoramic video stitching method, device, equipment, medium and product, and relates to the field of panoramic video stitching, and the method comprises the steps: obtaining an image set according to videos of all regions of a target fully mechanized coal mining face coal wall in a preset time period; respectively inputting the images in the image set into a neural network model to determine whether the coal machine roller exists in the images and the position of the coal machine roller in the images; verifying whether the coal machine roller exists in each image and the authenticity of the position of the coal machine roller in each image, and determining the image with the highest authenticity as a target image; inputting each image in the adjacent image set into a segmentation model to obtain a segmented image; inputting the segmented image into the trained generative adversarial network to obtain a spliced image of the target fully mechanized coal mining face; the panoramic video is obtained according to the spliced image of the target fully mechanized coal mining face at each moment in the preset time period, and the method does not need to modify the site, and is more convenient.

Description

Technical Field

[0001] The present application relates to the field of panoramic video stitching, and in particular to a method, device, equipment, medium and product for stitching panoramic videos of a fully mechanized mining face. Background Art

[0002] At present, although some coal mines have realized visual monitoring of the fully mechanized mining face, they still use the traditional monitoring mode, which can only present the split-lens picture monitoring of the local perspective, and cannot monitor and grasp the overall real-time of the large scene; lack of seamless integration and centralized display of multi-mode and multi-quantity image information. The specific problems are mainly reflected in three aspects: (1) Incomplete: It is impossible to see the full view of the fully mechanized mining face at one time. When the "three machines" are started, stopped, moved, retracted and released, there are problems such as whether the hydraulic support stroke is in place, and it is difficult to find the coal pile, coal volume and speed of the scraper conveyor. (2) Inaccurate: The upper drum collides with the guard plate: The guard plate of the upper drum front bracket is not retracted in place, resulting in a collision accident of the coal cutter. The upper drum collides with the front exploration beam: The front exploration beam of the upper drum front bracket is not retracted in place, resulting in a collision accident of the coal cutter. The lower drum does not probe the bottom in place: The lower drum rocker arm angle is not enough, resulting in the fluctuation of the bottom plate, making it difficult to push the scraper conveyor. (3) No sense of space: When different cameras transmit data to the remote control center from different angles, the lack of continuity in the video makes it impossible to observe the coal machine status and coal wall conditions in all directions, resulting in the remote control driver's lack of sense of space. Incomplete and inaccurate underground videos and lack of sense of space are key issues that restrict the remote control operation of equipment. To solve the above problems, Shaanxi Coal Group Shenmu Hongliulin Mining Co., Ltd. has developed a panoramic video stitching technology for the fully mechanized mining face, which fully considers safety, overall situation, practicality and maintainability, and fully restores the fully mechanized mining face scene, realizing time and space consistency perception and non-contact integrated perception, changing the current situation of "blind mining" that relies only on sensors and scattered cameras. However, this method requires on-site modification, which is inconvenient. Summary of the invention

[0003] The purpose of this application is to provide a method, device, equipment, medium and product for panoramic video stitching of a comprehensive mining working face, which does not require on-site modification and is more convenient.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for stitching panoramic videos of a fully mechanized mining face, comprising:

[0006] According to the video of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained; the image set includes images of all areas of the coal wall of the target fully mechanized mining working face at any time within the preset time period;

[0007] The images in the image set are respectively input into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, and determine that all images in the image set that have coal machine drums constitute a target image set;

[0008] Verify the authenticity of the coal machine drum in each image in the target image set, and determine the image with the highest authenticity as the target image;

[0009] Input each image in the adjacent image set into the segmentation model to obtain a segmented image corresponding to each image in the adjacent image set; the adjacent image set includes: the target image, images corresponding to N adjacent areas in the image set located before the target area, and images corresponding to N adjacent areas in the image set located after the target area; the target area is the area of ​​the target fully mechanized mining working face coal wall corresponding to the target image;

[0010] Input the segmented images corresponding to each image in the adjacent image set into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment;

[0011] A panoramic video of the target fully mechanized mining working face within the preset time period is obtained based on the spliced ​​images of the target fully mechanized mining working face at each moment within the preset time period.

[0012] Optionally, the authenticity of the coal machine drum in each image in the target image set is verified, specifically including:

[0013] If the images in the image set are images of all areas of the coal wall of the target fully mechanized mining working face at the initial moment within the preset time period, for any image in the target image set, the authenticity of the coal machine drum in the image is verified based on the image of the corresponding area of ​​the image at the moment before the initial moment and the front neighbor image and the rear neighbor image in the image set; the front neighbor image is the image of the area located before and adjacent to the corresponding area of ​​the image; the rear neighbor image is the image of the area located after and adjacent to the corresponding area of ​​the image;

[0014] If the image in the image set is not an image of all areas of the coal wall of the target comprehensive mining working face at the initial moment within the preset time period, for any image in the target image set, the authenticity of the coal machine drum in the image is verified based on the front neighbor image and the rear neighbor image in the image set, as well as the video of the corresponding area of ​​the image within the preset time period.

[0015] Optionally, the segmented images corresponding to each image in the adjacent image set are input into a trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment, specifically including:

[0016] The segmented images corresponding to each image in the adjacent image set are sequentially subjected to image compression, cropping and perspective transformation operations to obtain processed segmented images corresponding to each image in the adjacent image set;

[0017] Performing nearest neighbor interpolation processing on the processed segmented images corresponding to each image in the adjacent image set to obtain an interpolated segmented image corresponding to each image in the adjacent image set;

[0018] The interpolated segmented images corresponding to each image in the adjacent image set are input into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment.

[0019] Optionally, obtaining a panoramic video of the target fully mechanized mining working face within a preset time period according to the spliced ​​images of the target fully mechanized mining working face at each moment within a preset time period specifically includes:

[0020] Preprocessing the spliced ​​image of the target fully mechanized mining working face at the moment to obtain a panoramic image of the target fully mechanized mining working face at the moment;

[0021] A panoramic video of the target fully mechanized mining face within a preset time period is obtained based on the panoramic images of the target fully mechanized mining face at each moment within a preset time period.

[0022] Optionally, based on the videos of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained, specifically including:

[0023] The PTZ camera takes videos of various areas of the coal wall of the target fully mechanized mining face within a preset time period;

[0024] Based on the videos of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained.

[0025] Optionally, before inputting the images in the image set into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, the method further includes:

[0026] Store the image collection in a Redis compute cluster.

[0027] In the second aspect, the present application provides a panoramic video stitching device for a fully mechanized mining face, comprising:

[0028] An image set acquisition module is used to obtain an image set based on videos of various areas of the coal wall of the target fully mechanized mining working face within a preset time period; the image set includes images of all areas of the coal wall of the target fully mechanized mining working face at any time within the preset time period;

[0029] A position determination module, used for inputting the images in the image set into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image;

[0030] A position verification module is used to verify the authenticity of the coal machine drum in each image in the target image set, determine the image with the highest authenticity as the target image, and determine that all images in the image set that contain the coal machine drum constitute the target image set;

[0031] A segmentation module is used to input each image in the adjacent image set into a segmentation model to obtain a segmented image corresponding to each image in the adjacent image set; the adjacent image set includes: the target image, images corresponding to N adjacent areas in the image set located before the target area, and images corresponding to N adjacent areas in the image set located after the target area; the target area is an area of ​​the target fully mechanized mining working face coal wall corresponding to the target image;

[0032] A splicing module, used for inputting the segmented images corresponding to each image in the adjacent image set into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment;

[0033] The panoramic video module is used to obtain a panoramic video of the target comprehensive mining working face within a preset time period based on the spliced ​​images of the target comprehensive mining working face at each moment within the preset time period.

[0034] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for panoramic video stitching of a comprehensive mining working face.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for stitching panoramic videos of a comprehensive mining face.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for stitching panoramic videos of a comprehensive mining working face.

[0037] According to the specific embodiments provided in this application, this application has the following technical effects:

[0038] The present application provides a method, device, equipment, medium and product for stitching panoramic videos of a fully mechanized mining face. According to the videos of each area of ​​the coal wall of the target fully mechanized mining face within a preset time period, an image set is obtained; the images in the image set are respectively input into a neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image; the authenticity of the coal machine drum in each image in the target image set is verified, and the image with the highest authenticity is determined as the target image; each image in an adjacent image set is input into a segmentation model to obtain a segmented image corresponding to each image in the adjacent image set; the adjacent image set includes: the target image, images corresponding to N adjacent areas before the target area in the image set, and images corresponding to N adjacent areas after the target area in the image set; the target area is the area of ​​the coal wall of the target fully mechanized mining face corresponding to the target image; the segmented images corresponding to each image in the adjacent image set are input into a trained generative adversarial network to obtain a stitched image of the target fully mechanized mining face at the moment; the panoramic video of the target fully mechanized mining face within the preset time period is obtained according to the stitched images of the target fully mechanized mining face at each moment within the preset time period, and the images in the monitoring video can be directly obtained without the need to modify the site, which is more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flow chart of a panoramic video stitching method for a fully mechanized mining face provided in an embodiment of the present application;

[0041] Figure 2 This is the workflow diagram for the Redis computing cluster;

[0042] Figure 3 This is the software algorithm framework diagram corresponding to the panoramic video stitching method of the fully mechanized mining working face;

[0043] Figure 4 A schematic diagram of a panoramic video stitching method for a fully mechanized mining face provided in one embodiment of the present application;

[0044] Figure 5 This is the drum recognition result diagram;

[0045] Figure 6 is the image segmentation result map;

[0046] Figure 7Images before and after High Dynamic Range Imaging (HDR) processing;

[0047] Figure 8 To display the screen;

[0048] Fig. 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] In an exemplary embodiment, a method for stitching panoramic videos of a fully mechanized mining face is provided, such as Figure 1 and Figure 4 As shown, the following steps are included, wherein:

[0052] Step 201: Based on the videos of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained; the image set includes images of all areas of the coal wall of the target fully mechanized mining working face at any time within the preset time period. One area corresponds to a video within a preset time period.

[0053] Step 202: Input the images in the image set into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image. The result is as follows: Figure 5 As shown, it is determined that all images of coal machine drums in the image set constitute the target image set.

[0054] Step 203: verify the authenticity of the coal machine drum in each image in the target image set, and determine the image with the highest authenticity as the target image.

[0055] Step 204: Input each image in the adjacent image set into the segmentation model to obtain the segmented image corresponding to each image in the adjacent image set. The result is as follows: Figure 6As shown. The adjacent image set includes: the target image, images corresponding to N adjacent areas located before the target area in the image set, and images corresponding to N adjacent areas located after the target area in the image set; the target area is the area of ​​the target fully mechanized mining face coal wall corresponding to the target image. The segmentation model is the YOLACT model.

[0056] Step 205: Input the segmented images corresponding to each image in the adjacent image set into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment. When outputting images, using a generative adversarial network can avoid visual blur caused by unclear images.

[0057] Step 206: Obtain a panoramic video of the target fully mechanized mining face within the preset time period based on the stitched images of the target fully mechanized mining face at each moment within the preset time period.

[0058] In an exemplary embodiment, the neural network model may be a yolov8 model or a LSTM model.

[0059] This application uses the richer C2f structure of yolov8 gradient flow, and adjusts different numbers of channels for models of different scales, which belongs to the careful fine-tuning of the model structure. Adjust specific parts to meet the needs of specific tasks or data sets. To change the number of categories of COCO in the output layer, you need to change the size of the output layer according to the new number of categories. Here it is adjusted to the minimum value, and then the characteristics of the coal machine drum are added to make the convolution layer pay more attention to the characteristics of the drum and enhance the feature extraction ability of the model. Since the distance between the camera and the coal machine drum is basically fixed at 2-3m, the display size in the image is also basically fixed, because the anchor frame is fixed in the output layer, which is 200-300 pixels. When the pixels of the anchor frame are too large, they are discarded to filter out incorrect results. In order to reduce computational overhead and speed up training, freeze the first few layers and only train the features of the coal machine drum in the last few layers, retain most of the pre-trained feature results, and adapt to the specific needs of new pictures.

[0060] The substitutability of deep learning neural network algorithms. Recurrent Neural Network (RNN) is a type of neural network used to process sequence data. Similar to "human cognition is based on past experience and memory", by using a specific form of memory to simulate time-based dynamic changes, RNN can not only consider the current input, but also give the network a "memory" function for the previous content. This network structure can directly reflect the high nonlinearity and uncertainty in the dynamic change process of the system, so it is suitable for learning the rules of time series data and predicting future change trends, but this method has the problem of gradient disappearance and gradient explosion. LSTM can maintain long-term storage of previous information and is widely used in applications such as speech recognition, information retrieval, and text classification. It has now become the most effective sequence model in practical applications.

[0061] In an exemplary embodiment, based on the video of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained, which specifically includes:

[0062] The PTZ camera takes videos of various areas of the target coal wall in the pre-set time period. Specifically, the existing PTZ camera is installed with three hydraulic supports, and the distance between the cameras is about 1.75 meters. This results in many parts of the coal wall being missed when the camera is facing the coal wall, resulting in incomplete splicing of the fully-mechanized mining face. Therefore, under the condition of ensuring the complete splicing of the mining face, the camera is tilted towards the coal wall to ensure that the camera can obtain more image areas.

[0063] Based on the videos of each area of ​​the coal wall of the target comprehensive mining working face within the preset time period, an image collection is obtained. Specifically, first initialize the Hikvision SDK, prepare the IP address, service port, login username, and login password required to log in to the IPC, call the asynchronous login interface, and set the callback function; regardless of whether the login is successful or not, the callback function will return in an asynchronous manner, and you can verify and judge whether the login is successful in the asynchronous callback function; since the asynchronous callback function is still in the thread in the SDK, in order to avoid blocking the SDK thread, after judging that the login has been successful, an asynchronous message notification is initiated to the main window, and the actual IPC operation is performed in the asynchronous message notification. In this way, the video stream data of the pan-tilt camera can be obtained in real time. Parse the incoming video stream data into original image data. Perform single-frame processing on the original image data.

[0064] In an exemplary embodiment, verifying the authenticity of the coal machine drum in each image in the target image set specifically includes:

[0065] If the images in the image set are images of all areas of the coal wall of the target fully mechanized mining working face at the initial moment within the preset time period, for any image in the target image set, the authenticity of the coal machine drum in the image is verified based on the image of the corresponding area of ​​the image at the moment before the initial moment and the front neighbor image and the rear neighbor image in the image set; the front neighbor image is the image of the area located before and adjacent to the corresponding area of ​​the image; the rear neighbor image is the image of the area located after and adjacent to the corresponding area of ​​the image;

[0066] If the image in the image set is not an image of all areas of the coal wall of the target comprehensive mining working face at the initial moment within the preset time period, for any image in the target image set, the authenticity of the coal machine drum in the image is verified based on the front neighbor image and the rear neighbor image in the image set, as well as the video of the corresponding area of ​​the image within the preset time period.

[0067] In an exemplary embodiment, the authenticity of the coal machine drum in the image is verified based on the image of the corresponding area of ​​the image at a moment before the initial moment and the front neighbor image and the rear neighbor image in the image set, specifically including:

[0068] The image of the corresponding area of ​​the image at one moment before the initial moment, the front neighbor image and the back neighbor image in the image set are respectively input into the neural network model to obtain whether there is a coal machine drum in the image of the corresponding area of ​​the image at one moment before the initial moment, the front neighbor image and the back neighbor image in the image set, and the position of the coal machine drum in the image.

[0069] If there is a coal machine drum in the front neighbor image or the rear neighbor image in the image set, and there is also a coal machine drum in the image of the corresponding area of ​​the image at the moment before the initial moment, then the authenticity of the presence of the coal machine drum in the image and the position of the coal machine drum in the image is high.

[0070] For example: the area ID corresponding to the image to be verified in the target image set is 10. If the image of the area with area ID 9 in the image set contains a coal machine drum or the image of the area with area ID 11 contains a coal machine drum, and the area with area ID 10 has a drum in the image at the moment before the initial moment, then the authenticity of the presence of a coal machine drum in the image to be verified and the position of the coal machine drum in the image to be verified is high, and the image to be verified is input into the subsequent segmented image.

[0071] In an exemplary embodiment, based on the front neighbor image and the rear neighbor image in the image set and the video of the corresponding area of ​​the image within a preset time period, the authenticity of the coal machine drum in the image is verified, specifically including:

[0072] First, based on the front neighbor image and the back neighbor image in the image set, the authenticity of the coal machine drum in the image is verified to obtain a first authenticity result. (If there is a coal machine drum in the front neighbor image or the back neighbor image in the image set, the first authenticity result is high).

[0073] Then, according to the condition of identifying one out of every three images, the images to be identified are selected in the screening video, and the selected images are input into the neural network model to obtain whether there is a coal machine drum in each selected image and the position of the coal machine drum in the image; the screening video is the video of the area corresponding to the image within a preset time period.

[0074] The forward direction and speed of the coal machine drum are determined according to the time corresponding to the selected image with the coal machine drum and the position of the coal machine drum in the image.

[0075] The position of the coal machine drum at the moment corresponding to the image is calculated according to the forward direction and speed of the coal machine drum.

[0076] The authenticity of the coal mill drum in the image is verified based on the calculated position of the coal mill drum at the moment corresponding to the image, and a second authenticity result is obtained; if the calculated position of the coal mill drum at the moment corresponding to the image is the same as the position of the coal mill drum obtained by inputting the image into the neural network model, then the second authenticity result is high.

[0077] When the first authenticity result is high and the second authenticity result is also high, it is determined that the authenticity of the coal machine drum in the image is high. Recognition is performed on a 3090 graphics card with 24G video memory, but it is necessary to recognize about 10 pictures before and after at the same time. Each picture is compressed to 680P, and 15 pictures need to be processed per second. The GPU computing power requirements are also relatively high. At the same time, the maximum speed of the coal machine is 20m / min, which is not particularly fast. Therefore, the images are screened when imported in batches. Recognition is performed once every 3 pictures, and the other parts are performed in the normal way of reasoning. By obtaining the forward direction and forward speed of the coal machine, the pixel value of the rolling movement in the image is calculated, the position of the anchor frame is estimated, and the computing power pressure is reduced.

[0078] For example: the area ID corresponding to the image to be verified in the target image set is 10. If the image of the area with area ID 9 in the image set contains a coal machine drum or the image of the area with area ID 11 contains a coal machine drum, the first authenticity result is high.

[0079] Then, according to the condition of identifying one out of every three images, the image to be identified is selected from the video with area ID 10 within the preset time period, and the selected image is input into the neural network model to obtain whether there is a coal machine drum in each selected image and the position of the coal machine drum in the image.

[0080] The forward direction and speed of the coal machine drum are determined according to the time corresponding to the selected image with the coal machine drum and the position of the coal machine drum in the image.

[0081] The position of the coal machine drum at the moment corresponding to the image to be verified is calculated according to the forward direction and speed of the coal machine drum.

[0082] According to the calculated position of the coal mill drum at the moment corresponding to the image to be verified, the authenticity of the coal mill drum in the image to be verified is verified to obtain a second authenticity result; if the calculated position of the coal mill drum at the moment corresponding to the image to be verified is the same as the position of the coal mill drum obtained by inputting the image to be verified into the neural network model, then the second authenticity result is high.

[0083] When the first authenticity result is high and the second authenticity result is also high, it is determined that the authenticity of the coal machine drum in the image to be verified is high, and the image to be verified is input into the subsequent segmentation model.

[0084] In an exemplary embodiment, the training process of the generative adversarial network is generally as follows:

[0085] Two neural networks are trained to compete with each other to generate more realistic new data from a given training dataset. One network generates new data by taking the input coal mining machine drum sample and modifying it as much as possible. The other network tries to predict whether the generated data output belongs to the original dataset. The generative neural network trains the coal mining machine drum image and trains the data attributes of the entire image. The discriminative neural network trains the original data to distinguish independent attributes, randomly adds some gray coal ash noise to the generative neural network, and modifies the attribute characteristics of the entire image. The data of the generative neural network is passed to the discriminative neural network. The discriminative neural network calculates the probability that the generated output belongs to the original dataset. The discriminative neural network provides some guidance to the generative neural network to reduce the randomization of the noise vector in the next cycle, thereby improving the quality of the image. And filter out the original coal ash noise to improve the clarity of the original image.

[0086] In an exemplary embodiment, the segmented images corresponding to each image in the adjacent image set are input into a trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment, specifically including:

[0087] The segmented images corresponding to each image in the adjacent image set are sequentially subjected to image compression, cropping and perspective transformation operations to obtain processed segmented images corresponding to each image in the adjacent image set.

[0088] The processed segmented images corresponding to each image in the adjacent image set are processed by nearest neighbor interpolation to obtain the interpolated segmented images corresponding to each image in the adjacent image set; the present application uses nearest neighbor interpolation to quickly process the image size by finding the known pixel closest to the target pixel position and assigning its value to the target pixel. This will form an obvious jagged effect, but the image size can be quickly compressed.

[0089] The interpolated segmented images corresponding to each image in the adjacent image set are input into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment.

[0090] In an exemplary embodiment, obtaining a panoramic video of the target fully mechanized mining working face within a preset time period according to the spliced ​​images of the target fully mechanized mining working face at each moment within a preset time period specifically includes:

[0091] Preprocessing the spliced ​​image of the target fully mechanized mining working face at the moment to obtain a panoramic image of the target fully mechanized mining working face at the moment; Figure 7 Part (a) is the effect picture before HDR processing. Figure 7 Part (b) shows the effect after HDR processing.

[0092] A panoramic video of the target fully mechanized mining face within a preset time period is obtained based on the panoramic images of the target fully mechanized mining face at each moment within a preset time period.

[0093] In an exemplary embodiment, a panoramic video of the target fully mechanized mining working face within a preset time period is obtained according to the spliced ​​images of the target fully mechanized mining working face at each moment within a preset time period, and then the method further includes:

[0094] Display the panoramic video interface, such as Figure 8 As shown in the figure, the panoramic video is transmitted to the display terminal through the Redis computing cluster.

[0095] In an exemplary embodiment, preprocessing the stitched image of the target fully mechanized mining working face at the moment to obtain a panoramic image of the target fully mechanized mining working face at the moment specifically includes:

[0096] Use geometric characteristics to stack images. Because the position of each camera on the bracket is fixed and known, and the spacing between each shelf is 1.75m, the pixel value of the image can be obtained through early testing. In the original 720P image, the overlapping part of the image is about 120 pixels, which is cropped and modified to ensure that there is no overlap between images.

[0097] HDR wide dynamic adjustment is used to adjust the color difference between images as a whole. Multiple images of the same scene are taken, each image uses a different exposure time (short exposure captures highlights, long exposure captures shadows), and then these images are synthesized into an HDR image. Use weighted averaging or other synthesis methods to generate a single image that retains all the details in the scene. Images with different exposures are converted to a uniform brightness value by estimating the camera's illumination response function. This step is to ensure the brightness comparability of images with different exposures. Apply the same logarithmic transformation mapping function to the entire image, which is suitable for scenes with uniformly distributed details. Noise is introduced during the synthesis and processing process, especially in low-exposure images. Using noise suppression algorithms can improve the quality of the final image.

[0098] In an exemplary embodiment, before inputting the images in the image set into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, it also includes:

[0099] Store the image collection in a Redis computing cluster. The Redis computing cluster is a non-relational database that contains multiple data structures, a memory-based, optional persistent key-value storage system, and provides APIs in multiple languages.

[0100] Application scenarios: hot data cache, distributed session, counter, tag related issues, ranking related issues, distributed locks, other business immutable or low probability of change data. Figure 2 As shown in the figure, the Redis computing cluster workflow is:

[0101] 1. When the redis server starts, AE_READABLE will be registered with eventLoop (IO multiplexing).

[0102] 2. The client requests to establish a connection with the server. The server generates a Scoket (s1) channel and binds the AE_READABLE event.

[0103] 3. The client (s1) requests to execute set key value (write command). S1 triggers AE_READAB LE, and the command requester reads the key and value into the memory and modifies the data.

[0104] 4. After setting the value, associate s1 with the AE_WRITABLE event to trigger the write operation. If successful, the command responder outputs the result "OK" and returns the result to unbind the AE_WRITABLE event of s1 from the command reply processor.

[0105] IO multiplexing: I / O refers to network I / O multiplexing refers to multiple TCP connections (sockets or channels), and multiplexing refers to multiplexing one or several threads. That is, one or a group of threads handle multiple TCP connections. The biggest advantage is that it reduces system overhead, and there is no need to create too many processes / threads, nor to maintain these processes / threads.

[0106] Select mechanism: When the client operates the server, three types of file descriptors (fd for short) will be generated: writefds (write), readfds (read), and exceptfds (exception). Select will block and monitor the three types of file descriptors, and will return when there is data, readable, writable, exception or timeout; after returning, it will traverse the entire array of fdset to find the ready descriptor fd, and then perform the corresponding IO operation. Advantages: Supported on almost all platforms, with good cross-platform support. Disadvantages: Since the full disk scan is carried out in a polling manner, the performance will decrease as the number of file descriptors FD increases. Each time select() is called, the fd set needs to be copied from user state to kernel state and traversed (messages are passed from kernel to user space). The default limit for a single process to open FD is 1024, and the macro definition can be modified, but the efficiency is still slow.

[0107] epoll mechanism: There is no limit on the number of fds. Copying from user state to kernel state only needs one time, and is triggered by the time notification mechanism. Register fd through epoll_ctl. Once fd is ready, the corresponding fd will be activated through the callback mechanism to perform related io operations. The high performance of epoll is due to its three functions:

[0108] 1. When the epoll_create() system starts, a B+ tree structure file system is applied for in the Linux kernel and the epoll object is returned, which is also an fd.

[0109] 2. Every time a new connection is created, epoll_ctl() operates the epoll object through this function, modifies, adds, deletes the corresponding link fd in this object, and binds a callback function.

[0110] 3. epoll_wait() polls all callback sets and completes the corresponding IO operations.

[0111] Advantages: 1. No fd limit, the upper limit of the supported FD is the maximum number of file handles of the operating system, 1G memory supports about 100,000 handles. 2. Improved efficiency, using callback notification instead of polling, the efficiency will not decrease as the number of FDs increases. 3. The kernel and user space mmap the same memory (mmap is a method of memory mapping files, that is, mapping a file or other object to the address space of the process).

[0112] In order to avoid single point failure, the common practice is to replicate the database to multiple copies and deploy them on different servers, so that even if one server fails, other servers can continue to provide services. To this end, Redis provides a replication function, which can automatically synchronize the updated data to other databases when the data in one database is updated.

[0113] In the concept of replication, databases are divided into two categories, one is the master database and the other is the slave database. The master database can perform read and write operations, and when the write operation causes data changes, it will automatically synchronize the data to the slave database. The slave database is generally read-only and accepts data synchronized from the master database. A master database can have multiple slave databases, while a slave database can only have one master database.

[0114] The principle is as follows: the slave database connects to the master database and sends a SYNC command; after receiving the SYNC command, the master database starts to execute the BGSAVE command to generate an RDB file and uses the buffer to record all write commands executed thereafter; after the master database BGSAVE is executed, it sends snapshot files to all slave databases and continues to record the executed write commands during the sending period; after receiving the snapshot file, the slave database discards all old data and loads the received snapshot; after the master database snapshot is sent, it starts to send write commands in the buffer to the slave database; the slave database completes the loading of the snapshot, starts to receive command requests, and executes write commands from the master database buffer; (the slave database is initialized) the master database sends the same write command to the slave database every time it executes a write command, and the slave database receives and executes the received write command (the operation after the slave database is initialized) after the disconnection and reconnection occurs, the version after 2.8 will pass the commands during the disconnection to the reconnection database for incremental replication. When the master and slave are just connected, full synchronization is performed; after full synchronization is completed, incremental synchronization is performed. Of course, if necessary, the slave can initiate full synchronization at any time. Redis's strategy is that, no matter what, it will first try to perform incremental synchronization. If it fails, it will require the slave to perform full synchronization. 1. It supports master-slave replication. The host will automatically synchronize data to the slave, and read-write separation can be performed. 2. In order to share the read operation pressure of the Master, the Slave server can provide read-only operation services to the client. 3. The Slave can also accept connections and synchronization requests from other Slaves, which can effectively share the synchronization pressure of the Master. 4. The Master Server provides services to the Slaves in a non-blocking manner. Therefore, during the Master-Slave synchronization, the client can still submit queries or modification requests. 5. The Slave Server also completes data synchronization in a non-blocking manner. During synchronization, if a client submits a query request, Redis will return the data before synchronization.

[0115] In an exemplary embodiment, the software algorithm framework diagram corresponding to the panoramic video stitching method of the fully mechanized mining face provided in the embodiment of the present application is as follows: Figure 3 As shown, it includes a network camera, a camera IO computing cluster and a Redis cluster connected in sequence, as well as multiple deep learning computing units, an image stitching computing unit, a visualization computing unit and a large display screen connected to the visualization computing unit.

[0116] This application has the following technical effects:

[0117] (1) Using server cluster (Redis computing cluster) computing method to allocate resources and calculate, the hardware can be expanded at any time to meet the application needs of software computing resources. Through the structured data transmission image dictionary, the real-time requirements of data transmission can be better met.

[0118] (2) The YOLOv8 model is used to perform special processing on the drum of the underground coal machine, including setting a fixed anchor frame output value to improve the speed; setting the model characteristics of the frozen layer to specifically train the recognition of the drum part of the coal machine; at the same time, in order to improve the recognition speed, the coal machine is selectively identified according to its actual operating speed and direction, and the reasoning method is used to identify the coal machine to reduce resource consumption.

[0119] (3) When stitching images, a generative adversarial network is used to improve the clarity of the image and remove the rust, thereby enhancing the visual appearance. The HDR wide dynamic range adjustment method is also added to ensure the brightness consistency of the images taken under different lighting conditions, without causing obvious color differences due to uneven brightness.

[0120] (4) Use server clusters for coordinated computing:

[0121] 1. Easy to implement: Server cluster technology is mainly implemented through software. In terms of hardware, it can connect multiple low-performance and cheap servers together through cluster services to achieve the growth of the entire server system by multiples, dozens, or hundreds of times. Whether from the perspective of software and hardware composition costs or technology, it is much lower than other expansion methods.

[0122] 2. Super expansion: One of the advantages of clusters is its expansion technology. Generally, only about 20 CPUs can be expanded, and the expansion capacity is limited. However, a cluster system using cluster technology can expand multiple servers with thousands of CPUs, and the expansion capacity is obvious. Cluster services can also continuously adjust themselves to meet the growing application needs. If the load exceeds the limit of the cluster itself, additional nodes can be added.

[0123] 3. Easy to manage: Use the cluster manager to manage all server resources and applications of the cluster system, just as if they were all running on the same server. At the same time, you can monitor the status of the cluster from any node and resource in the network. If the failed server is connected back, it will automatically return to working status, and the cluster technology will automatically balance the load in the cluster without human intervention.

[0124] 4. Ultra-high availability: Use cluster services to have full access to the entire cluster system resources. For example, disk drives and IP addresses will be automatically transferred from the failed server to the available server. If a system or application in the cluster fails, the cluster software will restart the failed application on the available server, or distribute the work on the failed node to the remaining nodes.

[0125] (5) Using yolov8 and GAN output results:

[0126] 1. Higher accuracy and speed: The yolov8 model achieves a better balance between inference speed and accuracy. Efficient inference speed Even on mobile devices or embedded systems, yolov8 can provide fast and efficient inference, suitable for real-time application scenarios.

[0127] 2. Improved data enhancement technology: GAN has made improvements in data enhancement and introduced a variety of enhancement technologies, such as Mosaic, Mixup, random scaling and cropping, etc. These technologies help the model improve its generalization ability and robustness, and improve its performance on different data sets.

[0128] 3. Support multi-task learning: It not only supports target detection, but also integrates multiple tasks such as instance segmentation, key point detection and image classification. Users can complete the training and reasoning of multiple tasks on the same model to improve the flexibility of the application.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a comprehensive mining face panoramic video stitching device for implementing the comprehensive mining face panoramic video stitching method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more comprehensive mining face panoramic video stitching device embodiments provided below can refer to the limitations of the comprehensive mining face panoramic video stitching method above, and will not be repeated here.

[0130] In an exemplary embodiment, a panoramic video stitching device for a fully mechanized mining face is provided, comprising:

[0131] The image set acquisition module is used to obtain an image set based on the video of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period; the image set includes images of all areas of the coal wall of the target fully mechanized mining working face at any time within the preset time period.

[0132] The position determination module is used to input the images in the image set into the yolov8 model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, and to determine that all images in the image set with coal machine drums constitute a target image set.

[0133] The position verification module is used to verify the authenticity of the coal machine drum in each image in the target image set, and determine the image with the highest authenticity as the target image.

[0134] A segmentation module is used to input each image in an adjacent image set into a segmentation model to obtain a segmented image corresponding to each image in the adjacent image set; the adjacent image set includes: the target image, images corresponding to N adjacent areas in the image set located before the target area, and images corresponding to N adjacent areas in the image set located after the target area; the target area is the area of ​​the target fully mechanized mining working face coal wall corresponding to the target image.

[0135] The splicing module is used to input the segmented images corresponding to each image in the adjacent image set into the trained generative adversarial network to obtain a spliced ​​image of the target comprehensive mining working face at the said moment.

[0136] The panoramic video module is used to obtain a panoramic video of the target comprehensive mining working face within a preset time period based on the spliced ​​images of the target comprehensive mining working face at each moment within the preset time period.

[0137] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store panoramic video stitching data of the comprehensive mining working face. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a panoramic video stitching method for a comprehensive mining working face is implemented.

[0138] Those skilled in the art will understand that Fig. 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned method embodiments when executing the computer program.

[0139] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0140] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0142] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A panoramic video stitching method for a fully mechanized mining face, characterized in that: The panoramic video stitching method of the fully mechanized mining working face comprises: According to the video of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained; the image set includes images of all areas of the coal wall of the target fully mechanized mining working face at any time within the preset time period; The images in the image set are respectively input into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, and determine that all images in the image set that have coal machine drums constitute a target image set; Verify the authenticity of the coal machine drum in each image in the target image set, and determine the image with the highest authenticity as the target image; Input each image in the adjacent image set into the segmentation model to obtain a segmented image corresponding to each image in the adjacent image set; the adjacent image set includes: the target image, images corresponding to N adjacent areas in the image set located before the target area, and images corresponding to N adjacent areas in the image set located after the target area; the target area is the area of ​​the target fully mechanized mining working face coal wall corresponding to the target image; Input the segmented images corresponding to each image in the adjacent image set into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment; A panoramic video of the target fully mechanized mining working face within the preset time period is obtained based on the spliced ​​images of the target fully mechanized mining working face at each moment within the preset time period.

2. The method for stitching panoramic video of a fully mechanized mining face according to claim 1 is characterized in that: Verify the authenticity of the coal machine drum in each image in the target image set, including: If the images in the image set are images of all areas of the coal wall of the target fully mechanized mining working face at the initial moment within the preset time period, for any image in the target image set, the authenticity of the coal machine drum in the image is verified based on the image of the corresponding area of ​​the image at the moment before the initial moment and the front neighbor image and the rear neighbor image in the image set; the front neighbor image is the image of the area located before and adjacent to the corresponding area of ​​the image; the rear neighbor image is the image of the area located after and adjacent to the corresponding area of ​​the image; If the image in the image set is not an image of all areas of the coal wall of the target comprehensive mining working face at the initial moment within the preset time period, for any image in the target image set, the authenticity of the coal machine drum in the image is verified based on the front neighbor image and the rear neighbor image in the image set, as well as the video of the corresponding area of ​​the image within the preset time period.

3. The method for splicing panoramic videos of fully mechanized mining working faces according to claim 1 is characterized in that: The segmented images corresponding to each image in the adjacent image set are input into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment, specifically including: The segmented images corresponding to each image in the adjacent image set are sequentially subjected to image compression, cropping and perspective transformation operations to obtain processed segmented images corresponding to each image in the adjacent image set; Performing nearest neighbor interpolation processing on the processed segmented images corresponding to each image in the adjacent image set to obtain an interpolated segmented image corresponding to each image in the adjacent image set; The interpolated segmented images corresponding to each image in the adjacent image set are input into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment.

4. The method for stitching panoramic video of a fully mechanized mining face according to claim 1 is characterized in that: The panoramic video of the target fully mechanized mining face within the preset time period is obtained according to the spliced ​​images of the target fully mechanized mining face at each moment within the preset time period, specifically including: Preprocessing the spliced ​​image of the target fully mechanized mining working face at the moment to obtain a panoramic image of the target fully mechanized mining working face at the moment; A panoramic video of the target fully mechanized mining face within a preset time period is obtained based on the panoramic images of the target fully mechanized mining face at each moment within a preset time period.

5. The method for stitching panoramic video of a fully mechanized mining face according to claim 1 is characterized in that: According to the video of each area of ​​the coal wall of the target fully mechanized mining working face within the preset time period, an image set is obtained, which specifically includes: The PTZ camera takes videos of various areas of the coal wall of the target fully mechanized mining face within a preset time period; Based on the videos of each area of ​​the coal wall of the target fully mechanized mining working face within a preset time period, an image set is obtained.

6. The method for stitching panoramic video of a fully mechanized mining face according to claim 5 is characterized in that: Before inputting the images in the image set into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, the method further includes: Store the image collection in a Redis compute cluster.

7. A panoramic video splicing device for a fully mechanized mining face, characterized in that: The panoramic video splicing device of the fully mechanized mining working face comprises: An image set acquisition module is used to obtain an image set based on videos of various areas of the coal wall of the target fully mechanized mining working face within a preset time period; the image set includes images of all areas of the coal wall of the target fully mechanized mining working face at any time within the preset time period; A position determination module is used to input the images in the image set into the neural network model to determine whether there is a coal machine drum in each image in the image set and the position of the coal machine drum in the image, and to determine that all images in the image set containing coal machine drums constitute a target image set; A position verification module is used to verify the authenticity of the coal machine drum in each image in the target image set, and determine the image with the highest authenticity as the target image; A segmentation module is used to input each image in the adjacent image set into a segmentation model to obtain a segmented image corresponding to each image in the adjacent image set; the adjacent image set includes: the target image, images corresponding to N adjacent areas in the image set located before the target area, and images corresponding to N adjacent areas in the image set located after the target area; the target area is an area of ​​the target fully mechanized mining working face coal wall corresponding to the target image; A splicing module, used for inputting the segmented images corresponding to each image in the adjacent image set into the trained generative adversarial network to obtain a spliced ​​image of the target fully mechanized mining working face at the moment; The panoramic video module is used to obtain a panoramic video of the target comprehensive mining working face within a preset time period based on the spliced ​​images of the target comprehensive mining working face at each moment within the preset time period.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the panoramic video stitching method for a comprehensive mining working face as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for stitching a panoramic video of a comprehensive mining face as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for stitching a panoramic video of a comprehensive mining face as described in any one of claims 1 to 6 is implemented.

Images