Underground coal mining machine shielding recovery method and device, terminal and storage medium

Abstract

The invention provides an underground coal mining machine shielding recovery method and device, a terminal and a storage medium, and the method comprises the steps: inputting a to-be-processed original image of a coal mining machine into a trained instance segmentation model, and obtaining an unshielded segmented image; performing image addition processing on the preprocessed original image of the coal mining machine and the unshielded segmented image to obtain a preprocessed unshielded segmented image; inputting the preprocessed unshielded segmented image into a trained image restoration model to obtain a restored image; and obtaining a shielded target image according to the original image of the coal mining machine and the unshielded segmented image, obtaining a repaired position image in the repaired image, and replacing the shielded target image in the original image of the coal mining machine with the repaired position image to obtain a complete image of the coal mining machine. According to the method, the accuracy of virtual recovery of the shielded part of the coal mining machine on the image is improved, and the problem that the recovered image is inaccurate due to the fact that a semantic segmentation method or an image restoration method is independently used is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, terminal and storage medium for recovering an underground coal mining machine from occlusion. Background Art

[0002] Intelligent mining technology for coal mining faces is one of the important technologies for realizing intelligent coal mines. In the process of coordinated coal mining in the fully mechanized working face, the rotation of the shearer drum will generate a large amount of coal dust. At the same time, the shearer will also generate coal dust during the excavation and cutting process, causing the shearer to be often blocked by coal dust and fallen coal blocks. The blocking of coal may interfere with the normal positioning and communication of the shearer in the intelligent mining technology of the shearer, affecting its automated and intelligent operation.

[0003] Coal mining machine occlusion recovery is a complex problem. At present, there are relatively limited methods for coal mining machine occlusion recovery in mines, including methods based on semantic segmentation and methods using image restoration. For the task of coal mining machine occlusion recovery, the method based on semantic segmentation can only separate the unoccluded area of ​​the target, but cannot restore the occluded area; while the method that only uses image restoration may modify the background while restoring the occluded area. Therefore, the image obtained by the existing technology for coal mining machine occlusion recovery is not accurate.

[0004] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, device, terminal and storage medium for recovering occlusion of a coal mining machine in a mine are provided, aiming to solve the problem that the image obtained by recovering occlusion of a coal mining machine in the prior art is not accurate.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A method for recovering a coal mining machine from obstruction in a mine, wherein the method comprises:

[0008] Input the original image of the coal mining machine to be processed into the trained instance segmentation model to obtain an unoccluded segmented image;

[0009] Performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image;

[0010] Inputting the preprocessed unoccluded segmented image into a trained image restoration model to obtain a restored image;

[0011] An occluded target image is obtained according to the original image of the coal mining machine and the unoccluded segmented image, a repair position image is obtained in the repaired image, and the occluded target image in the original image of the coal mining machine is replaced by the repair position image to obtain a complete image of the coal mining machine.

[0012] In one embodiment of the present application, before performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain the preprocessed unobstructed segmented image, the method further includes:

[0013] Graying, contour detection and coal shearer area extraction are performed on the original image of the coal shearer to obtain a preprocessed original image of the coal shearer.

[0014] In one embodiment of the present application, the instance segmentation model is a YOLOv8-seg network, and the image restoration model is a FcF-Inpainting network.

[0015] In one embodiment of the present application, the training step of the instance segmentation model includes:

[0016] Acquire a training data set, wherein the training data set includes a plurality of training images and labels corresponding to each training image, wherein the training images include: unobstructed coal mining machine training images and obstructed coal mining machine training images;

[0017] A YOLOv8-seg network is obtained, and based on the unobstructed coal mining machine training image, the obstructed coal mining machine training image and corresponding labels, instance segmentation training is performed on the YOLOv8-seg network to obtain a trained instance segmentation model.

[0018] In one embodiment of the present application, the training step of the image restoration model includes:

[0019] Acquire a training data set, wherein the training data set includes a plurality of training images and labels corresponding to each training image, wherein the training images include: unobstructed coal mining machine training images and obstructed coal mining machine training images;

[0020] Obtain an FcF-Inpainting network, and train the FcF-Inpainting network based on the unobstructed shearer training image, the obstructed shearer training image, and a randomly generated mask to obtain a trained image restoration model.

[0021] In one embodiment of the present application, the step of acquiring the unobstructed coal mining machine training image includes:

[0022] Collect the first motion video of the coal mining machine in an unobstructed scene;

[0023] Using OpenCV to capture several frames in the first motion video, to obtain several unobstructed scene images;

[0024] The plurality of unobstructed scene images are scaled to a preset size, and the edges are filled with black pixels to obtain a plurality of unobstructed coal mining machine training images.

[0025] In one embodiment of the present application, the step of acquiring the occluded coal mining machine training image includes:

[0026] Collect the second motion video of the coal mining machine in the occlusion scene;

[0027] intercepting several frames in the second motion video through OpenCV to obtain several occlusion scene images;

[0028] After the occluded scene image is subjected to an image defogging operation, graying, contour detection and coal mining machine area extraction are performed to obtain an occluded coal mining machine training image.

[0029] The present application also provides a device for recovering a coal mining machine in a mine, wherein the device comprises:

[0030] The first input module is used to input the original image of the coal mining machine to be processed into the trained instance segmentation model to obtain an unobstructed segmented image;

[0031] An image processing module is used to perform image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image;

[0032] A second input module, used for inputting the preprocessed unoccluded segmented image into a trained image restoration model to obtain a restored image;

[0033] The image restoration module is used to obtain an occluded target image based on the original image of the coal mining machine and the unoccluded segmented image, obtain a repair position image in the repaired image, and replace the occluded target image in the original image of the coal mining machine with the repair position image to obtain a complete image of the coal mining machine.

[0034] The present application also provides a terminal, which includes: a memory, a processor, and an underground coal mining machine obstruction recovery program stored in the memory and executable on the processor, wherein the underground coal mining machine obstruction recovery program, when executed by the processor, implements the steps of the underground coal mining machine obstruction recovery method as described above.

[0035] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the underground coal mining machine obstruction recovery method as described above.

[0036] The present invention provides a method, device, terminal and storage medium for recovering coal mining machines in a mine. The method comprises: inputting the original image of the coal mining machine to be processed into a trained instance segmentation model to obtain an unobstructed segmented image; performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image; inputting the preprocessed unobstructed segmented image into a trained image restoration model to obtain a restored image; obtaining an occluded target image according to the original image of the coal mining machine and the unobstructed segmented image, obtaining a restored position image in the restored image, and replacing the occluded target image in the original image of the coal mining machine with the restored position image to obtain a complete image of the coal mining machine. The present invention improves the accuracy of virtual restoration of the obstructed part of the coal mining machine on the image by first inputting the original image of the coal mining machine into the instance segmentation model, performing image segmentation processing, and then inputting it into the image restoration model to perform image restoration, thereby avoiding the problem of inaccurate restored images caused by using semantic segmentation methods or image restoration methods alone. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a preferred embodiment of the method for recovering coal mining machines from obstruction in a mine in the present invention;

[0038] Figure 2 It is a flowchart of the training steps of the instance segmentation model and the image restoration model in the present invention;

[0039] Figure 3 It is a functional principle block diagram of a preferred embodiment of the shielding recovery device for coal mining machine in a mine in the present invention;

[0040] Figure 4 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] See also Figure 1 , Figure 1 Flowchart of the method for recovering coal mining machine from occlusion in the mine according to the present invention. Figure 1 As shown, the method for recovering the coal mining machine from obstruction in a mine according to an embodiment of the present invention includes:

[0043] Step S100: input the original image of the coal mining machine to be processed into the trained instance segmentation model to obtain an unobstructed segmented image.

[0044] In the embodiment of the present application, the instance segmentation model is a YOLOv8-seg network. Specifically, YOLOv8 is a model with high precision and high scalability in the YOLO series of target detection algorithms, and the YOLOv8-seg network is the image segmentation task. The YOLOv8-seg network structure is mainly divided into three parts: the backbone network (Backbone), the Neck layer and the Head layer. The backbone network is used to extract features from the input image, including modules such as CBS (Conv+BN+SiLU), C2f and SPPF. These modules gradually reduce the resolution of the feature map and increase the number of channels through convolution operations to extract key information from the image; the Neck layer is used to perform feature fusion on the feature maps of different scales output by the backbone network, and through operations such as upsampling and downsampling, the feature maps of different scales are fused together to enrich the semantic information of the feature map; the Head layer is used to generate the final prediction result. In the instance segmentation task, in addition to generating feature maps for target detection and classification, the Head layer also generates additional feature maps for Mask, which are used to represent the contours and shapes of the segmented areas.

[0045] The YOLOv8-seg adopted in the embodiment of the present application makes the model more flexible and accurate in predicting the target position, simplifies the complexity of the model, and improves the reasoning speed.

[0046] like Figure 1 As shown, the method for recovering the coal mining machine from obstruction in a mine described in this embodiment also includes:

[0047] Step S200: performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image.

[0048] In the embodiment of the present application, before step S200, the step also includes: graying, contour detection and coal shearer area extraction of the coal shearer original image to obtain a preprocessed coal shearer original image.

[0049] The embodiment of the present application performs grayscale processing on the original image of the coal mining machine. The grayscale image only contains brightness information and removes the color information of the red, green and blue channels in the color image, thereby simplifying the image information and reducing the amount of data for subsequent processing; and the calculation amount of the grayscale image is much smaller than that of the color image, and the processing speed is faster, which significantly improves the execution efficiency of the algorithm. The embodiment of the present application performs contour detection on the original image of the coal mining machine and can obtain the accurate shape information of the target. The embodiment of the present application performs coal mining machine area extraction on the original image of the coal mining machine to facilitate image segmentation.

[0050] like Figure 1 As shown, the method for recovering the coal mining machine from obstruction in a mine described in this embodiment also includes:

[0051] Step S300: input the pre-processed unoccluded segmented image into a trained image restoration model to obtain a restored image.

[0052] In an embodiment of the present application, the image restoration model is an FcF-Inpainting network. Specifically, the FcF-Inpainting network is an improved GANs network, which consists of two parts: a generator and a discriminator, which compete with each other during the training process to improve their respective performance. The FcF-Inpainting network focuses on the field of image restoration. It combines the advantages of StyleGAN2 and fast Fourier convolution to simultaneously improve the generation of structure and texture; in terms of the generator, the FcF model adopts a coarse-to-fine generation strategy similar to StyleGAN2, but in the upsampling process, the features from the encoder and generator are integrated through the FaF-Syn module to refine the texture; its discriminator is also similar to StyleGAN2.

[0053] The FcF-Inpainting network used in the embodiment of the present application achieves high-quality image restoration effects by combining structure and texture.

[0054] like Figure 1 As shown, the method for recovering the coal mining machine from obstruction in a mine described in this embodiment also includes:

[0055] Step S400, obtaining an occluded target image according to the original image of the coal mining machine and the unoccluded segmented image, obtaining a repair position image in the repaired image, and replacing the occluded target image in the original image of the coal mining machine with the repair position image to obtain a complete image of the coal mining machine.

[0056] Specifically, according to the feature of consistent image size, the position of the occluded target is obtained from the original image, and the repaired template is obtained from the repaired image to replace the position of the occluded target in the original image.

[0057] The embodiment of the present application realizes the virtual restoration of the obscured part of the coal mining machine blocked by coal and coal dust on the image, thereby improving the effectiveness and robustness of the restoration of the obscured object in a complex environment.

[0058] In an embodiment of the present application, the training step of the instance segmentation model includes:

[0059] Acquire a training data set, wherein the training data set includes a plurality of training images and labels corresponding to each training image, wherein the training images include: unobstructed coal mining machine training images and obstructed coal mining machine training images;

[0060] A YOLOv8-seg network is obtained, and based on the unobstructed coal mining machine training image, the obstructed coal mining machine training image and corresponding labels, instance segmentation training is performed on the YOLOv8-seg network to obtain a trained instance segmentation model.

[0061] Specifically, the input of the YOLOv8-seg network is: Input = I org +label, where I org is the original image, label is the label given to the original image.

[0062] In one embodiment of the present application, the training step of the image restoration model includes:

[0063] Acquire a training data set, wherein the training data set includes a plurality of training images and labels corresponding to each training image, wherein the training images include: unobstructed coal mining machine training images and obstructed coal mining machine training images;

[0064] Obtain an FcF-Inpainting network, and train the FcF-Inpainting network based on the unobstructed shearer training image, the obstructed shearer training image, and a randomly generated mask to obtain a trained image restoration model.

[0065] Specifically, the input of the FcF-Inpainting network is: Input = I mask +I org , where I org is the original image, I mask is a randomly generated mask.

[0066] In the embodiment of the present application, the step of acquiring the unobstructed coal mining machine training image includes:

[0067] Collect the first motion video of the coal mining machine in an unobstructed scene;

[0068] Using OpenCV to capture several frames in the first motion video, to obtain several unobstructed scene images;

[0069] The plurality of unobstructed scene images are scaled to a preset size, and the edges are filled with black pixels to obtain a plurality of unobstructed coal mining machine training images.

[0070] Specifically, when performing image acquisition, the first motion video of the coal mining machine in an unobstructed scene is acquired. The first motion video of the coal mining machine in an unobstructed scene needs to be obtained from a working camera on the coal mining face under good lighting conditions, and is a video shot from multiple angles. OpenCV (Open Source Computer Vision Library) is an open source cross-platform computer vision library that provides a large number of image and video processing functions. Scale the plurality of unobstructed scene images to 256*256 or 640*640 size.

[0071] The embodiment of the present application processes the first motion video into an unobstructed coal mining machine training image to facilitate the training of the YOLOv8-seg network and the FcF-Inpainting network.

[0072] In the embodiment of the present application, the step of obtaining the occluded coal mining machine training image includes:

[0073] Collect the second motion video of the coal mining machine in the occlusion scene;

[0074] intercepting several frames in the second motion video through OpenCV to obtain several occlusion scene images;

[0075] After the occluded scene image is subjected to an image defogging operation, graying, contour detection and coal mining machine area extraction are performed to obtain an occluded coal mining machine training image.

[0076] Specifically, when performing image acquisition, the second motion video of the coal mining machine in the occlusion scene is collected. The second motion video of the coal mining machine in the occlusion scene can be obtained from the monitoring of the coal mining machine during coal mining work, and part of it can be obtained from the Internet. The occluders of the occlusion scene are cables, coal, hydraulic supports and / or people. The defogging operation uses a dark channel image defogging algorithm, where the mathematical expression of the dark channel is:

[0077]

[0078] In the formula, J c (y) represents any color channel of image J, c represents the three channels of the color image, Ω(x) represents the window centered on pixel x, and the algorithm takes the grayscale values ​​of the three channels of pixel y on image J, and takes the minimum of these three values ​​as the final value to obtain a single-channel grayscale image, and then takes the minimum y value in the Ω(x) window centered on pixel x to replace the grayscale value x of the central pixel, so as to obtain J c (y) Dark channel image J dark (x).

[0079] The embodiment of the present application processes the second motion video into an occluded coal mining machine training image to facilitate the training of the YOLOv8-seg network and the FcF-Inpainting network.

[0080] The training process diagram of the instance segmentation model and image restoration model is as follows Figure 2 shown. Figure 2 The image normalization process is to scale the image to a preset size and fill the edges with black pixels.

[0081] Compared with the existing solution of obtaining the occluded part template from the original image based on the HSV image color space feature, the embodiment of the present application uses the YOLOv8 segmentation model and image contour. The following is the comparison result of the target object area size (pixel point) after processing the 6 images separately and scaling them to 256*256 size as shown in Table 1.

[0082] Original image size Method for extracting HSV features YOLOv8's approach 256*256 52 1876 256*256 0 12715 256*256 543 7240 256*256 385 10158 256*256 306 8394 256*256 316 7353

[0083] Table 1

[0084] Judging from the results, compared with the method of obtaining the occluded part template from the original image using only the HSV image color space features, the method provided in the embodiment of the present application has better acquisition effect on the target object and better robustness.

[0085] In terms of image restoration, the values ​​of the images generated using the FcF-Inpainting model are shown in Table 2.

[0086] FID mean LPIPS mean LPIPS std SSIM mean SSIM std total 19.7729 0.0159 0.0159 0.9864 0.0149

[0087] Table 2

[0088] Among them, FID mean represents the distance between the generated image and the real image, LPIPS std and LPIPS mean represent the standard deviation and variance of LPIPS, indicating the difference between the generated image and the real image; SSIM mean and SSIM std represent the standard deviation and variance of SSIM, indicating the structural similarity between the generated image and the real image. From the results, it can be considered that the image has relatively good occlusion recovery accuracy.

[0089] The embodiment of the present application proposes a coal mining machine occlusion recovery strategy and corresponding algorithm suitable for coal mining working faces in underground coal mines, which specifically includes the technical steps of using Yolo and FcF-Inpainting algorithms to obtain and repair from the occluded object data set in the application. This step can be used to automatically complete the coal mining machine (or its outline) obscured by dust in the monitoring camera during actual production.

[0090] In one embodiment, if Figure 3As shown, based on the above-mentioned underground coal mining machine obstruction recovery method, the present invention also provides an underground coal mining machine obstruction recovery device, including:

[0091] The first input module 100 is used to input the original image of the coal mining machine to be processed into the trained instance segmentation model to obtain an unobstructed segmented image;

[0092] An image processing module 200 is used to perform image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image;

[0093] A second input module 300, configured to input the pre-processed unoccluded segmented image into a trained image restoration model to obtain a restored image;

[0094] The image restoration module 400 is used to obtain an occluded target image based on the original image of the coal mining machine and the unoccluded segmented image, obtain a repair position image in the repaired image, and replace the occluded target image in the original image of the coal mining machine with the repair position image to obtain a complete image of the coal mining machine.

[0095] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0096] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0097] When the processor 502 executes the program, the underground coal mining machine occlusion recovery method provided in the above embodiment is implemented.

[0098] Furthermore, the terminal further includes:

[0099] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0100] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0101] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0103] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0104] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0105] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned method for recovering coal mining machines from obstruction in an underground mine is implemented.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0110] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0114] In summary, the present invention discloses a method, device, terminal and storage medium for recovering coal mining machines in a mine. The method comprises: inputting the original image of the coal mining machine to be processed into a trained instance segmentation model to obtain an unobstructed segmented image; performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image; inputting the preprocessed unobstructed segmented image into a trained image restoration model to obtain a restored image; obtaining an occluded target image according to the original image of the coal mining machine and the unobstructed segmented image, obtaining a restored position image in the restored image, and replacing the occluded target image in the original image of the coal mining machine with the restored position image to obtain a complete image of the coal mining machine. The present invention improves the accuracy of virtual restoration of the obstructed part of the coal mining machine on the image by first inputting the original image of the coal mining machine into the instance segmentation model, performing image segmentation processing, and then inputting it into the image restoration model to perform image restoration, thereby avoiding the problem of inaccurate restored images caused by using semantic segmentation methods or image restoration methods alone.

[0115] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for recovering coal mining machines in a mine, characterized in that: The method comprises: Input the original image of the coal mining machine to be processed into the trained instance segmentation model to obtain an unoccluded segmented image; Performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image; Inputting the preprocessed unoccluded segmented image into a trained image restoration model to obtain a restored image; An occluded target image is obtained according to the original image of the coal mining machine and the unoccluded segmented image, a repair position image is obtained in the repaired image, and the occluded target image in the original image of the coal mining machine is replaced by the repair position image to obtain a complete image of the coal mining machine.

2. The method for recovering coal mining machines from obstruction in a mine according to claim 1, characterized in that: Before performing image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain the preprocessed unobstructed segmented image, the method further includes: Graying, contour detection and coal shearer area extraction are performed on the original image of the coal shearer to obtain a preprocessed original image of the coal shearer.

3. The method for recovering a coal mining machine from obstruction in a mine according to claim 1, characterized in that: The instance segmentation model is a YOLOv8-seg network, and the image restoration model is a FcF-Inpainting network.

4. The method for recovering a coal mining machine from obstruction in a mine according to claim 3, characterized in that: The training steps of the instance segmentation model include: Acquire a training data set, wherein the training data set includes a plurality of training images and labels corresponding to each training image, wherein the training images include: unobstructed coal mining machine training images and obstructed coal mining machine training images; A YOLOv8-seg network is obtained, and based on the unobstructed coal mining machine training image, the obstructed coal mining machine training image and corresponding labels, instance segmentation training is performed on the YOLOv8-seg network to obtain a trained instance segmentation model.

5. The method for recovering coal mining machines from obstruction in a mine according to claim 3, characterized in that: The training steps of the image restoration model include: Acquire a training data set, wherein the training data set includes a plurality of training images and labels corresponding to each training image, wherein the training images include: unobstructed coal mining machine training images and obstructed coal mining machine training images; Obtain an FcF-Inpainting network, and train the FcF-Inpainting network based on the unobstructed shearer training image, the obstructed shearer training image, and a randomly generated mask to obtain a trained image restoration model.

6. The method for recovering a coal mining machine from obstruction in a mine according to claim 4 or 5, characterized in that: The step of acquiring the unobstructed coal mining machine training image comprises: Collect the first motion video of the coal mining machine in an unobstructed scene; Using OpenCV to capture several frames in the first motion video, to obtain several unobstructed scene images; The plurality of unobstructed scene images are scaled to a preset size, and the edges are filled with black pixels to obtain a plurality of unobstructed coal mining machine training images.

7. The method for recovering a coal mining machine from obstruction in a mine according to claim 4 or 5, characterized in that: The step of obtaining the occluded coal mining machine training image comprises: Collect the second motion video of the coal mining machine in the occlusion scene; intercepting several frames in the second motion video through OpenCV to obtain several occlusion scene images; After the occluded scene image is subjected to an image defogging operation, graying, contour detection and coal mining machine area extraction are performed to obtain an occluded coal mining machine training image.

8. A coal mining machine shielding recovery device in a mine, characterized in that: The device comprises: The first input module is used to input the original image of the coal mining machine to be processed into the trained instance segmentation model to obtain an unobstructed segmented image; An image processing module is used to perform image processing on the preprocessed original image of the coal mining machine and the unobstructed segmented image to obtain a preprocessed unobstructed segmented image; A second input module, used for inputting the preprocessed unoccluded segmented image into a trained image restoration model to obtain a restored image; The image restoration module is used to obtain an occluded target image based on the original image of the coal mining machine and the unoccluded segmented image, obtain a repair position image in the repaired image, and replace the occluded target image in the original image of the coal mining machine with the repair position image to obtain a complete image of the coal mining machine.

9. A terminal, characterized in that: include: A memory, a processor, and an underground coal mining machine obstruction recovery program stored in the memory and executable on the processor. When the underground coal mining machine obstruction recovery program is executed by the processor, the steps of the underground coal mining machine obstruction recovery method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the underground coal mining machine obstruction recovery method as described in any one of claims 1 to 7.

Images