Multi-feature fusion underground wall caving identification method

Abstract

The invention provides a multi-feature fusion underground wall caving recognition method, which comprises the following steps: acquiring a current image of a first region of a fully mechanized face, recognizing foreground pixels of a region of interest of the current image through a Gaussian mixture model, and acquiring a foreground image of the current image; performing connected domain analysis on the foreground image, and judging whether at least one suspected rib spalling frame exists or not; when at least one suspected rib caving frame exists, determining a target suspected rib caving frame based on the operation state of each suspected rib caving frame in the at least one suspected rib caving frame and whether the suspected rib caving frame coincides with a target object frame generated by a target structure of the fully-mechanized face; predicting the current image through a mixed model of a trained convolutional neural network CNN and a visual converter ViT to obtain coordinate information and confidence information of at least one large coal image; based on the coordinate information and the confidence degree information of the at least one large coal image and the target suspected wall caving frame, obtaining an identification result of the underground wall caving; and whether the wall caving phenomenon occurs on the fully-mechanized face can be accurately determined.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a multi-feature fusion downhole chip recognition method. Background Art

[0002] During the process of coal cutting by coal cutting machines in fully mechanized mining faces in coal mines, spalling incidents often occur. The so-called "coal mine spalling" refers to the phenomenon in mining engineering whereby the pressure is transferred to the coal wall due to the overhang in the rock layer of the roof of the goaf area of ​​the mining face. The pressure on the coal wall increases, forming a pressurized area. Under the action of the additional load, part of the coal is crushed and squeezed into the goaf. This phenomenon is called spalling. Among them, spalling can easily cause the following two hazards: 1. Spalling can easily cause collapse accidents: spalling can cause the exposed area of ​​the roof to increase, the amount of roof subsidence to increase, and it can easily cause roof collapse accidents. 2. Casualties or equipment damage: spalling destroys the safety of shafts and tunnels, sometimes causing casualties or damage to mining equipment, affecting normal production.

[0003] Therefore, in order to prevent the occurrence of coal mine disasters, it is necessary to provide a method for identifying the spalling phenomenon. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the purpose of this application is to propose a multi-feature fusion downhole spalling identification method to achieve downhole spalling identification.

[0006] To achieve the above purpose, the present application embodiment proposes a multi-feature fusion downhole chip recognition method, including:

[0007] Acquire a current image of a first area of ​​a fully mechanized mining face, where the first area is an area where spalling is likely to occur;

[0008] Identifying foreground pixels of a region of interest of the current image by using a mixed Gaussian model to obtain a foreground image of the current image;

[0009] Performing a connected domain analysis on the foreground image to determine whether there is at least one suspected chip frame; and when there is at least one suspected chip frame, determining a target suspected chip frame based on the running state of each suspected chip frame in the at least one suspected chip frame and whether it overlaps with a target object frame generated by a target structure of the fully mechanized mining face;

[0010] Predicting the current image through a hybrid model of a trained convolutional neural network (CNN) and a visual converter (ViT) to obtain coordinate information and confidence information of at least one large coal image;

[0011] Based on the coordinate information and confidence information of the at least one large coal image and the target suspected coal spalling frame, an underground coal spalling recognition result is obtained.

[0012] In some implementations, the step of identifying foreground pixels of a region of interest of the current image by using a mixed Gaussian model to obtain a foreground image of the current image comprises:

[0013] For each pixel in the region of interest in the current image, obtain multiple mixed Gaussian models of the pixel and its neighboring pixels;

[0014] Matching the pixel with the multiple mixed Gaussian models, if the number of matches is less than a matching threshold, determining that the pixel is a foreground pixel;

[0015] The obtained multiple foreground pixels of the region of interest in the current image constitute a foreground image of the current image.

[0016] In some implementations, the performing a connected domain analysis on the foreground image to determine whether there is at least one suspected frame includes:

[0017] Acquire a foreground binary image of the foreground image;

[0018] Performing a connected domain analysis on the foreground binary image to determine whether there is a foreground target frame that meets preset conditions;

[0019] If there is a foreground target frame that meets the preset conditions, the foreground target frame is determined as a suspected film edge frame.

[0020] In some implementations, determining a target suspected spalling frame based on the operating state of each suspected spalling frame in the at least one suspected spalling frame and whether it overlaps with a target object frame generated by a target structure of the fully mechanized mining face comprises:

[0021] For each suspected film side frame in the at least one suspected film side frame, obtaining trajectory information of the suspected film side frame based on a plurality of frames of images preceding the current image;

[0022] Based on the trajectory information of the suspected film edge frame, determining the state of the suspected film edge frame; wherein the state includes a moving state and a stopped state;

[0023] When the state of the suspected spalling frame is in the stopped state, determining whether the suspected spalling frame overlaps with a target object frame generated by a target structure of the fully mechanized mining face;

[0024] When the suspected spall frame does not overlap with the target object frame generated by the target structure of the fully mechanized mining face, the suspected spall frame is determined to be the target suspected spall frame.

[0025] In some implementations, the step of obtaining the trajectory information of the suspected film edge frame based on a plurality of frames of images preceding the current image includes:

[0026] The trajectory information of the suspected film edge frame is obtained based on the previous multiple frames of the current image by combining the Kalman filter algorithm with the Hungarian matching algorithm.

[0027] In some implementations, the determining the state of the suspected film edge frame based on the trajectory information of the suspected film edge frame includes:

[0028] Based on the trajectory information of the suspected film edge frame, a position mean and a variance of the suspected film edge frame are obtained;

[0029] Determine whether the position of the suspected frame in the current image minus the mean is less than the variance;

[0030] When the position of the suspected film edge frame in the current image minus the mean is smaller than the variance, it is determined that the state of the suspected film edge frame is a static state.

[0031] In some implementations, the target structure includes a rock guard plate and a coal cutter, and the step of determining whether the suspected rock spalling frame coincides with a target object frame generated by the target structure of the fully mechanized mining face includes:

[0032] Identify a first target object frame of the guard plate of the fully mechanized mining face by using a trained first convolutional neural network;

[0033] Identify a second target object frame of the coal cutter of the fully mechanized mining face by using a trained second convolutional neural network;

[0034] It is determined whether the suspected film support frame overlaps with the first target object frame or the second target object frame.

[0035] In some implementations, the current image is predicted by a hybrid model of a trained convolutional neural network (CNN) and a visual converter (ViT) to obtain coordinate information and confidence information of at least one large coal image; including:

[0036] Inputting the current image into the convolutional neural network (CNN) of the hybrid model to obtain local features encoded by the CNN;

[0037] Divide the current image into blocks and input them into the visual converter ViT of the hybrid model to obtain the global features encoded by ViT;

[0038] The feature fusion layer of the hybrid model acquires fusion features by fusing the local features and the global features to obtain a feature image with the same size as the current image;

[0039] The hybrid model generates five channels based on the feature image; four channels of the five channels include coordinate information of the at least one large lump of coal image, and the remaining channel of the five channels generates a confidence coefficient of the at least one large lump of coal image.

[0040] In some implementations, obtaining the identification result of underground coal spalling based on the coordinate information and confidence information of the at least one large coal image and the target suspected coal spalling frame includes:

[0041] Determine, from the at least one large coal image, an initial target large coal image having confidence information greater than a confidence threshold;

[0042] Based on the coordinate information of the initial target large lump coal image, it is determined whether there is a large lump coal image in the initial target large lump coal image that overlaps with the target suspected coal slag frame;

[0043] When there is a large coal lump image that overlaps with the target suspected coal spalling frame in the initial target large coal lump image, it is identified that underground coal spalling occurs.

[0044] The multi-feature fusion underground rock spalling identification method provided in this application can accurately identify whether rock spalling occurs in the fully mechanized mining face, thereby preventing further coal mine disasters from occurring.

[0045] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 A schematic flow chart of a multi-feature fusion downhole spalling identification method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0049] The following describes the multi-feature fusion downhole chip identification method of an embodiment of the present application with reference to the accompanying drawings.

[0050] Figure 1 The following is a flow chart of a multi-feature fusion downhole chip identification method provided in an embodiment of the present application. Figure 1As shown, the multi-feature fusion downhole chip recognition method includes the following steps:

[0051] Step S101, obtaining a current image of a first area of ​​a fully mechanized mining face, where the first area is an area where spalling is likely to occur.

[0052] As an implementation method, a camera installed at a position opposite to the comprehensive mining face is used to obtain the current image of the first area of ​​the comprehensive mining face. A camera is installed opposite to the comprehensive mining face, and the area prone to spalling is displayed in the picture captured by the camera.

[0053] Step S102, identifying foreground pixels of the region of interest of the current image by using a mixed Gaussian model, and obtaining a foreground image of the current image.

[0054] First of all, it should be explained that after the camera is fixed opposite to the fully mechanized mining face, a region of interest (ROI) will be drawn in the picture captured by the camera.

[0055] As an implementation method, a method for obtaining a foreground image of the current image by identifying foreground pixels of a region of interest of the current image through a mixed Gaussian model includes:

[0056] For each pixel in the region of interest in the current image, multiple mixed Gaussian models of the pixel and its neighboring pixels are obtained;

[0057] Match the pixel with multiple mixed Gaussian models. If the number of matches is less than the matching threshold, the pixel is judged to be a foreground pixel.

[0058] The obtained multiple foreground pixels of the region of interest in the current image constitute a foreground image of the current image.

[0059] As an implementation method, Gaussian background modeling is performed on the region of interest, and three background mixed Gaussian models are established for each pixel in the region of interest in the Gaussian background modeling; the pixels of the region of interest of the current image are matched with the three background mixed Gaussian models and the background mixed Gaussian models of the pixels adjacent to the pixel. If the match is successful, the pixel is a background pixel, and if the match fails, the pixel is a foreground pixel.

[0060] For example, the 3*3 neighborhood of the current pixel is also taken into account, so there are a total of 3*3*3 mixed Gaussian models. Here, a matching threshold number M is set, and the new pixel of the current image is matched with these 27 mixed Gaussian models. If the number of matches exceeds M, it is considered to be a background pixel. If the number of matches is less than or equal to M, it is considered to be a foreground pixel.

[0061] Since the color of the coal wall is very similar to the color of the coal wall of the large piece of coal after it falls off, this embodiment also adds the neighborhood information of the current pixel, so that through the improved mixed Gaussian model, it is possible to use spatial information to determine whether the pixel of the current image is a foreground pixel, so as to adapt to the detection of large pieces of coal falling off.

[0062] Step S103, performing connected domain analysis on the foreground image to determine whether there is at least one suspected chip frame; and when there is at least one suspected chip frame, determining the target suspected chip frame based on the operating status of each suspected chip frame in the at least one suspected chip frame and whether it overlaps with the target object frame generated by the target structure of the comprehensive mining face.

[0063] As an implementation method, a method for performing a connected domain analysis on a foreground image to determine whether there is at least one suspected film-joint frame; comprising: obtaining a foreground binary image of the foreground image; performing a connected domain analysis on the foreground binary image to determine whether there is a foreground target frame that meets preset conditions; if there is a foreground target frame that meets the preset conditions, determining the foreground target frame as a suspected film-joint frame.

[0064] It can be understood that after acquiring the foreground image, foreground detection analysis is performed on the region of interest ROI. Once the spalling phenomenon occurs, a foreground target frame will be generated in the foreground binary image through the connected domain analysis. This foreground target frame is considered to be a suspected spalling frame generated after a large piece of coal falls off the coal wall.

[0065] As an implementation method, based on the running status of each suspected spalling frame in at least one suspected spalling frame and whether it overlaps with the target object frame generated by the target structure of the fully mechanized mining face, determining the target suspected spalling frame; including:

[0066] For each suspected chip-joint frame in at least one suspected chip-joint frame, the trajectory information of the suspected chip-joint frame is obtained based on the previous multiple frames of the current image; based on the trajectory information of the suspected chip-joint frame, the state of the suspected chip-joint frame is judged; wherein the state includes a moving state and a stopped state; when the state of the suspected chip-joint frame is a stopped state, it is judged whether the suspected chip-joint frame coincides with the target object frame generated by the target structure of the comprehensive mining face; when the suspected chip-joint frame does not coincide with the target object frame generated by the target structure of the comprehensive mining face, the suspected chip-joint frame is determined to be the target suspected chip-joint frame.

[0067] It can be understood that in order to analyze whether a suspected spalling frame actually has large pieces of coal falling off, this embodiment uses a target tracking strategy. The main purpose of tracking is to determine whether the suspected spalling frame is in a static state or in motion through the trajectory of the center point of the suspected spalling frame. If it is in a static state, it is very likely that spalling has occurred; if it is in motion, it is believed that the suspected spalling frame corresponds to a large piece of coal, and the large piece of coal is still in motion.

[0068] As an implementation method, the trajectory information of the suspected film edge frame is obtained based on the previous multiple frames of the current image; including: using a Kalman filter algorithm combined with a Hungarian matching algorithm to obtain the trajectory information of the suspected film edge frame based on the previous multiple frames of the current image.

[0069] It can be understood that in the tracking algorithm, the Kalman filter algorithm is first used to predict the position of the current suspected film frame in the previous frame, and then a unique ID match is made with the position of the suspected film frame in the current image through the Hungarian matching algorithm, thereby realizing the tracking process of the suspected film frame. The combination of the Kalman filter algorithm and the Hungarian matching algorithm to realize target tracking belongs to the existing technology and will not be described in detail here.

[0070] As an implementation method, the state of the suspected film-joint frame is determined based on the trajectory information of the suspected film-joint frame; including: obtaining the position mean and variance of the suspected film-joint frame based on the trajectory information of the suspected film-joint frame; determining whether the position of the suspected film-joint frame in the current image minus the mean is less than the variance; when the position of the suspected film-joint frame in the current image minus the mean is less than the variance, determining that the state of the suspected film-joint frame is a stationary state.

[0071] Exemplarily, the first thirty frames of the current image are obtained, and for the current suspected chip-joint frame, the mean and variance of the center position of the suspected chip-joint frame are calculated according to the motion trajectory of the current suspected chip-joint frame in the thirty frames. If the position of the current frame minus the mean is less than the variance, it is considered that the current suspected chip-joint frame is in a stationary state and is likely to be a chip. If it is greater than the variance, it is considered that the suspected chip-joint frame corresponds to a large piece of coal, which is still in motion.

[0072] In some embodiments, the target structure includes a side guard plate and a coal cutter, and determining whether a suspected side guard frame coincides with a target object frame generated by the target structure of the comprehensive mining face comprises: identifying a first target object frame of the side guard plate of the comprehensive mining face by a trained first convolutional neural network; identifying a second target object frame of the coal cutter of the comprehensive mining face by a trained second convolutional neural network; and determining whether the suspected side guard frame coincides with the first target object frame or the second target object frame.

[0073] It can be understood that after a suspected rock spalling frame is generated, due to the complexity of the scene inside the fully mechanized mining face of the coal mine, there is both movement information of the rock guard plate and movement information of the coal cutter. These two types of information can easily lead to false alarms. Therefore, it is necessary to use these two markers in the fully mechanized mining face for annotation, and use the convolutional neural network CNN for training to generate convolutional neural networks corresponding to different markers; for example, for the recognition of the rock guard plate, the first convolutional neural network is trained; for the recognition of the coal cutter, the second convolutional neural network is trained; and when these two markers are detected by the first convolutional neural network and the second convolutional neural network, if a suspected rock spalling frame is generated at this time, and this suspected rock spalling frame coincides with the frame identified by the first convolutional neural network or the second convolutional neural network, it is considered to be a false alarm, and no rock spalling phenomenon has occurred, and no alarm is required.

[0074] Step S104, predicting the current image through a hybrid model of the trained convolutional neural network CNN and the visual converter ViT, to obtain coordinate information and confidence information of at least one large coal image.

[0075] It can be understood that in order to further verify whether a large piece of coal has really fallen off the suspected spalling frame, the present invention further determines whether a large piece of coal has been produced in the suspected spalling frame. If a large piece of coal is really produced, the spalling is considered to have really occurred. Since the color of the large piece of coal is very similar to the color of the surrounding small pieces of coal, in order to solve this problem, the present invention uses a hybrid model that combines a convolutional neural network CNN and a visual converter ViT. The hybrid model aims to make full use of the respective advantages of CNN and ViT, especially in the task of classifying coals of similar colors, by finding category-specific decision boundaries to improve performance. CNN extracts these detailed information by using convolutional layers.

[0076] It should be noted that the convolution layer is a mathematical operation that helps the model focus on local features in the image, such as edges, corners, and textures. Therefore, a convolutional neural network CNN has a limitation, that is, it is only good at capturing local information of the image, but not good at understanding the global structure or overall background of the entire image. This means that when dealing with more complex visual tasks, especially when it is necessary to understand how different parts of the entire image are related to each other, the performance of CNN may not be as expected. In order to overcome the limitations of CNN, the present invention introduces a visual converter ViT, which is a model that uses a self-attention mechanism to process images. Simply put, the self-attention mechanism is a method that allows the model to "pay attention" to different parts of the image and understand the relationship between them. Unlike CNN, ViT does not understand the image through local features, but looks at the image in a global way, which can better understand the global context of the image and the relationship between different parts. Therefore, CNN and ViT are combined to construct a hybrid model that can simultaneously utilize the local feature extraction ability of CNN and the global feature capture ability of ViT to perform well in different types of data and tasks.

[0077] As an implementation method, a method for predicting a current image through a hybrid model of a trained convolutional neural network CNN and a visual converter ViT to obtain coordinate information and confidence information of at least one large lump of coal image includes: inputting the current image into the convolutional neural network CNN of the hybrid model to obtain local features encoded by CNN; dividing the current image into blocks and inputting the visual converter ViT of the hybrid model to obtain global features encoded by ViT; acquiring a feature fusion layer of the hybrid model by fusing local features and global features to obtain a fused feature to obtain a feature image of the same size as the current image; the hybrid model generates five channels based on the feature image; four of the five channels include coordinate information of at least one large lump of coal image, and the remaining channels of the five channels generate a confidence coefficient of at least one large lump of coal image.

[0078] It can be understood that the current image is input into two networks at the same time, one network is CNN, and the other network is ViT, and the networks CNN and ViT are in parallel. Among them, the image sent to the ViT network needs to be divided into blocks, such as 16*16, to form a feature vector. The feature layer generated after encoding by the CNN and ViT networks re-enters the feature fusion. The information in the feature fusion includes both the local information encoded by CNN and the ViT global context index information. After the feature fusion layer, upsampling is performed to achieve the same size as the original input image, generating a total of 5 channels. The first four channels contain the coordinate positions of large pieces of coal, and the fifth channel generates the confidence coefficient of large pieces of coal. By judging the confidence coefficient, it can be determined whether a piece of coal is generated. For example, the target box corresponding to the coordinate position with a confidence coefficient exceeding 8 is judged to have a piece of coal.

[0079] Step S105, based on the coordinate information and confidence information of at least one large coal image and the target suspected coal spalling frame, obtain the recognition result of the underground coal spalling.

[0080] It can be understood that whether chipping occurs is determined by two methods, and when chipping occurs is identified by both methods, it is determined that chipping occurs and an alarm is triggered.

[0081] As an implementation method, based on the coordinate information and confidence information of at least one large lump of coal image and a target suspected lump of coal frame, an identification result of underground lump of coal is obtained; including: determining an initial target large lump of coal image whose confidence information is greater than a confidence threshold from at least one large lump of coal image; based on the coordinate information of the initial target large lump of coal image, judging whether there is a large lump of coal image in the initial target large lump of coal image that overlaps with the target suspected lump of coal frame; when there is a large lump of coal image in the initial target large lump of coal image that overlaps with the target suspected lump of coal frame, identifying that underground lump of coal has occurred.

[0082] The multi-feature fusion underground coal spalling identification method of the embodiment of the present application uses two methods to identify whether coal spalling occurs based on the image of the fully-mechanized mining face. One method is to obtain a suspected coal spalling frame by performing a connected domain analysis on the acquired foreground image. Based on the operating status of the suspected coal spalling frame and excluding the misjudgment of the target object, it is identified whether large pieces of coal have indeed fallen off the suspected coal spalling frame. The other method is to identify the image of the fully-mechanized mining face through a hybrid model of a convolutional neural network and a visual converter to determine whether there are image features of large pieces of coal falling off. The identification results of the two methods are then combined to identify whether coal spalling occurs. It is possible to accurately identify whether coal spalling occurs in the fully-mechanized mining face, thereby preventing further coal mine disasters.

[0083] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0085] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0086] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0087] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0088] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0090] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-feature fusion downhole spalling identification method, characterized in that: The following steps are involved: Acquire a current image of a first area of ​​a fully mechanized mining face, where the first area is an area where spalling is likely to occur; Identifying foreground pixels of a region of interest of the current image by using a mixed Gaussian model to obtain a foreground image of the current image; Performing a connected domain analysis on the foreground image to determine whether there is at least one suspected chip frame; and when there is at least one suspected chip frame, determining a target suspected chip frame based on the running state of each suspected chip frame in the at least one suspected chip frame and whether it overlaps with a target object frame generated by a target structure of the fully mechanized mining face; Predicting the current image through a hybrid model of a trained convolutional neural network (CNN) and a visual converter (ViT) to obtain coordinate information and confidence information of at least one large coal image; Based on the coordinate information and confidence information of the at least one large coal image and the target suspected coal spalling frame, an identification result of underground coal spalling is obtained.

2. The method according to claim 1, characterized in that The method of identifying foreground pixels of the region of interest of the current image by using a mixed Gaussian model to obtain a foreground image of the current image comprises: For each pixel in the region of interest in the current image, obtain multiple mixed Gaussian models of the pixel and its neighboring pixels; Matching the pixel with the multiple mixed Gaussian models, if the number of matches is less than a matching threshold, determining that the pixel is a foreground pixel; The obtained multiple foreground pixels of the region of interest in the current image constitute a foreground image of the current image.

3. The method according to claim 1, characterized in that The method of performing a connected domain analysis on the foreground image to determine whether there is at least one suspected frame; comprises: Acquire a foreground binary image of the foreground image; Performing a connected domain analysis on the foreground binary image to determine whether there is a foreground target frame that meets preset conditions; If there is a foreground target frame that meets the preset conditions, the foreground target frame is determined as a suspected film edge frame.

4. The method according to claim 1, characterized in that: The step of determining a target suspected spalling frame based on the running state of each suspected spalling frame in the at least one suspected spalling frame and whether it overlaps with a target object frame generated by a target structure of the fully mechanized mining face comprises: For each suspected film side frame in the at least one suspected film side frame, obtaining trajectory information of the suspected film side frame based on a plurality of frames of images preceding the current image; Based on the trajectory information of the suspected film edge frame, determining the state of the suspected film edge frame; wherein the state includes a moving state and a stopped state; When the state of the suspected spalling frame is in the stopped state, determining whether the suspected spalling frame overlaps with a target object frame generated by a target structure of the fully mechanized mining face; When the suspected spall frame does not overlap with the target object frame generated by the target structure of the fully mechanized mining face, the suspected spall frame is determined to be the target suspected spall frame.

5. The method according to claim 4, characterized in that The step of obtaining the trajectory information of the suspected film edge frame based on the previous multiple frames of the current image comprises: The trajectory information of the suspected film edge frame is obtained based on the previous multiple frames of the current image by combining the Kalman filter algorithm with the Hungarian matching algorithm.

6. The method according to claim 4, characterized in that The step of determining the state of the suspected film edge frame based on the trajectory information of the suspected film edge frame comprises: Based on the trajectory information of the suspected film edge frame, a position mean and a variance of the suspected film edge frame are obtained; Determine whether the position of the suspected frame in the current image minus the mean is less than the variance; When the position of the suspected film edge frame in the current image minus the mean is smaller than the variance, it is determined that the state of the suspected film edge frame is a static state.

7. The method according to claim 4, characterized in that The target structure includes a rock guard plate and a coal cutter, and the step of judging whether the suspected rock spalling frame coincides with a target object frame generated by the target structure of the fully mechanized mining face includes: Identify a first target object frame of the guard plate of the fully mechanized mining face by using a trained first convolutional neural network; Identify a second target object frame of the coal cutter of the fully mechanized mining face by using a trained second convolutional neural network; It is determined whether the suspected film support frame overlaps with the first target object frame or the second target object frame.

8. The method according to claim 1, characterized in that The obtaining of the current image of the first area of ​​the fully mechanized mining face comprises: The current image of the first area of ​​the fully mechanized mining face is obtained by a camera installed at a position opposite to the fully mechanized mining face.

9. The method according to claim 1, characterized in that: The method predicts the current image by using a hybrid model of a trained convolutional neural network (CNN) and a visual converter (ViT) to obtain coordinate information and confidence information of at least one large coal image; comprising: Inputting the current image into the convolutional neural network (CNN) of the hybrid model to obtain local features encoded by the CNN; Divide the current image into blocks and input them into the visual converter ViT of the hybrid model to obtain the global features encoded by ViT; The feature fusion layer of the hybrid model acquires fusion features by fusing the local features and the global features to obtain a feature image with the same size as the current image; The hybrid model generates five channels based on the feature image; four channels of the five channels include coordinate information of the at least one large lump of coal image, and the remaining channel of the five channels generates a confidence coefficient of the at least one large lump of coal image.

10. The method according to claim 1, characterized in that The step of obtaining the identification result of underground coal spalling based on the coordinate information and confidence information of the at least one large coal image and the target suspected coal spalling frame comprises: Determine, from the at least one large coal image, an initial target large coal image having confidence information greater than a confidence threshold; Based on the coordinate information of the initial target large lump coal image, it is determined whether there is a large lump coal image in the initial target large lump coal image that overlaps with the target suspected coal slag frame; When there is a large coal lump image that overlaps with the target suspected coal spalling frame in the initial target large coal lump image, it is identified that underground coal spalling occurs.

Images