Coal mine production control method and system based on law enforcement recorder, electronic equipment and storage medium

Abstract

The invention relates to the technical field of coal mine production, in particular to a coal mine production control method and system based on a law enforcement recorder, electronic equipment and a storage medium, and the method comprises the steps: obtaining video data of a target production area under different visual angles, and converting the video data into a YIQ color space; performing three-dimensional semantic segmentation on the video data converted into the YIQ color space, and extracting a three-dimensional semantic segmentation result; constructing a production control model based on a three-dimensional semantic segmentation result; and analyzing the space-time relationship among different semantic objects by using the production control model, extracting a control strategy of the target production area and issuing the control strategy to underground workers. The objective of the invention is to realize accurate and intelligent control for coal mine production, reduce the influence caused by uncertain factors in the generation of a coal mine production control strategy, and improve the accuracy of the control strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mine production, and in particular to a coal mine production control method, system, electronic equipment and storage medium based on a law enforcement recorder. Background Art

[0002] In the field of coal mining production, the coal mining machine is a core equipment, and its operating efficiency and safety are directly related to the benefits of the entire coal mining operation. The coal mining machine control strategy is the key to ensuring the efficient and stable operation of the coal mining machine; accurate control strategy is crucial to ensure the safety of coal mining operations, and can avoid safety accidents such as cutting drum overload and traction unit failure caused by improper operation of the coal mining machine, ensuring the safety of underground workers and the continuity of coal mining production.

[0003] At present, the traditional coal mining machine control strategy generation method is mainly based on empirical formulas and preset parameters. For example, according to the average hardness of the coal seam and the expected output, the fixed cutting speed and traction speed of the coal mining machine are set in advance, and the upper limit of the cutting depth is determined according to the design load-bearing capacity of the coal mining machine. Although this empirical method can ensure the normal operation of the coal mining machine to a certain extent, it lacks adaptability to the real-time changes in the coal seam; especially in the face of the complex and changeable production environment of coal mines, when the method based on empirical formulas and preset parameters is used, due to the natural unevenness of the geological conditions of the coal seam, the actual coal seam hardness, thickness and interlayer distribution may have a large deviation from the preset situation, making the preset control strategy unable to meet the actual mining needs; affected by these uncertain factors, the accuracy and effectiveness of the existing coal mining machine control strategy are restricted. Summary of the invention

[0004] In order to achieve precise and intelligent control of coal mining machines, reduce the impact of uncertainty factors in the generation of coal mining machine control strategies, and improve the accuracy of control strategies, the present invention provides a coal mine production control method, system, electronic equipment and storage medium based on law enforcement recorders. The technical solutions adopted are as follows:

[0005] The technical solution of the first aspect of the present invention provides a coal mine production control method based on a law enforcement recorder, the method comprising:

[0006] Obtain video data from different viewing angles of the target production area and convert it to the YIQ color space;

[0007] Perform 3D semantic segmentation on the video data converted to the YIQ color space and extract the 3D semantic segmentation results;

[0008] Build a production control model based on 3D semantic segmentation results;

[0009] The production control model is used to analyze the spatiotemporal relationship between different semantic objects and extract the control strategy of the target production area.

[0010] Furthermore, obtaining video data of the target production area at different viewing angles and converting it into a YIQ color space includes:

[0011] Obtain the side video data of the coal mining machine and the video data of the bottom of the coal seam after cutting and perform preprocessing;

[0012] The preprocessed side video data and bottom video data of the coal mining machine are converted into the YIQ color space respectively.

[0013] Further, obtaining the video data of the side of the coal mining machine and the video data of the bottom of the coal seam after cutting and preprocessing includes:

[0014] Preprocess the side video data based on geometric correction perspective transformation, including:

[0015] Based on the initial static state of the coal mining machine, the key feature points on the side of the coal mining machine are marked and the image coordinates without perspective distortion are extracted;

[0016] Solve the perspective transformation matrix based on the least squares method;

[0017] For each frame of the image in the side video data, its perspective transformation matrix is ​​transformed to obtain a coordinate matrix after perspective transformation and reconstruct the side image.

[0018] Furthermore, obtaining the video data of the side of the coal mining machine and the video data of the bottom of the coal seam after cutting and preprocessing it also includes:

[0019] The coal seam bottom video data is preprocessed based on histogram equalization, including:

[0020] Extract the grayscale histogram of the video frame at the bottom of the coal seam;

[0021] Calculate the cumulative distribution function based on the grayscale histogram and perform normalization;

[0022] Based on the normalized cumulative distribution function, the original gray value of each pixel in the video frame of the coal seam bottom is mapped to a new gray value;

[0023] The linear stretching method is used to enhance the contrast of the entire image based on the mapped new grayscale values.

[0024] Furthermore, the video data converted to the YIQ color space is subjected to three-dimensional semantic segmentation, and the three-dimensional semantic segmentation results are extracted, including:

[0025] Use 3D convolutional neural network to build a 3D semantic segmentation model and extract the features of video data in the spatiotemporal dimension;

[0026] For the feature graph output by each convolutional layer, the spatial attention weight is calculated based on the relative importance and mutual correlation of each component in the spatial layout of the coal mining machine, and the temporal attention weight is calculated based on the phased nature of the coal mining machine's workflow and the characteristics of the coal seam changing over time during the mining process;

[0027] Annotate the preprocessed video data of the side and bottom of the coal mining machine and train a 3D semantic segmentation model;

[0028] The real-time video data is input into the trained 3D semantic segmentation model. Based on the spatiotemporal features and semantic information learned during the training process, the semantic category of each pixel in 3D space-time is output to obtain a complete 3D semantic segmentation result.

[0029] Furthermore, constructing a production control model based on the 3D semantic segmentation results includes:

[0030] Based on the 3D semantic segmentation results, the shearer kinematic model and geological analysis method are combined to extract the shearer state characteristics and coal seam multi-dimensional characteristics respectively.

[0031] Fuzzy sets are designed for the state characteristics of coal mining machine and coal seam characteristics respectively, and membership function is used to perform fuzzy processing on the state characteristics of coal mining machine and coal seam characteristics.

[0032] The multi-layer feedforward neural network is used to process the fuzzified coal mining machine state characteristics and coal seam characteristics, and learn the causal relationship between the fuzzy characteristics.

[0033] A Bayesian network is constructed with the fuzzy characteristics of coal mining machines and coal seams as nodes, and the conditional probability table of each node in the Bayesian network under the condition of its parent node is calculated using the maximum likelihood estimation method.

[0034] Furthermore, the production control model is used to analyze the spatiotemporal relationship between different semantic objects, and the control strategies for the target production area are extracted, including:

[0035] Based on the conditional probability table, analyze the relationship between different semantic objects in the time and space dimensions;

[0036] Perform fuzzy processing based on the results of spatiotemporal relationship analysis and convert them into spatiotemporal relationship parameters;

[0037] The spatiotemporal relationship parameters between different semantic objects after fuzzification are set as state space, the control strategies that can be adopted are set as action space, and the reward function is constructed according to the working efficiency index and equipment status information of the coal mining machine.

[0038] Through reinforcement learning, the optimal action control strategy is selected according to the current state.

[0039] The technical solution of the second aspect of the present invention provides a coal mine production control method system based on a law enforcement recorder, which adopts the coal mine production control method based on a law enforcement recorder described in the technical solution of the first aspect of the present invention, and the system includes:

[0040] A video image acquisition module, configured to acquire video data of a target production area at different viewing angles and convert the video data into a YIQ color space;

[0041] A three-dimensional semantic segmentation module, configured to perform three-dimensional semantic segmentation on the video data converted into the YIQ color space, and extract the three-dimensional semantic segmentation result;

[0042] A production control module configured to build a production control model based on the three-dimensional semantic segmentation results;

[0043] The control strategy generation module is configured to analyze the spatiotemporal relationship between different semantic objects using the production control model and extract the control strategy of the target production area.

[0044] The technical solution of the third aspect of the present invention provides an electronic device, comprising: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the coal mine production control method based on the law enforcement recorder described in the technical solution of the first aspect of the present invention.

[0045] The technical solution of the fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for implementing a coal mine production control method based on a law enforcement recorder. The program for implementing a coal mine production control method based on a law enforcement recorder is executed by a processor to implement the steps of the coal mine production control method based on a law enforcement recorder described in the technical solution of the first aspect of the present invention.

[0046] The present invention has the following beneficial effects:

[0047] The coal mine production control method based on the law enforcement recorder provided by the present invention can accurately distinguish the relevant features of different semantic objects such as various components of the coal mining machine and the coal seam from the complex video data by performing three-dimensional semantic segmentation on the video data after conversion to the YIQ color space and extracting the three-dimensional semantic segmentation results; then, a production control model is constructed based on the three-dimensional semantic segmentation results, and the spatiotemporal relationship between different semantic objects is analyzed to extract the control strategy of the target production area, so as to dynamically and accurately adjust the control strategy of the coal mining machine according to the actual working state of the coal mining machine and the real-time situation of the coal seam, effectively improve the working efficiency of the coal mining machine, make the coal mining machine control more suitable for the complex and changeable production environment of the coal mine, and promote the development of coal mine production towards intelligence and efficiency; on the other hand, the spatiotemporal relationship between different semantic objects is analyzed by using the production control model, and the actual working state of the coal mining machine and the real-time situation of the coal seam are comprehensively considered, which can effectively avoid the uncertainty caused by a single factor or simple relationship judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A method flow chart of a coal mine production control method based on a law enforcement recorder provided by one embodiment of the present invention;

[0050] Figure 2 A schematic structural diagram of a coal mine production control system based on a law enforcement recorder provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the coal mine production control method, system, electronic device and storage medium based on the law enforcement recorder proposed by the present invention, its specific implementation method, structure, characteristics and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0052] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0053] The following is a detailed description of a specific scheme of a coal mine production control method, system, electronic device and storage medium based on a law enforcement recorder provided by the present invention in conjunction with the accompanying drawings.

[0054] See also Figure 1 , which shows a method flow chart of a coal mine production control method based on a law enforcement recorder provided by an embodiment of the present invention, the method comprising:

[0055] Step S100: acquiring video data of the target production area at different viewing angles and converting it into a YIQ color space;

[0056] Step S100 specifically includes:

[0057] Step S110: Obtain the video data of the side of the coal mining machine and the video data of the bottom of the coal seam after cutting and pre-process them; specifically, use the law enforcement recorder with high-definition shooting capability, wide-angle lens and good night vision function, and adjust it to the best angle and height to cover the target area; the geometric correction algorithm can be used to correct the possible perspective distortion; at the same time, in order to adapt to the uneven lighting conditions underground, the automatic exposure or HDR mode of the law enforcement recorder will be enabled, and lighting compensation and contrast enhancement processing will be performed when necessary; in actual use, it is also necessary to ensure that the law enforcement recorder is equipped with sufficient protection measures to resist dust and vibration in the mine, and establish a stable data transmission mechanism for real-time or regular transmission and storage of video data; due to the installation position and angle of the law enforcement recorder, the collected video of the side of the coal mining machine may have perspective distortion. In order to eliminate this distortion, it is necessary to use perspective transformation for geometric correction; due to the uneven lighting underground, the video data of the bottom of the coal seam after the coal mining machine cuts may be locally too bright or too dark, and it is necessary to perform lighting compensation and contrast enhancement;

[0058] Step S111: pre-processing the side video data based on the geometric correction perspective transformation, including: marking the key feature points on the side of the coal mining machine when the coal mining machine is in the initial static state and extracting the coordinates of the image without perspective distortion; specifically, manually marking at least 4 non-collinear feature points on the side of the coal mining machine, including the name, position, motion state (such as rotation, swing, translation, etc.) of each component of the coal mining machine and the connection relationship between the components. These feature points should be distributed at the connection point between the rocker arm and the fuselage of the coal mining machine and the edge point of the cutting drum; according to the actual size and design drawings of the coal mining machine, determine the coordinates of these feature points in the ideal image without perspective distortion; solve the perspective transformation matrix based on the least squares method; for each frame of the image in the side video data, transform its perspective transformation matrix, obtain the coordinate matrix after perspective transformation and reconstruct the side image;

[0059] Step S112: Preprocessing the coal seam bottom video data based on histogram equalization, including: extracting the grayscale histogram of the coal seam bottom video frame, H(k), k represents the grayscale value, H(k) represents the number of pixels with a grayscale value of k; calculating the cumulative distribution function based on the grayscale histogram And normalize it to get N is the total number of pixels in the image; for each pixel (x, y) in the image, its original grayscale value is g(x, y), and the grayscale value after illumination compensation can be expressed as g′(x, y) = N·CDF norm (g(x,y)), and then use the linear stretching method to enhance the contrast of the entire image based on the mapped new grayscale value, define the minimum grayscale value and maximum grayscale value of the image, and the grayscale value g″(x,y) after contrast enhancement can be expressed as: Through illumination compensation and contrast enhancement, the quality of the video data at the bottom of the coal seam can be improved, making the details of the bottom of the coal seam clearer.

[0060] Step S120: convert the pre-processed side video data and bottom video data of the coal mining machine into the YIQ color space respectively; specifically, for the side video data of the coal mining machine, when converting to the YIQ space, analyze the characteristic performance of different parts of the coal mining machine (such as metal shell, connecting parts, etc.) on each YIQ component. Since the color of the coal mining machine parts is relatively single and has a certain metallic luster, focus on the reflection of the Y (brightness) component on the contour and surface details of the parts, and the potential value of the I and Q components in distinguishing different materials or wear areas; for the video data of the bottom of the coal seam, consider the subtle differences in color between the coal seam and gangue in the YIQ space. According to the color characteristics of the coal seam and the possible color changes of the gangue, the YIQ space can be used to better separate the coal seam and gangue information, so as to provide more favorable color data for subsequent semantic segmentation.

[0061] Step S200: performing three-dimensional semantic segmentation on the video data converted into the YIQ color space, and extracting the three-dimensional semantic segmentation result;

[0062] Step S200 specifically includes:

[0063] Step S210: construct a three-dimensional semantic segmentation model using a three-dimensional convolutional neural network, and extract the features of the video data in the spatiotemporal dimension; specifically, when constructing the three-dimensional convolutional neural network, the network parameters such as the depth, convolution kernel size and step length are determined according to the spatiotemporal continuity of the movement of the coal mining machine and the spatial distribution law of the geological characteristics of the coal seam; a first branch is set in the network to extract the features of the key components of the coal mining machine, such as the rotation features of the cutting drum and the swing angle features of the rocker arm, and then the motion mode and spatiotemporal relationship of the components are learned by analyzing the position and posture of the components in different time frames using the three-dimensional convolutional neural network; at the same time, a feature extraction path is configured for the video data of the bottom of the coal seam, and attention is paid to the changes in the thickness of the coal seam and the performance of the geological characteristics of the interlayer distribution in the spatiotemporal dimension, and the texture and structural feature information of the coal seam at different depths and time points is obtained through multi-layer convolution and pooling operations;

[0064] Step S220: For the feature map output by each convolution layer, the spatial attention weight is calculated based on the relative importance and mutual correlation of each component in the spatial layout of the coal mining machine, and the temporal attention weight is calculated based on the phased nature of the work process of the coal mining machine and the characteristics of the coal seam changing over time during the mining process; specifically, when calculating the spatial attention weight, a higher spatial attention weight should be given to the area near the connection point between the cutting drum and the rocker arm, because these areas are crucial for the normal operation and control of the coal mining machine; for the coal seam characteristic map, the spatial attention weight is calculated based on the gradient of the coal seam thickness change and the uneven distribution of interlayers, and a higher weight is given to the area where the coal seam thickness changes dramatically or interlayers appear, so as to better capture the spatial change information of the geological characteristics of the coal seam, for subsequent The semantic segmentation and analysis of the coal mining machine provide key focus areas; on the other hand, the time attention weight is based on the working process of the coal mining machine and the changes in the coal seam. The rotation cycle of the coal mining machine cutting drum and the change cycle of the coal mining machine's propulsion speed are taken into consideration. When calculating the time attention weight, a higher weight is given to the time points related to these motion cycles to highlight the characteristic changes of the coal mining machine in different working stages, which is helpful to identify the working status of the coal mining machine and possible abnormal situations. For the video data of the bottom of the coal seam, the time attention weight is determined according to the time law of the thickness change during the coal seam mining process and the frequency change of the occurrence of gangue. A high weight is given in the time period when the coal seam thickness changes suddenly or the gangue frequently appears, so as to better capture the dynamic changes of the coal seam geological conditions in the time dimension;

[0065] Step S230: annotate the pre-processed video data of the side and bottom of the coal mining machine and train the 3D semantic segmentation model; specifically, based on step S111: manually annotate the feature points of the video data of the side of the coal mining machine, including the name, position, motion state (such as rotation, swing, translation, etc.) of each component of the coal mining machine and the connection relationship between the components; and the coal seam thickness, the position and size of the gangue, and the geological information of the coal seam stratification annotated in the video data of the bottom of the coal seam; consider the characteristics of the coal mining machine and the coal seam data when training the 3D semantic segmentation model, because the coal mining machine components and the coal seam are The proportion of coal seams in video data may vary. In the training process, data balancing technology is used to appropriately oversample coal seam data or undersample coal mining machine component data to avoid overfitting of the model to a certain type of data. For the segmentation of coal mining machine components, the loss weights of component connection points and motion boundaries are increased to improve the segmentation accuracy of these key areas. For coal seam segmentation, the corresponding loss weights are adjusted according to the influence of coal seam thickness measurement errors and misjudgment of gangue, so that the model pays more attention to the accuracy of coal seam feature segmentation that has an important impact on coal mining machine control.

[0066] Step S240: Input the real-time video data into the trained 3D semantic segmentation model, and output the semantic category of each pixel in 3D space-time based on the spatiotemporal features and semantic information learned during the training process, so as to obtain a complete 3D semantic segmentation result; specifically, after inputting the real-time video data of the side and bottom of the coal mining machine into the model, the semantic category output by the model should accurately reflect the working status of the coal mining machine components (such as normal operation, fault warning, etc.) and the geological conditions of the coal seam (such as high-quality coal seam, coal seam containing gangue, etc.). By semantically classifying each pixel, a complete semantic description of the coal mining machine and the coal seam in 3D space-time is constructed, providing detailed and accurate input information for the subsequent production control model.

[0067] Step S300: constructing a production control model based on the 3D semantic segmentation result;

[0068] Step S300 specifically includes:

[0069] Step S310: Based on the three-dimensional semantic segmentation results, the state characteristics of the coal mining machine and the multi-dimensional characteristics of the coal seam are extracted in combination with the kinematic model of the coal mining machine and the geological analysis method; specifically, the motion parameters of each component of the coal mining machine are calculated from the three-dimensional semantic segmentation results, including at least the speed and acceleration of the drum, the rotation speed stability of the cutting drum, the frequency and amplitude changes of the rocker arm swing, and the traction speed and traction force changes of the traction part. By analyzing the time series of these parameters, the motion characteristics of the coal mining machine in different working stages are obtained, which provides a basis for evaluating the working efficiency and stability of the coal mining machine; secondly, the wear of the coal mining machine components is further analyzed. By analyzing the wear of the pick teeth, the wear of the pick teeth is analyzed. Dynamic monitoring of the wear index and analysis of changes in the surface texture of other components, combined with the operating time and load conditions of the coal mining machine, identify possible component wear and fault characteristics. For example, observe the relationship between the degree of wear of the pick and the cutting effect. When the wear reaches a certain threshold and the cutting efficiency is significantly reduced, it is marked as a potential fault point. Specifically, the position information of the coal mining machine cutting drum, rocker arm, traction part, etc. at different time points is obtained from the three-dimensional semantic segmentation results. The coordinates of the center of the cutting drum in three-dimensional space are assumed to be (x1, y1, z1) at time t1 and (x2, y2, z2) at time t2. Calculate its velocity and acceleration, which can be expressed as:

[0070]

[0071] In the formula, v represents the drum speed; a represents the drum acceleration; at the same time, the motion characteristics such as the swing angle change rate of the rocker arm are analyzed to quantify the motion state of the coal mining machine;

[0072] Through the semantic segmentation results of the cutting drum pick, the visibility or integrity of the pick at different time points is counted. Suppose the standard visible area of ​​the pick is S1, and the visible area obtained by semantic segmentation at time t is S t , define the pick wear index, which can be expressed as:

[0073]

[0074] In the formula, w represents the pick wear index. By combining the wear indexes of all picks, the overall wear condition of the picks of the coal mining machine can be obtained.

[0075] Regarding the multi-dimensional characteristics of coal seams, in addition to the thickness of coal seams and the volume proportion of interlayers, the geological parameters such as the hardness distribution of coal seams, the flatness of coal seam stratification, and the contact relationship between coal seams and roof and floor plates are further analyzed. By analyzing the distribution and changes of these parameters in three-dimensional space, the mining difficulty and potential risks of coal seams can be fully understood, and more detailed coal seam condition information can be provided for the formulation of coal mining machine control strategies. Specifically, for the multi-dimensional characteristics of coal seams, the thickness h(x, y) of the coal seams at different positions is first determined based on the three-dimensional semantic segmentation results of the video data at the bottom of the coal seams, and the average change rate of the coal seam thickness in the target space area is calculated, which can be expressed as:

[0076]

[0077] In the formula, Δh R represents the average rate of change of coal seam thickness; R represents the target spatial area;

[0078] Secondly, identify the location, size and distribution of gangue in the coal seam, and calculate the volume percentage of gangue to quantify the impact of gangue on the coal mining process.

[0079] Step S320: Design fuzzy sets for the state characteristics of the coal mining machine and the coal seam characteristics respectively, and use the membership function to fuzzify the state characteristics of the coal mining machine and the coal seam characteristics; use the extracted state characteristics of the coal mining machine and the coal seam characteristics as the input of the fuzzy logic system. For example, for the wear index w of the coal mining machine pick, define the fuzzy sets "low wear", "medium wear", and "high wear", and use the membership function to represent the degree of different fuzzy sets; similarly, fuzzify other characteristics such as coal mining machine speed, acceleration, coal seam thickness change rate, and volume proportion of interlayer gangue; establish fuzzy rules based on the professional knowledge and experience of coal mine production. For example, if the pick wear index is "high wear" and the volume proportion of interlayer gangue in the coal seam is "high", the risk of coal mining machine failure is "high"; for the hardness of the coal seam, it can be divided into fuzzy sets such as "soft coal seam", "medium hard coal seam", and "hard coal seam"; for the change of coal seam thickness, set fuzzy sets such as "stable coal seam", "gradual thickness coal seam", and "sudden thickness change coal seam". The membership functions of these fuzzy sets are determined through geological exploration data and field mining experience to better handle the uncertainty of coal seam characteristics; it should be noted that when formulating fuzzy rules, the interaction between the working state of the coal mining machine and the geological conditions of the coal seam is fully considered. For example, if the coal seam is a "hard coal seam" and the cutting drum of the coal mining machine has a "normal speed" but "moderate wear on the cutting teeth", the failure risk of the coal mining machine is "high". These rules combine the fuzzy characteristics of the coal mining machine and the coal seam, reflecting the operating status and potential problems of the coal mining machine under different working conditions, and providing a basis for subsequent model decision-making.

[0080] Step S330: Use a multi-layer feedforward neural network to process the fuzzified coal mining machine state characteristics and coal seam characteristics, and learn the causal relationship between the fuzzified characteristics; specifically, construct a multi-layer feedforward neural network, the number of nodes in its input layer is the same as the number of fuzzified input characteristics, and the hidden layer preferably has 10-20 neurons and ReLU activation function; the number of nodes in the output layer is determined according to the number of coal mining machine parameters that need to be controlled, such as outputting the optimal cutting speed, cutting depth, traction speed and other control parameters of the coal mining machine; use historical coal mine production data and expert-annotated data, use the fuzzified input characteristics as the input of the neural network, and use the corresponding optimal control parameters as the output to construct a training data set; use the training data set to train the neural network, optimize the weights and biases of the network, and the training process can use the back propagation algorithm and Adam optimizer to minimize the root mean square error between the predicted output and the actual output;

[0081] Step S340: construct a Bayesian network with the fuzzy characteristics of the coal mining machine and the coal seam as nodes, and use the maximum likelihood estimation method to calculate the conditional probability table of each node in the Bayesian network under the condition of its parent node; specifically, construct a Bayesian network with the fuzzy characteristics of the coal mining machine and the coal seam as nodes, analyze the causal relationship between the various state characteristics of the coal mining machine and the various characteristics of the coal seam, such as taking the degree of wear of the pick of the coal mining machine, the rotation speed of the drum, the traction speed, etc. as nodes, and taking the hardness, thickness, and gangue of the coal seam as nodes, and determining the directed edges between the nodes according to the production principle of coal mines, such as the degree of wear of the pick will affect the cutting efficiency of the coal mining machine, and then be associated with the hardness of the coal seam and the traction speed of the coal mining machine, so there is a directed edge from the node of the degree of wear of the pick to the node of the cutting efficiency, and then to the nodes related to the hardness of the coal seam and the traction speed;

[0082] Then, through data analysis of the coal mine production process, the Bayesian network structure is optimized, focusing on the possible indirect causal relationship and feedback mechanism between the coal mining machine and the coal seam, adding necessary intermediate nodes or adjusting the direction of the edges, so that the network structure can more accurately reflect the complex causal relationship in coal mine production; for example, if it is found that changes in coal seam thickness will affect the cutting depth of the coal mining machine, thereby changing the load condition of the coal mining machine and ultimately affecting the degree of wear of the cutting teeth, intermediate nodes reflecting the load condition can be added to the network, and the directions of related edges can be adjusted to construct a network structure that is more in line with the actual situation.

[0083] Finally, a large amount of coal mining machine operation data and coal seam geological data are collected, and the coal mining machine and coal seam characteristic information obtained by three-dimensional semantic segmentation are combined to organize and classify the data. It is necessary to ensure that the data covers different coal mining machine models, different coal seam geological conditions and different working stages. The conditional probability of each node in the Bayesian network under the condition of its parent node is calculated using statistical analysis methods; at the same time, in the calculation process, taking into account the dynamic changes and uncertainty of coal mine production data, a dynamic update mechanism is adopted to recalculate the conditional probability regularly or according to the acquisition of new data to ensure that the conditional probability table can accurately reflect the actual probability relationship in the current coal mine production.

[0084] In step S300, the probability information in the conditional probability table can help determine the degree of association between the coal mining machine state characteristics and the coal seam characteristics under different semantic objects. If the conditional probability table shows that under the condition that a certain coal mining machine state characteristic appears, the probability of a certain coal seam characteristic appearing is higher, then in the spatiotemporal relationship analysis, the two characteristics can be regarded as having a strong correlation. In this way, the relationship between different characteristics in time and space can be analyzed more accurately, providing more valuable information for subsequent control strategy extraction; secondly, based on the probability distribution in the conditional probability table, more reasonable fuzzification rules can be formulated. For example, for those event combinations with higher conditional probabilities, higher weights or stricter fuzzy division criteria can be given during fuzzification processing. In this way, the fuzzified spatiotemporal relationship parameters can more accurately reflect the actual situation and reduce the impact of uncertainty on control strategy extraction.

[0085] Step S400: Analyze the spatiotemporal relationship between different semantic objects using the production control model to extract the control strategy of the target production area.

[0086] Step S400 specifically includes:

[0087] Step S410: Based on the conditional probability table, analyze the mutual relationship between different semantic objects in the time and space dimensions; Step S410 specifically includes:

[0088] Step S411: Spatiotemporal interaction analysis between coal mining machine and coal seam:

[0089] Matching analysis of cutting depth and coal seam thickness: The cutting depth of the cutting drum of the coal mining machine at different times (calculated by the relative position of the cutting drum and the coal seam, such as the vertical distance between the lowest point of the cutting drum and the bottom of the coal seam) and the coal seam thickness (the plane coordinates of the coal seam) are obtained from the three-dimensional semantic segmentation results, and the matching degree of the cutting depth and the coal seam thickness is calculated: when the matching degree of the coal seam thickness is close to 1, it means that the cutting depth and the coal seam thickness are well matched; when it is far less than 1, there is a problem of too shallow cutting or the coal seam thickness changes greatly and the cutting depth is not adjusted in time; in this way, the matching degree of the cutting depth and the coal seam thickness under different probability conditions can be more accurately evaluated, because in the actual coal mining process, some combinations of cutting depth and coal seam thickness may be more common or more conducive to coal mining, and conditional probability can reflect this information;

[0090] Step S412: Analysis of the adaptability of the shearer speed to coal seam changes: Given the moving speed of the shearer in three-dimensional space, analyze its spatiotemporal relationship with the coal seam texture changes and the distribution of interlayers. Calculate the coal seam texture change rate (by calculating the texture features of the coal seam semantic segmentation image, such as the contrast change of the grayscale co-occurrence matrix) and the density of interlayers, and define the adaptability of the shearer speed to coal seam changes: when the adaptability to coal seam changes is within a reasonable range, the shearer speed adapts to coal seam changes; if it is too high or too low, the shearer speed may need to be adjusted; Step S412 more reasonably considers the adaptability of the shearer speed to coal seam changes under different probability conditions, because different combinations of coal seam changes and shearer speeds may have different probabilities of occurrence and impacts on coal mining results;

[0091] Step S413: Spatiotemporal coordination analysis between the components of the coal mining machine: coordination between the cutting drum and the rocker arm: obtain the rotation speed of the cutting drum and the swing angle of the rocker arm and its rate of change, and calculate the coordination coefficient between the cutting drum and the rocker arm: when it is close to the design value of the coordination coefficient, the two are well coordinated; otherwise, there is a problem of mechanical failure or improper control; by introducing these conditional probabilities, the coordination relationship between the cutting drum and the rocker arm in actual work can be analyzed more accurately, because their coordination effect may be affected by various conditional factors;

[0092] Step S414: Synchronicity between the traction part and the cutting part: Calculate the synchronization coefficient between the traction part and the cutting part according to the traction speed of the coal mining machine and the cutting speed of the cutting drum (calculated by the speed and radius of the cutting drum): When the synchronization coefficient is stable in a reasonable range, the two are well synchronized; if there is a large fluctuation, it may affect the coal mining efficiency and equipment life; Step S414 can more accurately evaluate the synchronization of the traction part and the cutting part under different probability conditions, because in the actual coal mining process, different traction speed and cutting speed combinations may have different probabilities and effects on the overall performance of the coal mining machine;

[0093] Step S420: Fuzzification is performed based on the results of the spatiotemporal relationship analysis, and converted into spatiotemporal relationship parameters; the spatiotemporal relationship parameters such as matching degree, fitness, coordination coefficient, synchronization coefficient, etc. obtained by the above calculations are fuzzified; for example, for the matching degree of the cutting depth and the coal seam thickness, the fuzzy sets "good match", "medium match", and "poor match" are defined, and the fuzzy state is represented by the membership function. Similarly, other parameters are fuzzified; based on coal mine production knowledge and experience, a fuzzy rule base is established. For example, if the matching degree of the cutting depth and the coal seam thickness is "poor match" and "low fitness", the cutting depth and the speed of the coal mining machine are adjusted. The fuzzy reasoning can adopt the Mamdani method or other suitable methods. Through fuzzy reasoning, fuzzy control decisions are obtained, such as "substantially increase the cutting depth", "appropriately reduce the speed of the coal mining machine", etc.;

[0094] For the adaptability of the shearer speed to the coal seam change, the fuzzy sets "good adaptability", "general adaptability", "poor adaptability" and "unfitness" are set, and the membership function is determined according to the frequency of occurrence of different fitness values ​​in the conditional probability table and the corresponding working state of the shearer. For example, if the value of the adaptability of the shearer speed to the coal seam change under a certain conditional probability is related to the stable operation of the shearer and efficient coal mining, then the membership of this value in the "good adaptability" fuzzy set is relatively large;

[0095] For the coordination coefficient between the cutting drum and the rocker arm and the synchronization coefficient of the cutting part, appropriate fuzzy sets are defined, such as "good coordination", "general coordination", "poor coordination", "stable synchronization", "general synchronization", "abnormal synchronization", etc. According to the probability distribution of these coefficients in the conditional probability table under different working conditions of the coal mining machine, the membership function is determined to more accurately describe the fuzziness of the coordination and synchronization between the components;

[0096] Step S430: Set the spatiotemporal relationship parameters between different semantic objects after fuzzification processing as the state space, set the control strategies that can be adopted as the action space, and construct a reward function according to the working efficiency index and equipment status information of the coal mining machine; possible control strategies (such as adjusting the cutting depth, speed, rocker arm angle, etc.) are used as the action space; define the reward function, for example, when the working efficiency of the coal mining machine is improved (such as increased coal mining volume and reduced energy consumption) and the equipment state is stable (monitoring equipment wear and failure through three-dimensional semantic segmentation), a positive reward is given; when the risk of failure increases or the coal mining efficiency decreases, a negative reward is given;

[0097] Step S440: Select the optimal action control strategy according to the current state through reinforcement learning; Specifically, the spatiotemporal relationship parameters such as the matching degree of cutting depth and coal seam thickness after fuzzy processing, the adaptability of coal mining machine speed and coal seam change, the coordination coefficient and synchronization coefficient of coal mining machine components are used as the dimensions of the state space, and the value of each dimension is determined according to the fuzzification result; The control strategy in the action space includes adjusting the cutting speed, cutting depth, traction speed, and rocker arm swing angle operation of the coal mining machine. The action space can be expressed as; Then, according to the working efficiency of the coal mining machine (such as coal mining volume, coal mining quality), equipment status (such as component wear degree, fault condition) and the probability of different state and action combinations in the conditional probability table, a reward function is constructed. For example, if the coal mining machine takes a certain action in a certain state, the coal mining volume increases and the equipment state remains stable, and the probability of this state-action combination in the conditional probability table is high, then a higher positive reward is given. In this way, the reward function comprehensively considers the working and equipment status of the coal mining machine and the probability rationality of different control strategies under the current spatiotemporal relationship.

[0098] Finally, a reinforcement learning algorithm suitable for processing continuous state and action space is selected, such as the deep deterministic policy gradient (DDPG) algorithm, which can effectively process the complex spatiotemporal relationship parameters (state space) and control strategies (action space) in the coal mining machine control problem; according to the characteristics of coal mining machine control, the parameters of the reinforcement learning algorithm are adjusted; specifically, the learning rate, discount factor, exploration rate and other parameters are optimized according to factors such as the coal mining machine working cycle, control adjustment frequency and data uncertainty; the discount factor is determined according to the long-term impact of the coal mining machine control decision, so that the algorithm pays more attention to long-term rewards; the exploration rate is adjusted according to the stability of the coal mining machine operating environment, and the exploration rate is appropriately reduced in a stable environment, and increased in a complex and changing environment, so as to balance the relationship between exploring new strategies and using existing experience. During the training process, the reinforcement learning algorithm is used to select the optimal action as the control strategy based on the current state space.

[0099] For example, at a certain moment, according to the spatiotemporal relationship parameters such as the matching degree between the current cutting depth and the thickness of the coal seam, the speed of the coal mining machine and the adaptability of the coal seam change, the reinforcement learning algorithm determines whether to increase the cutting speed, adjust the cutting depth, or change the swing angle of the rocker arm to achieve optimal control. At the same time, as the coal mining machine continues to operate and new data is continuously acquired, the policy network and value network in the reinforcement learning algorithm are continuously updated according to the new state-action-reward feedback.

[0100] In summary, the coal mine production control method based on the law enforcement recorder provided by the present invention can accurately extract the features of the various components of the coal mining machine and the coal seam from the complex video through three-dimensional semantic segmentation using a three-dimensional convolutional neural network combined with spatial and temporal attention weight calculation, which makes the basis for the generation of the control strategy more accurate and reduces the policy deviation caused by inaccurate feature extraction; when constructing the production control model, the features are extracted from the three-dimensional semantic segmentation results based on the kinematics and geological analysis of the coal mining machine, thereby ensuring the scientificity and comprehensiveness of the features; and through fuzzification processing, the fuzzy factors in the characteristics of the coal mining machine and the coal seam are reasonably quantified, avoiding the policy uncertainty caused by fuzzy boundaries; the mining of the causal relationship of the fuzzy features by the multi-layer feedforward neural network and the construction of the conditional probability table by the Bayesian network reveal the complex internal connection between the coal mining machine and the coal seam from different levels, so that the control strategy can fully consider the mutual influence and probability of various factors, and enhance the ability of the strategy to cope with complex working conditions. Finally, the production control model is used to analyze the spatiotemporal relationship and generate a control strategy. By combining the real-time working status of the coal mining machine and the real-time situation of the coal seam, the optimal action is selected based on rigorous spatiotemporal relationship analysis and reasonable fuzzy processing, as well as reinforcement learning guided by the reward function. This realizes dynamic adjustment of the control strategy and effectively responds to various uncertainties in the operation of the coal mining machine, such as component wear and coal seam changes. It significantly improves the working efficiency, stability and safety of the coal mining machine, making the coal mining machine control more intelligent and precise, and adapting to the complex and changeable production environment of coal mines.

[0101] See also Figure 2 The technical solution of the second aspect of the present invention provides a coal mine production control method system based on a law enforcement recorder, which adopts the coal mine production control method based on a law enforcement recorder described in the technical solution of the first aspect of the present invention, and the system includes:

[0102] A video image acquisition module is configured to acquire video data of the target production area at different viewing angles and convert it into a YIQ color space; specifically, the video image acquisition module uses a law enforcement recorder;

[0103] A three-dimensional semantic segmentation module, configured to perform three-dimensional semantic segmentation on the video data converted into the YIQ color space, and extract the three-dimensional semantic segmentation result;

[0104] A production control module configured to build a production control model based on the three-dimensional semantic segmentation results;

[0105] The control strategy generation module is configured to use the production control model to analyze the spatiotemporal relationship between different semantic objects and extract the control strategy of the target production area; specifically, the control strategy generation module is also used to send the extracted control strategy of the target production area to underground workers.

[0106] The technical solution of the third aspect of the present invention provides an electronic device, comprising: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the coal mine production control method based on the law enforcement recorder described in the technical solution of the first aspect of the present invention.

[0107] The technical solution of the fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for implementing a coal mine production control method based on a law enforcement recorder. The program for implementing a coal mine production control method based on a law enforcement recorder is executed by a processor to implement the steps of the coal mine production control method based on a law enforcement recorder described in the technical solution of the first aspect of the present invention.

[0108] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A coal mine production control method based on a law enforcement recorder, characterized in that: The method comprises: Obtain video data from different viewing angles of the target production area and convert it to the YIQ color space; Perform 3D semantic segmentation on the video data converted to the YIQ color space and extract the 3D semantic segmentation results; Build a production control model based on 3D semantic segmentation results; The production control model is used to analyze the spatiotemporal relationship between different semantic objects and extract the control strategy of the target production area.

2. The coal mine production control method based on the law enforcement recorder according to claim 1 is characterized in that: Obtaining video data from different viewing angles of the target production area and converting it to the YIQ color space includes: Obtain the side video data of the coal mining machine and the video data of the bottom of the coal seam after cutting and perform preprocessing; The preprocessed side video data and bottom video data of the coal mining machine are converted into the YIQ color space respectively.

3. The coal mine production control method based on the law enforcement recorder as claimed in claim 2 is characterized in that: Obtaining the video data of the side of the coal mining machine and the video data of the bottom of the coal seam after cutting and preprocessing includes: Preprocess the side video data based on geometric correction perspective transformation, including: Based on the initial static state of the coal mining machine, the key feature points on the side of the coal mining machine are marked and the image coordinates without perspective distortion are extracted; Solve the perspective transformation matrix based on the least squares method; For each frame of the image in the side video data, its perspective transformation matrix is ​​transformed to obtain a coordinate matrix after perspective transformation and reconstruct the side image.

4. The coal mine production control method based on the law enforcement recorder as claimed in claim 3 is characterized in that: Acquiring the video data of the side of the coal mining machine and the video data of the bottom of the coal seam after cutting and preprocessing it also includes: The coal seam bottom video data is preprocessed based on histogram equalization, including: Extract the grayscale histogram of the video frame at the bottom of the coal seam; Calculate the cumulative distribution function based on the grayscale histogram and perform normalization; Based on the normalized cumulative distribution function, the original gray value of each pixel in the video frame of the coal seam bottom is mapped to a new gray value; The linear stretching method is used to enhance the contrast of the entire image based on the mapped new grayscale values.

5. The coal mine production control method based on the law enforcement recorder according to any one of claims 1 to 4, characterized in that: Perform 3D semantic segmentation on the video data after conversion to the YIQ color space, and extract the 3D semantic segmentation results including: Use 3D convolutional neural network to build a 3D semantic segmentation model and extract the features of video data in the spatiotemporal dimension; For the feature graph output by each convolutional layer, the spatial attention weight is calculated based on the relative importance and mutual correlation of each component in the spatial layout of the coal mining machine, and the temporal attention weight is calculated based on the phased nature of the coal mining machine's workflow and the characteristics of the coal seam changing over time during the mining process; Annotate the preprocessed video data of the side and bottom of the coal mining machine and train a 3D semantic segmentation model; The real-time video data is input into the trained 3D semantic segmentation model. Based on the spatiotemporal features and semantic information learned during the training process, the semantic category of each pixel in 3D space-time is output to obtain a complete 3D semantic segmentation result.

6. The coal mine production control method based on the law enforcement recorder according to claim 5 is characterized in that: The production control model based on 3D semantic segmentation results includes: Based on the 3D semantic segmentation results, the shearer kinematic model and geological analysis method are combined to extract the shearer state characteristics and coal seam multi-dimensional characteristics respectively. Fuzzy sets are designed for the state characteristics of coal mining machine and coal seam characteristics respectively, and membership function is used to perform fuzzy processing on the state characteristics of coal mining machine and coal seam characteristics. The multi-layer feedforward neural network is used to process the fuzzified coal mining machine state characteristics and coal seam characteristics, and learn the causal relationship between the fuzzy characteristics. A Bayesian network is constructed with the fuzzy characteristics of coal mining machines and coal seams as nodes, and the conditional probability table of each node in the Bayesian network under the condition of its parent node is calculated using the maximum likelihood estimation method.

7. The coal mine production control method based on the law enforcement recorder according to claim 6 is characterized in that: The production control model is used to analyze the spatiotemporal relationship between different semantic objects, and the control strategies for the target production area are extracted, including: Based on the conditional probability table, analyze the relationship between different semantic objects in the time and space dimensions; Perform fuzzy processing based on the results of spatiotemporal relationship analysis and convert them into spatiotemporal relationship parameters; The spatiotemporal relationship parameters between different semantic objects after fuzzification are set as state space, the control strategies that can be adopted are set as action space, and the reward function is constructed according to the working efficiency index and equipment status information of the coal mining machine. Through reinforcement learning, the optimal action control strategy is selected according to the current state.

8. A coal mine production control method system based on law enforcement recorder, characterized in that: The coal mine production control method based on the law enforcement recorder according to any one of claims 1 to 7 is adopted, and the system comprises: A video image acquisition module, configured to acquire video data of a target production area at different viewing angles and convert the video data into a YIQ color space; A three-dimensional semantic segmentation module, configured to perform three-dimensional semantic segmentation on the video data converted into the YIQ color space, and extract the three-dimensional semantic segmentation result; A production control module configured to build a production control model based on the three-dimensional semantic segmentation results; The control strategy generation module is configured to analyze the spatiotemporal relationship between different semantic objects using the production control model and extract the control strategy of the target production area.

9. An electronic device, characterized in that: The electronic device includes: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the coal mine production control method based on the law enforcement recorder as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing a coal mine production control method based on a law enforcement recorder, and the program for implementing a coal mine production control method based on a law enforcement recorder is executed by a processor to implement the steps of the coal mine production control method based on a law enforcement recorder as described in any one of claims 1 to 7.

Images