Mining behavior classification method in mine operation scene

Abstract

The invention provides a mining behavior classification method in a mine operation scene, and the method comprises the steps: firstly receiving first video data, and then carrying out the processing of the first video data, and obtaining second video data; wherein the image quality of the second video data is higher than that of the first video data. And finally, inputting the second video data into a preset action behavior classification model for processing to obtain an action behavior classification result. The second video data is obtained by processing the first video data, and the image quality of the second video data is higher than that of the first video data, so that the accuracy of action behavior classification is improved, and the problem that in the prior art, under the application scene of intelligent coal mining behavior analysis, the accuracy of action behavior classification is improved is solved. And the technical problem that the action behavior classification of the actor is not accurate enough is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of action classification, and in particular to a method for classifying mining behaviors in a mining operation scenario. Background Art

[0002] Intelligent coal mining behavior analysis refers to the use of artificial intelligence, data mining and Internet of Things technologies to monitor, analyze and identify the behavior of workers and equipment at the coal mining site in real time to improve the safety, efficiency and sustainability of coal mine production. Through intelligent coal mining behavior analysis, it is possible to monitor the behavior of coal mine operators and equipment, detect abnormal situations in a timely manner and take corresponding measures, thereby reducing accident risks, improving production efficiency and ensuring worker safety.

[0003] In the prior art, in the application scenario of intelligent coal mining behavior analysis, there is a technical problem of inaccurate classification of the actor's action behavior. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a mining behavior classification method in a mining operation scenario to solve the technical problem of inaccurate classification of the actor's action behavior in the application scenario of intelligent coal mine mining behavior analysis.

[0005] The present invention adopts the following technical solutions: The present invention provides a mining behavior classification method in a mining operation scenario, the method comprising: Receive first video data; wherein the first video data is video data used to represent a mining operation scene; Processing the first video data to obtain second video data; wherein the image quality of the second video data is higher than the image quality of the first video data; The second video data is input into a preset action behavior classification model for processing to obtain an action behavior classification result.

[0006] Beneficial effects: The present invention first receives first video data, and then processes the first video data to obtain second video data; wherein the image quality of the second video data is higher than the image quality of the first video data. Finally, the second video data is input into a preset action behavior classification model for processing to obtain an action behavior classification result. The present application processes the first video data to obtain the second video data, and the image quality of the second video data is higher than the first video data, thereby improving the accuracy of action behavior classification, and thus solving the technical problem in the prior art that the action behavior classification of the actor is not accurate enough in the application scenario of intelligent coal mine mining behavior analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram of a scenario example provided by an embodiment of the present invention; Figure 2 It is a flowchart of a method for classifying mining behaviors in a mining operation scenario provided by an embodiment of the present invention; Figure 3 It is a flow chart of a method for classifying mining behaviors in a mining operation scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0008] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0009] Overview: In related technologies, action classification technology refers to extracting features and attributes of the actors in received video data, and classifying and categorizing the features and attributes, so as to classify the actions of the actors.

[0010] At present, some coal mine workers are not well-educated and lack the necessary safety awareness and skills, and illegal operations often occur. Although most coal mining companies have formulated safety management systems, due to the lack of effective implementation and supervision, these illegal operations may bring risks to production safety. In some solutions, by collecting video data of coal mine workers' on-site operations, and then using motion classification technology to classify the behaviors of coal mine workers, the purpose of real-time monitoring, analysis and identification is achieved.

[0011] However, in mining operation scenes, whether it is an open-pit mine operation scene or an underground mine operation scene, there is often a lot of dust and smoke, which often makes the video data collected on site unclear. In addition, mining operation scenes usually have poor lighting conditions. For example, in underground mines or at night in open-pit mines, the light may be insufficient or uneven, resulting in insufficient brightness or shadows in the video data, affecting the image quality of the video data. In addition to smoke and dust, there may also be a lot of dust and water vapor at the mining operation site. These particles will deposit on the surface of the camera equipment, affecting the clarity of the lens, causing the video data to be blurred or stained.

[0012] Therefore, because the image quality of the collected video data is not high enough, it often affects the accuracy of subsequent action classification. Therefore, in the prior art, in the application scenario of intelligent coal mining behavior analysis, there is a technical problem of inaccurate classification of the action behavior of the actor.

[0013] In response to this technical problem, the present invention first receives first video data, and then processes the first video data to obtain second video data; wherein the image quality of the second video data is higher than that of the first video data. Finally, the second video data is input into a preset action behavior classification model for processing to obtain an action behavior classification result. The present invention processes the first video data to obtain the second video data, and the image quality of the second video data is higher than that of the first video data, thereby improving the accuracy of action behavior classification, and thus solving the technical problem in the prior art that the action behavior classification of the actor is not accurate enough in the application scenario of intelligent coal mine mining behavior analysis.

[0014] Example scenario: like Figure 1 As shown, before describing the embodiments of the present invention in detail, a specific scenario example is used to reveal the scenario in which the technical solution of the present invention can be used. In this scenario example, multiple edge computing terminals and service terminals may be included.

[0015] Multiple edge computing terminals can be set up at the site of the mine operation scene to receive and process the initial video data collected by the video acquisition device. The video acquisition device is also set up at the site of the mine operation scene to collect initial video data.

[0016] The edge computing terminal can identify the target image frame in the initial video data, and then compress or delete the target image frame to obtain the first video data. It is understandable that the edge computing terminal can be configured with a corresponding algorithm to identify the target image frame in the initial video data.

[0017] The server can be communicatively connected with the multiple edge computing terminals to receive the first video data transmitted by the multiple edge computing terminals, and then process the first video data to obtain the second video data; wherein the image quality of the second video data is higher than the image quality of the first video data; finally, the second video data is input into a preset action behavior classification model for processing to obtain an action behavior classification result. It is understandable that the corresponding algorithm can be configured in the server to process the first video data, so that the image quality of the second video data is higher than the image quality of the first video data. The action behavior classification model can be configured in the server.

[0018] Example method: According to an embodiment of the present invention, a mining behavior classification method in a mining operation scenario is provided. The method can be applied to a server. Figure 2 ,include: Step S102: Receive first video data; wherein the first video data is video data used to characterize a mining operation scene.

[0019] In this embodiment, the receiving action can be represented as the server receiving the first video data transmitted from other devices. The other device can be a video acquisition device deployed in a mine operation scene. It is understandable that the video acquisition device can be deployed at a mine operation site to collect video data of the personnel performing operations at the operation site. The other device can also be an edge computing terminal deployed in a mine operation scene. It is understandable that the edge computing terminal can be deployed at a mine operation site to process the video data collected by the video acquisition device, such as performing compression to reduce the size of the data, thereby improving transmission efficiency.

[0020] The mining operation scene may be an open-pit mine operation scene and an underground mine operation scene. It is understandable that an open-pit mine refers to a mine that mines ore on the surface, and an underground mine refers to a mine that performs mining operations underground. For both the open-pit mine operation scene and the underground mine operation scene, a video acquisition device may be provided on site to acquire the first video data. The video acquisition device may be a camera. For the open-pit mine operation scene, the first video data may also be acquired by a video acquisition device configured on a drone.

[0021] Step S104: Process the first video data to obtain second video data; wherein the image quality of the second video data is higher than the image quality of the first video data.

[0022] In some cases, such as in an underground mine operation scene, there is often a lot of dust and smoke, which may affect the image quality of the video data collected by the video acquisition device. Correspondingly, the light in the underground mine operation scene may not be sufficient, which may also affect the image quality of the video data collected by the video acquisition device. It is understandable that video data with lower image quality will affect subsequent processing.

[0023] In this embodiment, the processing action can be represented by performing at least one of the following processing methods on the first video data, so that the image quality of the second video data obtained after processing is higher than the image quality of the first video data. The processing method may include: reducing noise, improving contrast, adjusting brightness, etc. It can be understood that the video data is composed of multiple image frames, and therefore, the image quality of the second video data refers to the image quality of the image frame in the second video data. Therefore, the image quality of the second video data is higher than the image quality of the first video data, which can be represented by the image quality reflected by all the image frames in the second video data as a whole is higher than the image quality reflected by all the image frames in the first video data as a whole. In some embodiments, the image quality of the second video data is higher than the image quality of the first video data, which can also be represented by the image quality of each image frame in the second video data is higher than the image quality of the corresponding image frame of the first video data.

[0024] In this embodiment, the image quality can be measured by indicators such as image clarity, image contrast, and image noise. Specifically, image clarity can be measured using parameters such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). Image noise can be measured using parameters such as root mean square error (RMSE) and noise power spectral density (PSD) to measure the noise level in the image.

[0025] Step S106: Input the second video data into a preset action behavior classification model for processing to obtain an action behavior classification result.

[0026] In this embodiment, the action behavior classification model is pre-trained and used to classify the action types of the actor in the second video data. Specifically, the action behavior classification model can be pre-trained by the following method. First, a data set is constructed, and the data set may include multiple video data and labels corresponding to the video data. The multiple video data may be video data of the actor performing various illegal actions in a mining operation scene. The label may include various illegal action categories. For example, lying or sleeping underground, taking off the safety helmet when resting underground, and not patrolling the surroundings of the coal mining machine before starting the coal mining machine, etc. Specifically, a video data may be a video data of an actor performing a complete illegal action. Accordingly, considering that the data features of the data can fit the real scene, multiple video data can be collected for a type of illegal action. It can be understood that for a specific illegal action, its implementation method may be diverse. For example, for the illegal action of taking off the safety helmet when resting underground, the actor takes off the safety helmet in a variety of ways when resting underground. In other words, when constructing a data set, a large amount of video data should be collected so that the data features of these video data can fit the real scene. Then, the data set is divided into a training set and a test set according to a certain ratio. For example, the training set and the test set are divided according to a ratio of 4:1. Then, the video data and corresponding labels in the training set are used to train the initial model, and after the training is completed, the trained model is obtained. The trained model can be tested with the test set. The test results are obtained, for example, accuracy, precision or recall rate, etc. If the test result does not reach the threshold, for example, the threshold of accuracy can be 90%. Then the training set is used to continue training the trained model until it passes the test of the test set. Finally, after passing the test of the test set, the preset action behavior classification model is obtained. The initial model can use a CNN (Convolutional Neural Network) model as the initial model, an RNN (Recurrent Neural Network) model as the initial model, or an LSTM (Long Short-Term Memory) as the initial model.

[0027] In the present embodiment, the action behavior classification result may include the action category of the actor in the video data. The action category specifically includes various illegal action categories. When the actor in the video data does not perform illegal actions, the action behavior classification result may output the category of no illegal actions. It is understandable that the action behavior classification model is pre-trained using the video data and labels in the training set, and the labels are various illegal action categories. Therefore, in step S106, if the actor in the second video data does not perform illegal actions, the confidence of the action behavior classification model in predicting various illegal action categories is low, and the category of no illegal actions is output. In the present embodiment, it is not necessary to consider the classification of the category of no illegal actions, and only the category of illegal actions is considered.

[0028] To sum up, the present invention processes the first video data to obtain the second video data, and the image quality of the second video data is higher than that of the first video data, and then inputs the second video data into a preset action behavior classification model for processing to improve the accuracy of action behavior classification, thereby solving the technical problem in the prior art that the action behavior classification of the actor is not accurate enough in the application scenario of intelligent coal mine mining behavior analysis.

[0029] In some cases, the actor, such as a coal mine worker, may leave the current work area, for example, to take a break or have a meal, or may leave the work surface to go to the equipment maintenance area for equipment maintenance and maintenance work, etc. In other words, the video acquisition device may not have the actor in the video data collected within a certain period of time. First of all, it is understandable that for this video data without the actor, the image quality does not affect the accuracy of the actor's action behavior classification. Secondly, it is understandable that in the mining operation scene, real-time monitoring and detection of the actor's action behavior is very critical. Timely discovery and identification of illegal operation behaviors can take timely measures to intervene and correct, thereby reducing safety risks. Therefore, the action behavior classification needs to have high real-time performance, that is, it can timely identify and classify the actor's actions. In the coal mine operation scene, the network bandwidth is usually limited, especially in underground mines and other environments, the network connection may be unstable or the bandwidth is low. If the transmission volume of video data is too large, it will occupy a large amount of network bandwidth, resulting in the obstruction of the transmission of other important data, and even affecting the stability of the entire system.

[0030] Based on this, in some implementations, before the step of receiving the first video data, the method further includes: Collecting initial video data; wherein the initial video data is collected on-site at a mining operation scene.

[0031] In this embodiment, the acquisition action is represented by setting up a video acquisition device at the mining operation scene to acquire initial video data. The video acquisition device can be a fixed camera, a spherical pan-tilt camera, or an infrared camera, etc.

[0032] A target image frame in the initial video data is identified.

[0033] In this embodiment, the target image frame in the initial video data can be identified based on a preset target detection algorithm; wherein the preset target detection algorithm is configured in an edge computing terminal, and the edge computing terminal is set at the site of the mining operation scene. The edge computing terminal is in communication connection with the server and the video acquisition device.

[0034] Specifically, an edge computing terminal can be set up on site to perform the identification action. Specifically, a target detection algorithm can be preset in the edge computing terminal to detect whether there is a person in each image frame, and the image frame without the person is the target image frame.

[0035] In this embodiment, the edge computing terminal may be an edge server set up on site, or an edge computing gateway.

[0036] In this embodiment, the preset target detection algorithm may be a target detection algorithm of the YOLO (You Only Look Once) series, such as the YOLOv5 target detection algorithm. It is understandable that the preset target detection algorithm is also pre-trained and configured in the edge computing terminal.

[0037] The target image frame is compressed or deleted to obtain the first video data.

[0038] In this embodiment, the compression or deletion action can be performed based on the edge computing terminal. Specifically, for the identified target image frame, the target image frame can be compressed according to a preset compression algorithm, or directly deleted. This can reduce the amount of data and improve transmission efficiency. This ensures that only the target image frame containing the actor is transmitted to the server, reducing unnecessary data transmission and processing.

[0039] In this implementation, the preset compression algorithm may be an image compression algorithm based on discrete cosine transform or an image compression algorithm based on wavelet transform, etc.

[0040] In the above-mentioned implementation, by identifying and processing the target image frame of the initial video data on site, the amount of video data to be transmitted can be reduced, unnecessary data transmission can be reduced, and bandwidth and transmission costs can be saved. Secondly, compressing or deleting the target image frame can reduce the size of the data and improve the efficiency of data transmission. The compressed first video data can be transmitted to the server more quickly, reducing transmission delay and improving real-time performance.

[0041] In some implementations, the step of processing the first video data to obtain the second video data further includes: The first video data is input into a preset image enhancement model or a preset image restoration model for processing to obtain the second video data.

[0042] In this embodiment, the image enhancement model and the image restoration model can both improve the image quality in the video data, and the image enhancement model can use at least one image processing algorithm to improve the clarity, contrast or color accuracy of the image.

[0043] In this embodiment, the image enhancement model may adopt an image denoising model, an image contrast enhancement model, or an image defogging model. The image restoration model may adopt a GAN (Generative Adversarial Network) model.

[0044] Specifically, the image denoising model is to remove the noise effect by analyzing the noise distribution and characteristics in the image, so as to make the image clearer and more realistic. The image denoising model can be a denoising model based on mean filtering, median filtering or wavelet transform denoising algorithm. The image contrast enhancement model is used to increase the brightness difference between different areas in the image and improve the contrast of the image. The image contrast enhancement model can be an image contrast enhancement model based on algorithms such as histogram equalization or adaptive histogram equalization. The image defogging model can be used to deal with image blur and degradation problems caused by atmospheric interference such as haze and smoke. For example, an image defogging model based on a dark channel prior algorithm can be used.

[0045] In this embodiment, when the image restoration model adopts a generative adversarial model, the generative adversarial model can also be pre-trained. Specifically, an initial generative adversarial model is first constructed. The initial generative adversarial model may include a generator network and a discriminator network. The generator network is responsible for generating and outputting high-quality image data, and its input is low-quality image data affected by noise or smoke interference. The generator network can adopt a convolutional neural network structure, or a deconvolutional network (Deconvolutional Network) or a fully connected network (FullyConnected Network) structure; high-quality image data can be generated by learning the mapping relationship from input to output. The discriminator network is responsible for determining whether the input image is a real image or a false image generated by the generator network, and the discriminator network can adopt a convolutional neural network structure.

[0046] Then a data set is constructed. The data set may include two types of image data. The first type of image data may include multiple image data that are not affected by interference factors such as noise, haze or smoke. These image data may be clear images of targets such as equipment, workers, ores, and environments in mining operation scenes. The first type of image data is used to train the discriminator network to help it learn the features and structure of real images. The second type of image data may include multiple image data with noise, haze or smoke. Similarly, these image data with noise, haze or smoke are also image data of targets such as equipment, workers, ores, and environments in mining operation scenes.

[0047] Finally, adversarial training is performed between the generator network and the discriminator network. Through adversarial learning, the generator network continuously optimizes its ability to generate realistic images, and the discriminator network also continuously improves its accuracy in judging authenticity. During the training process, the generator network and the discriminator network are trained alternately. After the generator network generates an image, the discriminator network classifies the generated image and the real image, calculates the loss and updates the parameters. The generator network then generates more realistic images based on the feedback from the discriminator network, and the cycle iterates until convergence.

[0048] In the above implementation, the preset image enhancement model or image restoration model can help remove noise, haze or other interference factors, thereby improving the quality and clarity of the image data. By processing the first video data in this way, clearer second video data can be obtained.

[0049] In some cases, in a mining operation scene, the actor may be blocked by other people or mechanical devices, resulting in that some actions cannot be accurately analyzed and identified. Based on this, in some embodiments, the step of inputting the second video data into a preset action behavior classification model for processing to obtain an action behavior classification result is as follows: Figure 3 As shown, it may include: Step S1062: Determine whether there is at least one person obscured in the second video data.

[0050] In this embodiment, the judgment action can be expressed as judging whether at least one person is obscured in the second video data based on a preset algorithm. For example, the person in the image data can be detected based on a target detection algorithm. If the position of the person is detected, it can be judged by the position and size of the detection frame whether other objects or people obstruct the person. It is also possible to detect edge and contour information in the image data through an edge detection algorithm. By detecting the edge information of the person in the video frame, it can be judged whether other objects or people obstruct the person.

[0051] Step S1064: When at least one person is obscured, extract third video data; wherein the third video data is video data used to characterize a mining operation scene, and an acquisition angle of the third video data is different from an acquisition angle of the first video data.

[0052] In this embodiment, the third video data is acquired by another video acquisition device relative to the first video data. And the acquisition angle of the third video data is different from the acquisition angle of the first video data. In a possible scenario example, in a mining operation scenario, in a specific coal mine working face, two video acquisition devices are set, and the video acquisition devices are relatively set and used to comprehensively acquire video data in the current scenario.

[0053] In this embodiment, it is understandable that if it is detected that there is no person being blocked in the second video data, it is not necessary to extract the third video data, and the second video data is directly input into the preset action behavior classification model for processing.

[0054] In this embodiment, the extraction action can be expressed as extracting the third video data from the database. In other words, after the third video data is collected on site, it is transmitted to the database via the network. The database and the server are communicatively connected. When the server determines that at least one actor is obscured in the second video data, the server can query and extract the corresponding third video data from the database based on predefined rules. In this embodiment, the predefined rules can be expressed as the conditions and standards used by the server to query and extract the third video data. Specifically, in an example of a possible scenario, different video acquisition devices set on the same coal mine working face can be set with associated identity identifiers, and accordingly, the video data collected by these video acquisition devices can also carry these identity identifiers. Therefore, the server can query and extract the corresponding third video data from the database based on the identity identifier.

[0055] Step S1066: jointly use the second video data and the third video data, and process them based on a preset action behavior classification model to obtain an action behavior classification result.

[0056] In this embodiment, the preset action behavior classification model includes a feature extraction module, a timing information learning module, a mapping module and a classification module connected in sequence; wherein, the feature extraction module is used to extract the spatiotemporal features of the second video data and the third video data, the timing information learning module is used to extract timing features based on the spatiotemporal features, the mapping module is used to map the timing features into the same feature space, and the classification module is used to classify the features in the feature space.

[0057] Specifically, the preset action behavior classification model may include a feature extraction module, a timing information learning module, a mapping module and a classification module connected in sequence. The feature extraction module is used to extract the spatiotemporal features of the second video data and the third video data. The timing information learning module is used to receive the spatiotemporal features output by the feature extraction module and learn the timing information in the spatiotemporal features. More specifically, video data is essentially a series of temporally continuous frames. Although the feature extraction module can extract the spatiotemporal features in the video frame, these spatiotemporal features are mainly concentrated in a local time window. Therefore, the present embodiment sets the timing information learning module to deeply mine the timing information between frames, so as to better understand the progress and changes of actions in the video data. The mapping module is used to receive the timing features output by the timing information learning module, and map the received timing features in the same feature space. The classification module estimates the probability of each action category, outputs the predicted probability of each category, and the final recognition result of the action category will be determined based on the category with the highest probability.

[0058] More specifically, the feature extraction module can use a convolutional neural network, a ResNet network (Residual Neural Network), a 3DResNet network or a recurrent neural network. 3DResNet is a three-dimensional residual neural network based on the ResNet structure. Based on ResNet, 3DResNet extends the convolution operation from two dimensions to three dimensions for processing video data or other data with a time dimension. The feature extraction module can extract the spatiotemporal features of video data at different acquisition angles. The spatiotemporal features can be expressed as features with a spatiotemporal relationship extracted from the video data, which can include features of motion trajectories, features of motion patterns, features of optical flows, and the like. The spatiotemporal features can characterize the spatial information and temporal information in the video data. For spatial information, the spatiotemporal features can describe information such as the position, shape, and size of key points or objects in the video in space. For example, the motion path and position change of an object can be understood through the motion trajectory, and the displacement and motion direction of a pixel can be understood through the optical flow. For temporal information, the spatiotemporal features can describe the temporal changes and movements of key points or objects in the video. For example, motion trajectory or optical flow can be used to understand the speed and acceleration of an object, and action density can be used to understand the distribution and density of actions in a video. These features are mainly used to capture action and motion information in a video. Accordingly, these features can reflect the actions and motion patterns of the person in the video.

[0059] In this embodiment, the spatiotemporal feature can be represented as a multi-dimensional vector. Each spatiotemporal feature can be regarded as a feature dimension, and multiple spatiotemporal features are combined to form a multi-dimensional vector.

[0060] The temporal information learning module can further extract temporal features from the spatiotemporal features. Compared with the spatiotemporal features, the temporal features can be expressed as features with temporal relationships extracted from time series data, which contain information about the changes and evolution of the data at different time points. For example, it can include action frame difference features, action speed change features, action frequency features, and action acceleration features, etc. The action frame difference features can reflect the changes and motion trajectories of the action. The action speed change features describe the speed changes of the action at different time points, and can capture the speed changes of the action. The action frequency features describe the frequency and rhythm of the action, and can reflect the speed of the action. The action acceleration features describe the acceleration changes of the action over time.

[0061] Similarly, the time series feature can be represented as a multi-dimensional vector. Each time series feature can be regarded as a feature dimension, and multiple time series features are combined together to form a multi-dimensional vector.

[0062] The temporal information learning module may adopt an LSTM network, a Bidirectional Long Short-Term Memory (BiLSTM) network or a Gated Recurrent Unit (GRU) network.

[0063] The mapping module may adopt a fully connected layer or a combination of multiple fully connected layers.

[0064] The classification module may adopt a SoftMax classifier.

[0065] Therefore, in this embodiment, the second video data and the third video data can be input into the above-mentioned action behavior classification model to perform the above-mentioned processing in sequence, so as to obtain the action behavior classification result.

[0066] In some implementations, the step of jointly using the second video data and the third video data to perform processing based on a preset action behavior classification model to obtain an action behavior classification result may also be: Inputting the second video data and the third video data into a preset action behavior classification model for processing respectively to obtain respective action behavior classification results; The action behavior classification result with a high confidence level among the action behavior classification results is used as the final action behavior classification result.

[0067] The preset action behavior classification model can be a trained convolutional neural network, and two of them can be set. One convolutional neural network is used to receive the second video data, and the other convolutional neural network is used to receive the third video data. The action behavior classification result with high confidence output by the convolutional neural network is used as the final action behavior classification result. This method can make full use of the feature information of the second video data and the third video data, and process them through two independent convolutional neural networks, so as to obtain a more accurate action behavior classification result.

[0068] In some embodiments, the mining behavior classification method in a mining operation scenario further includes: The second video data and the corresponding action behavior classification results are displayed.

[0069] In this embodiment, the displayed action can be represented by displaying the second video data and the corresponding action behavior classification results through a display device. The display device can be a computer screen, a tablet computer or other visualization device. On the display device, the second video data will be displayed in the form of a video. Users can understand the mining behavior in mining operations by watching the video. At the same time, the display device will display the predicted action behavior classification results. These results can be presented in text, charts or other forms so that users can intuitively understand the classification of the model.

[0070] Based on the user's operation, the action behavior classification result is corrected to obtain a corrected action behavior classification result.

[0071] On the display device, interactive UI elements may be provided, such as buttons, sliders, text boxes, etc. These UI elements may be used by the user to perform UI operations, such as modifying the action behavior classification result.

[0072] The action behavior classification model is incrementally learned using the second video data and the corrected action behavior classification result to update the action behavior classification model.

[0073] In this embodiment, by displaying the second video data and the corresponding action behavior classification results, the user can intuitively understand the classification of the model and correct the classification errors or inaccurate results. The action behavior classification model is incrementally learned using the second video data and the corrected action behavior classification results, which can achieve continuous optimization and updating of the model. By continuously introducing new data and correction results, the model can gradually adapt to new scenarios and changes, and improve the generalization ability and accuracy of the model.

[0074] In some embodiments, the mining behavior classification method in a mining operation scenario further includes: The action behavior classification result is matched with a preset risk disposal library to obtain target risk disposal measures.

[0075] In this embodiment, a plurality of risk treatment measures may be preset in the risk treatment library, for example, a warning message or reminder message may be issued to the target personnel. A power outage or emergency shutdown may also be performed on the relevant equipment. The relevant equipment may be mining equipment or conveying equipment.

[0076] Implement the target risk treatment measures as described.

[0077] In this embodiment, on the one hand, by matching the action behavior classification results with the preset risk disposal library, potential risk situations can be quickly and accurately identified. According to the matching results, the target risk disposal measures can be obtained, so that corresponding measures can be taken in time to reduce the possibility of risk occurrence. On the other hand, the preset risk disposal library can be customized according to specific mining operation scenarios and needs. By matching the action behavior classification results with the risk disposal library, personalized risk disposal measures can be obtained according to the actual situation to adapt to different risk situations and needs.

[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0080] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0081] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0082] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A mining behavior classification method in a mining operation scenario, characterized in that: The method comprises: Receive first video data; wherein the first video data is video data used to represent a mining operation scene; Processing the first video data to obtain second video data; wherein the image quality of the second video data is higher than the image quality of the first video data; The second video data is input into a preset action behavior classification model for processing to obtain an action behavior classification result.

2. The method according to claim 1, characterized in that Before the step of receiving the first video data, the method further includes: Collecting initial video data; wherein the initial video data is collected on-site at a mine operation scene; Identifying a target image frame in the initial video data; The target image frame is compressed or deleted to obtain the first video data.

3. The method according to claim 1, characterized in that The step of processing the first video data to obtain the second video data, the method further comprises: The first video data is input into a preset image enhancement model or a preset image restoration model for processing to obtain the second video data.

4. The method according to claim 3, characterized in that The image enhancement model adopts an image denoising model, an image contrast enhancement model, or an image defogging model, and the image restoration model is a generative adversarial model.

5. The method according to claim 1, characterized in that The step of inputting the second video data into a preset action behavior classification model for processing to obtain an action behavior classification result includes: Determining whether at least one person is blocked in the second video data; In the case where at least one person is blocked, extracting third video data; wherein the third video data is video data used to characterize a mining operation scene, and the acquisition angle of the third video data is different from the acquisition angle of the first video data; The second video data and the third video data are used together and processed based on a preset action behavior classification model to obtain an action behavior classification result.

6. The method according to claim 5, characterized in that The step of inputting the second video data into a preset action behavior classification model for processing to obtain an action behavior classification result includes: Inputting the second video data and the third video data into a preset action behavior classification model for processing respectively to obtain respective action behavior classification results; The action behavior classification result with a high confidence level among the action behavior classification results is used as the final action behavior classification result.

7. The method according to claim 5, characterized in that The preset action behavior classification model includes a feature extraction module, a timing information learning module, a mapping module and a classification module connected in sequence; wherein the feature extraction module is used to extract the spatiotemporal features of the second video data and the third video data, the timing information learning module is used to extract timing features based on the spatiotemporal features, the mapping module is used to map the timing features to the same feature space, and the classification module is used to classify the features in the feature space.

8. The method according to claim 7, characterized in that The feature extraction module adopts the ResNet network, the temporal information learning module adopts the LSTM network, the mapping module adopts the fully connected layer, and the classification module adopts the SoftMax classifier.

9. The method according to claim 1, characterized in that: The method further comprises: Displaying the second video data and the corresponding action behavior classification results; Based on the user's operation, the action behavior classification result is modified to obtain a modified action behavior classification result; The action behavior classification model is incrementally learned using the second video data and the corrected action behavior classification result to update the action behavior classification model.

10. The method according to claim 1, characterized in that The method further comprises: Matching the action behavior classification results with a preset risk disposal library to obtain target risk disposal measures; Implement the target risk treatment measures as described.

Images