Mine image recognition method, device and system

Abstract

The invention relates to a mine image recognition method, device and system. The method comprises the following steps: acquiring an underground road image, performing feature extraction processing on the underground road image to obtain a first feature map, inputting the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model, and calculating a loss function corresponding to the first image recognition result by using a self-supervised learning algorithm, sending the underground road image and the first image recognition result to a cloud server when the loss value does not meet a first preset condition, receiving a fusion recognition result sent by the cloud server, and outputting the fusion recognition result. The fusion recognition result is obtained by fusing the second image recognition result and the first image recognition result through the cloud server, and the second image recognition result is obtained by recognizing the underground road image through the cloud server through a second recognition model. According to the scheme, the accuracy of identifying the underground road image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular to a mine image recognition method, device and system. Background Art

[0002] Among the related technologies, realizing "unmanned" mining of fully mechanized working faces is one of the important contents of intelligent construction of coal mines. Automatic driving of auxiliary transportation in coal mines is an important part of the daily production and operation of coal mines. In addition to coal transportation, personnel, materials, equipment, and supplies are all transported through auxiliary transportation systems. Automatic driving in mines requires the use of semantic segmentation technology to identify underground roads and lanes, identify the boundaries of roads to detect obstacles, and thus avoid miners and obstacles and plan reasonable underground travel routes. However, the automatic driving environment in coal mines is complex and diverse, with the characteristics of multiple perception scenes, variable and irregular optical fibers, and complex perception environments, resulting in different perception scenes having different visual features, which increases the difficulty of semantic segmentation. The computing power resources in coal mines are limited, and it is difficult to perform fine recognition of mine images. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a mine image recognition method, device and system.

[0004] According to a first aspect of an embodiment of the present disclosure, a mine image recognition method is provided, which is applied to a mine edge server, comprising:

[0005] Acquire underground road images;

[0006] Performing feature extraction processing on the underground road image to obtain a first feature map;

[0007] Inputting the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model;

[0008] Calculating a loss function corresponding to the first image recognition result using a self-supervised learning algorithm to obtain a loss value;

[0009] When the loss value does not satisfy the first preset condition, sending the underground road image and the first image recognition result to a cloud server;

[0010] receiving a fusion recognition result sent by a cloud server, and outputting the fusion recognition result; the fusion recognition result is obtained by the cloud server fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server recognizing the underground road image using a second recognition model;

[0011] When the loss value satisfies a first preset condition, the first image recognition result is output.

[0012] According to a second aspect of an embodiment of the present disclosure, a mine image recognition method is provided, which is applied to a cloud server and includes:

[0013] receiving an underground road image and a first image recognition result sent by a mine edge server; the first image recognition result is obtained by the mine edge server recognizing the underground road image using a pre-trained first recognition model; a loss value corresponding to the first image recognition result does not meet a first preset condition; the loss value is obtained by the mine edge server calculating a loss function corresponding to the first image recognition result using a self-supervised learning algorithm;

[0014] Using a second recognition model to recognize the underground road image, to obtain a second image recognition result;

[0015] Fusing the first image recognition result with the second image recognition result to obtain a fused recognition result;

[0016] The fusion recognition result is sent to the mine edge server, so that the mine edge server outputs the fusion recognition result.

[0017] According to a third aspect of an embodiment of the present disclosure, a mine image recognition device is provided, which is applied to a mine edge server, including:

[0018] An acquisition unit, used for acquiring an underground road image;

[0019] A feature extraction unit, used to perform feature extraction processing on the underground road image to obtain a first feature map;

[0020] A first recognition unit, configured to input the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model;

[0021] A calculation unit, configured to calculate a loss function corresponding to the first image recognition result by using a self-supervised learning algorithm to obtain a loss value;

[0022] A first sending unit, configured to send the underground road image and the first image recognition result to a cloud server when the loss value does not satisfy a first preset condition;

[0023] A first receiving unit is used to receive a fusion recognition result sent by a cloud server and output the fusion recognition result; the fusion recognition result is obtained by the cloud server fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server recognizing the underground road image using a second recognition model;

[0024] An output unit is used to output the first image recognition result when the loss value meets a first preset condition.

[0025] According to a fourth aspect of an embodiment of the present disclosure, a mine image recognition device is provided, which is applied to a cloud server, including:

[0026] A second receiving unit is used to receive an underground road image and a first image recognition result sent by a mine edge server; the first image recognition result is obtained by the mine edge server recognizing the underground road image using a pre-trained first recognition model; the loss value corresponding to the first image recognition result does not meet a first preset condition; the loss value is obtained by the mine edge server calculating a loss function corresponding to the first image recognition result using a self-supervised learning algorithm;

[0027] A second recognition unit, used to recognize the underground road image using a second recognition model to obtain a second image recognition result;

[0028] a fusion unit, configured to fuse the first image recognition result with the second image recognition result to obtain a fused recognition result;

[0029] The second sending unit is used to send the fusion recognition result to the mine edge server, so that the mine edge server outputs the fusion recognition result.

[0030] According to a fifth aspect of an embodiment of the present disclosure, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspect is implemented, or the method described in any one of the second aspect is implemented.

[0031] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method as described in any one of the first aspect is implemented, or the method as described in any one of the second aspect is implemented.

[0032] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements a method as described in any one of the first aspect, or implements a method as described in any one of the second aspect.

[0033] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: by acquiring an underground road image; performing feature extraction processing on the underground road image to obtain a first feature map; inputting the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model; using a self-supervised learning algorithm to calculate the loss function corresponding to the first image recognition result to obtain a loss value; when the loss value does not meet the first preset condition, sending the underground road image and the first image recognition result to a cloud server; receiving the fusion recognition result sent by the cloud server, and outputting the fusion recognition result; the fusion recognition result is obtained by the cloud server after fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image; when the loss value meets the first preset condition, outputting the first image recognition result. By calculating the loss value, the recognition result of the first model is divided into recognition results that meet the requirements and recognition results that do not meet the requirements, and further processed by using the second model configured on the cloud server, thereby ensuring the recognition speed while improving the accuracy of the recognition result, and solving the problem of inaccurate recognition of underground road images due to the complex underground environment.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0036] Figure 1 The figure is a flow chart of a mine image recognition method according to an exemplary embodiment.

[0037] Figure 2 FIG. 4 is a flowchart of a data set expansion according to the fundamental disclosed embodiment.

[0038] Figure 3 The figure is a flow chart of another mine image recognition method according to an exemplary embodiment.

[0039] Figure 4 The figure is a block diagram of a mine image recognition device according to an exemplary embodiment.

[0040] Figure 5 It is a block diagram of another mine image recognition device according to an exemplary embodiment.

[0041] Figure 6 It is a block diagram of a device for a mine image recognition method according to an exemplary embodiment. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0043] The terms used in the disclosed embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the disclosed embodiments. The singular forms "a", "an" and "the" used in the disclosed embodiments and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0044] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the disclosed embodiments, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the disclosed embodiments, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at" or "when" or "in response to determination".

[0045] In addition, various forms of processes shown in the embodiments of the present disclosure may be used to reorder, add or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present disclosure can be achieved, and this document does not limit this.

[0046] Among the related technologies, realizing "unmanned" mining of fully mechanized working faces is one of the important contents of intelligent construction of coal mines. Automatic driving of auxiliary transportation in coal mines is an important part of the daily production and operation of coal mines. In addition to coal transportation, personnel, materials, equipment, and supplies are all transported through auxiliary transportation systems. Automatic driving in mines requires the use of semantic segmentation technology to identify underground roads and lanes, identify the boundaries of roads to detect obstacles, and thus avoid miners and obstacles and plan reasonable underground travel routes. However, the automatic driving environment in coal mines is complex and diverse, with the characteristics of multiple perception scenes, variable and irregular optical fibers, and complex perception environments, resulting in different perception scenes having different visual features, which increases the difficulty of semantic segmentation. The computing power resources in coal mines are limited, and it is difficult to perform fine recognition of mine images.

[0047] At present, the research on autonomous driving environment perception and recognition using machine vision methods mainly uses target detection or semantic segmentation technology to segment objects on underground roads. The research on these technologies is mainly based on deep learning methods, and a large amount of sample data is required to train network models. In the research on the application of machine vision technology to autonomous driving road perception semantic segmentation, since the mine autonomous driving car is the data collected by mobile edge devices, it will be affected by location changes, and the data collected by monitoring cameras will be affected by time changes. The distribution and characteristics of the data collected by the IoT edge devices are not static. In the real mine IoT edge environment, data drift and data heterogeneity are common. Data drift is the phenomenon that the statistical characteristics of data change over time. This change may affect the performance of the machine learning model, because the model is trained based on historical data. If the distribution of the input data changes, the model may not be able to generalize well to the new data. Data heterogeneity refers to data coming from different sources or having different formats, structures and qualities. Data drift and data heterogeneity will have a great impact on the intelligent perception ability of the mine IoT edge devices, and in severe cases, it may even cause casualties and business damage. In addition, there is a lack of a large amount of labeled image data, which limits the training effect of the deep learning network model. Therefore, the problem of insufficient image data needs to be solved.

[0048] In order to solve the above problems, the present disclosure provides a mine image recognition method, device and system, which obtains an underground road image; performs feature extraction processing on the underground road image to obtain a first feature map; inputs the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model; uses a self-supervised learning algorithm to calculate the loss function corresponding to the first image recognition result to obtain a loss value; when the loss value does not meet the first preset condition, sends the underground road image and the first image recognition result to a cloud server; receives the fusion recognition result sent by the cloud server, and outputs the fusion recognition result; the fusion recognition result is obtained by the cloud server after the second image recognition result and the first image recognition result are fused; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image; when the loss value meets the first preset condition, the first image recognition result is output. The recognition result of the first model is divided into a recognition result that meets the requirements and a recognition result that does not meet the requirements by calculating the loss value, and further processed by the second model configured on the cloud server, thereby ensuring the recognition speed while improving the accuracy of the recognition result, and solving the problem of inaccurate recognition of the underground road image due to the complex underground environment.

[0049] Figure 1 is a flow chart of a mine image recognition method according to an exemplary embodiment. Figure 1 As shown, it should be noted that the mine image recognition method of the embodiment of the present disclosure is applied to the mine edge server. Figure 1 As shown, the method may include the following steps:

[0050] Step 101, acquiring an underground road image.

[0051] In the embodiments of the present disclosure, an image acquisition device such as a camera may be used to acquire images of underground roads.

[0052] Step 102: Perform feature extraction processing on the underground road image to obtain a first feature map.

[0053] It is understandable that in the underground environment, there are variable and irregular light conditions, weak illumination underground, and different lighting conditions in permanent tunnels, temporary tunnels, intersections, etc. In addition, the perception environment underground is also relatively complex, with small differences in surface features of underground tunnels, high environmental humidity, many ups and downs, multiple spray dust reduction areas, and water accumulation on the road surface. The complex environment makes different perception scenes have different visual features, which increases the difficulty of semantic segmentation of underground road images.

[0054] Therefore, before image recognition, it is necessary to perform feature extraction processing on the underground road image, so as to achieve the effect of denoising the underground road image.

[0055] In some embodiments of the present disclosure, step 102 may specifically include the following steps:

[0056] Inputting the underground road image into the pre-trained Transformer model to obtain a first intermediate feature map obtained after the Transformer model performs long-distance feature extraction on the underground road image;

[0057] The intermediate feature map is processed with a hole rate to obtain a first feature map.

[0058] In the embodiment of the present disclosure, the above-mentioned inputting of the underground road image into the pre-trained Transformer model to obtain the first intermediate feature map obtained after the Transformer model performs long-distance feature extraction on the underground road image may specifically include the following steps:

[0059] The underground road image is input into the pre-trained Transformer model, and the Vision Transformer's Patch is used to segment the underground road image into multiple small blocks, and a corresponding feature vector is generated for each small block. Let the input image be S, and the segmented Patch be P = {p1, p2, ..., p n}.., each p i Each corresponds to a feature vector

[0060] For the self-attention mechanism, the query Q (the current focus), key K (i.e. the other elements in the sequence), and value V (i.e. the specific information of each element) are calculated, which are obtained by linear transformation of the input feature vector:

[0061] Q=Z0W Q , K=Z0W K , V=Z0W V

[0062] Among them, W Q , W K , W V is a learnable weight matrix;

[0063] Calculate the similarity between Q and K using the dot product

[0064] Normalize the similarity weights so that the sum of the weight values ​​of each position and all positions is 1;

[0065] Use the normalized weight values ​​to weighted sum all V values ​​to get the final attention output;

[0066] Using a multi-head attention mechanism, multiple copies of the self-attention mechanism are processed in parallel, and then the results are merged; the model processes the same input data through multiple different linear transformations (i.e. different "heads"), and each head learns information in a different representation subspace of the input data. Finally, the attention values ​​of each small block (i.e. each "head") are merged, usually by concatenating these values ​​and then fusing them through an additional linear layer to produce the final merged result;

[0067] The output of the self-attention is residually connected to the input and layer normalized.

[0068] The final output is a feature representation Z (also known as the first feature map mentioned above) that is rich in contextual information and long-distance dependencies. This representation can be used for further image analysis and understanding tasks.

[0069] In some embodiments of the present disclosure, performing hole rate processing on the intermediate feature map to obtain the first feature map may specifically include the following steps:

[0070] Obtain multiple preset void rates;

[0071] For each of the multiple dilation rates, convolution processing is performed on the first intermediate feature map using a dilation convolution layer corresponding to the dilation rate to obtain a second intermediate feature map;

[0072] The second intermediate feature maps corresponding to the plurality of hole rates are fused to obtain a first feature map.

[0073] In the disclosed embodiment, the intermediate feature maps can be cascaded using dilated convolution layers with different dilation rates to capture multi-scale features. Without changing the size of the convolution kernel, the receptive field can be expanded by cascading dilated convolutions to obtain more feature information. The feature maps processed with different dilation rates are fused, which can be done in a cascade (series) or parallel manner.

[0074] For example, we can assume that Z is the first feature map processed by the Transformer model, which contains the long-distance dependency information of the underground road image. n Let A(d) represent the dilated convolution operation with dilated rate d, then: Z1=A(d1)(Z), Z2=A(d2)(Z), ..., Z n =A(d n )(Z), where Z1, Z2, ..., Z nThey are the second intermediate feature maps processed with different void ratios. The second intermediate feature maps processed with different void ratios are fused. This is done in parallel. Let F be the feature fusion operation, then the first feature map Z out =Concat(Z1, Z2, ..., Z n ).

[0075] Step 103: input the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model.

[0076] In one embodiment, the first image recognition result may be a segmentation mask for the underground road image, and the segmentation mask may be used to distinguish regions corresponding to different objects in the underground road image.

[0077] In the embodiment of the present disclosure, the first recognition model may be an RFNet model, that is, the RFNet model may be used to generate the first image recognition result in the following manner:

[0078] Four independent encoders (T1, T1ce, Flair, and T2) are used to extract features from the four modalities, and these encoders do not share weights; there can be multiple underground road images, which are obtained by different acquisition forms, such as infrared laser photography, radar scanning photography, and camera photography. Each underground road image corresponds to a first feature map, and each first feature map (that is, each modality) uses an independent encoder for feature extraction;

[0079] RFM is used to embed the normalized global features into the modality-level attention weights through two fully connected layers, a Leaky ReLU layer, and a Sigmoid activation function.

[0080] RFM generates attention weights for each region separately, and then applies these weights to the segmentation features of the corresponding regions;

[0081] To pass these regional features to the decoder, RFM uses a concatenation operation followed by a convolutional bottleneck layer. In addition, a skip connection similar to residual learning is also used;

[0082] RFNet uses a weight-sharing decoder to segment images of each modality, and regularization uses corresponding weighted cross entropy loss and Dice loss;

[0083] The decoder Dfuse is used to predict the final segmentation mask based on the fused features, and the weighted cross entropy loss and Dice loss are used to align the prediction with the corresponding ground truth segmentation map;

[0084] The total loss function is composed of the loss of the prediction graph, the regularization loss and the fusion loss, so as to complete the image semantic segmentation inference of the sample (that is, complete the recognition of the first feature graph and obtain the first image recognition result).

[0085] Step 104: Calculate the loss function corresponding to the first image recognition result using a self-supervised learning algorithm to obtain a loss value.

[0086] It should be noted that the above self-supervised learning algorithm can be implemented using existing technologies, which will not be elaborated here.

[0087] In the disclosed embodiment, the above loss value is used to determine the degree of difference between the first image recognition result and the actual result.

[0088] Step 105: When the loss value does not satisfy the first preset condition, the underground road image and the first image recognition result are sent to the cloud server.

[0089] In one embodiment, the loss value is compared with a preset threshold value. When the loss value is greater than or equal to the preset threshold value, it is determined that the loss value does not meet the first preset condition, and the underground road image and the first image recognition result are sent to the cloud server. That is, the first image recognition result does not meet the requirements, and the mine edge server is unable to accurately recognize the underground road image, so it is necessary to use the cloud server for further processing.

[0090] Step 106, receiving the fusion recognition result sent by the cloud server, and outputting the fusion recognition result.

[0091] Among them, the fusion recognition result is obtained by the cloud server after fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image.

[0092] In the disclosed embodiment, the mine edge server sends the underground road image and the first image recognition result to the cloud server. The cloud server recognizes the underground road image using the second recognition model to obtain the second image recognition result. The cloud server fuses the second image recognition result with the first image recognition result to obtain a fused recognition result.

[0093] In one embodiment, the second recognition model may be a visual large model SAM, and the SAM model may be trained in the following manner:

[0094] Use the pre-trained Vision Transformer (ViT) as the image encoder to process the input image and output image embedding;

[0095] For a given segmentation prompt, including points, boxes, text, or existing segmentation masks, different methods are used to convert these prompts into embedding vectors;

[0096] The mask decoder combines the image embedding and the hint embedding to predict the segmentation mask through a series of Transformer decoder blocks;

[0097] If the cue may correspond to multiple valid objects, SAM is designed to be able to predict multiple output masks to handle ambiguous cases;

[0098] SAM can predict multiple masks for a single prompt, and usually predicts three masks by default to cover multiple possible reasonable segmentation situations;

[0099] Use a linear combination of focal loss and Dice loss to supervise the model's mask prediction and train it;

[0100] During training, we simulate an interactive segmentation setting by randomly sampling hints and iteratively refining the segmentation mask;

[0101] By building a data engine, iteratively using models to assist data collection, and using newly collected data to improve the model;

[0102] After training, SAM can segment new downstream tasks through zero-shot transfer without additional training;

[0103] For each input image and prompt, SAM finally outputs one or more segmentation masks, which can be used in various downstream tasks.

[0104] In one embodiment, the preliminary segmentation mask (i.e., the first image recognition result proposed in some embodiments of the present disclosure) can be compared and analyzed with the mask obtained by a higher precision or more comprehensive analysis in the cloud (i.e., the second image recognition result proposed in some embodiments of the present disclosure); then, through a certain fusion strategy, such as weighted average, voting mechanism, or confidence-based fusion method, the information of the two is integrated to improve the accuracy and robustness of the segmentation; and finally, an optimized cloud inference result (i.e., the fused recognition result proposed in some embodiments of the present disclosure) is generated that combines the fast response of edge computing and the advantages of deep analysis in the cloud. This method can make full use of the low latency of edge computing and the powerful processing power of cloud computing to achieve better image segmentation effects.

[0105] In some embodiments of the present disclosure, after receiving the fusion recognition result sent by the cloud server in step 106, the method may further include: training a first recognition model using the underground road image and the fusion recognition result to obtain a trained first recognition model.

[0106] It should be noted that data drift is the phenomenon that the statistical characteristics of data change over time. This change may affect the performance of machine learning models, because the model is trained based on historical data. If the distribution of input data changes, the model may not generalize well to new data.

[0107] It is understandable that when the underground environment changes and the data distribution changes, the reasoning accuracy of the originally trained RFNet model will drop significantly when facing the first feature map after the data distribution changes. The reasoning accuracy of the RF Net model is improved by continuously training the RF Net model on the newly distributed data to adapt to the data drift phenomenon. The fused recognition results and their corresponding underground road images can be stored. When the number of stored fused recognition results exceeds a certain value, the fused recognition results can be used to retrain the RF Net model, thereby improving the ability of the mine edge server to deal with data drift problems.

[0108] Step 107: When the loss value satisfies the first preset condition, output the first image recognition result.

[0109] In one embodiment, the loss value is compared with a preset threshold value, and when the loss value is less than the preset threshold value, it is determined that the loss value meets the first preset condition, and the first image recognition result is output. That is, the first image recognition result meets the requirements, and the mine edge server is capable of accurately recognizing the underground road image.

[0110] In some embodiments of the present disclosure, Figure 2 As shown, since the number of original data sets used to train the first recognition model and the second recognition model is limited, the data sets can be expanded and processed in the following manner before training the first recognition model and the second recognition model:

[0111] The input is: underground road image T. Based on the generative adversarial network deep learning model, the generator (G) accepts random noise as input and converts it into samples (Fake) close to real data, which are used to expand the original data set; the discriminator receives fake samples from the generator and real samples from the original data set, and then tries to distinguish them. If the discriminator can successfully distinguish between true and false samples, it will output a high probability value indicating that the sample is real; if the discriminator thinks that the sample is fake, it will output a low probability value. The generator and the discriminator compete with each other during the training process. The generator tries to generate more and more realistic samples, while the discriminator tries to distinguish between true and false samples more accurately. Through this adversarial training, the generator gradually learns to generate higher quality fake samples, while the discriminator improves its recognition ability. When the generator is trained well enough, it can generate a large number of fake samples, which can be added to the original data set to expand the data set. The expanded data set is K.

[0112] According to the mine image recognition method proposed in the embodiment of the present disclosure, an underground road image is obtained; a feature extraction process is performed on the underground road image to obtain a first feature map; the first feature map is input into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model; a loss function corresponding to the first image recognition result is calculated using a self-supervised learning algorithm to obtain a loss value; when the loss value does not meet the first preset condition, the underground road image and the first image recognition result are sent to a cloud server; a fusion recognition result sent by the cloud server is received, and a fusion recognition result is output; the fusion recognition result is obtained by the cloud server after the second image recognition result and the first image recognition result are fused; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image; when the loss value meets the first preset condition, the first image recognition result is output. The recognition result of the first model is divided into a recognition result that meets the requirements and a recognition result that does not meet the requirements by calculating the loss value, and further processed by the second model configured on the cloud server, thereby ensuring the recognition speed while improving the accuracy of the recognition result, and solving the problem of inaccurate recognition of the underground road image due to the complex underground environment.

[0113] Figure 3 is a flow chart of another mine image recognition method according to an exemplary embodiment. Figure 3 As shown, it should be noted that the mine image recognition method of the embodiment of the present disclosure is applied to a cloud server. Figure 3 As shown, the method may include the following steps:

[0114] Step 301: Receive an underground road image and a first image recognition result sent by a mine edge server.

[0115] Among them, the first image recognition result is obtained by the mine edge server using a pre-trained first recognition model to recognize the underground road image; the loss value corresponding to the first image recognition result does not meet the first preset condition; the loss value is obtained by the mine edge server using a self-supervised learning algorithm to calculate the loss function corresponding to the first image recognition result.

[0116] In the disclosed embodiment, the mine edge server obtains an underground road image, performs feature extraction processing on the underground road image to obtain a first feature map, inputs the first feature map into a pre-trained first recognition model, obtains a first image recognition result output by the first recognition model, and uses a self-supervised learning algorithm to calculate a loss function corresponding to the first image recognition result to obtain a loss value. When the loss value does not meet a first preset condition, the underground road image and the first image recognition result are sent to a cloud server.

[0117] Step 302: Use the second recognition model to recognize the underground road image to obtain a second image recognition result.

[0118] Step 303: Fusing the first image recognition result with the second image recognition result to obtain a fused recognition result.

[0119] In some embodiments of the present disclosure, step 303 may specifically include the following steps:

[0120] Determining a degree of difference between the first image recognition result and the second image recognition result;

[0121] Determine a first preset weight value corresponding to the first image recognition result and a second preset weight value corresponding to the second image recognition result by using the degree of difference;

[0122] A weighted average function is calculated using the first weight value, the second weight value, the first image recognition result, and the second image recognition result to obtain a fusion recognition result.

[0123] In one embodiment, the first recognition model uses the first feature map to quickly generate a preliminary segmentation mask M e (That is, the first image recognition result proposed in some embodiments of the present disclosure):

[0124] M e =f e (I)

[0125] Where I is the first feature map, f e is the segmentation function of the first recognition model.

[0126] The second recognition model uses more powerful computing resources to perform in-depth analysis and obtain the segmentation mask M c(That is, the second image recognition result proposed in some embodiments of the present disclosure):

[0127] M c =f c (I)

[0128] Among them, f c is the segmentation function of the second recognition model.

[0129] Compare M e With M c The consistency and difference between the two can be achieved by calculating the difference D between the two: D = |M e -M c |;

[0130] According to the confidence of the edge mask and the cloud mask, the weighted average method is used for fusion:

[0131] M f =w e ·M e +w c ·M c

[0132] w c =1-w e

[0133]

[0134] Among them, w e is the first weight value, w c is the second weight value, and α is an adjustment parameter used to control the influence of the difference D on the weight, which can be set according to actual needs.

[0135] Step 304: Send the fusion recognition result to the mine edge server, so that the mine edge server outputs the fusion recognition result.

[0136] In one embodiment, the second recognition model may be a visual large model SAM, and the SAM model may be trained in the following manner:

[0137] Use the pre-trained Vision Transformer (ViT) as the image encoder to process the input image and output image embedding;

[0138] For a given segmentation prompt, including points, boxes, text, or existing segmentation masks, different methods are used to convert these prompts into embedding vectors;

[0139] The mask decoder combines the image embedding and the hint embedding to predict the segmentation mask through a series of Transformer decoder blocks;

[0140] If the cue may correspond to multiple valid objects, the SAM model is designed to be able to predict multiple output masks to handle ambiguous cases;

[0141] The SAM model can predict multiple masks for a single prompt, usually predicting three masks by default to cover multiple possible reasonable segmentation situations;

[0142] Use a linear combination of focal loss and Dice loss to supervise the model's mask prediction and train it;

[0143] During training, we simulate an interactive segmentation setting by randomly sampling hints and iteratively refining the segmentation mask;

[0144] By building a data engine, iteratively using models to assist data collection, and using newly collected data to improve the model;

[0145] After training, SAM can segment new downstream tasks through zero-shot transfer without additional training;

[0146] For each input image and prompt, SAM finally outputs one or more segmentation masks, which can be used in various downstream tasks.

[0147] In one embodiment, the preliminary segmentation mask (i.e., the first image recognition result proposed in some embodiments of the present disclosure) can be compared and analyzed with the mask obtained by a higher precision or more comprehensive analysis in the cloud (i.e., the second image recognition result proposed in some embodiments of the present disclosure); then, through a certain fusion strategy, such as weighted average, voting mechanism, or confidence-based fusion method, the information of the two is integrated to improve the accuracy and robustness of the segmentation; and finally, an optimized cloud inference result (i.e., the fused recognition result proposed in some embodiments of the present disclosure) is generated that combines the fast response of edge computing and the advantages of deep analysis in the cloud. This method can make full use of the low latency of edge computing and the powerful processing power of cloud computing to achieve better image segmentation effects.

[0148] According to the mine image recognition method proposed in the embodiment of the present disclosure, an underground road image is obtained; a feature extraction process is performed on the underground road image to obtain a first feature map; the first feature map is input into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model; a loss function corresponding to the first image recognition result is calculated using a self-supervised learning algorithm to obtain a loss value; when the loss value does not meet the first preset condition, the underground road image and the first image recognition result are sent to a cloud server; a fusion recognition result sent by the cloud server is received, and a fusion recognition result is output; the fusion recognition result is obtained by the cloud server after the second image recognition result and the first image recognition result are fused; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image; when the loss value meets the first preset condition, the first image recognition result is output. The recognition result of the first model is divided into a recognition result that meets the requirements and a recognition result that does not meet the requirements by calculating the loss value, and further processed by the second model configured on the cloud server, thereby ensuring the recognition speed while improving the accuracy of the recognition result, and solving the problem of inaccurate recognition of the underground road image due to the complex underground environment.

[0149] Figure 4 is a block diagram of a mine image recognition device according to an exemplary embodiment. Figure 4 The device is applied to a mine edge server and includes an acquisition unit 401, a feature extraction unit 402, a first recognition unit 403, a calculation unit 404, a first sending unit 405, a first receiving unit 406 and an output unit 407.

[0150] Wherein, the acquisition unit 401 is used to acquire the underground road image;

[0151] A feature extraction unit 402 is used to perform feature extraction processing on the underground road image to obtain a first feature map;

[0152] A first recognition unit 403 is used to input the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model;

[0153] A calculation unit 404 is used to calculate a loss function corresponding to the first image recognition result by using a self-supervised learning algorithm to obtain a loss value;

[0154] The first sending unit 405 is used to send the underground road image and the first image recognition result to the cloud server when the loss value does not meet the first preset condition;

[0155] The first receiving unit 406 is used to receive the fusion recognition result sent by the cloud server and output the fusion recognition result; the fusion recognition result is obtained by the cloud server after fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server recognizing the underground road image using the second recognition model;

[0156] The output unit 407 is used to output the first image recognition result when the loss value meets the first preset condition.

[0157] In some embodiments of the present disclosure, the feature extraction unit 402 may be specifically used for:

[0158] Inputting the underground road image into the pre-trained Transformer model to obtain a first intermediate feature map obtained after the Transformer model performs long-distance feature extraction on the underground road image;

[0159] The intermediate feature map is processed with a hole rate to obtain a first feature map.

[0160] In some embodiments of the present disclosure, the feature extraction unit 402 may also be used to:

[0161] Obtain multiple preset void rates;

[0162] For each of the multiple dilation rates, convolution processing is performed on the first intermediate feature map using a dilation convolution layer corresponding to the dilation rate to obtain a second intermediate feature map;

[0163] The second intermediate feature maps corresponding to the plurality of hole rates are fused to obtain a first feature map.

[0164] In some embodiments of the present disclosure, the device may further include:

[0165] The training unit is used to train the first recognition model using the underground road image and the fusion recognition result to obtain the trained first recognition model.

[0166] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0167] According to the mine image recognition device proposed in the embodiment of the present disclosure, the underground road image is obtained; the underground road image is subjected to feature extraction processing to obtain a first feature map; the first feature map is input into the pre-trained first recognition model to obtain the first image recognition result output by the first recognition model; the loss function corresponding to the first image recognition result is calculated by using the self-supervised learning algorithm to obtain the loss value; when the loss value does not meet the first preset condition, the underground road image and the first image recognition result are sent to the cloud server; the fusion recognition result sent by the cloud server is received, and the fusion recognition result is output; the fusion recognition result is obtained by the cloud server after the second image recognition result and the first image recognition result are fused and processed; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image; when the loss value meets the first preset condition, the first image recognition result is output. The recognition result of the first model is divided into the recognition result that meets the requirements and the recognition result that does not meet the requirements by calculating the loss value, and the second model configured in the cloud server is used for further processing, so as to improve the accuracy of the recognition result while ensuring the recognition speed, and solve the problem of inaccurate recognition of the underground road image due to the complex underground environment.

[0168] Figure 5 is a block diagram of another mine image recognition device according to an exemplary embodiment, referring to Figure 5 The device is applied to a cloud server and includes a second receiving unit 501, a second identification unit 502, a fusion unit 503 and a second sending unit 504.

[0169] Among them, the second receiving unit 501 is used to receive the underground road image and the first image recognition result sent by the mine edge server; the first image recognition result is obtained by the mine edge server using the pre-trained first recognition model to recognize the underground road image; the loss value corresponding to the first image recognition result does not meet the first preset condition; the loss value is obtained by the mine edge server using the self-supervised learning algorithm to calculate the loss function corresponding to the first image recognition result;

[0170] The second recognition unit 502 is used to recognize the underground road image using the second recognition model to obtain a second image recognition result;

[0171] A fusion unit 503 is used to fuse the first image recognition result with the second image recognition result to obtain a fused recognition result;

[0172] The second sending unit 504 is used to send the fusion recognition result to the mine edge server, so that the mine edge server outputs the fusion recognition result.

[0173] In some embodiments of the present disclosure, the fusion unit 503 may be specifically used for:

[0174] Determining a degree of difference between the first image recognition result and the second image recognition result;

[0175] Determine a first preset weight value corresponding to the first image recognition result and a second preset weight value corresponding to the second image recognition result by using the degree of difference;

[0176] A weighted average function is calculated using the first weight value, the second weight value, the first image recognition result, and the second image recognition result to obtain a fusion recognition result.

[0177] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0178] According to the mine image recognition device proposed in the embodiment of the present disclosure, the underground road image is obtained; the underground road image is subjected to feature extraction processing to obtain a first feature map; the first feature map is input into the pre-trained first recognition model to obtain the first image recognition result output by the first recognition model; the loss function corresponding to the first image recognition result is calculated by using the self-supervised learning algorithm to obtain the loss value; when the loss value does not meet the first preset condition, the underground road image and the first image recognition result are sent to the cloud server; the fusion recognition result sent by the cloud server is received, and the fusion recognition result is output; the fusion recognition result is obtained by the cloud server after the second image recognition result and the first image recognition result are fused and processed; the second image recognition result is obtained by the cloud server using the second recognition model to recognize the underground road image; when the loss value meets the first preset condition, the first image recognition result is output. The recognition result of the first model is divided into the recognition result that meets the requirements and the recognition result that does not meet the requirements by calculating the loss value, and the second model configured in the cloud server is used for further processing, so as to improve the accuracy of the recognition result while ensuring the recognition speed, and solve the problem of inaccurate recognition of the underground road image due to the complex underground environment.

[0179] Figure 6 6 is a block diagram of a device for a mine image recognition method according to an exemplary embodiment. For example, the device 600 may be an electronic device, such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0180] Reference Figure 6, the device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0181] The processing component 602 generally controls the overall operation of the device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0182] The memory 604 is configured to store various types of data to support operations on the device 600. Examples of such data include instructions for any application or method operating on the device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0183] The power component 606 provides power to the various components of the device 600. The power component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 600.

[0184] The multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0185] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the device 600 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0186] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0187] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the device 600. For example, the sensor assembly 614 can detect the open / closed state of the device 600, the relative positioning of components, such as the display and keypad of the device 600, and the sensor assembly 614 can also detect the position change of the device 600 or a component of the device 600, the presence or absence of user contact with the device 600, the orientation or acceleration / deceleration of the device 600, and the temperature change of the device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0188] The communication component 616 is configured to facilitate wired or wireless communication between the device 600 and other devices. The device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0189] In an exemplary embodiment, the apparatus 600 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0190] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of the device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0191] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, which implements the above method when executed by the processor 620 of the device 600 .

[0192] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0193] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A mine image recognition method, characterized in that: Applied to mine edge servers, including: Acquire underground road images; Performing feature extraction processing on the underground road image to obtain a first feature map; Inputting the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model; Calculating a loss function corresponding to the first image recognition result using a self-supervised learning algorithm to obtain a loss value; When the loss value does not satisfy the first preset condition, sending the underground road image and the first image recognition result to a cloud server; receiving a fusion recognition result sent by a cloud server, and outputting the fusion recognition result; the fusion recognition result is obtained by the cloud server fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server recognizing the underground road image using a second recognition model; When the loss value satisfies a first preset condition, the first image recognition result is output.

2. The mine image recognition method according to claim 1, characterized in that: The step of performing feature extraction processing on the underground road image to obtain a first feature map includes: Inputting the underground road image into a pre-trained Transformer model to obtain a first intermediate feature map obtained after the Transformer model performs long-distance feature extraction on the underground road image; The intermediate feature map is subjected to hole rate processing to obtain the first feature map.

3. The mine image recognition method according to claim 2, characterized in that: The step of performing hole rate processing on the first intermediate feature map to obtain the first feature map includes: Obtain multiple preset void rates; For each of the multiple dilation rates, using a dilation convolution layer corresponding to the dilation rate to perform convolution processing on the first intermediate feature map to obtain a second intermediate feature map; The second intermediate feature maps corresponding to the plurality of hole rates are fused to obtain the first feature map.

4. The mine image recognition method according to claim 1, characterized in that: After receiving the fusion recognition result sent by the cloud server, the method further includes: The first recognition model is trained using the underground road image and the fusion recognition result to obtain the trained first recognition model.

5. A mine image recognition method, characterized in that: Applied to cloud servers, including: receiving an underground road image and a first image recognition result sent by a mine edge server; the first image recognition result is obtained by the mine edge server recognizing the underground road image using a pre-trained first recognition model; a loss value corresponding to the first image recognition result does not meet a first preset condition; the loss value is obtained by the mine edge server calculating a loss function corresponding to the first image recognition result using a self-supervised learning algorithm; Using a second recognition model to recognize the underground road image, to obtain a second image recognition result; Fusing the first image recognition result with the second image recognition result to obtain a fused recognition result; The fusion recognition result is sent to the mine edge server, so that the mine edge server outputs the fusion recognition result.

6. The mine image recognition method according to claim 5, characterized in that: The fusing the first image recognition result and the second image recognition result to obtain a fused recognition result includes: Determining a degree of difference between the first image recognition result and the second image recognition result; Determining a first preset weight value corresponding to the first image recognition result and a second preset weight value corresponding to the second image recognition result by using the difference degree; A weighted average function is calculated using the first weight value, the second weight value, the first image recognition result, and the second image recognition result to obtain the fusion recognition result.

7. A mine image recognition device, characterized in that: Applied to mine edge servers, including: An acquisition unit, used for acquiring an underground road image; A feature extraction unit, used to perform feature extraction processing on the underground road image to obtain a first feature map; A first recognition unit, configured to input the first feature map into a pre-trained first recognition model to obtain a first image recognition result output by the first recognition model; A calculation unit, configured to calculate a loss function corresponding to the first image recognition result by using a self-supervised learning algorithm to obtain a loss value; A first sending unit, configured to send the underground road image and the first image recognition result to a cloud server when the loss value does not satisfy a first preset condition; A first receiving unit is used to receive a fusion recognition result sent by a cloud server and output the fusion recognition result; the fusion recognition result is obtained by the cloud server fusing the second image recognition result with the first image recognition result; the second image recognition result is obtained by the cloud server recognizing the underground road image using a second recognition model; An output unit is used to output the first image recognition result when the loss value meets a first preset condition.

8. A mine image recognition device, characterized in that: Applied to cloud servers, including: A second receiving unit is used to receive an underground road image and a first image recognition result sent by a mine edge server; the first image recognition result is obtained by the mine edge server recognizing the underground road image using a pre-trained first recognition model; the loss value corresponding to the first image recognition result does not meet a first preset condition; the loss value is obtained by the mine edge server calculating a loss function corresponding to the first image recognition result using a self-supervised learning algorithm; A second recognition unit, used to recognize the underground road image using a second recognition model to obtain a second image recognition result; a fusion unit, configured to fuse the first image recognition result with the second image recognition result to obtain a fused recognition result; The second sending unit is used to send the fusion recognition result to the mine edge server, so that the mine edge server outputs the fusion recognition result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented, or the method according to any one of claims 5 to 6 is executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented, or the method according to any one of claims 5 to 6 is executed.

Images