Multi-modal lithology identification method and system

Abstract

The invention discloses a multi-modal lithology identification method and system. The method comprises the following steps: acquiring a rock sample image and a rock mesoscopic image; preprocessing the obtained images, and inputting the preprocessed rock sample images and rock mesoscopic images into the trained fusion model for real-time identification; outputting a lithology identification result by the fusion model, and naming the image according to the identification result; according to the method, the lithology of the surrounding rock is identified by establishing the fusion model based on deep learning, the fusion model can fuse the color, texture, shape and other features of the rock sample image with the crystal structure, pore distribution, particle size and other features of the rock mesoscopic image, mapping is formed by corresponding rock relations, and therefore the purpose of automatic identification is achieved; the lithology intelligent identification method based on deep learning is high in identification speed, low in cost, high in accuracy and high in automation degree, and the requirements for professional knowledge and identification experience of constructors are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of geotechnical engineering and tunnel engineering, and in particular to a multi-modal lithology identification method and system. Background Art

[0002] Tunnel construction often encounters unfavorable geological bodies such as faults, karst caves, and underground rivers, which induce large-scale geological disasters such as water inrush and mud inrush. Unfavorable geological conditions such as faults and karst are the biggest culprits and challenges for tunnel disasters. Through geological forecasting, abnormal conditions can be discovered in a timely manner, providing a basis for adjusting the construction plan, effectively preventing catastrophic accidents in tunnels, and ensuring construction safety.

[0003] Lithology identification is a basic and important issue in the fields of geology, resource exploration, tunnel construction, rock mechanics and engineering, and directly affects the selection of construction plans and the reliability of reservoir evaluation. Traditional lithology identification methods include: (1) visual observation, (2) thin section identification and (3) physical testing, which rely on professional experience and are time-consuming. With the advancement of computer technology, artificial intelligence technology has begun to be applied to lithology identification, and intelligent lithology identification methods based on data mining and image recognition have continued to emerge. In recent years, domestic and foreign scholars have conducted a series of studies on lithology identification with the help of image recognition technology, but a relatively complete intelligent lithology identification method in tunnels has not yet been established, and there is a lack of portable lithology identification equipment for engineering applications. At the same time, the accuracy and speed of lithology identification of surrounding rocks need to be improved. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a multimodal lithology identification method and system, which can perform real-time identification based on collected rock samples and rock mesoscopic images, and realize efficient storage, sharing and collaboration of field data through a remote data sharing platform, which is conducive to improving the speed and accuracy of field lithology identification.

[0005] The technical solution of the present invention is as follows:

[0006] In a first aspect of the present invention, a multimodal lithology identification method is provided, comprising the following steps:

[0007] Obtain rock sample images and rock mesoscopic images;

[0008] Preprocess the acquired images, and input the preprocessed rock sample images and rock mesoscopic images into the trained fusion model for real-time recognition;

[0009] The fusion model outputs the lithology identification results and names the images according to the identification results;

[0010] The establishment process of the fusion model is as follows: an image classification algorithm is adopted, with Shufflenet as the framework, a convolutional neural network based on an attention mechanism is used to automatically extract the features of color, texture and shape for rock sample images, and a convolutional neural network is used to automatically extract the features of crystal structure, pore distribution and particle size for rock mesoscopic images, and the weighted feature splicing method is used to fuse the two features, and then the SoftMax activation function is used for multi-category classification to obtain a CNN model that integrates feature extraction and fusion.

[0011] In some embodiments of the present invention, the attention mechanism-based convolutional neural network introduces an attention mechanism in the convolutional layer, so that the network can pay more attention to the color and texture information in the image when extracting features, and at the same time strengthen the feature map representing color and texture by increasing weights.

[0012] In some embodiments of the present invention, the weighted feature concatenation method assigns weights to the extracted color and texture features and then concatenates all the features to obtain a unified feature representation that focuses on color and texture features. The specific formula is as follows:

[0013] x concat =[x 1 ;x 2 ;…;x n ]

[0014] [;] represents the connection operation, which connects x horizontally. 1 , x 2 …,x n Spliced ​​together to form a new feature vector x concat .

[0015] In some embodiments of the present invention, the training process of the fusion model includes: using the data in the training set to train the fusion model, and using the back propagation algorithm and the gradient descent optimization algorithm to update the model parameters, and then using the data in the test set to evaluate the performance of the trained model, and tuning the model according to the evaluation results, and finally applying the optimized convolutional neural network model to the actual lithology identification task to realize the automatic identification of rock types.

[0016] In some embodiments of the present invention, the image preprocessing process is: adjusting the brightness, contrast and color balance of the image.

[0017] In some embodiments of the present invention, the rock sample image is acquired through a camera, and the rock mesoscopic image is acquired through an electronic magnifying glass.

[0018] In a second aspect of the present invention, a multimodal lithology identification system is provided, comprising:

[0019] The image acquisition module is configured to: acquire rock sample images and rock mesoscopic images;

[0020] The lithology recognition module is configured to: pre-process the acquired images, and input the pre-processed rock sample images and rock mesoscopic images into the trained fusion model for real-time recognition;

[0021] The data interaction module is configured to: output the lithology identification result of the fusion model and name the image according to the identification result;

[0022] The establishment process of the fusion model is as follows: an image classification algorithm is adopted, with Shufflenet as the framework, a convolutional neural network based on an attention mechanism is used to automatically extract the features of color, texture and shape for rock sample images, and a convolutional neural network is used to automatically extract the features of crystal structure, pore distribution and particle size for rock mesoscopic images, and the weighted feature splicing method is used to fuse the two features, and then the SoftMax activation function is used for multi-category classification to obtain a CNN model that integrates feature extraction and fusion.

[0023] In some embodiments of the present invention, the image acquisition module, lithology identification module and data interaction module are arranged on a handheld box-type device, wherein the image acquisition module includes a camera and an electronic magnifying glass, the lithology identification module adopts a Nano core board and a carrier board, and the data interaction module adopts a display screen and an HDMI driver board.

[0024] In some embodiments of the present invention, the Nano core board and carrier board are used to provide computing power, and the lithology identification function is realized by deploying the trained fusion model to the Nano development board, wherein the specific deployment steps are as follows:

[0025] Environment configuration, complete the basic environment configuration of Nano development board;

[0026] Model conversion: convert the original model trained by PyTorch into a TensorRT model suitable for the Nano development board;

[0027] Model call: use Python or C++ API to call the TensorRT model to implement model reasoning.

[0028] In some embodiments of the present invention, a WIFI module is further included, and the WIFI module connects the handheld box-type device with a remote data sharing platform.

[0029] One or more technical solutions of the present invention have the following beneficial effects:

[0030] (1) The present invention establishes a deep learning model based on rock images to identify the lithology of the surrounding rock, which can automatically extract rock features and form a mapping corresponding to the lithology relationship, thereby achieving the purpose of automatic identification. This method has high recognition accuracy, fast speed, and low cost, and reduces the requirements for professional knowledge and identification experience of construction personnel.

[0031] (2) The present invention uses a feature fusion method to fuse the color, texture, and shape features of rock sample images with the crystal structure, pore distribution, and particle size features of rock mesoscopic images, thereby providing more comprehensive and integrated geological information, facilitating in-depth analysis of the composition and structure of rocks, and improving the robustness of the model, making the model more robust and reliable when faced with diverse data.

[0032] (3) The present invention integrates the multimodal rock type identification system into a portable handheld device, which does not affect TBM construction during operation and can reduce the impact of adverse working environments such as dimness, humidity and dust in the tunnel on identification, providing a lightweight, flexible, accurate and efficient rock type identification technology for on-site construction, thus realizing automated identification.

[0033] (4) The present invention utilizes a remote data sharing platform to achieve real-time storage, sharing and collaboration of on-site images, allowing data resources to be fully utilized, reducing repetitive work and corresponding costs such as image acquisition and data collection, and expanding the application scope of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the multimodal lithology identification method of the present invention;

[0035] Figure 2 is a structural diagram of the multi-modal lithology identification system of the present invention;

[0036] FIG3(a) is a top view of the handheld box-type device, FIG3(b) is a perspective view of the handheld box-type device, and FIG3(c) is a front view of the handheld box-type device;

[0037] Figure 4 is a schematic diagram of a metal housing of a handheld box-type device of the present invention;

[0038] Figure 5 It is a diagram of the modules in the handheld box-type device of the present invention.

[0039] In the picture: 1. Metal shell; 2. Camera; 3. Nano core board and carrier board; 4. Display screen and HDMI driver board; 5. Battery and charging and discharging board; 6. WIFI module; 7. A shell; 8. B shell; 9. Sealing ring; 10. Screws; 11. Rib; 12. USB interface. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] Example 1

[0042] In a typical embodiment of the present invention, a multimodal lithology identification method is proposed, such as Figure 1 As shown, the following steps are included:

[0043] Obtain rock sample images and rock mesoscopic images;

[0044] Preprocess the acquired images, and input the preprocessed rock sample images and rock mesoscopic images into the trained fusion model for real-time recognition;

[0045] The fusion model outputs the lithology identification results and names the images according to the identification results;

[0046] The establishment process of the fusion model is as follows: an image classification algorithm is adopted, with Shufflenet as the framework, a convolutional neural network based on an attention mechanism is used to automatically extract the features of color, texture and shape for rock sample images, and a convolutional neural network is used to automatically extract the features of crystal structure, pore distribution and particle size for rock mesoscopic images, and the weighted feature splicing method is used to fuse the two features, and then the SoftMax activation function is used for multi-category classification to obtain a CNN model that integrates feature extraction and fusion.

[0047] Furthermore, the attention mechanism-based convolutional neural network introduces an attention mechanism in the convolutional layer, so that the network can pay more attention to the color and texture information in the image when extracting features, and at the same time strengthen the feature map representing color and texture by increasing weights.

[0048] Furthermore, the weighted feature concatenation method assigns weights to the extracted color and texture features, and then concatenates all the features to obtain a unified feature representation that focuses on color and texture features. The specific formula is as follows:

[0049] x concat =[x 1 ;x 2 ;…;x n ]

[0050] [;] represents the connection operation, which connects x horizontally. 1 , x 2 …,x n Spliced ​​together to form a new feature vector x concat .

[0051] Furthermore, the training process of the fusion model includes: using the data in the training set to train the fusion model, and using the back propagation algorithm and gradient descent optimization algorithm to update the model parameters, and then using the data in the test set to evaluate the performance of the trained model, and tuning the model according to the evaluation results, and finally applying the optimized convolutional neural network model to the actual lithology identification task to realize the automatic identification of rock types.

[0052] In this embodiment, the image preprocessing process is as follows: the lithology recognition module automatically processes the image using traditional image processing technology, and makes the rock features in the image more clearly visible by increasing or decreasing the overall brightness of the image; by increasing the local contrast of the image or adjusting the curve of the image, the contrast between the rock and the background in the image is enhanced to highlight the details and texture features of the rock; by removing the color cast in the image or adjusting the color distribution, the hue, saturation and color temperature of the image are adjusted to make the color of the rock more realistic and accurately reflect the actual situation. The standard of preprocessing is to make the processed image have sufficient clarity and contrast, and to highlight the structure, texture and color information of the rock, so that the lithology recognition module can accurately extract and analyze the rock features while reducing the influence of noise and background interference.

[0053] In this embodiment, the rock sample image is acquired through a camera, and the rock mesoscopic image is acquired through an electronic magnifying glass.

[0054] Example 2

[0055] In a typical embodiment of the present invention, a multi-modal lithology identification system is proposed, such as Figure 2 As shown, including:

[0056] The image acquisition module is configured to: acquire rock sample images and rock mesoscopic images;

[0057] The lithology recognition module is configured to: pre-process the acquired images, and input the pre-processed rock sample images and rock mesoscopic images into the trained fusion model for real-time recognition;

[0058] The data interaction module is configured to: output the lithology identification result of the fusion model and name the image according to the identification result;

[0059] The establishment process of the fusion model is as follows: an image classification algorithm is adopted, with Shufflenet as the framework, a convolutional neural network based on an attention mechanism is used to automatically extract the features of color, texture and shape for rock sample images, and a convolutional neural network is used to automatically extract the features of crystal structure, pore distribution and particle size for rock mesoscopic images, and the weighted feature splicing method is used to fuse the two features, and then the SoftMax activation function is used for multi-category classification to obtain a CNN model that integrates feature extraction and fusion.

[0060] like Figure 3(a)-3(c) as well as Figure 4 As shown, the image acquisition module, lithology recognition module and data interaction module are arranged on a handheld box-type device, wherein the image acquisition module includes a camera 2 and an electronic magnifying glass, and uses a camera to obtain rock sample images, and an electronic magnifying glass to obtain rock mesoscopic images, and transmits the images to the lithology recognition module and stores them. The lithology recognition module uses a Nano core board and a carrier board 3 to provide computing power, and realizes the lithology recognition function by deploying the trained fusion model to the Nano development board. The data interaction module uses a display screen and an HDMI driver board 4, which can display the recognition results and named images, store and upload images and recognition results, and also includes manually modifying the recognition results, deleting images and adding remark information to realize the human-computer interaction function. It also includes a core control module for controlling the operating status and instruction transmission of other modules.

[0061] Specifically, the housing of the handheld box-type device is a metal housing, which is divided into an A shell 7 and a B shell 8. The A and B shells are sealed with a sealing ring 9 and connected by screws 10, wherein the sealing ring 9 is placed in the groove of the A shell, and the rear shell Rib11 positively deforms the sealing ring 9. The B shell 8 is designed as a double-wall structure, and the sealing Rib11 is the inner wall, and its parting surface is a plane, which is conducive to sealing. A USB interface 12 is designed on the outside of the B shell 8 for connecting external devices such as an electronic magnifying glass. The USB interface is designed with a soft rubber plug, and the contour is deformed by extrusion to achieve a sealing effect.

[0062] In this embodiment, the display screen and shell of the handheld box-type device are made of high-strength materials, and the device is ensured to have waterproof, dust-resistant and drop-proof properties through sealing and anti-penetration technologies.

[0063] Furthermore, the rock sample image is captured by camera 2, which is designed as an automatic zoom camera equipped with a flash, and takes into account properties such as waterproof, dust-resistant and drop-resistant, to ensure normal shooting in the dim environment of the tunnel. The "take a photo" command is sent through the core control module, and camera 2 automatically takes pictures.

[0064] Furthermore, the Nano core board and the carrier board 3 are combined into a Nano development board for providing computing power. The lithology identification function is realized by deploying the trained fusion model to the Nano development board. The specific deployment steps are as follows:

[0065] Environment configuration, complete the basic environment configuration of Nano development board;

[0066] Model conversion: convert the original model trained by PyTorch into a TensorRT model suitable for the Nano development board;

[0067] Model call: use Python or C++ API to call the TensorRT model to implement model reasoning.

[0068] The lithology identification module has built-in multiple trained fusion models, including built-in models, cloud models and local models. The built-in models are installed with the software and do not support separate upgrades. The cloud models are the latest models released on the server side and support separate upgrades. The local models are models imported through the import model function.

[0069] Furthermore, a display screen and an HDMI driver board 4 are used to display the camera shooting images and recognition results in real time and realize the human-computer interaction function. A battery and a charge-discharge board 5 are used to ensure that the portable rock property recognition device can work continuously in the tunnel.

[0070] Furthermore, it also includes a WIFI module 6, which connects the handheld box-type device with a remote data sharing platform.

[0071] The module layout of Nano core board and carrier board 2, camera 3, display screen and HDMI driver board 4, battery and charge and discharge board 5 and WIFI 6 is as follows Figure 5 shown.

[0072] The data interaction module adopts a remote data sharing platform, which integrates the uploading, storage and Internet-based remote transmission technology of images and recognition results, thereby realizing the sharing and processing of images and recognition results by multiple users.

[0073] The method for using the multi-modal portable handheld box-type device provided in this embodiment is as follows:

[0074] A. The core control module sends instructions to the controller to control the camera 3 to start up and take rock sample images, and to control the electronic magnifying glass connected to the USB interface 12 to start up and take rock surface images, and the images are transmitted to the lithology identification module.

[0075] B. Preprocess the collected images and optimize their quality by adjusting the brightness, contrast and color balance of the images. Use traditional image processing technology to process the images, increase or decrease the overall brightness of the image to make the rock features in the image more visible; increase the local contrast of the image or adjust the curve of the image to enhance the contrast between the rock and the background in the image to highlight the details and texture features of the rock; adjust the hue, saturation and color temperature of the image by removing the color cast in the image or adjusting the color distribution, so that the color of the rock more realistically and accurately reflects the actual situation, thereby achieving the purpose of optimizing image quality.

[0076] C. Import the trained cloud model and local model or load the built-in model for lithology identification, and feed the identification results back to the data interaction module.

[0077] D. After the recognition is completed, the image is named according to the recognition result. If the recognition is inaccurate, the recognition result can be manually modified.

[0078] E. Store and upload images and recognition results; click Delete to delete images and recognition results.

[0079] F. Repeat steps AE until the lithology of the surrounding rock is identified.

[0080] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal lithology identification method, characterized in that: The following steps are involved: Obtain rock sample images and rock mesoscopic images; Preprocess the acquired images, and input the preprocessed rock sample images and rock mesoscopic images into the trained fusion model for real-time recognition; The fusion model outputs the lithology identification results and names the images according to the identification results; The establishment process of the fusion model is as follows: an image classification algorithm is adopted, with Shufflenet as the framework, a convolutional neural network based on an attention mechanism is used to automatically extract the features of color, texture and shape for rock sample images, and a convolutional neural network is used to automatically extract the features of crystal structure, pore distribution and particle size for rock mesoscopic images, and the weighted feature splicing method is used to fuse the two features, and then the SoftMax activation function is used for multi-category classification to obtain a CNN model that integrates feature extraction and fusion.

2. The multimodal lithology identification method according to claim 1, characterized in that: The attention-based convolutional neural network introduces an attention mechanism in the convolutional layer, so that the network can pay more attention to the color and texture information in the image when extracting features, and at the same time strengthen the feature map representing color and texture by increasing weights.

3. The multimodal lithology identification method according to claim 2, characterized in that: The weighted feature concatenation method assigns weights to the extracted color and texture features, and then concatenates all the features to obtain a unified feature representation that focuses on color and texture features. The specific formula is as follows: x concat =[x1;x2;…;x n ] [;] represents the connection operation, which connects x1, x2, ..., x n Spliced ​​together to form a new feature vector x concat .

4. The multimodal lithology identification method according to claim 1, characterized in that: The training process of the fusion model includes: using the data in the training set to train the fusion model, and using the back propagation algorithm and gradient descent optimization algorithm to update the model parameters, and then using the data in the test set to evaluate the performance of the trained model, and tuning the model according to the evaluation results. Finally, the optimized convolutional neural network model is applied to the actual lithology identification task to realize the automatic identification of rock types.

5. The multimodal lithology identification method according to claim 1, characterized in that: The image preprocessing process is to adjust the brightness, contrast and color balance of the image.

6. The multimodal lithology identification method according to claim 1, characterized in that: The rock sample image is acquired through a camera, and the rock mesoscopic image is acquired through an electronic magnifying glass.

7. A multimodal lithology identification system, characterized in that: include: The image acquisition module is configured to: acquire rock sample images and rock mesoscopic images; The lithology recognition module is configured to: pre-process the acquired images, and input the pre-processed rock sample images and rock mesoscopic images into the trained fusion model for real-time recognition; The data interaction module is configured to: output the lithology identification result of the fusion model and name the image according to the identification result; The establishment process of the fusion model is as follows: an image classification algorithm is adopted, with Shufflenet as the framework, a convolutional neural network based on an attention mechanism is used to automatically extract the features of color, texture and shape for rock sample images, and a convolutional neural network is used to automatically extract the features of crystal structure, pore distribution and particle size for rock mesoscopic images, and the weighted feature splicing method is used to fuse the two features, and then the SoftMax activation function is used for multi-category classification to obtain a CNN model that integrates feature extraction and fusion.

8. The multimodal lithology identification system according to claim 7, characterized in that: The image acquisition module, lithology identification module and data interaction module are arranged on a handheld box-type device, wherein the image acquisition module includes a camera and an electronic magnifying glass, the lithology identification module adopts a Nano core board and a carrier board, and the data interaction module adopts a display screen and an HDMI driver board.

9. The multimodal lithology identification system according to claim 8, characterized in that: The Nano core board and carrier board are used to provide computing power. The lithology identification function is realized by deploying the trained fusion model to the Nano development board. The specific deployment steps are as follows: Environment configuration, complete the basic environment configuration of Nano development board; Model conversion: convert the original model trained by PyTorch into a TensorRT model suitable for the Nano development board; Model call: use Python or C++ API to call the TensorRT model to implement model reasoning.

10. The multimodal lithology identification system according to claim 8, characterized in that: It also includes a WIFI module, which connects the handheld box-type device with a remote data sharing platform.

Images