Mine scene segmentation method and system based on graph neural network and contrastive learning

Abstract

The embodiment of the application provides a mine scene segmentation method and system based on a graph neural network and contrast learning, which can call a segmentation model after obtaining a mine area image. The encoder of the segmentation model extracts features in the mine area image to obtain feature representation of the mine area image. And the decoder of the segmentation model expands the feature representation to the original dimension of the mine area image to generate and output a segmentation result image. The encoder of the segmentation model further comprises an LORA fine-tuning module, which can adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during the training process. The method can improve the efficiency and accuracy of mine monitoring by applying deep learning to the field of mine identification. The automatic image analysis and interpretation process is used to quickly identify the mine boundary and improve the segmentation efficiency.

Description

Technical Field

[0001] This application relates to the field of image segmentation technology, and in particular to a method and system for segmenting mining scenes based on graph neural networks and contrastive learning. Background Technology

[0002] Mine scene segmentation refers to the image segmentation process of separating mining areas, such as mining sites, water pools, stockpiles, and auxiliary buildings, from remote sensing images of mining areas through image feature recognition. Mine scene segmentation is a coarse-grained research object, characterized by complex and diverse features, with features from different regions easily confounding each other, making large-scale accurate identification difficult. Therefore, for mine scenes and related downstream tasks, manual field surveys or manual interpretation of remote sensing images are often used to identify and segment mine scenes, but this image segmentation method is inefficient.

[0003] To improve the efficiency of mine scene segmentation, image processing algorithms such as deep learning models can also be used for mine scene segmentation. Image processing algorithms can extract corresponding feature regions from mine scene images by reading features such as color, texture, edge, and shape, and performing operations such as region segmentation, region merging, region splitting, and boundary adjustment.

[0004] However, due to limitations in the amount of data available for mine segmentation and shortcomings in model design, even with the powerful image processing capabilities of deep learning models, effective classification and segmentation of complex scenes like mines remains difficult due to significant intra-class discrepancies. This hinders efficient and rapid identification of mine boundaries. Furthermore, the over-reliance on expert knowledge and the need for extensive manual visual interpretation further contribute to the low efficiency of mine scene image segmentation. Summary of the Invention

[0005] In view of this, embodiments of this application provide a mining scene segmentation method and system based on graph neural networks and contrastive learning to solve the problem of low efficiency in mining scene image segmentation.

[0006] According to one aspect of this application, a mining scene segmentation method based on graph neural networks and contrastive learning is provided, the method comprising:

[0007] Acquire images of the mining area;

[0008] The segmentation model is invoked, which is a deep learning model obtained by contrastive learning training through an additional output head; the segmentation model includes an encoder and a decoder, the encoder includes a pre-trained SAM backbone network and a low-rank adaptive LORA fine-tuning module; the LORA fine-tuning module is configured to adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training; the decoder includes a feature propagation tool based on a graph neural network.

[0009] The encoder extracts features from the image of the mining area to obtain a feature representation of the image of the mining area.

[0010] The decoder expands the feature representation to the original dimensions of the mining area image to generate a segmentation result image;

[0011] Output the segmentation result image.

[0012] Optionally, the method further includes:

[0013] Obtain a training dataset, which includes a large-scale dataset and a mining scene dataset; the large-scale dataset and the mining scene dataset are labeled with segmentation result labels;

[0014] The training dataset is input into the segmentation model to obtain the training result image output by the segmentation model;

[0015] The comparative learning loss is calculated based on the training result image and the segmentation result label;

[0016] The contrastive learning loss is used to optimize the model parameters of the segmentation model.

[0017] Optionally, the training dataset is input into the segmentation model to obtain the training result image output by the segmentation model, including:

[0018] The SAM backbone network is pre-trained using the large-scale dataset.

[0019] The mining scene dataset is input into the pre-trained SAM backbone network to obtain a set of feature representations.

[0020] Based on the low-rank matrix, graph feature aggregation is performed on the feature representation set to obtain aggregated image data;

[0021] The decoder performs upsampling on the aggregated image data to generate the training result image.

[0022] Optionally, a contrastive learning loss is calculated based on the training result image and the segmentation result label, including:

[0023] The feature representation extracted by the encoder is enhanced by the additional output head;

[0024] Obtain the predicted mask and corresponding segmentation result label output in each round of training;

[0025] The training dataset is divided into a positive sample set and a negative sample set based on the segmentation result label. The positive sample set includes pixels of the same segmentation category as the segmentation result label, and the negative sample set includes pixels of different segmentation categories than the classification label.

[0026] The contrastive learning loss is calculated based on the positive sample set and the negative sample set.

[0027] Optionally, the contrastive learning loss is calculated based on the positive sample set and the negative sample set, including:

[0028] Based on the predicted mask and the segmentation result label corresponding to the predicted mask, a sampled pixel set is constructed, wherein the sampled pixel set includes pixels whose segmentation result label is positive and whose predicted mask is negative;

[0029] Multiple random samples are taken from the set of sampled pixels;

[0030] The contrastive learning loss is calculated based on the positive and negative sample sets, which are anchored by multiple random sampling results.

[0031] Optionally, the encoder performs feature extraction on the mining area image to obtain a feature representation of the mining area image, including:

[0032] Raw data is read from the image of the mining area, the raw data including the pixel values ​​of the pixels in the image of the mining area;

[0033] Obtain the downsampling weight matrix and downsampling bias vector of the SAM backbone network;

[0034] Based on the downsampling weight matrix and the downsampling bias vector, the original data is mapped to the target feature space to obtain the hidden layer output data;

[0035] Based on the linear rectifier unit activation function, the hidden layer output data is transformed into activated output data. The linear rectifier unit activation function is used to set the negative values ​​in the hidden layer output data to zero, while keeping the positive values ​​unchanged.

[0036] The feature representation of the mining area image is generated based on the activated output data.

[0037] Optionally, generating a feature representation of the mining area image based on the activated output data includes:

[0038] Obtain the upsampling weight matrix and the upsampling bias vector;

[0039] The output vector is calculated based on the activation output data, the upsampling weight matrix, and the upsampling bias vector to obtain the feature representation of the mining area image.

[0040] Optionally, the method further includes:

[0041] Obtain the feature representation and two-dimensional coordinates of the first pixel and the second pixel, where the first pixel and the second pixel are two pixels in the image of the mining area;

[0042] Calculate the cosine similarity between the first pixel and the second pixel based on the feature representation;

[0043] Calculate the coordinate distance between the first pixel and the second pixel based on the two-dimensional coordinates;

[0044] Learnable parameters are obtained, and an adjacency matrix is ​​constructed based on the learnable parameters, the cosine similarity, and the coordinate distance.

[0045] An edge set is constructed based on the adjacency matrix, and a graph vector is constructed based on the edge set, the graph vector including a node set and an edge set.

[0046] Optionally, the method further includes:

[0047] A diagonal matrix is ​​constructed based on the adjacency matrix, wherein the diagonal elements of the diagonal matrix are the sum of the elements of the corresponding rows in the adjacency matrix;

[0048] The adjacency matrix is ​​normalized based on the diagonal matrix to generate a normalized matrix, which is obtained by multiplying the adjacency matrix by the inverse square root of the diagonal matrix.

[0049] The normalized matrix is ​​multiplied by the original feature matrix to perform node feature aggregation;

[0050] A new feature representation is obtained by linearly transforming the node feature aggregation result using a weight matrix.

[0051] According to another aspect of this application, a mine scene segmentation system based on graph neural networks and contrastive learning is provided, the system comprising:

[0052] The image acquisition module is used to acquire images of the mining area;

[0053] The model invocation module is used to invoke a segmentation model, which is a deep learning model trained through contrastive learning with an additional output head. The segmentation model includes an encoder and a decoder. The encoder includes a pre-trained SAM backbone network and a low-rank adaptive LORA fine-tuning module. The LORA fine-tuning module is configured to adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training. The decoder includes a feature propagation tool based on a graph neural network.

[0054] The feature extraction module is used to extract features from the image of the mining area through the encoder to obtain a feature representation of the image of the mining area.

[0055] The segmentation image generation module is used to expand the feature representation to the original dimensions of the mine area image through the decoder to generate a segmentation result image;

[0056] The result output module is used to output the segmentation result image.

[0057] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described mining scene segmentation method based on graph neural networks and contrastive learning.

[0058] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described mining scene segmentation method based on graph neural networks and contrastive learning.

[0059] Based on the above technical solutions, this application provides a method and system for mine scene segmentation based on graph neural networks and contrastive learning. The method can call a segmentation model after acquiring an image of the mine area. The encoder of the segmentation model extracts features from the mine area image to obtain a feature representation of the mine area image. The decoder of the segmentation model expands the feature representation to the original dimension of the mine area image to generate and output the segmentation result image. The encoder of the segmentation model also includes a LORA fine-tuning module, which can adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training. This method can improve the efficiency and accuracy of mine monitoring by applying deep learning to the field of mine identification. It utilizes an automated image analysis and interpretation process to quickly identify mine boundaries, improving segmentation efficiency. Furthermore, by using graph convolutional networks to capture complex topological structures and pixel relationships in the image, and combining contrastive loss to optimize distance relationships in the feature space, the model's segmentation accuracy and robustness under different terrain conditions can be improved, adapting to the complex environment and varied landform features of mines.

[0060] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0061] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0062] Figure 1 This is a schematic diagram of the mining scene segmentation process provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the client-server connection relationship provided in an embodiment of this application;

[0064] Figure 3 A schematic diagram of a mining scene segmentation method based on graph neural networks and contrastive learning provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the output segmentation result image flow provided in the embodiments of this application;

[0066] Figure 5 This is a schematic diagram of the segmentation model training process provided in an embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the graph feature aggregation process provided in an embodiment of this application;

[0068] Figure 7 A schematic diagram of the loss calculation process provided in the embodiments of this application;

[0069] Figure 8 This is a schematic diagram of the image conversion process provided in an embodiment of this application;

[0070] Figure 9 This is a schematic diagram of the feature transfer process provided in an embodiment of this application;

[0071] Figure 10 This is a schematic diagram of a mine scene segmentation system based on graph neural networks and contrastive learning, provided as an embodiment of this application. Detailed Implementation

[0072] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0073] In this embodiment, the mine scene segmentation refers to the image processing process of segmenting mine scene areas such as mining sites, water pools, stockpiles, and auxiliary buildings from a mine area image by performing pixel feature recognition. Figure 1 As shown.

[0074] The raw data for mine scene segmentation is the mine area image. A mine area image refers to image data generated from remote sensing monitoring of a mine area. Mine area images can be obtained by remote sensing equipment such as high-resolution satellites capturing images of the mine area. Alternatively, mine area images can be obtained by stitching together multiple local images captured by detection equipment such as visual radar and remote sensing aircraft, based on the geographical locations of the corresponding areas in the local images.

[0075] The result data obtained from the segmentation of the mine scene is the segmentation result image. The segmentation result image is an image with the same parameters characterizing the image specifications, such as size and resolution, as the image of the mine area. To characterize the segmentation result, the segmented regions can be labeled in the segmentation result image using color, boundaries, and markers. Pixels within each segmented region can have the same labeling method. For example, if the segmentation result image is a binarized image, then pixels in the mining area are set to the foreground color, while pixels outside the mining area are set to the background color.

[0076] In this embodiment, the Graph Neural Network (GNN) is a deep learning framework specifically designed for processing graph-structured data. GNNs can transform graph-structured data into a standardized representation by formulating certain strategies on nodes and edges in the graph, and then input this representation into various different neural networks for training. In a GNN, nodes and edges are the basic elements constituting the graph. Nodes represent entities, which can be pixels, sets of pixels, or the entire image. A node can be associated with a feature vector, which can contain the node's attribute information. In a GNN, nodes can act as the center of information processing; that is, nodes receive information from neighboring nodes and update their own feature representations. When performing classification tasks, the nodes themselves are the targets of classification, and the GNN needs to predict the class label for each node.

[0077] Edges are used to represent relationships or interactions between nodes, such as feature distance, actual distance, and correlation between pixels. In Generative Neural Networks (GNNs), edges define the paths through which information is transmitted between nodes. That is, a node can obtain information about its neighbors through edges. Edges can also have weights, representing the strength or probability of the relationship; these weights can be used to adjust the intensity of information transmission. Edges can also have directionality, determining the direction of information flow. In a directed graph, the direction of an edge indicates the direction of information flow; in an undirected graph, an edge represents a bidirectional information flow relationship between nodes.

[0078] In Generative Neural Networks (GNNs), nodes and edges collectively define the structure of a graph. By aggregating information from neighboring nodes, nodes can capture the structural features of their local neighborhoods, while edges provide the paths for this aggregation. This structured information transfer and update mechanism enables GNNs to effectively process graph data and achieve excellent performance in various graph-related tasks.

[0079] In this embodiment, the contrastive learning is an unsupervised learning method that learns feature representations of data by comparing the similarities and differences between different samples. Contrastive learning can focus on constructing positive and negative sample pairs. A positive sample pair refers to two samples with similar features, such as different angles or different lighting conditions of the same object in an image. A negative sample pair refers to two samples with different features, such as images of different objects.

[0080] Contrastive learning can improve a model's understanding of data by reducing the distance between positive sample pairs and increasing the distance between negative sample pairs. Therefore, it can learn meaningful feature representations even without labels. By comparing sample pairs, high-quality feature representations can be extracted and applied to tasks such as classification, clustering, and retrieval.

[0081] Therefore, in this embodiment of the application, the mine scene segmentation method based on graph neural networks and contrastive learning is an image segmentation approach that integrates graph neural networks and contrastive learning. It is used to segment images of mine areas to determine various scene regions from the images. After image segmentation, segmentation result images can be obtained, which can be used to perform subsequent related calculations, such as mine area display, setting up security zones, industrial land planning, and geological risk assessment.

[0082] It should be noted that the mine scene segmentation method based on graph neural networks and contrastive learning can run on devices with data processing capabilities, such as computers, mobile terminals, servers, industrial hosts, and smart wearable devices. The device with data processing capabilities can be a single device or a combination of multiple devices. That is, in a feasible implementation, the mine scene segmentation method can run on a single device, meaning all steps of the mine scene segmentation method can be executed by a single device. For example, the mine scene segmentation method can be integrated into an application installed on a personal computer; following user interaction commands, the personal computer can execute the corresponding steps of the method according to the application.

[0083] In another feasible implementation, the mine scene segmentation method can be run on multiple devices, meaning it requires the coordinated operation of multiple devices to execute all the steps corresponding to the mine scene segmentation method. For example, Figure 2 As shown, a server and multiple clients connected to the server can be configured. These clients can be operated by different users, and can send control commands or application data to the server via the network. The server can then respond to the control commands by further processing, distributing, and storing the application data, and feeding back the processing results to the clients, thereby realizing the image segmentation function corresponding to the described mine scene segmentation method.

[0084] For ease of description, in some embodiments of this application, one or more devices performing the mine scene segmentation method are collectively referred to as data processing devices. It should be understood that the data processing device refers to a combination of one or more devices, and other implementation methods conceived by those skilled in the art based on the data processing device described in this application by changing the combination of devices also fall within the scope of protection of this application.

[0085] like Figure 3 As shown in the embodiments of this application, a mine scene segmentation method based on graph neural networks and contrastive learning includes:

[0086] S101. Obtain images of the mining area.

[0087] When performing mine scene segmentation, the data processing equipment first needs to acquire images of the mine area. These images can be obtained through remote sensing monitoring of the mine area. To acquire these images, the data processing equipment can establish a communication connection with the remote sensing monitoring equipment or its corresponding data storage device.

[0088] The images of the mining area acquired by the data processing equipment can be real-time images. Taking remote sensing monitoring equipment as an example, when acquiring images of the mining area, the data processing equipment can send an image acquisition request to the remote sensing monitoring equipment. After receiving the image acquisition request, the remote sensing monitoring equipment can take pictures of the mining area to obtain images. The captured images of the mining area are then sent to the data processing equipment.

[0089] The images of the mining area acquired by the data processing equipment can also be historical images. Taking a data storage device as an example, the data storage device can establish communication connections with both the data processing equipment and the remote sensing monitoring equipment. During remote sensing monitoring, the captured images of the mining area can be sent to the data storage device for storage. When the data processing equipment needs to acquire images of the mining area, it can send an image acquisition request to the data storage device. Upon receiving the image acquisition request, the data storage device then sends the mining area images back to the data processing equipment.

[0090] In some embodiments, after acquiring images of the mining area, the data processing device can also preprocess the acquired images. Specifically, the data processing device can perform preprocessing such as grayscale conversion, noise removal, contrast enhancement, and histogram equalization on the acquired images of the mining area. Preprocessing can reduce the amount of data required for subsequent image segmentation, remove noise and interference from the image, make the image clearer, and improve the image's contrast and brightness.

[0091] S102, Invoke the segmentation model.

[0092] After acquiring images of the mining area, the data processing device can invoke a segmentation model. The data processing device can invoke the segmentation model in different ways. In some embodiments, the data processing device includes a server and a client. The segmentation model can be stored on the server, so after acquiring the mining image, the client can send a model invocation request to the server. Upon receiving the model invocation request, the server can send model parameters to the client, enabling the client to construct a segmentation model based on the acquired model parameters. The server can also receive the model invocation request and the mining area image sent by the client, run the segmentation model on the server, and perform scene segmentation processing on the mining area image using the segmentation model.

[0093] In some embodiments, the segmentation model can be stored in the local memory of the data processing device. Then, after acquiring the mine image, the data processing device can directly call and run the application corresponding to the segmentation model from the local memory, and use the segmentation model to perform scene segmentation processing on the mine area image.

[0094] To perform mine scene segmentation, an additional output head can be added to the segmentation model, allowing it to be trained through contrastive learning using this additional output head to obtain a deep learning model suitable for scene segmentation. The segmentation model includes an encoder and a decoder. The encoder comprises a pre-trained Segment Anything Model (SAM) backbone network and a Low-Rank Adaptation (LORA) fine-tuning module. The LORA fine-tuning module is configured to adjust the model parameters of the SAM backbone network during training by introducing a low-rank matrix. The decoder includes a feature propagation tool based on a graph neural network.

[0095] That is, Figure 4 As shown, the segmentation model can use an encoder-decoder structure as the model structure for mine segmentation. The encoder uses a pre-trained SAM backbone network combined with LORA fine-tuning for feature extraction. The decoder uses a graph neural network as a feature propagation tool to enhance the model's decoding capability. During training, a contrastive learning extra output head can also be used to enhance the model's feature perception capability. The LORA fine-tuning module adjusts the model's weights by introducing an additional low-rank matrix, thereby achieving rapid adaptation to new tasks. LORA fine-tuning ensures that the weight changes during task adaptation are low-rank, meaning they can be represented as the product of two smaller matrices. For example, the original weight matrix W0 remains unchanged, while the weight update ΔW is decomposed into the product of two low-rank matrices B and A, i.e., W = W0 + BA, where B ∈ R. d×r and A∈R r×k r is the rank of these two matrices, and r << min(d, k).

[0096] During training, W0 remains constant, and only B and A are trained parameters. Compared to full parameter tuning, LoRA fine-tuning can significantly reduce the number of parameters that need to be trained, thereby reducing computational and storage costs. Furthermore, during forward propagation, the input x is transformed through the original weight matrix W0, and simultaneously through the low-rank matrices A and B; the final output is a weighted sum of the two.

[0097] In image segmentation models, the LoRA fine-tuning module can be used to fine-tune the SAM encoder to adapt it to specific downstream tasks. By introducing the LoRA fine-tuning module into the SAM encoder, the model can be adjusted to suit new tasks without significantly increasing its size, thereby improving the model's flexibility and adaptability. LoRA fine-tuning is suitable for handling large-scale datasets and complex tasks, reducing the need for training resources while maintaining model performance.

[0098] To perform scene segmentation in mines based on graph neural networks and contrastive learning, a segmentation model needs to be pre-trained. Therefore, as... Figure 5 As shown, in some embodiments, the data processing device can acquire a training dataset. This training dataset includes sample images and training labels for the sample images. During image segmentation, the sample images are the original images, and the training labels are the segmentation result labels, i.e., the segmentation masks corresponding to the sample images. The training dataset can include large-scale datasets and mining scene datasets, both of which are labeled with segmentation result labels.

[0099] A large-scale dataset is a dataset used for image segmentation tasks. It can include raw images and segmentation masks from various domains, allowing models to be trained on a wide range of data and adapted to specific tasks. For example, the large-scale dataset could be the SA-18 large-scale dataset. The SA-18 large-scale dataset can include 100 million masks applied to 11 million images.

[0100] The mine scene dataset is also a dataset for image segmentation tasks. It includes sample images of mine scenes and corresponding segmentation masks. The mine scene dataset can make trained models more suitable for the technical field of mine scene segmentation.

[0101] After acquiring the training dataset, the data processing device can input the training dataset into the segmentation model to obtain the training result image output by the segmentation model. In some embodiments, such as Figure 6 As shown, to obtain the training result images, after inputting the training dataset into the segmentation model, the data processing device can perform feature extraction and mask generation based on the encoder and decoder in the segmentation model, that is, pre-train the SAM backbone network using the large-scale dataset. For example, the SA-18 large-scale dataset can be used for model pre-training. The SA-18 large-scale dataset can contain images of various scenes and can be used to train the model to recognize and segment different objects. During the pre-training stage, the network parameters based on the SAM backbone network can be iteratively modified to enable the encoder to recognize and understand the basic features and objects in the image.

[0102] After pre-training, the data processing device can input the mine scene dataset into the pre-trained SAM backbone network to obtain a set of feature representations. Since the amount of mine scene data is relatively small, it falls under the category of data-scarce tasks. Therefore, for tasks with insufficient data, the LORA fine-tuning module can be used to fine-tune the SAM backbone network to adapt it to the needs of these tasks. Specifically, the data processing device can perform graph feature aggregation on the feature representation set based on the low-rank matrix to obtain aggregated image data. After pre-training, the encoder based on the SAM backbone network can extract image features and perform feature aggregation to form a high-level understanding of the image content.

[0103] The feature map processed by the SAM encoder is upsampled by an upsampling layer in the decoder to restore the resolution of the original image. Therefore, the data processing device can perform upsampling on the aggregated image data through the decoder to generate the training result image.

[0104] After obtaining the training result image, the data processing device can calculate the contrastive learning loss based on the training result image and the segmentation result label. That is, after acquiring the training result image, the data processing device can compare the training result image with the segmentation result label to determine the difference between the training result image and the segmentation result label, thereby evaluating the output accuracy of the current segmentation model. To evaluate the accuracy of the current segmentation model, the training loss between the training result image and the segmentation result label can be calculated using a loss function. This loss function can be based on one or more combinations of algorithms such as cross-entropy loss, Dice loss, Focal loss, Tversky loss, structural similarity loss, mean squared error loss, and Hausdorff distance loss to obtain the training loss.

[0105] In some embodiments, the training loss can be a contrastive learning loss. To calculate this loss, the data processing device can enhance the feature representation extracted by the encoder using the additional output head. Then, the prediction mask and corresponding segmentation result label output from each training round are obtained. Based on the segmentation result labels, the training dataset is divided into a positive sample set and a negative sample set. The positive sample set includes pixels of the same segmentation category as the segmentation result label; the negative sample set includes pixels of different segmentation categories than the segmentation label. Finally, the contrastive learning loss is calculated based on the positive and negative sample sets.

[0106] During model training, the segmentation model can use an additional output head to enhance the features extracted by the model encoder. LoRA is applied to fine-tune the SAM Encoder, and when outputting the training result image, an upsampling module can be used to embed the high-dimensional features back to the original image size while preserving their feature dimensions.

[0107] When calculating the contrastive learning loss, a sampled pixel set can be constructed based on the predicted mask and the segmentation result label corresponding to the predicted mask. The sampled pixel set includes pixels whose segmentation result label is positive and whose predicted mask is negative.

[0108] like Figure 7 As shown, for each training round, the SAM Encoder fine-tuned using LoRA will have two sets of outputs: one set is the predicted mask P. mask The other set consists of F-type images of the same size as the original image, output by the feature decoder. decoder According to P mask And the corresponding segmentation result label. A pixel-level correspondence is designed for comparative learning. For pixel i, its positive samples are pixels with the same segmentation result label, and its negative samples are pixels with different labels. Thus, the following loss function is obtained:

[0109]

[0110] Among them, P i Let N be the set of positive samples corresponding to pixel i. i Let be the set of negative samples, and τ be a hyperparameter used to control the convergence of the model. According to P... mask And the corresponding Label, let Label be positive and P mask The set of negative pixels is U notreacll =Label-P mask .

[0111] This subset of pixels should be the focus of optimization; therefore, multiple random samplings are performed from this subset during each training iteration. The results of these multiple random samplings are used as the positive and negative sample sets for anchor points to calculate the contrastive learning loss. By performing multiple random samplings on the subset of pixels and incorporating the results as positive and negative samples for anchor points into the calculation, the model's ability to perceive the features of these pixels can be enhanced. Alternatively, the distribution of pixels in the feature space can be clustered, using the cluster centers as anchor points, and outliers can be treated as hard samples and included in the loss function calculation.

[0112] After calculating the contrastive learning loss as provided in the above embodiments, the data processing device can use the contrastive learning loss to optimize the model parameters of the segmentation model. That is, iterative optimization is performed using the contrastive learning loss. In each iteration, the data processing device compares the contrastive learning loss with a preset loss condition. If the contrastive learning loss meets the preset loss condition (e.g., the contrastive learning loss is less than or equal to a loss threshold), the segmentation model can be output. If the contrastive learning loss does not meet the preset loss condition (e.g., the contrastive learning loss is greater than the loss threshold), iterative optimization can continue until the contrastive learning loss meets the preset loss condition.

[0113] After the segmentation model is trained using the model training method provided in the above embodiments, the data processing device can apply the segmentation model to perform scene segmentation. Therefore, after calling the segmentation model, the data processing device can input the image of the mining area into the segmentation model, so that the segmentation model can extract features from the image of the mining area and generate a segmentation result image.

[0114] S103. The encoder extracts features from the image of the mining area to obtain a feature representation of the image of the mining area.

[0115] After inputting the image of the mining area into the segmentation model, the encoder in the segmentation model can first extract features from the image of the mining area. Since the SAM backbone network in the encoder can be fine-tuned by LoRA, introducing an additional low-rank matrix through LoRA fine-tuning can achieve effective adjustment of the model with relatively small parameter costs.

[0116] like Figure 8 As shown, in some embodiments, the data processing device can read raw data from the image of the mining area. The raw data includes the pixel values ​​of pixels in the image of the mining area. Then, the downsampling weight matrix and downsampling bias vector of the SAM backbone network are obtained. Based on the downsampling weight matrix and the downsampling bias vector, the raw data is mapped to the target feature space to obtain the hidden layer output data.

[0117] After acquiring the image of the mining area, the pixels in the image can be iterated to obtain the original data x. in Then obtain the downsampling weight matrix W. down and downsampling bias vector b down Then calculate the hidden layer output data x according to the following formula. hid ,Right now:

[0118] x hid =D down (x in ) = W down ×x in +bdown

[0119] After calculating the hidden layer output data, the data processing device further transforms the hidden layer output data into activated output data based on the linear rectifier unit activation function. The linear rectifier unit activation function is used to set the negative values ​​in the hidden layer output data to zero, while keeping the positive values ​​unchanged.

[0120] The Rectified Linear Unit (ReLU) activation function can map linearly inseparable data to a high-dimensional space, making it linearly separable. The ReLU activation function can transform the hidden layer output data x... hid Negative values ​​in the hidden layer are set to zero, while positive values ​​remain unchanged, thus changing the output data x of the hidden layer. hid Convert into activated output data x act ,Right now:

[0121] x act =ReLU(x hid ) = max(0, x hid )

[0122] Output data x from the hidden layer hid Convert into activated output data x act Subsequently, the data processing device can generate a feature representation of the mining area image based on the activated output data. By obtaining the upsampling weight matrix and the upsampling bias vector, an output vector can be calculated based on the activated output data, the upsampling weight matrix, and the upsampling bias vector to obtain the feature representation of the mining area image.

[0123] Obtaining activation output data x act Afterwards, the data processing device can obtain the upsampling weight matrix W. up and upsampling bias vector b up Then, the output vector x is calculated according to the following formula. out ,Right now:

[0124] x out =D up (x act ) = W up ×x act +b up

[0125] S104. The feature representation is extended to the original dimension of the mining area image by the decoder to generate a segmentation result image.

[0126] After obtaining the feature representation of the mining area image, the data processing equipment can upsample the feature representation using a decoder to expand it to the original dimensions of the mining area image, thereby generating a segmentation result image. The segmentation result image maintains the same parameters as the mining area image, such as image size and resolution.

[0127] Because the decoder includes a feature propagation tool based on graph neural networks, the graph neural network model has a complex topological structure. Therefore, using graph neural networks to model open-pit mines can make the output more adaptable to the complex structure of the mine.

[0128] A mining area can be further subdivided into several regions, such as open-pit mines, water pools, stockpiles, and ancillary buildings. In image segmentation tasks, these regions are grouped into a single class. However, the features of these regions can vary significantly. Therefore, relying solely on complex prompts (task descriptions) to guide the model across features for segmentation will result in substantial intra-class feature discrepancies. Thus, a graph model's message passing and feature aggregation mechanisms can be used to highlight the features of the mining area, enhancing the model's segmentation reasoning ability for this region.

[0129] That is, Figure 9 As shown, in some embodiments, when performing a mine scene segmentation task, the data processing device can acquire the feature representations and two-dimensional coordinates of a first pixel and a second pixel. The first pixel and the second pixel are two pixels in the image of the mine area. For example, after acquiring the feature representation output by the encoder, the decoder can read the data in the feature representation to obtain the feature representation and two-dimensional coordinates. That is, F i F is the feature of the first pixel (i). j Pos is a feature of the second pixel (j). i Pos represents the two-dimensional coordinates of the first pixel (i) on the feature map. j Let j be the two-dimensional coordinates of the second pixel (j point) on the feature map.

[0130] The cosine similarity between the first pixel and the second pixel is calculated based on the feature representation, and the coordinate distance between the first pixel and the second pixel is calculated based on the two-dimensional coordinates. For example, the cosine similarity algorithm can be used to calculate the cosine similarity based on the feature F of the first pixel (i). i The feature F of the second pixel (j point) j Calculate the cosine similarity (cos(F)) between points i and j before considering their features. i F j And based on the two-dimensional coordinates Pos of the first pixel (i) on the feature map. iThe two-dimensional coordinates Pos of the second pixel (j) on the feature map. j Calculate the L1 distance between the coordinates of points i and j on the feature map, i.e.:

[0131] L1 = ||Pos i Pos j ||1

[0132] Next, learnable parameters, such as γ and λ, are obtained, and an adjacency matrix A is constructed based on the learnable parameters, the cosine similarity, and the coordinate distance, i.e.:

[0133]

[0134] Then, an edge set is constructed based on the adjacency matrix, and a graph vector is constructed based on the edge set. The graph vector includes a node set and an edge set. For example, the feature distance and actual distance between nodes in the feature graph are fused as the edge weights. For the image segmentation task of a mining area, feature points are the nodes, and the edge set is constructed based on the above formula. Then, for a graph G = (V, E), there are N nodes v i ∈V, and the corresponding edge (v i v j )∈E.

[0135] When constructing a feature map, the weights of edges between nodes should encompass two meanings: the distance between pixel features and the actual topological distance between pixels on the graph. Therefore, edge values ​​between nodes can be adaptively constructed, with larger edge weights assigned to the parts that are actually connected in the topological relationship, thereby increasing the information transmission capacity of this part.

[0136] To perform feature transfer, in some embodiments, the data processing device can construct a diagonal matrix based on the adjacency matrix. The diagonal elements of the diagonal matrix are the sum of the elements of corresponding rows in the adjacency matrix. For example, for adjacency matrix A, self-loops (connections between nodes) can be created in the graph to transform adjacency matrix A into an adjacency matrix containing self-loops. Then calculate the adjacency matrix. Sum the elements of each row in the matrix to construct a diagonal matrix. That is, a diagonal matrix Each diagonal element It is an adjacency matrix The sum of the i-th row is the degree of node i.

[0137] The adjacency matrix is ​​then normalized based on the diagonal matrix to generate a normalized matrix. This normalized matrix is ​​obtained by multiplying the adjacency matrix by the inverse square root of the diagonal matrix. After obtaining the diagonal matrix, its inverse square root can be calculated. Then multiply the adjacency matrix by the inverse square root of the diagonal matrix to obtain the normalized matrix.

[0138] The normalized matrix is ​​then multiplied by the original feature matrix to perform node feature aggregation. The aggregation result is then linearly transformed using a weight matrix to obtain a new feature representation. Specifically, the new feature representation X′ is calculated according to the following formula:

[0139]

[0140] in, Let X be the adjacency matrix containing self-loops, W be the input feature matrix, and W be the network parameters. Let be the diagonal matrix of the graph. Feature propagation can then follow the basic GCN propagation method, utilizing multiple convolutional layers to learn the weights for information propagation.

[0141] S105. Output the segmentation result image.

[0142] After generating the segmentation result image, the data processing device can acquire the segmentation result image generated by the segmentation model and output the segmentation result image to downstream tasks or other applications. For example, the output segmentation result image can be displayed through a visualization application to show the distribution location of various scenes in the mining area.

[0143] By applying the technical solution of this embodiment, the method can call a segmentation model after acquiring an image of a mining area. The encoder of the segmentation model extracts features from the mining area image to obtain a feature representation of the mining area image. The decoder of the segmentation model then expands the feature representation to the original dimensions of the mining area image to generate and output the segmented result image. The encoder of the segmentation model also includes a LORA fine-tuning module, which can adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training. This method can improve the efficiency and accuracy of mine monitoring by applying deep learning to the field of mine identification. Utilizing an automated image analysis and interpretation process, it can quickly identify mine boundaries, improving segmentation efficiency. Furthermore, by using graph convolutional networks to capture complex topological structures and pixel relationships in the image, and combining contrastive loss to optimize distance relationships in the feature space, the model's segmentation accuracy and robustness under different terrain conditions can be improved, adapting to the complex environment and varied geomorphological features of mines.

[0144] Furthermore, as a specific implementation of the mine scene segmentation method based on graph neural networks and contrastive learning described in the above embodiments, this application provides a mine scene segmentation system based on graph neural networks and contrastive learning, such as... Figure 10 As shown, the system includes:

[0145] The image acquisition module is used to acquire images of the mining area;

[0146] The model invocation module is used to invoke a segmentation model, which is a deep learning model trained through contrastive learning with an additional output head. The segmentation model includes an encoder and a decoder. The encoder includes a pre-trained SAM backbone network and a low-rank adaptive LORA fine-tuning module. The LORA fine-tuning module is configured to adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training. The decoder includes a feature propagation tool based on a graph neural network.

[0147] The feature extraction module is used to extract features from the image of the mining area through the encoder to obtain a feature representation of the image of the mining area.

[0148] The segmentation image generation module is used to expand the feature representation to the original dimensions of the mine area image through the decoder to generate a segmentation result image;

[0149] The result output module is used to output the segmentation result image.

[0150] The mine scene segmentation system based on graph neural networks and contrastive learning provided in the above embodiments can improve the efficiency and accuracy of mine monitoring by applying deep learning to the field of mine identification. Furthermore, by utilizing automated image analysis and interpretation processes, it can quickly identify mine boundaries, reducing time and manpower costs compared to manual interpretation methods, while also lowering the false positive rate. Moreover, by using graph convolutional networks (GCNs) to capture complex topological structures and pixel relationships in images, and combining contrastive loss to optimize distance relationships in the feature space, the system can significantly improve segmentation accuracy and robustness under different terrain conditions, better adapting to the complex environment and varied landforms of mines. In addition, the system is not only suitable for monitoring mine boundaries but can also be extended to other types of complex scenes, improving the versatility and expandability of the technology.

[0151] It should be noted that other corresponding descriptions of the functional units involved in the mine scene segmentation system based on graph neural networks and contrastive learning provided in the embodiments of this application can be found in the corresponding descriptions in the mine scene segmentation method based on graph neural networks and contrastive learning provided in the above embodiments, and will not be repeated here.

[0152] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0153] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0154] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0155] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0157] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0158] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0159] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0160] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for segmenting mine scenes based on graph neural networks and contrastive learning, characterized in that, The method includes: Acquire images of the mining area; The segmentation model is invoked, which is a deep learning model obtained by contrastive learning training through an additional output head; the segmentation model includes an encoder and a decoder, the encoder includes a pre-trained SAM backbone network and a low-rank adaptive LORA fine-tuning module; the LORA fine-tuning module is configured to adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training; the decoder includes a feature propagation tool based on a graph neural network. The encoder extracts features from the mining area image to obtain a feature representation of the mining area image, including: reading raw data from the mining area image, the raw data including pixel values ​​of pixels in the mining area image; obtaining the downsampling weight matrix and downsampling bias vector of the SAM backbone network; mapping the raw data to a target feature space according to the downsampling weight matrix and the downsampling bias vector to obtain hidden layer output data; converting the hidden layer output data into activated output data based on a linear rectified unit activation function, the linear rectified unit activation function being used to set negative values ​​in the hidden layer output data to zero and retain positive values ​​unchanged; and generating a feature representation of the mining area image based on the activated output data. The decoder expands the feature representation to the original dimensions of the mining area image to generate a segmentation result image; Output the segmentation result image.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a training dataset, which includes a large-scale dataset and a mining scene dataset; the large-scale dataset and the mining scene dataset are labeled with segmentation result labels; The training dataset is input into the segmentation model to obtain the training result image output by the segmentation model; The comparative learning loss is calculated based on the training result image and the segmentation result label; The contrastive learning loss is used to optimize the model parameters of the segmentation model.

3. The method according to claim 2, characterized in that, The training dataset is input into the segmentation model to obtain the training result image output by the segmentation model, including: The SAM backbone network is pre-trained using the large-scale dataset. The mining scene dataset is input into the pre-trained SAM backbone network to obtain a set of feature representations. Based on the low-rank matrix, graph feature aggregation is performed on the feature representation set to obtain aggregated image data; The decoder performs upsampling on the aggregated image data to generate the training result image.

4. The method according to claim 2, characterized in that, The contrastive learning loss is calculated based on the training result image and the segmentation result label, including: The feature representation extracted by the encoder is enhanced by the additional output head; Obtain the predicted mask and corresponding segmentation result label output in each round of training; The training dataset is divided into a positive sample set and a negative sample set based on the segmentation result label. The positive sample set includes pixels of the same segmentation category as the segmentation result label, and the negative sample set includes pixels of different segmentation categories than the segmentation result label. The contrastive learning loss is calculated based on the positive sample set and the negative sample set.

5. The method according to claim 4, characterized in that, The contrastive learning loss is calculated based on the positive sample set and the negative sample set, including: Based on the predicted mask and the segmentation result label corresponding to the predicted mask, a sampled pixel set is constructed, wherein the sampled pixel set includes pixels whose segmentation result label is positive and whose predicted mask is negative; Multiple random samples are taken from the set of sampled pixels; The contrastive learning loss is calculated based on the positive and negative sample sets, which are anchored by multiple random sampling results.

6. The method according to claim 1, characterized in that, The feature representation of the mining area image is generated based on the activated output data, including: Obtain the upsampling weight matrix and the upsampling bias vector; The output vector is calculated based on the activation output data, the upsampling weight matrix, and the upsampling bias vector to obtain the feature representation of the mining area image.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the feature representation and two-dimensional coordinates of the first pixel and the second pixel, where the first pixel and the second pixel are two pixels in the image of the mining area; Calculate the cosine similarity between the first pixel and the second pixel based on the feature representation; Calculate the coordinate distance between the first pixel and the second pixel based on the two-dimensional coordinates; Learnable parameters are obtained, and an adjacency matrix is ​​constructed based on the learnable parameters, the cosine similarity, and the coordinate distance. An edge set is constructed based on the adjacency matrix, and a graph vector is constructed based on the edge set, the graph vector including a node set and an edge set.

8. The method according to claim 7, characterized in that, The method further includes: A diagonal matrix is ​​constructed based on the adjacency matrix, wherein the diagonal elements of the diagonal matrix are the sum of the elements of the corresponding rows in the adjacency matrix; The adjacency matrix is ​​normalized based on the diagonal matrix to generate a normalized matrix, which is obtained by multiplying the adjacency matrix by the inverse square root of the diagonal matrix. The normalized matrix is ​​multiplied by the original feature matrix to perform node feature aggregation; A new feature representation is obtained by linearly transforming the node feature aggregation result using a weight matrix.

9. A mine scene segmentation system based on graph neural networks and contrastive learning, characterized in that, The system includes: The image acquisition module is used to acquire images of the mining area; The model invocation module is used to invoke a segmentation model, which is a deep learning model trained through contrastive learning with an additional output head. The segmentation model includes an encoder and a decoder. The encoder includes a pre-trained SAM backbone network and a low-rank adaptive LORA fine-tuning module. The LORA fine-tuning module is configured to adjust the model parameters of the SAM backbone network by introducing a low-rank matrix during training. The decoder includes a feature propagation tool based on a graph neural network. The feature extraction module is used to extract features from the mining area image through the encoder to obtain a feature representation of the mining area image. The module includes: reading raw data from the mining area image, the raw data including pixel values ​​of pixels in the mining area image; obtaining the downsampling weight matrix and downsampling bias vector of the SAM backbone network; mapping the raw data to a target feature space based on the downsampling weight matrix and the downsampling bias vector to obtain hidden layer output data; converting the hidden layer output data into activated output data based on a linear rectified unit activation function, where the linear rectified unit activation function sets negative values ​​in the hidden layer output data to zero and retains positive values ​​unchanged; and generating a feature representation of the mining area image based on the activated output data. The segmentation image generation module is used to expand the feature representation to the original dimensions of the mine area image through the decoder to generate a segmentation result image; The result output module is used to output the segmentation result image.

Images