Graph data augmentation method and device, computer device and readable storage medium

Abstract

The application discloses a kind of graph data augmentation method, device, computer equipment and readable storage medium, related to internet technology and digital medical field, define multiple candidate augmentation strategies to simulate the noise or anomaly existing in medical graph data.By executing candidate augmentation strategy, modify the node, edge, image feature or label of medical original image to increase the diversity and complexity of augmented medical graph data.The method comprises: defining multiple candidate augmentation strategies;In response to the graph data augmentation instruction, extract the statistical characteristics of the original image, perform strategy screening on the multiple candidate augmentation strategies according to the statistical characteristics, and obtain the augmentation strategy vector;Iterate through each element in the augmentation strategy vector, determine all specified candidate augmentation strategies according to the element value of each element, and execute, modify or retain the node, edge, image feature or label of the original image to obtain the target image;The target image and the original image are input into the graph machine learning model for model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet and digital medicine, and particularly relates to a graph data augmentation method and device, a computer device and a readable storage medium. BACKGROUND

[0002] With the continuous development of the Internet and digital medical fields, more and more medical devices or medical service APPs with image analysis functions have appeared. An image analysis model obtains medical images of patients for health analysis to output health analysis results. The image analysis model needs to use a large amount of medical image data in the process of model training, and the medical image data is often sensitive and difficult to obtain. Therefore, a graph data augmentation technology appears, which is a technology for increasing the amount of training data by modifying or generating graph data to improve the generalization ability and performance of a graph machine learning model. The existing label graph data augmentation method is to expand the graph data by pseudo-labeling or data mixing, but it ignores the influence of different degrees and types of noise on graph data augmentation. Therefore, there is an urgent need for a graph data augmentation method that can simulate and introduce various noises, abnormalities or attack situations that may exist in the graph data in the augmentation process, thereby improving the robustness and generalization ability of the graph machine learning model for complex and real-world data. SUMMARY

[0003] Therefore, the present application provides a graph data augmentation method, device, computer device and readable storage medium, which mainly aims to solve the problem that the existing label graph data augmentation method ignores the influence of different degrees and types of noise on graph data augmentation.

[0004] According to a first aspect of the present application, a graph data augmentation method is provided, which comprises:

[0005] defining a plurality of candidate augmentation strategies, the plurality of candidate augmentation strategies being used to simulate noises, abnormalities or attacks existing in the graph data, including but not limited to a node discarding strategy, an edge perturbation strategy, a feature perturbation strategy and a label smoothing strategy;

[0006] in response to a graph data augmentation instruction, extracting statistical features of an original image, performing strategy screening on the plurality of candidate augmentation strategies according to the statistical features, and obtaining an augmentation strategy vector;

[0007] traversing each element in the augmentation strategy vector, determining all specified candidate augmentation strategies according to an element value of each element, and performing modification or reservation on nodes, edges, image features or labels of the original image to obtain a target image;

[0008] inputting the target image and the original image into a graph machine learning model for model training.

[0009] According to a second aspect of the present application, a graph data augmentation device is provided, which comprises:

[0010] A setting module is configured to define a plurality of candidate augmentation strategies for simulating noise, anomalies or attacks existing in the graph data, including but not limited to a node discard strategy, an edge perturbation strategy, a feature perturbation strategy and a label smoothing strategy.

[0011] A strategy selection module is configured to extract statistical features of an original image in response to a graph data augmentation instruction, perform strategy screening on the plurality of candidate augmentation strategies according to the statistical features, and obtain an augmentation strategy vector.

[0012] A strategy execution module is configured to traverse each element in the augmentation strategy vector, determine all specified candidate augmentation strategies according to an element value of each element, and perform modification or retention on nodes, edges, image features or labels of the original image to obtain a target image.

[0013] A self-supervised learning module is configured to input the target image and the original image into a graph machine learning model for model training.

[0014] According to a third aspect of the present application, a computer device is provided, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method of any one of the first aspect when executing the computer program.

[0015] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the method of any one of the first aspect when executed by a processor.

[0016] By the technical scheme, the method, the device, the computer equipment and the readable storage medium are provided, the application firstly defines a plurality of candidate augmentation strategies for simulating noise, anomaly or attack existing in the graph data. Next, in response to the graph data augmentation instruction, the statistical features of the original image are extracted. Then, the plurality of candidate augmentation strategies are strategy screened according to the statistical features, and the augmentation strategy vector is obtained. Further, each element in the augmentation strategy vector is traversed, and according to the element value of each element, all specified candidate augmentation strategies are determined and executed, and the nodes, edges, image features or labels of the original image are modified or retained to obtain the target image. Finally, the target image and the original image are input into the graph machine learning model for model training. The plurality of candidate augmentation strategies defined in the embodiment of the application are used to simulate and introduce various noise, anomaly or attack conditions that may exist in the graph data. By executing these candidate augmentation strategies, the nodes, edges, image features or labels of the original image are modified to increase the diversity and complexity of the augmented graph data. When the medical image machine learning model is trained using these diverse and complex training samples, the robustness and generalization ability of the model in processing complex and real data can be improved, and the adaptability of the medical image machine learning model to different scenes is further improved.

[0017] The above description is only a summary of the technical scheme of the application. In order to enable the technical means of the application to be more clearly understood, the following detailed description of the preferred embodiments of the application can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the application more obvious and easy to understand, the following detailed description of the preferred embodiments of the application is provided. BRIEF DESCRIPTION OF DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the scope of the application. Moreover, the same reference numerals in the attached drawings indicate the same or similar components. In the drawings:

[0019] Figure 1 A graph data augmentation method flow diagram provided by an embodiment of the application is shown;

[0020] Figure 2A A graph data augmentation method flow diagram provided by an embodiment of the application is shown;

[0021] Figure 2B A graph data augmentation method flow diagram provided by an embodiment of the application is shown;

[0022] Figure 3 A structure diagram of a graph data augmentation device provided by an embodiment of the application is shown;

[0023] Figure 4A device structure schematic diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood, and the scope of the present application can be accurately conveyed to those skilled in the art.

[0025] An embodiment of the present application provides a graph data augmentation method, as shown in the figure, the method comprises: Figure 1

[0026] 101, define a plurality of candidate augmentation strategies, the plurality of candidate augmentation strategies are used to simulate the noise, anomaly or attack existing in the graph data, including but not limited to node discard strategy, edge disturbance strategy, feature disturbance strategy and label smoothing strategy.

[0027] ​With the continuous development of Internet technology and the field of digital medicine, there are more and more medical devices or medical service APPs with image analysis function. The image analysis model obtains the medical image of the patient for health analysis to output the health analysis result. The image analysis model needs to use a large amount of medical image data in the process of model training, and the medical image data is often sensitive and difficult to obtain. Therefore, there is a graph data augmentation technology, which is a technology for increasing the amount of training data by modifying or generating graph data to improve the generalization ability and performance of graph machine learning models. The existing label graph data augmentation method is to expand the graph data by pseudo-labeling or data mixing, but it ignores the influence of different degrees and types of noise on graph data augmentation. Therefore, the present application proposes a graph data augmentation method, device, computer equipment and readable storage medium, which first defines a plurality of candidate augmentation strategies for simulating the noise, anomaly or attack existing in the graph data. Next, in response to a graph data augmentation instruction, the statistical features of the original image are extracted. Then, the statistical features are used to select the candidate augmentation strategies to obtain an augmentation strategy vector. Further, each element in the augmentation strategy vector is traversed, and the specified candidate augmentation strategies are determined and executed according to the element value of each element, and the nodes, edges, image features or labels of the original image are modified or retained to obtain a target image. Finally, the target image and the original image are input into a graph machine learning model for model training. The present application defines a plurality of candidate augmentation strategies to simulate and introduce various noise, anomaly or attack conditions that may exist in the graph data. By executing these candidate augmentation strategies, the nodes, edges, image features or labels of the original image are modified to increase the diversity and complexity of the augmented graph data. When the medical image machine learning model is trained using these diverse and complex training samples, the robustness and generalization ability of the model in processing complex and real data can be improved, and the adaptability of the medical image machine learning model to different scenarios can be improved.

[0028] In the present application, a group of candidate augmentation strategies can be defined in advance by the technician, such as node dropping, edge perturbation, feature perturbation and label smoothing. Through these strategies, the nodes of the graph data can be randomly discarded, the weights of the edges in the graph data can be disturbed, the node features can be disturbed or the certainty of the labels can be reduced. Thus, the noise, anomaly or attack that may exist in the graph data is simulated, so that the medical image machine learning model can be exposed to more changes and challenges during the training process using the augmented graph data, and the graph machine learning model can better adapt to different actual situations, have stronger robustness, and thus effectively improve the performance and generalization ability of the model.

[0029] 102. In response to the image data augmentation instruction, extract statistical features of the original image, perform strategy screening on multiple candidate augmentation strategies according to the statistical features, and obtain an augmentation strategy vector.

[0030] In an embodiment of the present application, in order to improve the performance and generalization ability of the subsequent medical graph machine learning model, the training image dataset corresponding to the medical graph machine learning model can be augmented to increase the diversity and complexity of the image samples in the training image dataset, and the augmented target image and the original image in the training sample dataset are used as training samples to train the medical graph machine learning model. For example, a graph neural network (GNN) is used to predict the properties of each molecule, such as solubility, toxicity, etc. Each graph in the training image set represents a molecular structure, each node represents an atom, each edge represents a chemical bond, and each node and edge has corresponding features and labels, that is, the medical original image in the training image set is obtained as G = (V, E, X, Y), where V is a node set, E is an edge set, X is a feature matrix, and Y is a label vector. Finally, the statistical features f of the medical original image G (such as the number of nodes, the number of edges, the average degree, etc.) are obtained, and based on the statistical features, the optimal strategy combination is determined to obtain an augmentation strategy vector, so that V, E, X, and Y are modified by executing the candidate augmentation strategy to obtain the target image. Through graph data augmentation, medical graph machine learning models can better adapt to different medical image scenarios, improve the analysis and prediction capabilities of medical images, and thus provide more accurate and reliable support for medical diagnosis and decision-making.

[0031] 103. Traverse each element in the augmentation strategy vector, determine all specified candidate augmentation strategies according to the element value of each element, and execute them, modify or retain the nodes, edges, image features or labels of the original image to obtain the target image.

[0032] In the embodiment of the present application, the graph data augmentation system uses a generative adversarial network (GAN) as the augmentation strategy execution module. The GAN includes a generator and a discriminator, which work together to execute the augmentation strategy and generate the target image. Specifically, the generator receives the original image G and the random noise vector z as input, and executes all the specified candidate augmentation strategies according to the augmentation strategy vector to generate the target image. It should be noted that the goal of the generator is to generate realistic target images To deceive the discriminator as much as possible. The discriminator receives the original image G or the target image As input, it outputs a binary scalar to represent the authenticity of the input image. When the binary scalar output by the discriminator is close to 1, it means that the discriminator believes that the input is a real image. When the binary scalar is close to 0, the discriminator believes that the input is an image generated by the generator. During the training process, the generator and the discriminator are optimized through adversarial competition. The generator tries to generate realistic target images. The discriminator attempts to accurately determine the authenticity of the input image. The optimization process of the generator and discriminator is iterative. By repeatedly training and adjusting their parameters, the generator is ultimately able to produce more realistic target images, and the discriminator is able to more accurately distinguish between real and generated images. By using a generative adversarial network as the augmentation strategy execution module, the graph data augmentation system can generate target images based on the augmentation strategy vector and use feedback from the discriminator to guide the generator optimization process. This can further increase the diversity and complexity of the training image dataset and improve the performance and generalization of medical graph machine learning models.

[0033] 104. Input the target image and the original image into the graph machine learning model for model training.

[0034] In an embodiment of the present application, the target image and the original image can be added to the training set to expand the training samples of the graph machine learning model. During the actual operation, the target image and the original image will be used as input data for the training process of the model. According to the training objectives, the graph machine learning model will learn to extract valuable features from the input image and associate these features with corresponding labels. The training objectives can be various medical image analysis tasks (such as classification, segmentation, detection, etc.), which can be set according to specific application scenarios and needs. By incorporating augmented target images and original images into the training set, the graph machine learning model can learn from more diverse and complex image samples, improve its performance in the field of medical images, and provide more accurate and reliable support for tasks such as medical diagnosis and decision-making.

[0035] The method provided in the embodiments of the present application first defines a plurality of candidate augmentation strategies for simulating noise, anomalies or attacks existing in the graph data. Next, in response to a graph data augmentation instruction, statistical features of the original image are extracted. Then, the plurality of candidate augmentation strategies are strategy-screened according to the statistical features to obtain an augmentation strategy vector. Further, each element in the augmentation strategy vector is traversed, and according to the element value of each element, all specified candidate augmentation strategies are determined and executed to modify or retain the nodes, edges, image features or labels of the original image to obtain a target image. Finally, the target image and the original image are input into a graph machine learning model for model training. The embodiments of the present application define a plurality of candidate augmentation strategies to simulate and introduce various noise, anomalies or attack situations that may exist in the graph data. By executing these candidate augmentation strategies, the nodes, edges, image features or labels of the original image are modified to increase the diversity and complexity of the augmented graph data. When the medical image machine learning model is trained using these diverse and complex training samples, the robustness and generalization ability of the model in processing complex and real data can be improved, and thus the adaptability of the medical image machine learning model to different scenarios is improved.

[0036] The embodiments of the present application provide a graph data augmentation method, as shown in Figure 2A The method comprises the following steps:

[0037] 201. Define a plurality of candidate augmentation strategies.

[0038] In the embodiments of the present application, a group of candidate augmentation strategies can be defined in advance, such as node dropping, edge perturbation, feature perturbation and label smoothing. Through these strategies, the nodes of the graph data can be randomly discarded, the weights of the edges in the graph data can be disturbed, the node features can be disturbed or the certainty of the labels can be reduced. Thus, the noise, anomalies or attacks that may exist in the graph data are simulated, so that the medical image machine learning model can be exposed to more changes and challenges during the training process using the augmented graph data, and thus the graph machine learning model can better adapt to different actual situations and have stronger robustness, thereby effectively improving the performance and generalization ability of the model.

[0039] 202. In response to a graph data augmentation instruction, extract statistical features of the original image.

[0040] In the embodiments of the present application, in order to improve the performance and generalization ability of the subsequent medical graph machine learning model, the training image dataset corresponding to the medical graph machine learning model can be subjected to graph data augmentation.

[0041] Specifically, the graph data augmentation system acquires a medical original image in a training image set as G=(V, E, X, Y), where V is a node set, E is an edge set, X is a feature matrix, and Y is a label vector, in response to a graph data augmentation instruction. The original image is converted into a to-be-processed image, and the to-be-processed image is segmented into a plurality of image regions by using an image segmentation method such as edge detection or threshold-based segmentation, where the to-be-processed image is used to indicate a grayscale image or a binary image. According to V, E, X, and Y of the original image G, the node number, the edge number, and the average degree of each image region in the plurality of image regions are acquired, and then statistical calculation is performed on the plurality of node numbers, the plurality of edge numbers, and the plurality of average degrees corresponding to the plurality of image regions. Specifically, the average value, the standard deviation, the maximum value, the minimum value, and the like of each statistical feature can be calculated to obtain the statistical feature f of the medical original image G. It should be noted that the statistical features include but are not limited to the node number, the edge number, and the average degree, and in actual operation, statistical features such as color features and texture features can be calculated according to actual needs.

[0042] 203. Strategy screening is performed on the plurality of candidate augmentation strategies according to the statistical features to obtain an augmentation strategy vector.

[0043] In the embodiments of the present application, after the graph data augmentation system acquires the statistical features of the original image, the optimal strategy combination can be determined according to the statistical features to obtain an augmentation strategy vector, so as to modify V, E, X, and Y by executing the candidate augmentation strategy to obtain a target image Through graph data augmentation, the medical image machine learning model can better adapt to different medical image scenarios, improve the analysis and prediction ability of the medical image, and thus provide more accurate and reliable support for medical diagnosis and decision-making.

[0044] Specifically, the augmentation strategy selection module in the graph data augmentation system uses a meta-learner, that is, a multi-layer perceptron (MLP) as a selector, and the meta-learner takes the statistical features f (such as the node number, the edge number, and the average degree) of the original graph G as input, performs strategy screening on the plurality of candidate augmentation strategies according to the statistical features, and outputs an augmentation strategy vector s. The system can also use a machine learning model to learn the mapping relationship from the statistical features to the augmentation strategy selection. In actual operation, the model can be trained using an existing data set, and then the statistical features are taken as input to predict the best augmentation strategy. It should be noted that the input layer and the output layer of the meta-learner are shown in the following formula 1 and formula 2, respectively.

[0045] Formula 1:

[0046] Formula 2: s = σ (W2·ReLU (W1·f + b1) + b2) ∈ {0, 1} k

[0047] where φ is a graph statistical feature extraction function; d is the dimension of the graph statistical feature; σ is a sigmoid function; W1, W2, b1 and b2 are learnable parameters of the model; ReLU is an activation function; and k is the number of candidate augmentation strategies. The meta-learner is trained using reinforcement learning, and the reward function is the performance of the downstream task (such as node classification, edge prediction, etc.) on the new graph after augmentation, as shown in the following formula 3. The meta-learner updates the parameters using the policy gradient method, and the parameter update formula is shown in the following formula 4. The performance of the above formula is obtained by indicators such as accuracy, AUC, etc., and the reward function formula is shown in the following formula 3. The meta-learner updates the parameters using the policy gradient method, and the parameter update formula is shown in the following formula 4.

[0048] Formula 3:

[0049] Formula 4:

[0050] where L is the loss function of the downstream task, θ is the model parameter of the downstream task, and α is the learning rate. The meta-learner can select a candidate augmentation strategy that meets the preset correlation condition as the specified candidate augmentation strategy by calculating the correlation between each of the multiple candidate augmentation strategies and the statistical features. Further, according to the specified candidate augmentation strategy, an augmentation strategy vector is constructed. Specifically, the multiple candidate augmentation strategies can be sorted according to a preset sorting rule or a preset strategy priority. Then, a specified sorting sequence number corresponding to the specified candidate augmentation strategy is determined, and the multiple candidate augmentation strategies are converted into a vector, i.e., a vector to be optimized, according to the specified sorting sequence number and the weight corresponding to the specified candidate augmentation strategy. For example, the number of specified candidate augmentation strategies is 3, among which the first strategy A and the third strategy C are selected as specified candidate augmentation strategies, and the execution vector 【1, 0, 1】 can be set to represent that the first strategy and the third strategy are the selected execution strategies. Further, the execution vector associated with the weight vector 【0.4, 0, 0.6】 is set as the vector to be optimized. Further, in order to fine-tune and optimize the augmentation strategy vector s according to the type and target of the downstream task, so that the augmentation strategy is more suitable for different types and targets of graph machine learning tasks, and the performance of the model on the new graph view is improved, an attention conditional random field is added to the output layer of the meta-learner, a preset attention conditional random field is adopted, and the weight corresponding to the specified candidate augmentation strategy is adjusted in the vector to be optimized according to the training target of the graph machine learning model, to obtain the augmentation strategy vector. Specifically, the formula of the attention conditional random field is shown in the following formula 5:

[0051] Formula 5:

[0052]

[0053] Where ACRF is the attention conditional random field function, T is the type and target information of the downstream task, such as node classification, edge prediction, etc., e is an embedding layer used to convert discrete information into a continuous vector, LSTM is a long short-term memory network used to capture the temporal dependency between information, h, g, f are three multi-layer perceptrons (MLPs) used to respectively calculate the attention weights between nodes or edges, the transfer probability between labels, and the optimized label sequence.

[0054] 204. Traverse each element in the augmentation strategy vector, determine all designated candidate augmentation strategies based on the element value of each element, and execute them, modify or retain the nodes, edges, image features or labels of the original image, and obtain the target image.

[0055] In the embodiment of the present application, the graph data augmentation system uses a generative adversarial network (GAN) as the augmentation strategy execution module. The GAN includes a generator and a discriminator, which work together to execute the augmentation strategy and generate the target image. The generator receives the original image G and the random noise vector z as input, and executes all the specified candidate augmentation strategies according to the augmentation strategy vector to generate the target image. It should be noted that the goal of the generator is to generate realistic target images To deceive the discriminator as much as possible. The discriminator receives the original image G or the target image As input, it outputs a binary scalar representing the authenticity of the input image. The optimization process of the generator and discriminator is iterative. By repeatedly training and adjusting their parameters, the generator will eventually be able to generate more realistic target images and the discriminator will be able to more accurately distinguish between real and generated images. The objective function of the adversarial network is shown in the following formula 6:

[0056]

[0057] where p data (G) is the data distribution of the original graph G, p z (z) is the prior distribution of the random noise vector z. In order to enable the generator G to perform corresponding operations according to the augmentation strategy vector s. The mask layer of the generator output layer can selectively retain or modify nodes, edges, features or labels according to the value of the augmentation strategy vector s. The mask layer formula is shown in the following formula 7-11:

[0058] Formula 7:

[0059] Formula 8:

[0060] Formula 9:

[0061] Formula 10:

[0062] Formula 11: where M is a mask layer function, is a Hadamard product, 1 is an all-one matrix, s i is the i-th element of the augmented policy vector s, indicating whether the i-th augmented policy is selected and its weight, m i is a randomly generated mask matrix or vector, indicating the probability that the corresponding node, edge, feature or label is kept or modified. For each element in the augmented policy vector, the execution state corresponding to the element is determined according to the element value of the element. If the execution state indicates execution, the specified candidate augmented policy corresponding to the query element and the specified weight are queried, the specified weight is mapped to a specified probability, and the specified candidate augmented policy is executed to modify or keep the nodes, edges, image features or labels of the original image according to the specified probability. For example, if the first element of s is 1, it indicates that the node dropping policy is selected. The mask layer selects the first specified node according to the specified probability, sets the rows and columns of the adjacency matrix of the first specified node to zero, and sets the rows of the feature matrix of the first specified node to zero; and / or, when the specified candidate augmented policy is the edge perturbation policy, the second specified node is selected according to the specified probability, and the adjacency matrix of the second specified node is modified. Finally, each element in the augmented policy vector is traversed to execute all specified candidate augmented policies to obtain the target image. It should be noted that in order to make the augmented policy more suitable for different types and target graph machine learning tasks, and improve the performance of the model on new graph views, an attention conditional random field is also added to the input layer of the generator. A preset attention conditional random field is used to adjust the graph performance of the original image in the vector to be optimized according to the training target of the graph machine learning model. Specifically, the formula of the attention conditional random field is shown in the following formulas 12-16:

[0063] Formula 12:

[0064] Formula 13:

[0065] Formula 14:

[0066] Formula 15:

[0067] Formula 16:

[0068] Among them, ACRF is the attention conditional random field function, T is the type and target information of the downstream task, such as node classification, edge prediction, etc.; E is an embedding layer, which is used to convert discrete information into a continuous vector; LSTM is a long short-term memory network, which is used to capture the temporal dependency between information; H, G, F are three multi-layer perceptrons (MLPs), which are used to calculate the attention weights between nodes or edges, the transfer probability between labels, and the optimized label sequence respectively.

[0069] 205. Input the target image and the original image into the graph machine learning model for model training.

[0070] In the embodiments of the present application, considering that for complex tasks and models, it is sometimes difficult for labeled data to cover all changes and situations, which can easily lead to overfitting. Self-supervised training can be trained with a large amount of unlabeled data, thereby reducing the risk of overfitting and improving the generalization ability of the model. Therefore, the augmented representation learning module of the graph data augmentation system seamlessly integrates and collaborates with graph representation learning methods of different types and levels, and uses the augmented new graph views for effective self-supervised training.

[0071] like Figure 2B As shown, after the original image is input into the meta-learner and the adversarial generative network generates the target image, the original image G and the target image Input to the graph representation learning method, extract the graph representation of the original image and the target image, and obtain the first graph representation h G And the second figure shows And input the first image representation and the second image representation into the encoder of the autoencoder to obtain the first potential representation z G , the second potential representation Furthermore, the first potential representation and the second potential representation are input into the decoder of the autoencoder to obtain the first reconstructed representation and the second reconstructed representation Subsequently, contrastive learning is used to optimize the latent representation z G and similarity between the first and second reconstructed representations. The first and second graph representations are optimized using the first and second reconstructed representations. Specifically, a consistency measure is defined, and a first contrastive sample is determined for the first graph representation and a second contrastive sample is determined for the first reconstructed representation. The first consistency loss between the first graph representation and the first contrastive sample is computed using the consistency measure, the second consistency loss between the first reconstructed representation and the second contrastive sample is computed using the consistency measure, and the consistency between the first graph representation and the first reconstructed representation is optimized by minimizing the first and second consistency losses. A third contrastive sample is determined for the second graph representation and a fourth contrastive sample is determined for the second reconstructed representation. The third consistency loss between the second graph representation and the third contrastive sample is computed using the consistency measure, the fourth consistency loss between the second reconstructed representation and the fourth contrastive sample is computed using the consistency measure, and the consistency between the second graph representation and the second reconstructed representation is optimized by minimizing the third and fourth consistency losses. Next, the first and second graph representations are optimized using a supervised loss of the graph machine learning model G and the second graph representation and inputting the optimized first and second graph representations to the graph machine learning model.

[0072] It should be noted that the encoder and decoder of the autoencoder are as shown in the following formulas 17-20:

[0073] Formula 17:

[0074] Formula 18:

[0075] Formula 19:

[0076] Formula 20:

[0077] where E G and D G are the encoder and decoder of the autoencoder, i.e., two multi-layer perceptrons (MLPs), m is the dimension of the latent representation, and n is the dimension of the graph representation. The objective function of the autoencoder is as shown in the following formulas 21-23:

[0078] Formula 21: L CAE = L rec + L con

[0079] Formula 22:

[0080] Formula 23:

[0081] where L CAEis the total loss function of the contrastive autoencoder, L rec is the reconstruction loss function, L con is the contrastive loss function, sim is a similarity function, such as cosine similarity, τ is a temperature parameter, K is a batch size, (z i , z i' ) is a pair of positive samples, i.e., the latent representations of the original graph and the new graph, (z i , z j' ) is a pair of negative samples, i.e., the latent representations between different graphs.

[0082] The method provided by the embodiments of the present application first defines a plurality of candidate augmentation strategies for simulating the noise, anomalies or attacks present in the graph data. Next, in response to a graph data augmentation instruction, the statistical features of the original image are extracted. Then, the plurality of candidate augmentation strategies are strategy-screened according to the statistical features to obtain an augmentation strategy vector. Further, each element in the augmentation strategy vector is traversed, and according to the element value of each element, all specified candidate augmentation strategies are determined and executed to modify or retain the nodes, edges, image features or labels of the original image, thereby obtaining a target image. Finally, the target image and the original image are input into a graph machine learning model for model training. The embodiments of the present application define a plurality of candidate augmentation strategies to simulate and introduce various noise, anomalies or attack situations that may exist in the graph data. By executing these candidate augmentation strategies, the nodes, edges, image features or labels of the original image are modified to increase the diversity and complexity of the augmented graph data. When the medical image machine learning model is trained using these diverse and complex training samples, the robustness and generalization ability of the model in processing complex and real data can be improved, thereby improving the adaptability of the medical image machine learning model to different scenarios.

[0083] Further, as a specific implementation of the method, the embodiments of the present application provide a graph data augmentation device, as shown in Figure 1 The device includes a setting module 301, a strategy selection module 302, a strategy execution module 303, and a self-supervised learning module 304. Figure 3 The setting module 301 is used to define a plurality of candidate augmentation strategies, which are used to simulate the noise, anomalies or attacks present in the graph data, including but not limited to node discard strategy, edge perturbation strategy, feature perturbation strategy and label smoothing strategy.

[0084]

[0085] ​The strategy selection module 302 is configured to extract statistical features of the original image in response to a graph data augmentation instruction, perform strategy screening on the plurality of candidate augmentation strategies according to the statistical features, and obtain an augmentation strategy vector.

[0086] The strategy execution module 303 is configured to traverse each element in the augmentation strategy vector, determine all specified candidate augmentation strategies and perform according to an element value of each element, modify or retain nodes, edges, image features or labels of the original image to obtain a target image.

[0087] The self-supervised learning module 304 is configured to input the target image and the original image into a graph machine learning model for model training.

[0088] In a specific application scenario, the strategy selection module 302 is configured to convert the original image into a to-be-processed image, and segment the to-be-processed image into a plurality of image regions, the to-be-processed image being used to indicate a grayscale image or a binary image; count a node number, an edge number and an average degree number corresponding to each image region in the plurality of image regions, and perform statistical calculation on a plurality of node numbers, a plurality of edge numbers and a plurality of average degree numbers corresponding to the plurality of image regions to obtain statistical features of the original image; calculate a correlation degree between each candidate augmentation strategy in the plurality of candidate augmentation strategies and the statistical features, and select a candidate augmentation strategy satisfying a preset correlation degree condition as a specified candidate augmentation strategy; and construct the augmentation strategy vector according to the specified candidate augmentation strategy.

[0089] In a specific application scenario, the strategy selection module 302 is configured to sort the plurality of candidate augmentation strategies according to a preset sorting rule or a preset strategy priority; determine a specified sorting sequence number corresponding to the specified candidate augmentation strategy, convert the plurality of candidate augmentation strategies into a vector according to the specified sorting sequence number and a weight corresponding to the specified candidate augmentation strategy; and adjust the weight corresponding to the specified candidate augmentation strategy in the vector according to a training target of the graph machine learning model by using a preset attention conditional random field to obtain the augmentation strategy vector.

[0090] In a specific application scenario, the strategy execution module 303 is configured to determine an execution state corresponding to each element in the augmentation strategy vector according to an element value of the element; if the execution state indicates execution, query a specified candidate augmentation strategy and a specified weight corresponding to the element, map the specified weight into a specified probability, and perform the specified candidate augmentation strategy to modify or retain nodes, edges, image features or labels of the original image according to the specified probability; and traverse each element in the augmentation strategy vector, execute all the specified candidate augmentation strategies, and obtain the target image.

[0091] In a specific application scenario, the strategy execution module 303 is configured to, when the specified candidate augmented strategy is a node dropping strategy, select a first specified node according to the specified probability, set rows and columns of an adjacency matrix of the first specified node to zero, and set rows of a feature matrix of the first specified node to zero; and / or, when the specified candidate augmented strategy is an edge perturbation strategy, select a second specified node according to the specified probability, and modify an adjacency matrix of the second specified node.

[0092] In a specific application scenario, the self-supervised learning module 304 is configured to extract graph representations of the original image and the target image to obtain a first graph representation and a second graph representation, input the first graph representation and the second graph representation into an encoder of a self-encoder to obtain a first latent representation and a second latent representation, input the first latent representation and the second latent representation into a decoder of the self-encoder to obtain a first reconstructed representation and a second reconstructed representation, optimize the first graph representation and the second graph representation using the first reconstructed representation and the second reconstructed representation, optimize the first graph representation and the second graph representation using a supervised loss of the graph machine learning model, and input the optimized first graph representation and the optimized second graph representation into the graph machine learning model.

[0093] In a specific application scenario, the self-supervised learning module 304 is configured to define a consistency metric, determine a first contrastive sample for the first graph representation, and determine a second contrastive sample for the first reconstructed representation, calculate a first consistency loss between the first graph representation and the first contrastive sample and a second consistency loss between the first reconstructed representation and the second contrastive sample using the consistency metric, and optimize consistency between the first graph representation and the first reconstructed representation by minimizing the first consistency loss and the second consistency loss, determine a third contrastive sample for the second graph representation, and determine a fourth contrastive sample for the second reconstructed representation, calculate a third consistency loss between the second graph representation and the third contrastive sample and a fourth consistency loss between the second reconstructed representation and the fourth contrastive sample using the consistency metric, and optimize consistency between the second graph representation and the second reconstructed representation by minimizing the third consistency loss and the fourth consistency loss.

[0094] The device provided by the embodiment of the application first defines a plurality of candidate augmentation strategies for simulating noise, anomalies or attacks existing in the graph data. Next, in response to a graph data augmentation instruction, statistical features of an original image are extracted. Then, the plurality of candidate augmentation strategies are strategy-screened according to the statistical features, and an augmentation strategy vector is obtained. Further, each element in the augmentation strategy vector is traversed, and according to an element value of each element, all specified candidate augmentation strategies are determined and executed, and nodes, edges, image features or labels of the original image are modified or retained to obtain a target image. Finally, the target image and the original image are input into a graph machine learning model for model training. The embodiment of the application defines a plurality of candidate augmentation strategies to simulate and introduce various noise, anomalies or attack situations that may exist in the graph data. By executing these candidate augmentation strategies, the nodes, edges, image features or labels of the original image are modified to increase the diversity and complexity of the augmented graph data. When the medical image machine learning model is trained using these diverse and complex training samples, the robustness and generalization ability of the model in processing complex and real data can be improved, and the adaptability of the medical image machine learning model to different scenarios is further improved.

[0095] It should be noted that other corresponding descriptions of the functions of the device for augmenting graph data provided by the embodiments of the application can be referred to Figure 1 and Figure 2A to Figure 2B for corresponding descriptions, which will not be repeated here.

[0096] Based on the above method as shown in Figure 1 , Figure 2A to Figure 2B Correspondingly, the embodiment also provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the graph data augmentation method.

[0097] Based on such understanding, the technical solution of the application can be embodied in the form of a software product. The to-be-identified software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation scenarios of the application.

[0098] Based on the above method as shown in Figure 1 , Figure 2A to Figure 2B and Figure 3 The embodiment of the graph data augmentation device, in order to achieve the above purpose, in the exemplary embodiment, see Figure 4The device also includes a communication bus, a processor, a memory, and a communication interface, and can further include an input / output interface and a display device, wherein the communication between the various functional units can be completed through the bus. The memory stores a computer program, and the processor is configured to execute the program stored in the memory to execute the graph data augmentation method in the above embodiments.

[0099] Optionally, the entity device can further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface can include a display screen, an input unit such as a keyboard, and the like. Optionally, the user interface can further include a USB interface, a card reader interface, and the like. The network interface can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and the like.

[0100] Those skilled in the art can understand that the entity device structure provided by the embodiment of the graph data augmentation does not constitute a limitation on the entity device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0101] The storage medium can also include an operating system and a network communication module. The operating system is a program for managing the hardware of the entity device and the to-be-identified software resources, supporting the running of the information processing program and other to-be-identified software and / or programs. The network communication module is used to realize the communication between the components in the storage medium, and the communication with other hardware and software in the information processing entity device.

[0102] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software with a necessary general hardware platform, or by hardware. By applying the technical solutions of the present application, first, a plurality of candidate augmentation strategies for simulating the noise, anomalies or attacks existing in the graph data are defined. Next, in response to a graph data augmentation instruction, the statistical features of the original image are extracted. Then, the plurality of candidate augmentation strategies are strategy screened according to the statistical features, and an augmentation strategy vector is obtained. Further, each element in the augmentation strategy vector is traversed, and according to the element value of each element, all specified candidate augmentation strategies are determined and executed, and the nodes, edges, image features or labels of the original image are modified or retained to obtain a target image. Finally, the target image and the original image are input into a graph machine learning model for model training. Compared with the prior art, the embodiments of the present application define a plurality of candidate augmentation strategies to simulate and introduce various noise, anomalies or attack situations that may exist in the graph data. By executing these candidate augmentation strategies, the nodes, edges, image features or labels of the original image are modified to increase the diversity and complexity of the augmented graph data. When the medical image machine learning model is trained using these diverse and complex training samples, the robustness and generalization ability of the model in processing complex and real data can be improved, and thus the adaptability of the medical image machine learning model to different scenarios is improved.

[0103] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred implementation scenario, and the modules or flows in the drawings are not necessarily required for implementing the present application.

[0104] Those skilled in the art can understand that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the description of the implementation scenario, or can be changed and located in one or more devices different from the implementation scenario. The modules of the above implementation scenario can be combined into one module, or can be further split into a plurality of sub-modules.

Claims

1. A graph data augmentation method, characterized in that: include: Defining multiple candidate augmentation strategies, wherein the multiple candidate augmentation strategies are used to simulate noise, anomalies, or attacks in the graph data, including but not limited to a node dropping strategy, an edge perturbation strategy, a feature perturbation strategy, and a label smoothing strategy; In response to the image data augmentation instruction, extracting statistical features of the original image, and performing strategy screening on the plurality of candidate augmentation strategies according to the statistical features to obtain an augmentation strategy vector; Traversing each element in the augmentation strategy vector, determining and executing all designated candidate augmentation strategies according to the element value of each element, modifying or retaining the nodes, edges, image features or labels of the original image, and obtaining a target image; The target image and the original image are input into a graph machine learning model for model training.

2. The method according to claim 1, characterized in that The step of extracting statistical features of the original image in response to the image data augmentation instruction and performing strategy screening on the plurality of candidate augmentation strategies according to the statistical features to obtain an augmentation strategy vector includes: Converting the original image into an image to be processed, and dividing the image to be processed into a plurality of image regions, wherein the image to be processed is used to indicate a grayscale image or a binary image; Counting the number of nodes, the number of edges, and the average degree corresponding to each of the multiple image regions, and performing statistical calculations on the multiple numbers of nodes, the multiple numbers of edges, and the multiple average degrees corresponding to the multiple image regions to obtain statistical features of the original image; Calculating the correlation between each candidate augmentation strategy among the plurality of candidate augmentation strategies and the statistical feature, and selecting a candidate augmentation strategy whose correlation satisfies a preset correlation condition as a designated candidate augmentation strategy; The augmentation strategy vector is constructed according to the specified candidate augmentation strategy.

3. The method according to claim 2, characterized in that The constructing the augmentation strategy vector according to the specified candidate augmentation strategy includes: Sorting the multiple candidate augmentation strategies according to a preset sorting rule or a preset strategy priority; Determining a designated sorting sequence number corresponding to the designated candidate augmentation strategy, and converting the plurality of candidate augmentation strategies into a vector according to the designated sorting sequence number and a weight corresponding to the designated candidate augmentation strategy; Using a preset attention conditional random field, according to the training objective of the graph machine learning model, the weight corresponding to the specified candidate augmentation strategy is adjusted in the vector to obtain the augmentation strategy vector.

4. The method according to claim 1, wherein The traversing each element in the augmentation strategy vector, determining and executing all designated candidate augmentation strategies according to the element value of each element, modifying or retaining the nodes, edges, image features or labels of the original image to obtain the target image, includes: For each element in the augmented strategy vector, determining an execution state corresponding to the element according to the element value of the element; If the execution state indicates execution, querying a specified candidate augmentation strategy and a specified weight corresponding to the element, mapping the specified weight to a specified probability, and executing the specified candidate augmentation strategy to modify or retain the nodes, edges, image features, or labels of the original image according to the specified probability; Each element in the augmentation strategy vector is traversed, and all the specified candidate augmentation strategies are executed to obtain the target image.

5. The method according to claim 4, characterized in that The querying of the specified candidate augmentation strategy and the specified weight corresponding to the element, mapping the specified weight to a specified probability, and modifying or retaining the nodes, edges, image features, or labels of the original image according to the specified probability includes: When the designated candidate augmentation strategy is a node discarding strategy, selecting a first designated node according to the designated probability, setting the rows and columns of the adjacency matrix of the first designated node to zero, and setting the rows of the feature matrix of the first designated node to zero; and / or, When the designated candidate augmentation strategy is the edge perturbation strategy, a second designated node is selected according to the designated probability, and the adjacency matrix of the second designated node is modified.

6. The method according to claim 1, characterized in that The step of inputting the target image and the original image as training samples into a graph machine learning model includes: Extracting graph representations of the original image and the target image to obtain a first graph representation and a second graph representation, and inputting the first graph representation and the second graph representation into an encoder of an autoencoder to obtain a first latent representation and a second latent representation; Inputting the first latent representation and the second latent representation into a decoder of the autoencoder to obtain a first reconstructed representation and a second reconstructed representation; The first reconstructed representation and the second reconstructed representation are used to optimize the first graph representation and the second graph representation, and the first graph representation and the second graph representation are optimized using the supervised loss of the graph machine learning model, and the optimized first graph representation and the optimized second graph representation are input into the graph machine learning model.

7. The method according to claim 6, characterized in that The optimizing the first graph representation and the second graph representation by using the first reconstructed representation and the second reconstructed representation comprises: defining a consistency metric, and determining a first comparison sample for the first image representation and a second comparison sample for the first reconstructed representation; Calculating a first consistency loss between the first image representation and the first comparison sample using the consistency metric, calculating a second consistency loss between the first reconstructed representation and the second comparison sample, and optimizing the consistency between the first image representation and the first reconstructed representation by minimizing the first consistency loss and the second consistency loss; determining a third comparison sample for the second image representation and a fourth comparison sample for the second reconstructed representation; The consistency metric is used to calculate a third consistency loss between the second image representation and the third comparison sample, a fourth consistency loss is calculated between the second reconstructed representation and the fourth comparison sample, and the consistency between the second image representation and the second reconstructed representation is optimized by minimizing the third consistency loss and the fourth consistency loss.

8. A graph data augmentation device, characterized in that: include: A setting module is used to define multiple candidate augmentation strategies, wherein the multiple candidate augmentation strategies are used to simulate noise, anomalies or attacks in graph data, including but not limited to node dropping strategy, edge perturbation strategy, feature perturbation strategy and label smoothing strategy; a strategy selection module, configured to extract statistical features of the original image in response to the image data augmentation instruction, and perform strategy screening on the plurality of candidate augmentation strategies according to the statistical features to obtain an augmentation strategy vector; a strategy execution module, configured to traverse each element in the augmentation strategy vector, determine all designated candidate augmentation strategies based on the element value of each element, and execute them, modify or retain the nodes, edges, image features, or labels of the original image to obtain a target image; The self-supervised learning module is used to input the target image and the original image into a graph machine learning model for model training.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images