Training method of neural network, graph representation extraction method, and task processing method

Abstract

The present disclosure provides a neural network training method, a graph representation extraction method, a task processing method, an apparatus, a device and a medium. The neural network training method comprises: obtaining first sample data and second sample data; obtaining a multi-scale graph representation of the first sample data and a multi-scale graph representation of the second sample data; performing graph matching on the graph representations of the first sample data at the first and second scales and the graph representations of the second sample data at the first and second scales to obtain a first current matching result representing a matching degree at the first scale and a second current matching result representing a matching degree at the second scale; determining a loss value according to a target matching result and / or a target task processing result and the first current matching result and / or the second current matching result; and training a graph representation extraction network according to the loss value.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a neural network training method, a graph representation extraction method, a task processing method, a neural network training device, a graph representation extraction device, a task processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] When analyzing and processing unstructured data such as images, videos, audio, text, molecular structures, and protein sequences, the raw form of this data is often difficult to use directly to produce effective results. A more effective approach is to transform the unstructured data into a semi-structured intermediate representation, and then perform analysis on this intermediate representation. Therefore, determining a suitable intermediate representation for unstructured data and how to utilize such an intermediate representation for effective analysis and processing of unstructured data has become an urgent problem to be solved.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a method for training a neural network, a method for extracting graph representations, a method for processing tasks, a device for training a neural network, a device for extracting graph representations, a device for processing tasks, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a method for training a neural network is provided, comprising: acquiring first sample data and second sample data, wherein the first sample data and the second sample data are respectively one of image data, audio data, text data, molecular structure data, and sequence data; acquiring multi-scale graph representations of the first sample data and the second sample data respectively, wherein the multi-scale graph representations are determined using a graph representation extraction network, the multi-scale graph representations including graph representations at the first scale and graph representations at the second scale, wherein each scale graph representation in the multi-scale graph representations of the first sample data and the second sample data includes at least one node, the node includes attributes, the attributes of the node include vector type attributes, and the graph representation extraction network includes... A network module for determining the vector type attributes of nodes; performing graph matching on the graph representations of the first sample data at a first scale and the graph representations of the second sample data at a first scale to obtain a first current matching result representing the degree of matching at the first scale; performing graph matching on the graph representations of the first sample data at a second scale and the graph representations of the second sample data at a second scale to obtain a second current matching result representing the degree of matching at the second scale; acquiring the target matching result and / or the target task processing result of the first sample data and the second sample data; determining a loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and training a graph representation extraction network based on the loss value.

[0006] According to another aspect of this disclosure, a graph representation extraction method is provided, comprising: inputting target data into a graph representation extraction network to obtain a multi-scale graph representation of the target data, wherein the target data is one of image data, audio data, text data, molecular structure data, and sequence data, and the graph representation of the target data includes a first-scale graph representation and a second-scale graph representation, wherein the first-scale graph representation includes at least one node of the first scale, the first-scale node has attributes, the attributes of the first-scale node include vector-type attributes, the second scale is lower than the first scale, the second-scale graph representation includes at least one node of the second scale, the second-scale node has attributes, the attributes of the second-scale node include vector-type attributes, wherein the graph representation extraction network is trained using the above-described neural network training method.

[0007] According to another aspect of this disclosure, a task processing method is provided, comprising: acquiring first data and second data, the first data and the second data being one of image data, audio data, text data, molecular structure data, and sequence data, respectively; performing the above-described graph representation extraction method on the first data and the second data respectively to determine multi-scale graph representations of the first data and the second data, wherein the multi-scale graph representations of the first data and the second data respectively include graph representations at a first scale and graph representations at a second scale, the graph representation at the first scale including at least one node at the first scale, the node at the first scale having attributes, the attributes of the node at the first scale including vector type attributes, the second scale being lower than the first scale, the graph representation at the second scale including at least one node at the second scale, the node at the second scale having attributes, the attributes of the node at the second scale including vector type attributes; performing graph matching on the graph representations at the first scale of the first data and the graph representations at the first scale of the second data to obtain a first matching result; performing graph matching on the graph representations at the second scale of the first data and the graph representations at the second scale of the second data to obtain a second matching result; determining a multi-scale matching result based on the first matching result and the second matching result; and determining a task processing result based on the multi-scale matching result.

[0008] According to another aspect of this disclosure, a training apparatus for a neural network is provided, comprising: a first acquisition unit configured to acquire first sample data and second sample data, the first sample data and the second sample data being one of image data, audio data, text data, molecular structure data, and sequence data, respectively; and a second acquisition unit configured to acquire multi-scale graph representations of the first sample data and the second sample data respectively, wherein the multi-scale graph representations are determined using a graph representation extraction network, the multi-scale graph representations including graph representations at the first scale and graph representations at the second scale, wherein each scale graph representation of the first sample data and the second sample data includes at least one node, the node includes attributes, the attributes of the node include vector type attributes, and the graph representation extraction network includes a network for determining the vector type attributes of the nodes. The system comprises: a first graph matching unit configured to perform graph matching on graph representations of first sample data at a first scale and graph representations of second sample data at a first scale to obtain a first current matching result representing the degree of matching at the first scale; a second graph matching unit configured to perform graph matching on graph representations of first sample data at a second scale and graph representations of second sample data at a second scale to obtain a second current matching result representing the degree of matching at the second scale; a third acquisition unit configured to acquire the target matching result and / or target task processing result of the first sample data and the second sample data; a first determination unit configured to determine a loss value based on the target matching result and / or target task processing result, and the first current matching result and / or the second current matching result; and a training unit configured to train a graph representation extraction network based on the loss value.

[0009] According to another aspect of this disclosure, a graph representation extraction apparatus is provided, comprising: a graph representation extraction network configured to receive first data and output a multi-scale graph representation of the first data, wherein the first data is one of image data, audio data, text data, molecular structure data, and sequence data, the graph representation of the first data includes a first-scale graph representation and a second-scale graph representation, the first-scale graph representation including at least one first-scale node, the first-scale node having attributes, the attributes of the first-scale node including vector-type attributes, the second scale being lower than the first scale, the second-scale graph representation including at least one second-scale node, the second-scale node having attributes, the attributes of the second-scale node including vector-type attributes, wherein the graph representation extraction network is trained using the aforementioned neural network training apparatus.

[0010] According to another aspect of this disclosure, a task processing apparatus is provided, comprising: a fourth acquisition unit configured to acquire first data and second data, wherein the first data and second data are respectively one of image data, audio data, text data, molecular structure data, and sequence data; and a second determination unit configured to process the first data and second data respectively using the above-described graph representation extraction apparatus to determine multi-scale graph representations of the first data and the second data, wherein the multi-scale graph representations of the first data and the second data include graph representations at a first scale and graph representations at a second scale, wherein the graph representation at the first scale includes at least one node at the first scale, the node at the first scale has attributes, and the attributes of the node at the first scale include vector-type attributes. The graph representation of the first data at a second scale is lower than that of the first scale. The graph representation of the second scale includes at least one node of the second scale. The nodes of the second scale have attributes, including vector-type attributes. A third graph matching unit is configured to perform graph matching on the graph representation of the first data at the first scale and the graph representation of the second data at the first scale to obtain a first matching result. A fourth graph matching unit is configured to perform graph matching on the graph representation of the first data at the second scale and the graph representation of the second data at the second scale to obtain a second matching result. A third determining unit is configured to determine a multi-scale matching result based on the first matching result and the second matching result. A fourth determining unit is configured to determine a task processing result based on the multi-scale matching result.

[0011] According to one or more embodiments of this disclosure, by extracting the attributes of nodes at multiple scales and the corresponding vector types of the first and second data, a more general and representative multi-scale graph representation for each data can be obtained. Furthermore, the graph matching results between graph representations of different sample data are utilized during the training of the graph representation extraction network. On the one hand, graph representations contain richer detailed features than vector representations, and the matching results obtained by graph matching are more accurate and reliable than those obtained by vector matching, enabling efficient and full utilization of the rich information contained in the data. Graph representations at different scales can take into account both the overall scale and the local scale, and are more robust to incomplete, deformed, viewpoint-changing, occluded, and attacked samples, and have stronger interpretability. On the other hand, the introduction of graph matching during the training process of the graph representation extraction network, the determination of the loss value based on the graph matching results between sample data, and the use of this loss value to train the graph representation extraction network, makes the annotation of sample data simpler. Furthermore, according to the method of this embodiment, by training the graph representation extraction network with the loss value determined based on the graph matching result, the target matching result, and / or the target task processing result, the graph representation extraction network can be used to obtain an accurate multi-scale graph representation suitable for downstream tasks during the inference stage, thereby helping downstream tasks to obtain accurate task processing results.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A flowchart of a neural network training method according to an embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic diagram of a multi-scale graph representation according to an embodiment of the present disclosure is shown;

[0016] Figure 3 It shows Figure 1 The flowchart shown illustrates the method for determining the loss value;

[0017] Figure 4 It shows Figure 1 The flowchart shown illustrates the method for obtaining the first and second sample data.

[0018] Figure 5 A flowchart of a neural network training method according to an embodiment of the present disclosure is shown;

[0019] Figure 6 A flowchart of a graph representation extraction method according to an embodiment of the present disclosure is shown;

[0020] Figure 7 A flowchart of a task processing method according to an embodiment of the present disclosure is shown;

[0021] Figure 8 It shows Figure 7 The flowchart of the graph matching process at each scale in the method shown;

[0022] Figure 9 It shows Figure 8 The flowchart shown illustrates the process of determining the matching results of candidate matching point pairs in the method described.

[0023] Figure 10 It shows Figure 8 The flowchart shown illustrates the process of determining the matching results of candidate matching edge pairs in the method described.

[0024] Figure 11 A structural block diagram of a neural network training apparatus according to an embodiment of the present disclosure is shown;

[0025] Figure 12 A block diagram illustrating the structure of an extraction apparatus according to an embodiment of the present disclosure is shown.

[0026] Figure 13 A structural block diagram of a task processing apparatus according to embodiments of the present disclosure is shown; and

[0027] Figure 14 A structural block diagram of an electronic device for a server or client according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0030] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0031] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0032] Figure 1 A flowchart of a neural network training method 100 according to an embodiment of the present disclosure is shown. The method 100 includes:

[0033] Step 101: Obtain first sample data and second sample data, wherein the first sample data and second sample data are one of image data, audio data, text data, molecular structure data and sequence data, respectively;

[0034] Step 102: Obtain the multi-scale graph representations of the first sample data and the second sample data respectively. The multi-scale graph representations are determined using a graph representation extraction network. The multi-scale graph representations include graph representations of the first scale and graph representations of the second scale. Each scale graph representation in the multi-scale graph representations of the first sample data and the second sample data includes at least one node. The node includes attributes. The attributes of the node include vector type attributes. The graph representation extraction network includes a network module for determining the vector type attributes of the node.

[0035] Step 103: Perform graph matching on the graph representation of the first sample data at the first scale and the graph representation of the second sample data at the first scale to obtain a first current matching result that characterizes the degree of matching at the first scale;

[0036] Step 104: Perform graph matching between the graph representation of the first sample data at the second scale and the graph representation of the second sample data at the second scale to obtain a second current matching result that characterizes the degree of matching at the second scale;

[0037] Step 105: Obtain the target matching results and / or target task processing results of the first sample data and the second sample data;

[0038] Step 106: Determine the loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and

[0039] Step 107: Based on the loss value, the training graph representation is used to extract the network.

[0040] In step 101, the sample data is labeled data.

[0041] The first sample data can be one of the following complex data types: image data (including pictures and videos), audio data, text data, molecular structure data, and sequence data. Sequence data can be, for example, protein sequence data, gene sequence data, or other sequence formats.

[0042] The first and second sample data can be raw data or data obtained by specific processing of raw data. In some embodiments, image data can be the original image or a preprocessed image after preprocessing the original image; audio data can be the original sampled audio data or preprocessed data after preprocessing the original sampled data (e.g., a spectrogram obtained after preprocessing the original sampled data); text data can be multiple original strings or preprocessed data obtained after preprocessing the text data, without limitation.

[0043] In step 102, the graph representation extraction network used in the multi-scale graph representation of the first sample data and the second sample data is obtained, for example, the graph representation extraction network trained in the Nth round.

[0044] After obtaining the first sample data, it can be input into the graph representation extraction network to obtain the multi-scale graph representation corresponding to the first sample data. In the multi-scale graph representation, each scale of the graph representation can include at least one node. Nodes can have attributes, and node attributes can include vector-type attributes. Accordingly, the graph representation extraction network includes a network module for determining the vector-type attributes of the nodes.

[0045] In one specific implementation, a node may further include scalar type attributes. These scalar type attributes may further include categorical attributes (e.g., discrete numerical values) and numerical attributes (e.g., continuous numerical values). Numerical attributes may further include both continuous and discrete numerical values.

[0046] In an exemplary embodiment, a node in a graph representation at a certain scale can be, for example, multiple objects obtained by sparsifying the original data (or dense data as specifically described below, such as feature maps obtained after image feature extraction from the original data, or preprocessed data obtained by preprocessing the original data) through object detection, for example. The vector-type attributes of a node can include, for example, the feature vector corresponding to the object. When a node has numerical or categorical attributes, the numerical attributes of the node can include, for example, the object's coordinates, size, orientation field, gradient field, texture density, and shape of its neighborhood, as well as the node's saliency. The categorical attributes of the node can include, for example, the object's classification category. It is understood that different nodes can include different attributes.

[0047] Taking the multi-scale graph representation of the first sample data, which includes a graph representation at the first scale and a graph representation at the second scale, as an example, the graph representation at the first scale includes at least one node at the first scale, and the node at the first scale has attributes, including vector-type attributes. The second scale is lower than the first scale, and the graph representation at the second scale includes at least one node at the second scale, and the node at the second scale has attributes, including vector-type attributes.

[0048] It is understood that this disclosure does not limit the number of scales included in the multi-scale graph representation. In some embodiments, the multi-scale graph representation may include graph representations of two, three, or more scales, without limitation herein. For ease of description, this disclosure uses a first scale and a second scale below the first scale as examples to illustrate the morphology, generation method, matching method, etc., of the multi-scale graph representation, but it is not intended to limit the scope of this disclosure.

[0049] It should be noted that the scale can be understood as the emphasis of the corresponding graph representation on the overall or local data. For example, the scale can be measured by the size of the corresponding part of each node in the original data, the number of nodes in the graph representation at that scale, etc. For example, nodes in a high-scale graph representation may correspond to the entire image or a paragraph of text, while nodes in a low-scale graph representation may correspond to a part of an image, a word or phrase in text, etc.

[0050] Figure 2 A schematic diagram of a multi-scale graph representation according to an embodiment of the present disclosure is shown. Figure 2 As shown, graph representations 202, 204, and 206 at three scales from high to low constitute a multi-scale graph representation. Each graph representation includes multiple nodes. Optionally, at least one scale of graph representation has adjacent edges (as shown in graph representation 206).

[0051] In steps 103 and 104, it should be noted that the graph referred to in this embodiment of the invention is a graph in a broad sense, which may include single-node graphs or multi-node graphs. Graph matching can be matching the nodes contained in the graph, or it can be matching the nodes and edges contained in the graph. When the graph at a certain scale is a single-node graph, graph matching refers to the matching between the vectors corresponding to the nodes. When the graph at a certain scale is a multi-node graph, graph matching can be graph matching in the traditional sense, or graph matching using the attributes of nodes and edges (including attributes of vector type and scalar type), or it can include node pair checking (e.g., node pair checking through geometric relationship solving by projective transformation), and combinations of the above. Among them, graph matching using the attributes of nodes and edges will be described in detail later.

[0052] According to some embodiments, the scalar type attributes of a node may include the node's saliency, label, and other attributes; the vector type attributes of a node may include the node's feature vector; the scalar type attributes of an adjacent edge may include the adjacent edge's saliency, label, and other attributes; and the vector type attributes of an adjacent edge may include the adjacent edge's feature vector.

[0053] Graph matching can result in the similarity between two graph representations. The similarity can be the sum of the similarities of each node / edge, or the sum of the saliency * similarity of each node / edge. For example, the similarity of a node / edge can be determined based on its attributes. In this way, a supervision signal can be generated for each local feature (attribute of a node / edge), allowing training to be performed individually on that local feature.

[0054] In an exemplary embodiment, in the multi-scale graph representation generated by the graph representation extraction network, if the features of a certain node are inferior to those of other features (insufficient robustness), then in the graph matching of positive samples, the matching error will mainly come from the features of this node. Therefore, during training, the supervision signal will be focused on the features of this node to strengthen the robustness of that feature.

[0055] In step 105, the target matching result can be a match or a non-match, or a result representing the degree of matching (e.g., a matching degree of 99%); the target task can be a matching task, a retrieval task, a classification task, an identification task, a fill-in-the-blank generation task, or other data analysis and processing related tasks. When the target task is a matching task, the result of the target task is the target matching result.

[0056] In one specific implementation, the target matching result and / or the target task processing result can be determined based on the annotations of the sample data. For example, the annotation might be that two sample data are positive samples of each other, which means the target matching result is labeled as "match". For example, if the target task is a classification task that classifies sample images, the target task processing result can be labeled as category "1". In this way, the matching result and / or the final task result can be labeled without labeling the specific graph representation extracted by the graph representation extraction network.

[0057] In step 106, determining the loss value and the first current matching result and / or the second current matching result based on the target matching result includes: determining the loss value based on the first current matching result and / or the second current matching result, and the target matching result;

[0058] Determining the loss value based on the target task processing includes: determining the current task result based on the first current matching result and / or the second current matching result, and determining the loss value based on the current task result and the target task result.

[0059] The current task result can be determined based on the first current matching result and / or the second current matching result. For example, if the task is to classify the first sample data, the first sample data and multiple second sample data whose current matching result is "matched" (for example, the current matching result is determined based on the first current matching result and the second current matching result) can be input into the classification network to obtain the current classification result. The loss value is determined based on the target classification result and the current classification result.

[0060] Understandably, the loss value can also be determined based on the target matching result and / or the target task processing result.

[0061] According to one or more embodiments of this disclosure, by extracting decoupled features at multiple scales of the first and second data, a more general and representative multi-scale graph representation for each data can be obtained. Furthermore, the graph matching results between graph representations of different sample data are utilized during the training of the graph representation extraction network. On the one hand, graph representations contain richer detailed features than vector representations, and the matching results obtained by graph matching are more accurate and reliable than those obtained by vector matching, enabling efficient and full utilization of the rich information contained in the data. Graph representations at different scales can take into account both the overall scale and the local scale, and are more robust to incomplete, deformed, viewpoint-changing, occluded, and attacked samples, and have stronger interpretability. On the other hand, the introduction of graph matching during the training of the graph representation extraction network, the determination of the loss value based on the graph matching results between sample data, and the use of this loss value to train the graph representation extraction network, makes the annotation of sample data simpler. Furthermore, according to the method of this embodiment, by training the graph representation extraction network with the loss value determined based on the graph matching result, the target matching result, and / or the target task processing result, the graph representation extraction network can be used to obtain an accurate multi-scale graph representation suitable for downstream tasks during the inference stage, thereby helping downstream tasks to obtain accurate task processing results.

[0062] According to some embodiments, the loss value may include a matching loss value and / or a task loss value. For example... Figure 3 As shown, step 106, determining the loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result, may include: step 301, determining the matching loss value based on the first current matching result and / or the second current matching result; and / or, step 302, determining the current task result based on the first current matching result and / or the second current matching result, and determining the task loss value based on the target task processing result and the current task result.

[0063] In some embodiments, if the target matching results at one or more scales or between multi-scale graph representations can be directly obtained, the corresponding matching loss value can be determined based on the target matching results and the first current matching result and / or the second current matching result, thereby generating a supervision signal to train the network.

[0064] In some embodiments, such as in a fill-in-the-blank task, if the corresponding target task processing result can be obtained, the current task processing result can be determined based on the first current matching result and / or the second current matching result. Then, the corresponding task loss value can be determined based on the target task processing result and the current task processing result, thereby generating a corresponding supervision signal to train the network. For example, given multiple second sample data, the current matching result between the first and second sample data is determined based on the first and / or the second current matching result. The first sample data and the second sample data whose current matching result matches the first data are input into the fill-in-the-blank network to obtain the task processing result. At this point, the target matching result may not be known, but the target task processing result (e.g., the labeled target task processing result) is known. In this case, the supervision signal can be determined based on the target task processing result.

[0065] According to some embodiments, step 301, determining the matching loss value based on the first current matching result and / or the second current matching result, may include: determining the current matching result based on the first current matching result and / or the second current matching result; and determining the matching loss value based on the current matching result and the target matching result.

[0066] In some embodiments, if the target matching result can be directly obtained from the multi-scale map representation, the current matching result can be determined first based on the first current matching result and / or the second current matching result, and then the corresponding matching loss value can be determined based on the current matching result and the target matching result, thereby generating the corresponding supervision signal to train the network.

[0067] According to some embodiments, the graph representation extraction network may include a first network for extracting graph representations at a first scale. In some embodiments, step 106, determining the matching loss value based on a first current matching result and / or a second current matching result, may include: determining a first-scale matching loss value based on the target matching result and the first current matching result. Step 107, training the graph representation extraction network based on the loss value, may include: training the first network based on the first-scale matching loss value.

[0068] According to some embodiments, the graph representation extraction network may include a second network for extracting graph representations at a second scale. In some embodiments, step 106, determining the matching loss value based on a first current matching result and / or a second current matching result, may include: determining a second-scale matching loss value based on the target matching result and the second current matching result. Step 107, training the graph representation extraction network based on the loss value, may include: training the second network based on the second-scale matching loss value. Thus, the loss values ​​at the first scale and the second scale can be calculated separately, and the corresponding network models can be trained respectively.

[0069] According to some embodiments, as mentioned above, the target matching result and / or target task processing result may be determined based on the annotation of sample data, which may be determined based on one of the following: manual annotation, teacher model and / or pre-trained model, rule-based method.

[0070] Specifically, the target matching results and / or target task processing results can be manually labeled. It's understood that manual labeling can be done along the data dimension rather than the scale dimension. For example, it's possible to label whether the first and second data points match, without needing to label whether a specific scale in the first data points matches a specific scale in the second data points. In fact, knowing whether the first and second data points match also tells us whether they match at each scale. Thus, the scale dimension labels can be derived from the data dimension labels, significantly increasing the number of supervision signals.

[0071] In another specific implementation, the target matching result and / or target task processing result can be determined based on the teacher model and / or a pre-trained model. The teacher model and pre-trained model can be models with a certain reasoning ability obtained by prior training on a large amount of data, or they can be used for knowledge distillation to train the graph representation extraction network. For example, the teacher model / pre-trained model is used to extract multi-scale graph representations of the first data and the second scale. The matching result and / or task processing result are judged based on this multi-scale graph representation. The target matching result and / or target task processing result are then determined based on the matching result and / or task processing result judged based on the multi-scale graph representation (e.g., selecting matching results or task processing with high confidence as the target matching result and / or target task processing result).

[0072] In another specific implementation, the target matching result and / or target task processing result can be determined based on rules. The rules in the rule-based approach can be determined based on prior knowledge. For example, a first-scale data and a second-scale multi-scale graph representation are extracted based on specific rules, and the matching result and / or task processing result are determined based on the multi-scale graph representation, and the matching result and / or task processing result are used as the target matching result and / or target task processing result.

[0073] It is understandable that other methods can be used to obtain target matching results and / or target task processing results, which are not limited here.

[0074] According to some embodiments, sample data is determined from unlabeled data using a network trained for the Nth round, and the target matching result can be determined based on the network trained for the Nth round. For example... Figure 4 As shown, step 101, obtaining the first sample data and the second sample data, may include:

[0075] Step 401: Use the network trained in the Nth round to extract multi-scale graph representations of the first and second unlabeled data respectively;

[0076] Step 402: Perform graph matching on the graph representation of the first unlabeled data at the first scale and the graph representation of the second unlabeled data at the first scale to obtain the matching result of the first unlabeled data that represents the degree of matching at the first scale;

[0077] Step 403: Perform graph matching between the graph representation of the first unlabeled data at the second scale and the graph representation of the second unlabeled data at the second scale to obtain the matching result of the second unlabeled data that represents the degree of matching at the second scale;

[0078] Step 404: Determine the unlabeled data matching result based on the first unlabeled data matching result and / or the second unlabeled data matching result;

[0079] Step 405: In response to determining that the first unlabeled data and the second unlabeled data satisfy a first condition, the first unlabeled data and the second unlabeled data are identified as first sample data and second sample data that are positive samples of each other. The first condition being that the first unlabeled data and the second unlabeled data satisfy the first condition includes the unlabeled data matching result satisfying the first matching condition, and the positive sample target matching result indicating that the corresponding first sample data and second sample data match; and / or

[0080] Step 406: In response to determining that the first unlabeled data and the second unlabeled data satisfy the second condition, the first unlabeled data and the second unlabeled data are determined as first sample data and second sample data that are negative samples of each other. The first unlabeled data and the second unlabeled data satisfying the second condition includes the unlabeled data matching result satisfying the second matching condition, and the target matching result of the negative sample indicates that the corresponding first sample data and second sample data do not match.

[0081] The matching result for unlabeled data can be a floating-point number or an integer. For example, the matching result for the first unlabeled data is the similarity score, which is a floating-point number; the matching result for the second unlabeled data is the number of nodes / edges matched, which is an integer.

[0082] The first condition, second condition, first matching condition, and second matching condition can be set by the user. For example, the first matching condition could be that the first unlabeled data matching result is greater than 80% and the second unlabeled data matching result is greater than 5 nodes / edges. Understandably, the more stringent the first and second matching conditions are set, the more reliable the target matching results corresponding to the positive / negative samples generated from the unlabeled data will be.

[0083] In addition to the matching requirements, when determining the first and second unlabeled data as sample data, auxiliary conditions can also be imposed on the first and second conditions. These auxiliary conditions can include time and location conditions, expert secondary confirmation conditions, etc. For example, when the first and second unlabeled data are image data, their shooting time and space information can be used as auxiliary conditions to determine whether they are positive or negative samples. For instance, if two images have high similarity, a large number of matched nodes / edges, and are taken at similar times and locations, then the probability that they contain the same object is higher, and the probability that they are each other's positive samples is higher.

[0084] Therefore, by using the network generated after the Nth round to generate positive and / or negative samples in the above manner, the graph representation extraction network can use these samples for the (N+1)th round of training. Only a small amount of sample data needs to be labeled to obtain the model after the Nth round of training, and then more sample data obtained after the Nth round of training can be used for further training, greatly reducing the amount of labeling required during model training. Furthermore, by simultaneously generating positive and negative samples, comparative learning can be performed using these positive and negative samples to achieve the ability to extract accurate graph representations, while also reducing the cost of obtaining samples.

[0085] In step 404, the unlabeled data matching result can be determined based on one or both of the first and second unlabeled data matching results. In embodiments with relatively strict data quality requirements, the unlabeled data matching result is determined to be a match in response to the determination that both the first and second unlabeled data matching results indicate a successful match. In some embodiments, the unlabeled data matching result can be determined to be a match when the first and second scales meet specific matching conditions. For example, if the similarity at the first scale is greater than 80% and the second scale matches 5 nodes, the unlabeled data matching result is determined to be a match. In some cases, cross-validation can be performed between different scales to generate more supervisory signals. In some embodiments, since the matching results of lower-scale graph representations involve more detailed features and have higher credibility than macroscopic features, the unlabeled data matching result can be determined to be a match when the matching results of lower-scale graph representations indicate a successful match. In some embodiments, when the multi-scale graph representation includes three or more scales, the unlabeled data matching result can be determined to be a match when the matching results of the highest and lowest scale graph representations indicate a successful match. In a more data-tolerant embodiment, unlabeled data matching results can be identified as matches when the matching results indicate a successful match in a higher-scale graph representation.

[0086] It is understandable that cross-validation can be performed at different scales in other ways to generate supervisory signals, and no limitation is made here.

[0087] It is understandable that "Nth round of training" means that the network has undergone at least one round of training and thus has a certain inference ability, but it is not intended to limit the specific number of training rounds of the network.

[0088] According to some embodiments, the graph representation extraction network may include rule modules and network modules. Rule modules may be, for example, rule-based modules that utilize prior knowledge. Such modules can be used without training, but their accuracy is lower than that of trained network modules, and they are less robust, have strong limitations, and are generally difficult to train or optimize. While trained network modules can output accurate results and have a wider range of applications and stronger robustness, they are difficult to converge quickly when training is challenging.

[0089] According to some embodiments, such as Figure 5 As shown, the training method 500 further includes at least one of the following steps: step 501, in response to determining that a fifth preset condition is met, replacing the first rule module in the rule module with a network module; and / or, step 502, in response to determining that a sixth preset condition is met, adding a network module in the graph representation extraction network. Figure 5 The operations in steps 503-509 and Figure 1 The operations in steps 101-107 are similar and will not be limited here. Step 509, training the graph representation extraction network based on the loss value may include: training the network modules in the current graph representation extraction network based on the loss value.

[0090] In some embodiments, during the initial training phase, rule modules can be used in some parts of the graph representation extraction network, while network modules can be used in other parts to train these network modules. After these network modules converge, more network modules can be added, or the rule modules can be replaced with network modules and training can continue to improve the network's performance. In this way, prior knowledge can be fully utilized, and the training speed and effectiveness of the network can be improved.

[0091] In some embodiments, the fifth and sixth preset conditions may be, for example, a specific number of training epochs, the current matching accuracy of the network, or other preset conditions such as convergence speed or trend. It is understood that those skilled in the art can determine the fifth and sixth preset conditions according to their needs, and no limitation is made herein.

[0092] According to some embodiments, the graph representation at each scale in the multi-scale graph representation may include at least one node. Each node may include attributes, which may include scalar-type attributes and vector-type attributes. The graph representation at least one scale in the multi-scale graph representation may include at least one adjacent edge. Each adjacent edge represents the relative relationship between two nodes at the same scale. The adjacent edge has attributes, which may include scalar-type attributes and vector-type attributes. The scalar-type attributes of a node include the node's saliency, and / or the vector-type attributes of a node include the node's eigenvector, and / or the scalar-type attributes of an adjacent edge include the adjacent edge's saliency, and / or the vector-type attributes of an adjacent edge include the adjacent edge's eigenvector.

[0093] According to some embodiments, a graph representation extraction network may include at least one of the following: a network module for determining scalar-type attributes of nodes; a network module for determining vector-type attributes of nodes; a network module for determining scalar-type attributes of adjacent edges; and a network module for determining vector-type attributes of adjacent edges. Loss values ​​may be applied to the differentiable portions corresponding to these network modules, thereby enabling the training of these network modules.

[0094] According to some embodiments, at least one node at a first scale and at least one node at a second scale can be obtained by sparsifying the same dense data separately. That is, different degrees of sparsification can be applied to the same dense data to obtain nodes at different scales.

[0095] According to some embodiments, dense data can include multiple scales. Dense data including multiple scales can be feature maps at multiple scales in a feature pyramid. At least one node at a first scale and at least one node at a second scale can be obtained by sparsifying each of the two scales of the dense data separately. That is, multiple dense data at different scales can be acquired first, and then sparsified for each scale to obtain nodes at the corresponding scales. In an exemplary embodiment, the original image can be convolved with different downsampling factors to obtain feature maps of different sizes, i.e., dense data at different scales. Then, these dense data can be sparsified separately to obtain nodes at different scales. Both of the above methods can generate nodes at multiple scales in parallel after obtaining the dense data.

[0096] It is understandable that a graph representation extraction network may include a feature extraction network module that derives dense data from the raw data.

[0097] The following will describe the different modules in the graph representation extraction network with reference to the embodiments.

[0098] According to some embodiments, nodes at at least one scale can be obtained by sparsifying the dense data corresponding to that data.

[0099] Dense data, or dense graphs, can include, for example, an original image with dense pixels, a feature map containing dense feature vectors obtained by convolving the original image, audio data with dense sampling points (and a spectrogram containing dense pixels obtained by spectrating the audio data), text segments containing dense characters or words, and dense molecular structure data and sequence data. By sparsifying dense data, multiple decoupled nodes, i.e., sparse graphs, can be obtained. Each node can correspond to a portion of the dense data and has attributes. It can be understood that dense data can also include multiple nodes, such as pixels in an image, sampling points in audio data, characters or words in text data, etc. Each node can include label-type attributes (e.g., position in the dense data, classification category) and vector label-type attributes (e.g., feature vector).

[0100] In some embodiments, the saliency of dense nodes can be used to identify nodes in dense data. The scalar-type attribute of a dense node can include saliency. Saliency characterizes the importance of each dense node in the dense data and can be represented by a probability distribution across all dense nodes. In some embodiments, the saliency of a dense node can be determined based on the feature vectors of the dense nodes. In one exemplary embodiment, a saliency network can be used to process the feature vectors of all dense nodes to determine the saliency of each dense node.

[0101] Sparsification of dense data may include, for example, identifying nodes whose saliency satisfies a third preset condition from at least a subset of dense nodes as the sparsified nodes. It is understood that those skilled in the art can set the third preset condition according to their needs, and this is not limited thereto. In an exemplary embodiment, the third preset condition may be top-k, i.e., selecting the k dense nodes with the highest saliency as the sparsified nodes, and / or, the third preset condition may be nodes with saliency greater than a saliency threshold.

[0102] In some embodiments, in addition to saliency, attention scores generated by attention mechanisms or other measures of importance to dense nodes may be used as the basis for filtering nodes during the sparsification process, and these methods are all within the scope of this disclosure.

[0103] In some embodiments, a detection-based approach can be used to identify nodes in dense data. The detection-based approach may include keypoint detection, object detection, or other types of detection, which are not limited thereto.

[0104] Understandably, node sparsity can be achieved through a node sparsity network, which may include detection networks, saliency networks, etc. When the node sparsity network is a detection network, dense data is input into the detection network to obtain the sparsified nodes corresponding to the dense data and their corresponding confidence scores. When the node sparsity network is a saliency network, dense data and its corresponding feature vectors are input into the saliency network to obtain the saliency scores of each dense node corresponding to the dense data. The top k dense nodes with saliency scores greater than a saliency threshold and / or the highest saliency scores are selected as the sparsified nodes. In some embodiments, non-maximal suppression and other conditions can also be considered for comprehensive selection.

[0105] According to some embodiments, nodes at at least one scale can be obtained by sparsifying the portion of dense data corresponding to the locations of nodes at another scale obtained through sparsification. In an exemplary embodiment, nodes at a first scale can be determined, for example, through object detection. Each node at a first scale can correspond to a portion of the dense data (i.e., the detection bounding box output by object detection). Nodes at a second scale can then be obtained by sparsifying the portion of the dense data corresponding to each node at a first scale, thus obtaining second-scale nodes corresponding to each node at a first scale. In this way, more valuable second-scale nodes can be obtained, thereby improving the processing efficiency and accuracy of subsequent matching tasks and downstream tasks.

[0106] According to some embodiments, nodes of at least one scale can be obtained by merging dense data. In some exemplary embodiments, clustering or graph neural network methods can be used, for example, to sparsify the dense data to obtain low-scale nodes. These low-scale nodes can then be further processed to obtain high-scale nodes.

[0107] According to some embodiments, nodes at at least one scale can be obtained by merging the low-scale nodes obtained through sparsification. In some exemplary embodiments, clustering or graph neural network methods can be used to cluster the multiple low-scale nodes obtained through sparsification, or a subgraph containing multiple low-scale nodes can be input into a graph neural network to obtain higher-scale nodes and / or the attributes of the nodes can be processed by sparse nodes of another scale to obtain higher-scale nodes.

[0108] The above-mentioned merging of nodes can be based on the scalar type attributes of the nodes (e.g., location information), the vector type attributes of the nodes (e.g., feature vectors), or the scalar or vector type attributes of the adjacent edges in the graph representation (e.g., the co-occurrence probability, correlation, etc. of the two connected nodes), and there is no limitation here.

[0109] The attributes of these nodes can be determined before, simultaneously with, or after the locations of nodes at each scale and their correspondences with nodes or dense nodes at other scales are determined.

[0110] According to some embodiments, the attributes of the nodes obtained through sparsification can be determined based on the attributes of at least a subset of dense nodes corresponding to that node among a plurality of dense nodes. In some embodiments, the attributes of the node can be determined based on the attributes of neighboring nodes within a certain range of the dense node corresponding to the node's position in the dense data. For example, these neighboring nodes can be input into a feature extraction network to obtain the node's feature vector, or the average of these neighboring node feature vectors can be used to determine the node's feature vector, or a weighted average of these neighboring node feature vectors based on saliency can be used to determine the node's feature vector. In an exemplary embodiment, the node is determined by object detection, and the feature vector (vector-type attribute) corresponding to the node can be determined by inputting the attributes of the dense nodes within the detection box corresponding to the node in the dense data into a feature extraction network. In another exemplary embodiment, the node is determined by merging, and the attributes of the node can be determined by clustering or graph neural networks, etc., based on the attributes of all low-scale nodes used to merge to obtain the node.

[0111] According to some embodiments, nodes at at least one scale can be obtained by merging nodes at another scale obtained through sparsification, and the attributes of the merged nodes can be determined based on the attributes of nodes at the other scale that have a subordinate relationship with the node.

[0112] In some embodiments, the attributes of at least a subset of dense nodes corresponding to the node or nodes that are subordinate to the node can be further processed to obtain the node's attributes. In an exemplary embodiment, a graph neural network can be used to process the attributes of these nodes corresponding to the node to obtain the node's attributes. In addition to the methods described above, various attributes of a node can be determined in other ways, which are not limited here.

[0113] Multiscale graphs can include adjacent edges when the relative relationships between nodes are helpful in characterizing the data. Examples include the distance between two objects in an image, the interaction between two objects in an image, the association between consecutive words in speech, and the interaction between different groups in a sequence.

[0114] According to some embodiments, at least one adjacent edge can be determined based on the respective attributes of at least one node at the same scale. By analyzing the attributes of the nodes, pairs of nodes with related relationships can be identified in a single-scale graph representation to generate corresponding adjacent edges.

[0115] In some embodiments, adjacent edges can be generated based on rules. In some embodiments, adjacent edges can be generated between node pairs whose distance is less than a preset threshold and / or between the k nearest nodes. In some embodiments, adjacent edges can be generated only along a specific direction. It is understood that those skilled in the art can set the corresponding adjacent edge generation rules according to prior knowledge, and no limitation is made here.

[0116] In some embodiments, candidate adjacent edges can be generated first, and then adjacent edges can be selected from the candidate adjacent edges. According to some embodiments, at least one adjacent edge is determined by performing the following steps: determining at least one candidate adjacent edge based on at least one node of the same scale; determining the saliency of each of the at least one candidate adjacent edge based on the respective attributes of the at least one node of the same scale; and determining the adjacent edge whose saliency satisfies a fourth preset condition among the at least one candidate adjacent edge as at least one adjacent edge. By generating adjacent edges using saliency, the adjacent edge generation process can be optimized through training to improve the effectiveness of the generated adjacent edges. The fourth preset condition can be a saliency greater than a saliency threshold and / or the top k saliencies.

[0117] According to some embodiments, the attributes of each adjacent edge in at least one adjacent edge can be determined based on at least one of the attributes of the two nodes connected by the adjacent edge and the relative relationship between the two nodes. In an exemplary embodiment, the position, length, angle, interaction magnitude, etc., of the adjacent edge connecting the two nodes can be determined as the attributes of the adjacent edge based on the position / attributes of the two nodes. In some embodiments, prior knowledge can be used to determine the relative relationship between the two nodes based on rules, and the attributes of the adjacent edge can be determined based on the relative relationship.

[0118] According to some embodiments, at least one dependent edge can be directly determined based on the dependency relationship between nodes at two scales. In one exemplary embodiment, a first node at a first scale is obtained by performing object detection on dense data, and a second node at a second scale is obtained by performing further object detection on the region corresponding to the first node in the dense data. In this case, the first node and the second node have a dependency relationship, and a dependent edge can be generated between them. In another exemplary embodiment, nodes at a second scale are obtained by clustering dense data, and nodes at a first scale are obtained by merging nodes at the second scale. In this case, nodes at a second scale used to merge nodes at the first scale have a dependency relationship with those nodes at the first scale, and a dependent edge can be generated between these nodes at the second scale and the nodes at the first scale.

[0119] According to some embodiments, the attributes of a dependent edge can be determined based on the attributes of the two nodes connected to the dependent edge. As described above, the attributes of the dependent edge can be determined in various ways based on the vector type attributes and / or scalar type attributes of the two nodes connected to the dependent edge, and no limitation is made herein.

[0120] According to some embodiments, the aforementioned sparsification module for sparsifying dense data to obtain sparsified nodes and the merging of the low-scale nodes obtained by sparsification to obtain high-scale nodes are also implemented through neural networks. Accordingly, the graph representation extraction network includes at least one of the following: a sparsification module for sparsifying dense data to obtain sparsified nodes; and a merging module for merging the low-scale nodes obtained by sparsification to obtain high-scale nodes.

[0121] According to some embodiments, nodes are obtained by a sparsification module, nodes are connected to form adjacent edges, and adjacent edges with saliency greater than a threshold are determined as retained adjacent edges by a network module for determining the saliency attribute of adjacent edges. The attributes of node / edge vector type are extracted according to a module for determining node / edge vector type attributes.

[0122] According to some embodiments, both nodes and edges include a determination module and an attribute extraction module. The determination module is used to determine nodes / edges. The node determination module may include a sparsity module (e.g., a detection module, a saliency module) or a merging module, and the edge determination module may include a saliency module. The attribute extraction module may be a module that determines attributes other than saliency. According to some embodiments, these modules are all network modules.

[0123] Figure 6 A flowchart of a graph representation extraction method 600 according to an embodiment of the present disclosure is shown. The method 600 includes:

[0124] Step 601: Input the target data into the graph representation extraction network to obtain a multi-scale graph representation of the target data. The target data is one of image data, audio data, text data, molecular structure data, and sequence data. The graph representation of the target data includes a first-scale graph representation and a second-scale graph representation. The first-scale graph representation includes at least one node of the first scale, and the node of the first scale has attributes, including vector-type attributes. The second scale is lower than the first scale. The second-scale graph representation includes at least one node of the second scale, and the node of the second scale has attributes, including vector-type attributes. The graph representation extraction network is trained by the aforementioned method.

[0125] It is understandable that the target data is similar to the first and second sample data described above, and its multi-scale plot representation is similar to that of the first and second sample data, so it will not be elaborated here.

[0126] Figure 7 A flowchart of a task processing method 700 according to an embodiment of the present disclosure is shown below. The method 700 includes:

[0127] Step 701: Obtain first data and second data, wherein the first data and second data are one of image data, audio data, text data, molecular structure data and sequence data, respectively;

[0128] Step 702: Perform the graph representation extraction method described above on the first data and the second data respectively to determine the multi-scale graph representations of the first data and the second data. The multi-scale graph representations of the first data and the second data each include a first-scale graph representation and a second-scale graph representation. The first-scale graph representation includes at least one node of the first scale, and the first-scale node has attributes, including vector-type attributes. The second scale is lower than the first scale, and the second-scale graph representation includes at least one node of the second scale, and the second-scale node has attributes, including vector-type attributes.

[0129] Step 703: Perform graph matching on the graph representation of the first data at the first scale and the graph representation of the second data at the first scale to obtain the first matching result;

[0130] Step 704: Perform graph matching between the graph representation of the first data at the second scale and the graph representation of the second data at the second scale to obtain a second matching result;

[0131] Step 705: Based on the first matching result and the second matching result, determine the multi-scale matching result; and

[0132] Step 706: Determine the task processing result based on the multi-scale matching results.

[0133] According to the method of this embodiment, by extracting decoupled features at multiple scales from the first and second data, a more general and representative multi-scale graph representation of each data point can be obtained. Then, through graph matching of the multi-scale graph representation and determining the task processing result based on the graph matching result, the task processing can be carried out efficiently and fully utilize the rich information contained in the data to obtain accurate task processing results. Furthermore, graph matching of the multi-scale graph representation enhances robustness, better handling situations such as changes in image perspective, changes in text expression, and different speakers.

[0134] It is understandable that the first data and the second data are similar to the first sample data and the second sample data described above. Steps 701-704 involve obtaining the first data and its multi-scale graph representation, obtaining the second data and its multi-scale graph representation, and performing graph matching on graph representations at different scales. Figure 1 The operations of steps 101-105 are similar and will not be repeated here.

[0135] Therefore, by acquiring graph representations with different features, including scalars, vectors, and graphs, at different scales, more general and powerful intermediate representations can be obtained for various types of data. This improves the accuracy of downstream tasks such as matching, retrieval, classification, recognition, fill-in-the-blank generation, and other data analysis and processing tasks. Furthermore, by using dependent edges, the correlation between graph representations at different scales can be strengthened, thereby further enriching the information included in the multi-scale graph representations.

[0136] The method for obtaining the multi-scale matching result based on the first and second matching results in step 705 is similar to that in step 404. In step 705, the corresponding method and logic for determining the multi-scale matching result based on the first and second matching results can be set according to requirements. In some embodiments, when both the first and second matching results are successful matches, the multi-scale matching result is determined to be a successful match. In some embodiments, when the second matching result at a lower scale is a successful match, the multi-scale matching result is determined to be a successful match. In some embodiments, the first and second matching results can be, for example, the matching degree of the graph representation at the first and second scales, then the multi-scale matching result can be a calculation result based on the matching degree of the graph representation at the two scales, such as the average of the matching degree of the graph representation at the two scales. It is understood that the multi-scale matching result can also be determined in other ways, which are not limited here.

[0137] According to some embodiments, the task is a matching task. Step 107, based on the multi-scale matching results, determines the task processing result, which may include taking the multi-scale matching results as the final task result.

[0138] According to some embodiments, the second data may be obtained from a database. Step 606, determining the task processing result based on the multi-scale matching result may include: determining at least one second data that matches the first data based on the multi-scale matching result of the first data and multiple second data in the database; and determining the task processing result based on the at least one second data. Thus, through the above method, other types of tasks based on multi-scale graph representation can be converted into matching tasks based on multi-scale graph representation. In an exemplary embodiment, the final task may be an identification task, matching task, or search task implemented by matching means, in which case the at least one matched second data can be directly used as the search result. In an exemplary embodiment, the final task may be a classification task, in which case the first data and the at least one matched second data can be input into a model for the classification task, so that the model uses the at least one second data as a reference for classification to complete the classification of the first data. In an exemplary embodiment, the final task may be a generation task (e.g., filling in blanks in text or images), in which case the first data with partial gaps and the at least one matched second data can be input into a model for the generation task, so that the model uses the at least one second data as a reference for generation to complete the generation of the first data. In this way, tasks can be completed using data similar to the first set of data. Compared to inputting only the first set of data into the model, this provides the model with richer information, and more accurately obtains the results of classification and generation tasks without increasing model complexity. The following will illustrate multi-scale graph representation, graph matching, and task processing for different types of data with examples.

[0139] In one exemplary embodiment, both the first data and the second data can be image data. The dense data can be a feature map obtained based on the corresponding image data, and the multiple dense nodes in the dense data can be multiple pixels in the feature map. Multiple nodes at a second scale can be obtained by sparsifying the first data (e.g., based on saliency). The attributes of these nodes can include the node's position in the first data and the node's corresponding feature vector (e.g., determining the node's feature vector based on its neighborhood in the feature map, or determining the node's feature vector based on the local image corresponding to the node in the first data; the node's feature vector can be used to describe the attributes of the node's neighborhood, such as orientation field; if the first data is fingerprint data, the node's feature vector can be used to describe the texture density of the node's neighborhood, etc.). Similarly, nodes at a second scale of the second data can be obtained. Multiple nodes at a first scale can be obtained by merging the nodes at the second scale of the first data. The attributes of these first nodes can also include the node's position in the first data and the node's corresponding feature vector. Similarly, nodes at a first scale of the second data can be obtained. The second scale may also include adjacent edges, used to establish connections between multiple nodes of the second scale that are subordinate to nodes of the same first scale. Subordinate edges may also exist between nodes of the first and second scales that are subordinate to each other. The attributes of adjacent and subordinate edges may include, for example, the relative position and angle between the edge and its corresponding node, and the length of the edge (e.g., used to describe the magnitude of the force exerted on the node it connects to).

[0140] Multi-scale graph structures in image data can extract geometric information of targets in an image (e.g., the positional relationships between multiple targets in an image, or the positional relationships between different parts of the same target) while preserving rich detail information (e.g., node feature vectors). Graph representations at different scales take into account both the overall and local contexts, making them more robust to incomplete, deformed, viewpoint-shifting, occluded, and attack samples, and also offering greater interpretability. Using such multi-scale graph representations for graph matching, and leveraging graph matching to solve complex downstream tasks (e.g., image matching, image search, image classification, image generation), yields more accurate and reliable results. Furthermore, due to its multi-scale characteristics, in image data retrieval and comparison tasks, high-scale graph representations can be used for initial screening, followed by precise retrieval and comparison using low-scale graph representations, while constraints based on prior knowledge (e.g., geometric constraints) are applied to obtain accurate results.

[0141] In an exemplary embodiment, both the first data and the second data can be text data. The dense data can be a text paragraph, and the nodes in the dense data can be characters / words within that text paragraph. It is understood that the nodes in the dense data can also be text features corresponding to these characters / words. By sparsifying the first and second data, nodes at first and second scales can be obtained. These nodes can correspond to different scales of text segments in the text paragraph, such as sentences, clauses, phrases, and words. The attributes of these nodes can include, for example, the word embeddings of the corresponding text segment, and their positions within the text paragraph. Adjacency edges between nodes can be used to represent the relationships between different text segments, while dependency edges can be used to represent the dependency relationships between text segments at different scales.

[0142] Multi-scale graph representation of text data can extract structural and / or logical relationships between text segments at different scales, such as characters, words, phrases, clauses, sentences, and paragraphs, and can preserve the text feature vectors corresponding to these text elements, enabling better handling of various natural language processing tasks. Graph representations at different scales consider both the overall picture and the local context, and are more robust to incomplete, fragmented, distorted, or different languages. Downstream tasks can include text translation, text continuation, and automatic question answering. In an exemplary embodiment, both the first and second data can be audio data, and the dense data can be a spectrogram of the audio data, with nodes in the dense data being pixels in the spectrogram. Nodes at the first scale can be, for example, multiple segment regions obtained by dividing the spectrogram into segments along a time-delay direction, while nodes at the second scale can be, for example, feature points extracted from the spectrogram. Second-scale nodes can have adjacent edges to connect adjacent feature points.

[0143] Multi-scale graph representation of audio data can extract multiple segments along the time axis, along with multiple feature points within each segment and the relationships between these feature points (e.g., temporal distance, frequency distance), while preserving the corresponding feature vectors. This allows for the resolution of issues arising from the randomness of different speech sounds, intonations, speaking styles, and content when performing audio-related tasks, especially speech-related tasks. Furthermore, graph representations at different scales consider both the overall picture and local details, making them more robust to incomplete speech and noise. Downstream tasks can include speech translation, among others.

[0144] In some exemplary embodiments, the first and second data can also be various complex data such as molecules, genes, proteins, and sequences. In this case, nodes in the dense data can be the smallest units of the corresponding data type, such as atoms, base pairs, and amino acids. The nodes in the graph representation can be consistent with the dense data or can be units at higher scales, such as atomic groups, functional groups, fragments composed of multiple base pairs (e.g., coding and non-coding regions, or at lower scales, enhancers, promoters, exons, introns, terminators, etc.), amino acid sequences in proteins, peptide chains, etc. Adjacency and dependency edges between nodes can be used to represent various relationships between units at the same scale (e.g., chemical bonds, hydrogen bonds) and various relationships between units at different scales (e.g., dependency relationships). Furthermore, multi-scale graph representations can also represent the structures of these data at different scales, such as the primary, secondary, tertiary, and quaternary structures of proteins. Downstream tasks can include property / structure prediction of molecular structure data and sequence data.

[0145] Multiscale graph representations of these complex data can characterize their complex spatial structure and detailed information, and can reflect the different relationships between various units in the complex data. Therefore, using multiscale graph representations makes it possible to make full use of the above information of complex data for matching tasks or other downstream tasks.

[0146] In some embodiments, other types of data can be first converted into image data, and then a multi-scale graph representation can be generated based on the converted image data. For example, other types of data such as audio data and text data can be converted into image data, and then a multi-scale graph representation can be extracted from the image data, thereby completing various downstream tasks based on the graph representation.

[0147] In some embodiments, graph matching can also be performed between graph representations of different types of data to accomplish specific cross-modal tasks.

[0148] The following section details how to perform graph matching using node and edge attributes. In general, the main difference between graph matching using node and edge attributes and traditional graph matching lies in the fact that when calculating the similarity between nodes / edges, the attribute information of the nodes / edges can be utilized. This includes one or more of the following: similarity of vector type attributes, saliency, etc. For example, the similarity between node A and node B = saliency of node A * saliency of node B * similarity of attributes between node A and node B. Here, the similarity of attributes between node A and node B is determined based on the similarity of vector type attributes between node A and node B.

[0149] According to some embodiments, such as Figure 8 As shown, the graph matching process at each scale in the first and second scales may include:

[0150] Step 801: Determine candidate matching point pairs based on at least one node included in the graph representation of the first data at this scale and at least one node included in the graph representation of the second data at this scale, wherein the candidate matching point pairs include a first candidate matching node belonging to the graph representation of the first data at this scale and a second candidate matching node belonging to the graph representation of the second data at this scale.

[0151] Step 802: For candidate matching point pairs, determine the matching result of the candidate matching point pairs based on the feature vectors of the first candidate matching node included in the candidate matching point pair and the feature vectors of the second candidate matching node included in the candidate matching point pair.

[0152] Step 803: Based on the matching results of the candidate matching point pairs, determine the matching results of the graph representation of the first data at this scale and the graph representation of the second data at this scale.

[0153] It is understandable that the matching result of the graph representation of the first data at this scale and the graph representation of the second data at this scale can be determined based on the matching results of multiple candidate matching point pairs. When the graph representation at this scale includes adjacent edges, it can also be determined based on the matching results of multiple candidate matching edge pairs.

[0154] Therefore, by matching the graph structure formed by the nodes in the graph representation and the two dimensions included in the node's own attributes (such as feature vectors), the information contained in the data can be fully utilized for matching, thereby improving the accuracy of the matching results and the results of subsequent tasks.

[0155] In step 801, the similarity information of the structures presented by nodes (and optionally, adjacent edges between nodes) in the graph representations of different data can be used to determine the matching relationships between nodes in the graph representations of different data, so as to obtain candidate matching point pairs. Existing matching algorithms can be combined to perform node matching between graph representations of different data to obtain candidate matching point pairs.

[0156] In some embodiments, candidate matching point pairs can be obtained quickly using point-by-point matching.

[0157] In some exemplary embodiments, when the matching result of a candidate matching point pair is determined to be a match in step 802, a new candidate matching point pair can be determined (for example, based on the confirmed matching point pairs A and B, the nearest neighbor C of node A and the nearest neighbor D of node B are determined as new candidate matching point pairs), and then step 802 is performed on the new candidate matching point pairs until the new candidate matching point pairs do not match or cannot be determined. Then, in step 803, based on the matching results of all historical candidate matching point pairs in the graph representation at that scale, the matching result of the graph representation at that scale of the first data and the graph representation at that scale of the second data is determined.

[0158] In some exemplary embodiments, steps 802 and 803 can be executed each time a new candidate matching point pair is obtained, thereby determining whether to continue searching for more candidate matching point pairs based on the matching result of the currently obtained graph representation. If the matching result of the graph representation at this time can determine that two data matches (e.g., the matching score is greater than a preset threshold), the search can be stopped and the result returned; otherwise, the search can continue until no more candidate matching point pairs can be found.

[0159] In some embodiments, by combining tree growth algorithms and bundle search, a branch can be grown on the tree formed by the already matched nodes at each step of the recursion, and the scores of the newly grown leaves (i.e., all possible branches) are calculated. The best k leaves are selected as the branches for the next step to achieve point-to-point matching. It is understood that other methods can also be used to achieve point-to-point matching, which are not limited here.

[0160] In some embodiments, candidate matching pairs can be obtained using global matching (e.g., the Hungarian algorithm).

[0161] In some embodiments, dynamic programming can be used to obtain candidate matching point pairs. Dynamic programming can obtain globally optimal matching results. In an exemplary embodiment, the matching result may include multiple candidate matching point pairs. Step 802 can be performed on each candidate matching point pair to obtain the corresponding matching result, and in step 803, the matching result of the graph representation is determined based on the matching results of all candidate matching point pairs.

[0162] In step 802, the matching result of the two nodes can be determined in a variety of ways based on the attributes of the first candidate matching node and the attributes of the second candidate matching node.

[0163] In some embodiments, the matching result of a candidate matching point pair may be, for example, the similarity between the feature vectors of the first candidate matching node and the feature vectors of the second candidate matching node. In some embodiments, the matching result of a candidate matching point pair may also be the product of the saliency of the first candidate matching node, the saliency of the second candidate matching node, and the similarity between the feature vectors of the first and second candidate matching nodes. Such a numerical matching result may also be referred to as the node matching score.

[0164] In some embodiments, the scalar type attribute of the node can be used to determine the first point pair matching result, and then the first point pair matching result can be used to determine whether it is necessary to further use the vector type attribute of the node to determine the second point pair matching result. For example... Figure 9As shown, step 802, determining the matching result of the candidate matching point pair, may include: step 901, determining the first point pair matching result of the candidate matching point pair based on the scalar type attributes of the first candidate matching node and the second candidate matching node included in the candidate matching point pair; step 902, in response to determining that the first point pair matching result of the candidate matching point pair satisfies a first preset condition, determining the second point pair matching result of the candidate matching point pair based on the feature vectors of the first and second candidate matching nodes included in the candidate matching point pair; and step 903, determining the matching result of the candidate matching point pair based on the second point pair matching result. In this way, on the one hand, prior knowledge can be used to judge the matching result based on the scalar type attributes; on the other hand, the computational load can be reduced, and the calculation speed of the matching result can be improved.

[0165] In step 901, for example, the consistency or correlation of the category attributes included in the scalar type attribute can be determined as the first point-to-point matching result, or the difference, ratio or other calculation result of the numerical attributes included in the scalar type attribute can be determined as the first point-to-point matching result, or the first point-to-point matching result can be determined in other ways, which are not limited here.

[0166] In step 902, the first preset condition can correspond to the matching result in the first point mentioned above. For example, it can be that the category attributes are consistent, or that the difference in numerical attributes is less than a threshold. It is understood that those skilled in the art can set the first preset condition according to their needs, and no limitation is made here. The method for determining the matching result in the second point is similar to the method described above for determining the matching result of two nodes using their respective feature vectors, and will not be repeated here.

[0167] In step 903, the second point pair matching result can be directly determined as the matching result of the candidate matching point pair, or the matching result of the candidate matching point pair can be determined based on the first point pair matching result and the second point pair matching result. In an exemplary embodiment, the first point pair matching result is the ratio of the numerical attributes of the two nodes in the candidate matching point pair, and the second point pair matching result is the similarity of the feature vectors of the two nodes. The combined calculation result of the ratio and the similarity can then be determined as the matching result of the candidate matching point pair.

[0168] In some embodiments, after obtaining candidate matching point pairs, the scalar type attribute of the node can be used to filter the candidate matching point pairs, thereby filtering out some mismatched point pairs to obtain more accurate graph representation matching results and reducing the computational load of the graph representation matching result calculation process.

[0169] Back Figure 8In some embodiments, the matching results of the two candidate matching nodes can also be determined using the attributes of the neighboring nodes and neighboring adjacent edges of the first candidate matching node and the neighboring nodes and neighboring adjacent edges of the second candidate matching node. It is understood that the probability of a match between two nodes is high when their neighboring nodes are similar and the edges connecting them are similar.

[0170] In step 803, the matching result of the graph representation of the first and second data at this scale can, for example, be the sum of the matching scores of all candidate matching point pairs. It is understood that other methods can also be used to determine the matching result of the graph representation. In one embodiment, the final matching result can be determined by comparing the sum of the matching scores with a preset threshold. In one embodiment, each candidate matching point pair can have a weight, and the final matching result can, for example, be the weighted sum of the matching scores of all candidate matching point pairs. In one embodiment, the matching result of a candidate matching point pair indicates whether the attributes of the candidate matching point pair are consistent, and the matching result of the graph representation can be determined based on these binary judgment results.

[0171] When performing graph matching, the adjacent edges included in the graph representation can also be matched, and the matching result of the graph representation can be determined based on the matching result of the adjacent edges. In some embodiments, if the graph representation only contains nodes, matching can be performed based on nodes; if the graph representation includes nodes that are adjacent to each other, matching can be performed using both methods simultaneously.

[0172] According to some embodiments, such as Figure 8 As shown, the graph matching process at each scale in the first and second scales may further include:

[0173] Step 804: Determine candidate matching edge pairs based on at least one adjacent edge included in the graph representation of the first data at this scale and at least one adjacent edge included in the graph representation of the second data at this scale, wherein the candidate matching edge pairs include a first candidate matching adjacent edge belonging to the graph representation of the first data at this scale and a second candidate matching adjacent edge belonging to the graph representation of the second data at this scale.

[0174] Step 805: For each candidate matching edge pair, based on the attributes of the first candidate matching adjacent edge and the attributes of the second candidate matching adjacent edge included in the candidate matching edge pair, determine the matching result of the candidate matching edge pair; and

[0175] Step 806: Based on the matching results of the candidate matching edge pairs, determine the matching results of the graph representation of the first data at this scale and the graph representation of the second data at this scale.

[0176] Therefore, by matching the graph structure formed by nodes and adjacent edges in the graph representation and the attributes included in the adjacent edges, the information contained in the data can be fully utilized for matching, thereby improving the accuracy of the matching results and the results of subsequent tasks.

[0177] In some embodiments, step 804 can be performed simultaneously with step 801. That is, the method described above can be used to obtain candidate matching point pairs and candidate matching edge pairs simultaneously. In some embodiments, candidate matching point pairs can be determined first, and then candidate matching edge pairs can be determined based on the adjacent edges between the points included in these candidate matching point pairs.

[0178] It is understandable that the method for determining the matching results of candidate edge pairs is similar to the method for determining the matching results of candidate vertex pairs. The method for determining the matching results of the graph representation based on the matching results of candidate edge pairs is similar to the method for determining the graph representation based on the matching results of candidate vertex pairs, and will not be elaborated here.

[0179] In step 805, the matching result of the two adjacent edges can be determined in a variety of ways based on the attributes of the first candidate matching adjacent edge and the attributes of the second candidate matching adjacent edge.

[0180] In some embodiments, the matching result of a candidate matching edge pair may be, for example, the similarity between the feature vectors of the first candidate matching adjacent edge and the feature vectors of the second candidate matching adjacent edge. In some embodiments, the matching result of a candidate matching edge pair may be the saliency of the first candidate matching adjacent edge, the saliency of the second candidate matching adjacent edge, and the product of the similarity between the feature vectors of the first candidate matching adjacent edge and the feature vectors of the second candidate matching adjacent edge.

[0181] In some embodiments, the matching result of the first edge pair can be determined first using the scalar type attribute of the adjacent edges, and then it can be determined whether the matching result of the second edge pair needs to be further determined using the vector type attribute of the adjacent edges. For example... Figure 10 As shown, step 805, determining the matching result of the candidate matching edge pair may include: step 1001, determining the first edge pair matching result of the candidate matching edge pair based on the scalar type attribute of the first candidate matching adjacent edge included in the candidate matching edge pair and the scalar type attribute of the second candidate matching adjacent edge included in the candidate matching edge pair; step 1002, in response to determining that the first edge pair matching result of the candidate matching edge pair satisfies the second preset condition, determining the second edge pair matching result of the candidate matching edge pair based on the feature vector of the first candidate matching adjacent edge included in the candidate matching edge pair and the feature vector of the second candidate matching adjacent edge included in the candidate matching edge pair; and step 1003, determining the matching result of the candidate matching edge pair based on the second edge pair matching result.

[0182] It is understood that the operations on candidate matching edge pairs in steps 1001-1003 are similar to the operations on candidate matching point pairs in steps 901-903, and will not be described in detail here. Those skilled in the art can set the second preset conditions according to their needs, and there are no restrictions here.

[0183] Back Figure 8 In some embodiments, the matching results of the two candidate matching adjacent edges can also be determined using the neighbor nodes of the first candidate matching adjacent edge and the neighbor nodes of the second candidate matching adjacent edge.

[0184] In step 803, the matching result of the graph representation of the first data and the second data at this scale can be the sum of the matching scores of all candidate matching point pairs and / or the matching scores of all candidate matching edge pairs, or it can be obtained by other means based on the matching results of candidate matching point pairs and / or the matching results of candidate matching edge pairs, which is not limited here.

[0185] In some embodiments, in addition to the matching score, the matching result can also be determined based on the node / edge pairing check result. For example, the node / edge pairing check includes node / edge pairing checks that solve for geometric relationships through projective transformations, etc. It is understood that the matching of nodes in different graph representations can also correspond to transformation relationships in the geometric space between data. Explicit transformations can include projective transformations in scene matching and isometric transformations in fingerprint matching, while implicit transformations can include changes in the speaker and environment in speech-related tasks. The node / edge pairing check result can affect the matching result in two ways: First, during the process of node matching to obtain candidate matching point pairs, constraints can be added, and point / edge pairs that satisfy the constraints can be used as candidate matching point / edge pairs, thereby incorporating prior knowledge into the matching process and accelerating the matching process. Second, after obtaining the initial graph matching result based on the matching results of candidate matching point / edge pairs, the node / edge pairing check result can be used to determine the check result, and the final graph matching result can be determined jointly based on the initial graph matching result and the check result. For example, if the initial graph matching result shows a matching degree of 80%, but the check result shows a mismatch, a weighted average can be used to obtain a final graph matching result of, for example, 70%.

[0186] In the process of graph matching for multi-scale graph representation, graph matching at each scale can be performed independently, or graph matching can be performed at a certain scale first, and then the matching results at that scale can be used to determine whether to perform graph matching at other scales, or to adjust the graph matching strategy or graph matching parameters at other scales.

[0187] According to some embodiments, step 705, performing graph matching on the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, may include: in response to determining that the first matching result is a successful match, performing graph matching on the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result. Thus, by first performing graph matching at the first scale, which reflects overall information (less information), and then determining whether to perform graph matching at the second scale, which reflects local information (more information), based on the first-scale graph matching result, the number of graph matches at the second scale can be reduced. This reduces the overall time consumption of the matching process without affecting the matching result and subsequent task processing results, thereby improving task processing efficiency.

[0188] According to some embodiments, step 705, performing graph matching on the graph representation of the first data at a second scale and the graph representation of the second data at a second scale to obtain a second matching result, may include: in response to determining that the first matching result is a successful match, matching the first subgraph of the first data at a second scale and the second subgraph of the second data at a second scale. Wherein, the first matching result indicates that a first node in the graph representation of the first data at a first scale and a second node in the graph representation of the first data at a first scale have successfully matched. The first subgraph may include nodes in the graph representation of the first data at a second scale that have a subordinate relationship with the first node, and the second subgraph may include nodes in the graph representation of the second data at a second scale that have a subordinate relationship with the second node.

[0189] Therefore, by first performing graph matching at the first scale, and then matching the subgraphs of nodes that have been successfully matched as indicated by the graph matching results at the first scale, it is not necessary to match parts of the graph representation that are unlikely to match. This reduces the number of nodes and / or adjacent edges that need to be calculated for matching results, and further reduces the overall time consumption of the matching process without affecting the matching results and subsequent task processing results, thereby improving task processing efficiency.

[0190] Understandably, the description of graph matching in step 104 also applies to graph matching in step 104.

[0191] According to some embodiments, step 705, performing graph matching on the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, may include: determining the matching result of the current node based on the attributes of the current node and whether nodes at the first scale that have a subordinate relationship with the current node have successfully matched, wherein the current node is a node at the second scale. By considering vertical relationships in the low-scale graph matching process and using the matching results of high-scale nodes that have a subordinate relationship with the node as reference elements, the accuracy of the low-scale graph matching results can be improved.

[0192] In one embodiment, the current matching result can be obtained based on the matching degree of two graph representations. The matching degree of the graph representation can be expressed as the sum of the matching degree of all candidate matching point pairs and the matching degree of all candidate matching adjacent edge pairs. The matching degree of a candidate matching point pair is the product of the salience of the first candidate matching point, the salience of the second candidate matching point, and the similarity between the feature vectors of the first and second candidate matching points. The matching degree of a candidate matching adjacent edge pair is the product of the salience of the first and second candidate matching adjacent edges, and the similarity between the feature vectors of the first and second candidate matching adjacent edges. Therefore, through this method, unmatched nodes / edges are weakened, thus preserving stable and reliable local features at different scales.

[0193] Figure 11 A structural block diagram of a neural network training apparatus 1100 according to an embodiment of the present disclosure is shown. The apparatus 1100 includes: a first acquisition unit 1110 configured to acquire first sample data and second sample data, the first sample data and the second sample data being one of image data, audio data, text data, molecular structure data, and sequence data, respectively; and a second acquisition unit 1120 configured to acquire multi-scale graph representations of the first sample data and the second sample data respectively, wherein the multi-scale graph representations are determined using a graph representation extraction network, and the multi-scale graph representations include graph representations at the first scale and graph representations at the second scale. Each scale graph representation of the first sample data and the second sample data includes at least one node, the node includes attributes, the node attributes include vector-type attributes, and the graph representation extraction network includes attributes for determining the vector type of the node. The network module includes: a first graph matching unit 1130 configured to perform graph matching on the graph representations of the first sample data at a first scale and the graph representations of the second sample data at a first scale to obtain a first current matching result representing the degree of matching at the first scale; a second graph matching unit 1140 configured to perform graph matching on the graph representations of the first sample data at a second scale and the graph representations of the second sample data at a second scale to obtain a second current matching result representing the degree of matching at the second scale; a third acquisition unit 1150 configured to acquire the target matching result and / or target task processing result of the first sample data and the second sample data; a first determination unit 1160 configured to determine a loss value based on the target matching result and / or target task processing result, and the first current matching result and / or the second current matching result; and a training unit 1170 configured to train a graph representation extraction network based on the loss value.

[0194] It is understandable that the operations of units 1110-1170 in device 1100 are similar to the operations of steps 101-107 in method 100, and will not be described in detail here.

[0195] Figure 12 A structural block diagram of a graph representation extraction apparatus 1200 according to an embodiment of the present disclosure is shown. The apparatus 1200 includes: a graph representation extraction network 1210 configured to receive target data and output a multi-scale graph representation of the target data, wherein the target data is one of image data, audio data, text data, molecular structure data, and sequence data, and the graph representation of the target data includes a first-scale graph representation and a second-scale graph representation. The first-scale graph representation includes at least one node of the first scale, and the first-scale node has attributes, including vector-type attributes. The second scale is lower than the first scale, and the second-scale graph representation includes at least one node of the second scale, and the second-scale node has attributes, including vector-type attributes. The graph representation extraction network 1210 is trained using a neural network training device of the apparatus 1100.

[0196] Figure 13 A structural block diagram of a task processing apparatus 1300 according to an embodiment of the present disclosure is shown. The apparatus 1300 includes: a fourth acquisition unit 1310 configured to acquire first data and second data, the first data and second data being one of image data, audio data, text data, molecular structure data, and sequence data, respectively; and a second determination unit 1320 configured to process the first data and second data respectively using a graph representation extraction device 1200 to perform the aforementioned method, thereby determining a multi-scale graph representation of each of the first data and the second data, wherein the multi-scale graph representation of each of the first data and the second data includes a first-scale graph representation and a second-scale graph representation, the first-scale graph representation including at least one first-scale node, the first-scale node having attributes, and the attributes of the first-scale node... The attributes include vector type properties, the second scale is lower than the first scale, the graph representation of the second scale includes at least one node of the second scale, the node of the second scale has attributes, and the attributes of the node of the second scale include vector type properties; the third graph matching unit 1330 is configured to perform graph matching on the graph representation of the first data at the first scale and the graph representation of the second data at the first scale to obtain a first matching result; the fourth graph matching unit 1340 is configured to perform graph matching on the graph representation of the first data at the second scale and the graph representation of the second data at the second scale to obtain a second matching result; the third determining unit 1350 is configured to determine a multi-scale matching result based on the first matching result and the second matching result; and the fourth determining unit 1360 is configured to determine a task processing result based on the multi-scale matching result.

[0197] It is understandable that the operations of units 1310-1360 in device 1300 are similar to the operations of steps 701-706 in method 700, and will not be described in detail here.

[0198] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0199] In the following text, combined with Figure 14 Illustrative examples describing such electronic devices, non-transitory computer-readable storage media, and computer program products.

[0200] Figure 14 An example configuration of electronic device 1400 that can be used to implement the methods described herein is shown. Each of the above-described devices 1100, 1200, and 1300 can also be implemented wholly or at least partially by electronic device 1400 or similar devices or systems.

[0201] Electronic device 1400 can be a variety of different types of devices. Examples of electronic device 1400 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to display devices, game consoles), televisions or other display devices, automotive computers, and so on.

[0202] Electronic device 1400 may include at least one processor 1402, memory 1404, multiple communication interfaces 1406, display device 1408, other input / output (I / O) devices 1410, and one or more mass storage devices 1412 capable of communicating with each other, such as via system bus 1414 or other suitable connections.

[0203] Processor 1402 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 1402 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 1402 may be configured to fetch and execute computer-readable instructions stored in memory 1404, mass storage device 1412, or other computer-readable media, such as program code of operating system 1416, program code of application program 1418, program code of other program 1420, etc.

[0204] Memory 1404 and mass storage device 1412 are examples of computer-readable storage media for storing instructions executed by processor 1402 to perform the various functions described above. For example, memory 1404 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 1412 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 1404 and mass storage device 1412 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 1402 as a specific machine configured to perform the operations and functions described in the examples herein.

[0205] Multiple programs may be stored on mass storage device 1412. These programs include operating system 1416, one or more application programs 1418, other programs 1420, and program data 1422, and they may be loaded into memory 1404 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: method 100, method 500, method 600, and / or method 700 (including any suitable steps of method 100, method 500, method 600, and method 700), and / or other embodiments described herein.

[0206] Although Figure 14 The modules 1416, 1418, 1420, and 1422, or portions thereof, are illustrated as being stored in memory 1404 of electronic device 1400. However, modules 1416, 1418, 1420, and 1422 may be implemented using any form of computer-readable medium accessible by electronic device 1400. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0207] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by electronic devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0208] One or more communication interfaces 1406 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth™ interface, Near Field Communication (NFC) interface, etc. Communication interface 1406 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 1406 can also provide communication with external storage devices (not shown), such as storage arrays, network-attached storage, storage area networks, etc.

[0209] In some examples, a display device 1408, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 1410 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0210] The technologies described herein can be supported by these various configurations of electronic device 1400, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from electronic device 1400. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect electronic device 1400 to other electronic devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality may be implemented partly on electronic device 1400 and partly through the platform that abstracts the functionality of the cloud.

[0211] Although this disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and suggestive, not restrictive; this disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practice with respect to the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps not listed, the indefinite article "a" or "an" does not exclude a plurality, the term "a plurality" means two or more, and the term "based on" should be interpreted as "at least partially based on". The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. A method for training a neural network, the method comprising: Acquire first sample data and second sample data, wherein the first sample data and the second sample data are one of image data, audio data, text data, molecular structure data and sequence data, respectively, and the sequence data is one of protein sequence data and gene sequence data; The first sample data and the second sample data are each obtained as multi-scale graph representations. The multi-scale graph representations are determined using a graph representation extraction network. Each multi-scale graph representation includes a first-scale graph representation and a second-scale graph representation. Each scale of the multi-scale graph representations of the first and second sample data includes at least one node. Each node includes attributes, including vector-type attributes and scalar-type attributes. The scalar-type attributes of the node include the node's saliency. The graph representation extraction network includes a network module for determining the vector-type attributes of the node and a network module for determining the scalar-type attributes of the node. Graph matching is performed between the graph representation of the first sample data at the first scale and the graph representation of the second sample data at the first scale to obtain a first current matching result that characterizes the degree of matching at the first scale; Graph matching is performed between the graph representation of the first sample data at the second scale and the graph representation of the second sample data at the second scale to obtain a second current matching result that characterizes the degree of matching at the second scale; Obtain the target matching results and / or target task processing results of the first sample data and the second sample data; The loss value is determined based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and The graph representation extraction network is trained based on the loss value.

2. The method according to claim 1, wherein, The loss value includes the matching loss value and / or the task loss value. The determination of the loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result includes: The matching loss value is determined based on the first current matching result and / or the second current matching result and the target matching result; And / or, Determine the current task result based on the first current matching result and / or the second current matching result; and The task loss value is determined based on the target task processing result and the current task result.

3. The method according to claim 2, wherein, Determining the matching loss value based on the first current matching result and / or the second current matching result includes: Determine the current matching result based on the first current matching result and / or the second current matching result; and Determine the matching loss value based on the current matching result and the target matching result.

4. The method according to claim 2, wherein, The graph representation extraction network includes a first network for extracting graph representations at a first scale, wherein determining the matching loss value based on the first current matching result and / or the second current matching result includes: Based on the target matching result and the first current matching result, determine the first scale matching loss value; Training the graph representation extraction network based on the loss value includes: The first network is trained based on the loss value matched by the first scale. And / or, The graph representation extraction network includes a second network for extracting graph representations at a second scale, wherein determining the matching loss value based on the first current matching result and / or the second current matching result includes: Based on the target matching result and the second current matching result, determine the second scale matching loss value; Training the graph representation extraction network based on the loss value includes: The second network is trained based on the loss value matched by the second scale.

5. The method according to any one of claims 1-4, wherein, The target matching result and / or the target task processing result are determined based on one of the following: Methods based on manual annotation, teacher-based models and / or pre-trained models, auxiliary constraint information, and rules.

6. The method according to any one of claims 1-4, wherein, The target matching result is determined using a network trained for the Nth round, wherein obtaining the first sample data and the second sample data includes: The network trained for the Nth round is used to extract multi-scale graph representations of the first and second unlabeled data, respectively. Graph matching is performed between the first scale graph representation of the first unlabeled data and the first scale graph representation of the second unlabeled data to obtain the first unlabeled data matching result that characterizes the degree of matching at the first scale; Graph matching is performed between the second-scale graph representation of the first unlabeled data and the second-scale graph representation of the second unlabeled data to obtain the second unlabeled data matching result that characterizes the degree of matching at the second scale; The unlabeled data matching result is determined based on the first unlabeled data matching result and / or the second unlabeled data matching result; In response to determining that the first unlabeled data and the second unlabeled data satisfy a first condition, the first unlabeled data and the second unlabeled data are determined as first sample data and second sample data as positive samples, wherein satisfying the first condition includes the unlabeled data matching result satisfying a first matching condition, and the target matching result of the positive samples indicates that the corresponding first sample data and second sample data match; and / or In response to determining that the first unlabeled data and the second unlabeled data satisfy the second condition, the first unlabeled data and the second unlabeled data are determined as the first sample data and the second sample data as negative samples, wherein the first unlabeled data and the second unlabeled data satisfying the second condition includes the unlabeled data matching result satisfying the second matching condition, and the target matching result of the negative sample indicates that the corresponding first sample data and the second sample data do not match.

7. The method according to claim 6, wherein, Based on the first unlabeled data matching result and / or the second unlabeled data matching result, the unlabeled data matching result is determined as follows: In response to determining that the second unlabeled data matching result indicates that the second-scale graph representation of the first unlabeled data and the second-scale graph representation of the second unlabeled data have successfully matched, the unlabeled data matching result is determined to be a match.

8. The method according to any one of claims 1-4, wherein, The graph represents the extraction network, which includes a rule module and a network module. The method further includes at least one of the following steps: In response to determining that the fifth preset condition is met, the first rule module in the rule module is replaced with a network module; and In response to the determination that the sixth preset condition is met, a network module is added to the extraction network represented in the figure. Training the graph representation extraction network based on the loss value includes: The network module is trained based on the loss value.

9. The method according to any one of claims 1-4, wherein, The graph representation of at least one scale in the multi-scale graph representation of both the first and second sample data includes at least one adjacent edge. Each of the at least one adjacent edge is used to characterize the relative relationship between two nodes at the same scale. The adjacent edge has attributes, including scalar type attributes and / or vector type attributes. The graph representation extraction network includes at least one of the following: A network module used to determine the scalar type attributes of adjacent edges; and A network module used to determine the vector type attributes of adjacent edges.

10. The method according to claim 9, wherein, The vector-type attributes of a node include the node's feature vector, and / or the scalar-type attributes of adjacent edges include the saliency of the adjacent edges, and / or the vector-type attributes of adjacent edges include the feature vector of the adjacent edges.

11. The method according to any one of claims 1-4, wherein, The graph representation extraction network includes at least one of the following: A sparsification module used to sparsify dense data to obtain sparsified nodes. A merging module is used to merge the low-scale nodes obtained by sparsification to obtain high-scale nodes.

12. A graph representation extraction method, comprising: The target data is input into a graph representation extraction network to obtain a multi-scale graph representation of the target data. The target data is one of image data, audio data, text data, molecular structure data, and sequence data. The sequence data is one of protein sequence data and gene sequence data. The graph representation of the target data includes a first-scale graph representation and a second-scale graph representation. The first-scale graph representation includes at least one node of the first scale, and the first-scale node has attributes, including vector-type attributes. The second scale is lower than the first scale, and the second-scale graph representation includes at least one node of the second scale, and the second-scale node has attributes, including vector-type attributes. The graph representation extraction network is obtained by training using the method according to any one of claims 1-11.

13. The method according to claim 12, wherein, The graph representation of the target data at at least one scale includes at least one adjacent edge, each of the at least one adjacent edge being used to characterize the relative relationship between two nodes at the same scale, and the adjacent edge having attributes.

14. The method according to claim 12 or 13, wherein, The multi-scale graph representation of the target data includes at least one dependent edge, each of which is used to characterize the dependency relationship between two nodes at different scales, and the dependent edge has attributes.

15. The method according to claim 14, wherein, The attributes of the dependent edge are determined based on the attributes of the two nodes connected to the dependent edge.

16. The method according to claim 12 or 13, wherein, The graphical representation of at least one of the first and second scales includes at least one of the following: The attributes of nodes at this scale include scalar type attributes; The properties of adjacent edges at this scale include scalar type properties; The attributes of the dependent edges of this scale include scalar type attributes; The properties of adjacent edges at this scale include vector-type properties; and The attributes of the dependent edges at this scale include attributes of vector type.

17. A task processing method, comprising: Acquire first data and second data, wherein the first data and the second data are respectively one of image data, audio data, text data, molecular structure data and sequence data, and the sequence data is one of protein sequence data and gene sequence data; The method according to any one of claims 12-16 is performed on the first data and the second data respectively to determine the multi-scale graph representations of the first data and the second data, wherein the multi-scale graph representations of the first data and the second data respectively include a graph representation at a first scale and a graph representation at a second scale, the graph representation at the first scale including at least one node at the first scale, the node at the first scale having attributes, the attributes of the node at the first scale including vector type attributes, the second scale being lower than the first scale, the graph representation at the second scale including at least one node at the second scale, the node at the second scale having attributes, the attributes of the node at the second scale including vector type attributes. Graph matching is performed between the first-scale graph representation of the first data and the first-scale graph representation of the second data to obtain a first matching result; Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result; Based on the first matching result and the second matching result, determine the multi-scale matching result; and Based on the multi-scale matching results, the task processing result is determined.

18. The method according to claim 17, wherein, The node's vector-type attributes include feature vectors, where the graph matching process at each scale in the first and second scales includes: Candidate matching point pairs are determined based on at least one node included in the graph representation of the first data at that scale and at least one node included in the graph representation of the second data at that scale, wherein the candidate matching point pair includes a first candidate matching node belonging to the graph representation of the first data at that scale and a second candidate matching node belonging to the graph representation of the second data at that scale; For the candidate matching point pair, the matching result of the candidate matching point pair is determined based on the feature vector of the first candidate matching node included in the candidate matching point pair and the feature vector of the second candidate matching node included in the candidate matching point pair. Based on the matching results of the candidate matching point pairs, the matching results of the graph representation of the first data at this scale and the graph representation of the second data at this scale are determined; And / or, Wherein, the graph representation of each data at at least one scale includes at least one adjacent edge, each of the at least one adjacent edge being used to characterize the relative relationship between two nodes at the same scale, the adjacent edge having attributes, wherein the graph matching process for each scale in the first scale and the second scale includes: Candidate matching edge pairs are determined based on at least one adjacent edge included in the graph representation of the first data at that scale and at least one adjacent edge included in the graph representation of the second data at that scale, wherein the candidate matching edge pair includes a first candidate matching adjacent edge belonging to the graph representation of the first data at that scale and a second candidate matching adjacent edge belonging to the graph representation of the second data at that scale. For the candidate matching edge pair, based on the attributes of the first candidate matching adjacent edge included in the candidate matching edge pair and the attributes of the second candidate matching adjacent edge included in the candidate matching edge pair, the matching result of the candidate matching edge pair is determined; and Based on the matching results of the candidate matching edge pairs, the matching results of the graph representation of the first data at this scale and the graph representation of the second data at this scale are determined.

19. The method according to claim 18, wherein, The node's attributes also include scalar type attributes, wherein determining the matching result of the candidate matching point pair includes: Based on the scalar type attributes of the first candidate matching node included in the candidate matching point pair and the scalar type attributes of the second candidate matching node included in the candidate matching point pair, the first point pair matching result of the candidate matching point pair is determined. In response to determining that the first point pair matching result of the candidate matching point pair satisfies a first preset condition, the second point pair matching result of the candidate matching point pair is determined based on the feature vectors of the first candidate matching nodes included in the candidate matching point pair and the feature vectors of the second candidate matching nodes included in the candidate matching point pair; and Based on the matching result of the second point pair, the matching result of the candidate matching point pair is determined. and / or The attributes of adjacent edges include scalar type attributes and vector type attributes. The vector type attributes of adjacent edges include feature vectors. The matching result for determining the candidate matching edge pairs includes: Based on the scalar type attributes of the first candidate matching adjacent edge included in the candidate matching edge pair and the scalar type attributes of the second candidate matching adjacent edge included in the candidate matching edge pair, the first edge pair matching result of the candidate matching edge pair is determined; In response to determining that the first edge pair matching result of the candidate matching edge pair satisfies a second preset condition, the second edge pair matching result of the candidate matching edge pair is determined based on the feature vectors of the first candidate matching adjacent edges included in the candidate matching edge pair and the feature vectors of the second candidate matching adjacent edges included in the candidate matching edge pair; and Based on the matching result of the second edge pair, the matching result of the candidate matching edge pair is determined.

20. The method according to claim 18, wherein, The scalar type properties of a node include the saliency of the node, and / or the scalar type properties of adjacent edges include the saliency of the adjacent edges.

21. The method according to claim 20, wherein, Determining the matching result of the candidate matching point pair includes: The matching result of the candidate matching point pair is determined by the product of the salience of the first candidate matching node included in the candidate matching point pair, the salience of the second candidate matching node included in the candidate matching point pair, and the similarity between the feature vectors of the first candidate matching node and the feature vectors of the second candidate matching node. And / or, The matching result for the candidate matching edge pair includes: The salience of the first candidate matching adjacent edge included in the candidate matching edge pair, the salience of the second candidate matching adjacent edge included in the candidate matching edge pair, and the product of the similarity between the feature vectors of the first candidate matching adjacent edge and the feature vectors of the second candidate matching adjacent edge are determined as the matching result of the candidate matching edge pair.

22. The method according to any one of claims 17-21, wherein, Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, including: In response to determining that the first matching result is a successful match, a graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result.

23. The method according to any one of claims 17-21, wherein, Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, including: In response to determining that the first matching result is a successful match, the first subgraph at the second scale of the first data and the second subgraph at the second scale of the second data are matched, wherein the first matching result indicates that the first node in the graph representation at the first scale of the first data and the second node in the graph representation at the first scale of the second data are successfully matched, the first subgraph includes nodes in the graph representation at the second scale of the first data that have a subordinate relationship with the first node, and the second subgraph includes nodes in the graph representation at the second scale of the second data that have a subordinate relationship with the second node.

24. The method according to any one of claims 17-21, wherein, Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, including: The matching result of the current node is determined based on the attributes of the current node and whether the nodes of the first scale that are subordinate to the current node are successfully matched, wherein the current node is a node of the second scale.

25. The method according to any one of claims 17-21, wherein, The second data is obtained from the database, wherein determining the task processing result based on the multi-scale matching result includes: Based on the multi-scale matching results of the first data and multiple second data in the database, at least one second data matching the first data is determined; and The task processing result is determined based on the at least one second data.

26. The method according to any one of claims 17-21, wherein, Both the first data and the second data are image data.

27. A training device for a neural network, comprising: The first acquisition unit is configured to acquire first sample data and second sample data, wherein the first sample data and the second sample data are respectively one of image data, audio data, text data, molecular structure data and sequence data, and the sequence data is one of protein sequence data and gene sequence data. The second acquisition unit is configured to acquire multi-scale graph representations of the first sample data and the second sample data respectively, wherein the multi-scale graph representation is determined using a graph representation extraction network, and the multi-scale graph representation includes graph representations at a first scale and graph representations at a second scale. Each scale graph representation in the multi-scale graph representations of the first sample data and the second sample data includes at least one node, the node includes attributes, the attributes of the node include vector type attributes and scalar type attributes, the scalar type attributes of the node include the saliency of the node, and the graph representation extraction network includes a network module for determining the vector type attributes of the node and a network module for determining the scalar type attributes of the node. The first graph matching unit is configured to perform graph matching between the graph representation of the first sample data at a first scale and the graph representation of the second sample data at a first scale, so as to obtain a first current matching result representing the degree of matching at the first scale; The second graph matching unit is configured to perform graph matching between the second-scale graph representation of the first sample data and the second-scale graph representation of the second sample data to obtain a second current matching result characterizing the degree of matching at the second scale. The third acquisition unit is configured to acquire the target matching result and / or target task processing result of the first sample data and the second sample data; The first determining unit is configured to determine a loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and The training unit is configured to train the graph representation extraction network based on the loss value.

28. A graphic representation extraction device, comprising: A graph representation extraction network is configured to receive first data and output a multi-scale graph representation of the first data, wherein the first data is one of image data, audio data, text data, molecular structure data, and sequence data, and the sequence data is one of protein sequence data and gene sequence data. The graph representation of the first data includes a first-scale graph representation and a second-scale graph representation. The first-scale graph representation includes at least one node of the first scale, and the first-scale node has attributes, including vector-type attributes. The second scale is lower than the first scale. The second-scale graph representation includes at least one node of the second scale, and the second-scale node has attributes, including vector-type attributes. The diagram shows that the extraction network is trained using the neural network training apparatus according to claim 27.

29. A task processing apparatus, comprising: The fourth acquisition unit is configured to acquire first data and second data, wherein the first data and the second data are one of image data, audio data, text data, molecular structure data and sequence data, and the sequence data is one of protein sequence data and gene sequence data. The second determining unit is configured to process the first data and the second data respectively using the graph representation extraction apparatus according to claim 28 to determine the multi-scale graph representations of the first data and the second data, wherein the multi-scale graph representations of the first data and the second data respectively include a first-scale graph representation and a second-scale graph representation, the first-scale graph representation including at least one node of the first scale, the first-scale node having attributes, the attributes of the first-scale node including vector-type attributes, the second scale being lower than the first scale, the second-scale graph representation including at least one node of the second scale, the second-scale node having attributes, the attributes of the second-scale node including vector-type attributes. The third graph matching unit is configured to perform graph matching between the graph representation of the first data at a first scale and the graph representation of the second data at a first scale to obtain a first matching result; The fourth graph matching unit is configured to perform graph matching between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result; The third determining unit is configured to determine a multi-scale matching result based on the first matching result and the second matching result; and The fourth determining unit is configured to determine the task processing result based on the multi-scale matching result.

30. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-26.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-26.

32. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-26.

Images