Task processing method, neural network training method, device, equipment and medium

Abstract

The present disclosure provides a task processing method, a neural network training method, an apparatus, a device and a medium, and relates to the technical field of artificial intelligence. The task processing method comprises: obtaining first data and second data; obtaining a graph representation of a first scale and a graph representation of a second scale of the first data and the second data respectively, the second scale being lower than the first scale, the graph representation of each scale comprising nodes of the scale, each node of each scale comprising an attribute of a vector type, at least one node of each data of each scale being obtained by sparsifying dense data corresponding to the data, and the graph representation of at least one scale of each data comprising adjacent edges representing the relative relationship between the nodes of the scale; performing graph matching of the first scale and the second scale on the first data and the second data respectively to obtain a first matching result and a second matching result; and determining a multi-scale matching result based on the first matching result and / or the second matching result and further determining a task processing result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to a task processing method, a neural network training method, a task processing device, a neural network training device, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] When analyzing and processing unstructured data such as images, videos, speech, text, molecular structures, protein sequences, etc., the original form of the data is usually difficult to use directly to produce effective results, and a more effective method is to convert the unstructured data into a semi-structured intermediate representation, and then analyze on the intermediate representation. Therefore, determining a suitable intermediate representation form of unstructured data and how to effectively analyze and process unstructured data using such an intermediate representation has become a problem to be solved.

[0003] The methods described in this section can not necessarily be the methods previously conceived or adopted. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because it is included in this section. Similarly, unless otherwise indicated, matters discussed in this section should not be assumed to be prior to the application. SUMMARY

[0004] The present disclosure provides a task processing method, a neural network training method, a task processing device, a neural network training device, an electronic device, a computer readable storage medium, and a computer program product.

[0005] According to an aspect of the present disclosure, a task processing method is provided, including: obtaining first data and second data, the first data and the second data being one of image data, audio data, text data, and sequence data; obtaining a first-scale graph representation of the first data and the second data, the first-scale graph representation including at least one first-scale node, wherein the first-scale node has attributes, and the attributes of the first-scale node include vector-type attributes; obtaining a second-scale graph representation of the first data and the second data, the second scale being lower than the first scale, the second-scale graph representation including at least one second-scale node, wherein the second-scale node has attributes, and the attributes of the second-scale node include vector-type attributes, wherein the at least one scale node of each of the first data and the second data is obtained by sparsifying dense data corresponding to the data, and the at least one scale graph representation of each data includes at least one adjacent edge, each of the at least one adjacent edge being used to represent a relative relationship between two nodes of the same scale, and the adjacent edge has attributes; performing graph matching on the first-scale graph representation of the first data and the first-scale graph representation of the second data to obtain a first matching result; performing graph matching on the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result; determining a multi-scale matching result based on the first matching result and the second matching result; and determining a task processing result based on the multi-scale matching result.

[0006] According to an aspect of the present disclosure, a neural network training method is provided, including: obtaining first sample data and second sample data, the first sample data and the second sample data being one of image data, audio data, text data, molecular structure data, and sequence data; obtaining a multi-scale graph representation of the first sample data and the second sample data, wherein the multi-scale graph representation is determined by a graph representation extraction network, and the multi-scale graph representation includes a first-scale graph representation and a second-scale graph representation; performing graph matching on the first-scale graph representation of the first sample data and the first-scale graph representation of the second sample data to obtain a first current matching result representing a matching degree of the first scale; performing graph matching on the second-scale graph representation of the first sample data and the second-scale graph representation of the second sample data to obtain a second current matching result representing a matching degree of the second scale; obtaining a target matching result and / or a target task processing result of the first sample data and the second sample data; determining a loss value according to the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and training the graph representation extraction network according to the loss value.

[0007] According to another aspect of the present disclosure, there is provided a task processing apparatus, comprising: a first obtaining unit configured to obtain first data and second data, the first data and the second data being one of image data, audio data, text data, molecular structure data and sequence data, respectively; a second obtaining unit configured to obtain a first scale graph representation of each of the first data and the second data, the first scale graph representation comprising at least one first scale node, wherein the first scale node has attributes, and the attributes of the first scale node comprise attributes of a vector type; a third obtaining unit configured to obtain a second scale graph representation of each of the first data and the second data, the second scale being lower than the first scale, the second scale graph representation comprising at least one second scale node, wherein the second scale node has attributes, and the attributes of the second scale node comprise attributes of a vector type, wherein the at least one scale node of each of the first data and the second data is obtained by sparsifying dense data corresponding to the data, and the at least one scale graph representation of each data comprises at least one adjacent edge, each of the at least one adjacent edge being used to represent a relative relationship between two nodes of the same scale, and the adjacent edge has attributes; a first graph matching unit configured to perform graph matching on the first scale graph representation of the first data and the first scale graph representation of the second data to obtain a first matching result; a second graph matching unit configured to perform graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result; a first determining unit configured to determine a multi-scale matching result based on the first matching result and the second matching result; and a second determining unit configured to determine a task processing result based on the multi-scale matching result.

[0008] According to another aspect of the present disclosure, a training device of a neural network is provided. The method comprises: a fourth acquisition unit configured to acquire first sample data and second sample data, the first sample data and the second sample data being one of image data, audio data, text data, molecular structure data and sequence data respectively; a fifth acquisition unit configured to acquire a multi-scale graph representation of each of the first sample data and the second sample data, wherein the multi-scale graph representation is determined by a graph representation extraction network, and the multi-scale graph representation comprises a graph representation of a first scale and a graph representation of a second scale; a third graph matching unit configured to perform graph matching on the graph representation of the first scale of the first sample data and the graph representation of the first scale of the second sample data to obtain a first current matching result representing a matching degree of the first scale; a fourth graph matching unit configured to perform graph matching on the graph representation of the second scale of the first sample data and the graph representation of the second scale of the second sample data to obtain a second current matching result representing a matching degree of the second scale; a seventh acquisition unit configured to acquire a target matching result and / or a target task processing result of the first sample data and the second sample data; a third determination unit configured to determine a loss value according to the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and a training unit configured to train the graph representation extraction network according to the loss value.

[0009] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above method.

[0011] According to one or more embodiments of the present disclosure, by extracting decoupled features of multiple scales of the first data and the second data, more general and more representative multi-scale graph representations of each data can be obtained, and then graph matching is performed using the multi-scale graph representations, and a task processing result is determined according to a graph matching result. On the one hand, the graph representation contains more detailed features than the vector representation, and the matching result obtained by using graph matching is more accurate and reliable than that obtained by using vector matching. The graph representations of different scales can take into account the overall scale and the local scale, and are more robust to incomplete, deformed, perspective changes, occlusions, attack samples and the like, and are more interpretable. On the other hand, the complex task is converted into a graph matching task, so that a model meeting the accuracy requirement can be trained more easily, and an accurate task processing result can be obtained.

[0012] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the present disclosure but merely contemplate it to a preferred embodiment. Other features, aspects, and advantages of the present disclosure will become apparent from the following description, which is given by way of example in accordance with the principles of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in

[0014] Figure 1 A flowchart of a task processing method according to an embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic diagram of a multi-scale graph representation according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A flowchart of a graph matching process for each scale in the illustrated method is shown; Figure 1 A flowchart of a graph matching process for each scale in the illustrated method is shown;

[0017] Figure 4 A flowchart of a graph matching process for each scale in the illustrated method is shown; Figure 3 A flowchart of a matching result determination process for a candidate matching point pair in the illustrated method is shown;

[0018] Figure 5 A flowchart of a matching result determination process for a candidate matching edge pair in the illustrated method is shown; Figure 3 A flowchart of a matching result determination process for a candidate matching edge pair in the illustrated method is shown;

[0019] Figure 6 A flowchart of a training method of a neural network according to an embodiment of the present disclosure is shown;

[0020] Figure 7 A flowchart of a training method of a neural network according to an embodiment of the present disclosure is shown; Figure 6 A flowchart of a first sample data and second sample data acquisition process in the illustrated method is shown;

[0021] Figure 8 A flowchart of a loss value determination process in the illustrated method is shown; Figure 6 A flowchart of a loss value determination process in the illustrated method is shown;

[0022] Figure 9 A flowchart of a training method of a neural network according to an embodiment of the present disclosure is shown;

[0023] Figure 10 A block diagram of a task processing apparatus according to an embodiment of the present disclosure is shown;

[0024] Figure 11 A block diagram of a training apparatus of a neural network according to an embodiment of the present disclosure is shown; and

[0025] Figure 12 A structural block diagram of an electronic device of a server or a client according to an embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help the understanding of the present disclosure. These should be considered in the context of the overall description and they are not intended to limit the scope of the present disclosure, which is described in the appended claims. Therefore, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, the descriptions below are made with omission of descriptions of well-known functions and structures for the sake of clarity and conciseness.

[0027] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only and do not intend to limit the positional relationship, the chronological relationship, or the importance of the elements, and such terms are used only to distinguish one element from another element. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0028] The terms used in the description of various examples in the present disclosure are only for the purpose of describing the specific examples and are not intended to be limiting. Unless the number of elements is specifically limited, the element can be one or more than one, unless the context clearly indicates otherwise. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations thereof.

[0029] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0030] Figure 1 A flowchart of a task processing method 100 according to an embodiment of the present disclosure is shown, the method 100 comprising:

[0031] Step 101, obtaining first data and second data, the first data and the second data being one of image data, audio data, text data, molecular structure data, and sequence data, respectively;

[0032] Step 102, obtaining a first scale graph representation of each of the first data and the second data, the first scale graph representation comprising at least one first scale node, wherein the first scale node has an attribute, and the attribute of the first scale node comprises a vector type attribute;

[0033] In step 103, a second scale graph representation of the first data and the second data is obtained, the second scale being lower than the first scale, the second scale graph representation including at least one second scale node, wherein the second scale node has an attribute, and the attribute of the second scale node includes a vector type attribute, wherein the at least one scale node of each of the first data and the second data is obtained by sparsifying dense data corresponding to the data, and the at least one scale graph representation of each data includes at least one adjacent edge, each of the at least one adjacent edge being used to represent a relative relationship between two nodes of the same scale, and the adjacent edge has an attribute;

[0034] In step 104, the first scale graph representation of the first data and the first scale graph representation of the second data are graph matched to obtain a first matching result.

[0035] In step 105, the second scale graph representation of the first data and the second scale graph representation of the second data are graph matched to obtain a second matching result.

[0036] In step 106, a multi-scale matching result is determined based on the first matching result and the second matching result.

[0037] In step 107, a task processing result is determined based on the multi-scale matching result.

[0038] According to the method of the present embodiment, by extracting decoupled features of multiple scales of the first data and the second data, more general and more powerful multi-scale graph representations of each data can be obtained, and then by graph matching of the multi-scale graph representations and determining a task processing result based on the graph matching result, the rich information contained in the data can be efficiently and fully utilized for task processing to obtain an accurate task processing result. In addition, the graph matching of the multi-scale graph representations can enhance robustness to better cope with image view transformation, text expression mode change, different speakers of speech, etc.

[0039] The first data and the second data can be one of image data (including pictures, videos), audio data, text data, molecular structure data and sequence data. The sequence data may, for example, be protein sequence data, gene sequence data, or other sequence form data. The first data and the second data can be the same type of data or different types of data, which is not limited herein.

[0040] The first data and the second data can be original data or data obtained after specific processing. In some embodiments, the image data can be an original image or a preprocessed image obtained after preprocessing of the original image; the audio data can be original sampling data of the audio or preprocessed data obtained after preprocessing of the original sampling data (for example, a frequency spectrum obtained after preprocessing of the original sampling data); and the text data can be a plurality of original strings or preprocessed data obtained after preprocessing of the text data, which is not limited herein.

[0041] After obtaining the first data and the second data, a multi-scale graph representation corresponding to each data can be obtained. In the multi-scale graph representation, the graph representation of each scale can include at least one node. The node can have attributes, and the attributes of the node can include vector type attributes and scalar type attributes. The scalar type attributes can further include category attributes (for example, discrete numerical values) and numerical attributes (for example, continuous numerical values). In an exemplary embodiment, the node in the graph representation of a certain scale can be, for example, a plurality of objects obtained by performing object detection on original data (or dense data as described later, for example, a feature map obtained by performing feature extraction on the original data, or preprocessed data obtained by preprocessing the original data). The vector type attributes of the node can include, for example, feature vectors corresponding to the objects, and the numerical attributes of the node can include, for example, coordinates, sizes of the objects, direction fields, gradient fields, texture densities of the neighborhood where the objects are located, and saliencies of the nodes. The category attributes of the node can include, for example, classification categories of the objects. It can be understood that different nodes can include different attributes.

[0042] According to some embodiments, in the multi-scale graph representation, the graph representation of at least one scale can further include at least one adjacent edge. The adjacent edge can be used to represent the relative relationship between two nodes of the same scale. The adjacent edge can have attributes, and the attributes of the adjacent edge can include vector type attributes and scalar type attributes. In an exemplary embodiment, the vector type attributes of the adjacent edge can include, for example, feature vectors of the two nodes corresponding to the adjacent edge and / or further processing results of the two feature vectors, the numerical attributes of the adjacent edge can include, for example, position information and / or geometric information such as coordinates, lengths, angles, and the like of the adjacent edge, and / or saliencies of the adjacent edge, and the category attributes of the adjacent edge can include, for example, categories of the adjacent edge, such as different types of chemical bonds, different types of forces, and the like.

[0043] According to some embodiments, the multi-scale graph representation may further include at least one dependent edge. A dependent edge can be used to characterize the dependency relationship between two nodes at different scales. A dependent edge may have attributes, which may include vector-type attributes or scalar-type attributes. In an exemplary embodiment, a dependent edge may, for example, characterize the relationship between target detection objects at two scales; for example, a dependent edge may exist between a vehicle detected at a first scale and the wheels of that vehicle detected at a second scale. The vector-type attributes of the dependent edge may, for example, include the feature vectors of its two corresponding nodes and / or the results of further processing of these two feature vectors. The numerical attributes of the dependent edge may, for example, include positional and geometric information such as the coordinates, length, and angle of the dependent edge, as well as the correlation between the nodes connected to the dependent edge. The category attributes of the dependent edge may include the category attributes of the nodes connected to the dependent edge.

[0044] It should be noted that the term "graph" in the embodiments of this invention is a broad term, which can include single-node graphs or multi-node graphs. Graph matching can be matching the nodes contained in the graph, or it can be matching the nodes and edges contained in the graph. When a graph at a certain scale is a single-node graph, graph matching refers to the matching between the vectors corresponding to the nodes. When a graph at a certain scale is a multi-node graph, graph matching can include traditional graph matching, graph matching using the attributes of nodes and edges (including attributes of vector type and scalar type), graph matching including node / edge pairing checks (e.g., node / edge pairing checks through geometric relationship solving by projective transformation), and combinations of the above. Among them, graph matching using the attributes of nodes and edges will be described in detail later.

[0045] Figure 2 A schematic diagram of a multi-scale graph representation according to an embodiment of the present disclosure is shown. Figure 2 As shown, graph representations 202, 204, and 206 at three scales from high to low constitute a multi-scale graph representation. Each graph representation includes multiple nodes, and graph representation 206 includes multiple adjacent edges. Graph representations 202, 204, and 206 can be obtained by sparsifying dense data 208. Specifically, dense data 208 includes three dense data points corresponding to the three scales, and by sparsifying these three dense data points respectively, graph representations 202, 204, and 206 at the three scales can be obtained.

[0046] Thus, by obtaining graph representations of different features such as scalars, vectors, graphs, etc. at different scales, a more general and powerful intermediate representation of various types of data can be obtained, and the accuracy of the results of downstream matching tasks, retrieval tasks, classification tasks, identification tasks, generation tasks, and other various data analysis and processing related tasks can be improved. In addition, by using dependent edges, the relevance between graph representations at different scales can be strengthened, thereby further enriching the information included in the multi-scale graph representation.

[0047] It can be understood that the disclosure does not limit the number of scales included in the multi-scale graph representation. In some embodiments, the multi-scale graph representation can include graph representations of two scales, three scales, or more scales, which are not limited herein. For ease of expression, the disclosure uses the first scale and the second scale lower than the first scale as examples to describe the form, generation method, matching method, etc. of the multi-scale graph representation, but is not intended to limit the scope of the disclosure.

[0048] It should be noted that the high and low of the scale can be understood as the focus of the corresponding graph representation on the whole or part of the data, for example, the high and low of the scale can be measured by the size of the corresponding part of each node in the original data in the graph representation, the number of nodes in the graph representation at this scale, etc. For example, the nodes of the high-scale graph representation can correspond to the whole image, the text paragraph, and the nodes of the low-scale graph representation can correspond to the local image, the word or word in the text, etc.

[0049] How to obtain a multi-scale graph representation will be described below in conjunction with an embodiment.

[0050] According to some embodiments, the nodes of at least one scale can be obtained by sparsifying the dense data corresponding to the data.

[0051] The dense data, or dense graph, may, for example, include an original image including dense pixels, a feature map including dense feature vectors obtained by convolving the original image, audio data including dense sampling points (and a spectrum map including dense pixels obtained by spectrally processing the audio data), a text paragraph including dense words or words, and dense molecular structure data and sequence data, etc. By sparsifying the dense data, a plurality of nodes, i.e., a sparse graph, can be obtained. Each node can correspond to a part of the region in the dense data and have an attribute. It can be understood that the dense data can also include a plurality of nodes, such as pixels in an image, sampling points in audio data, words or words in text data, etc. Each node can include a label type attribute (such as position, category in the dense data) and a vector type attribute (such as a feature vector).

[0052] According to some embodiments, the at least one first-scale node and the at least one second-scale node can be obtained by respectively sparsifying the same dense data. That is, the same dense data can be sparsified to different degrees to obtain nodes of different scales.

[0053] According to some embodiments, the dense data can comprise multiple scales. The dense data comprising multiple scales can be feature maps of multiple scales in a feature pyramid. The at least one first-scale node and the at least one second-scale node can be obtained by respectively sparsifying each of two scales of the dense data. That is, multiple dense data of different scales can be obtained first, and then each of the dense data of different scales can be sparsified to obtain nodes of corresponding scales. In an exemplary embodiment, the original image can be convolved with different downsampling multiples to obtain feature maps of different sizes, i.e., dense data of different scales. Then, the dense data can be respectively sparsified to obtain nodes of different scales. Both of the above-mentioned ways can generate nodes of multiple scales in parallel after obtaining the dense data.

[0054] In some embodiments, the nodes can be determined in the dense data by using the saliency of the dense nodes. The attribute of the dense nodes in the scalar type can include the saliency. The saliency represents the importance of each dense node in the dense data, which can be represented by a probability distribution over all dense nodes. In some embodiments, the saliency of the dense nodes can be determined according to the feature vectors of the dense nodes. In an exemplary embodiment, a saliency network can be used to process the feature vectors of all dense nodes to determine the saliency of each dense node.

[0055] Sparsifying the dense data can for example comprise determining, as the sparsified nodes, at least a part of the multiple dense nodes whose saliency satisfies a third preset condition. It can be understood that those skilled in the art can set the third preset condition according to needs, which is not limited herein. In an exemplary embodiment, the third preset condition can be top-k, i.e., selecting the top-k dense nodes with the highest saliency as the sparsified nodes, and / or the third preset condition can be nodes with saliency greater than a saliency threshold.

[0056] In some embodiments, in addition to the saliency, the attention scores generated by the attention mechanism or other ways of measuring the importance of the dense nodes can also be used as the basis for screening the nodes in the sparsification process, which are all within the protection scope of the present disclosure.

[0057] In some embodiments, the nodes in the dense data can be determined using a detection-based approach. The detection-based approach can include keypoint detection, object detection, or other types of detection, without limitation.

[0058] It can be understood that the node sparsification can be performed by a node sparsification network, which can include a detection network, a saliency network, or the like. When the node sparsification network is a detection network, the dense data is input into the detection network to obtain the sparsified nodes corresponding to the dense data and the confidence scores of the nodes. When the node sparsification network is a saliency network, the dense data and the feature vectors corresponding to the dense data are input into the saliency network to obtain the saliency scores of the dense nodes corresponding to the dense data, and the first k dense nodes with saliency scores greater than a saliency threshold and / or the highest saliency scores are taken as the sparsified nodes. In some embodiments, a non-maximal suppression condition or the like can be considered for comprehensive screening.

[0059] According to some embodiments, the nodes of at least one scale can be obtained by sparsifying a portion of the dense data corresponding to the positions of the nodes of another scale, which can be obtained by sparsifying the dense data. In an exemplary embodiment, the nodes of a first scale can be determined by an object detection approach, and each node of the first scale can correspond to a region in the dense data (i.e., a detection box output by the object detection). Then, the nodes of a second scale can be obtained by sparsifying the portion of the dense data corresponding to each node of the first scale, so that each node of the first scale corresponds to a node of the second scale. In this way, more valuable nodes of the second scale can be obtained, thereby improving the processing efficiency and accuracy of subsequent matching tasks and downstream tasks.

[0060] According to some embodiments, the nodes of at least one scale can be obtained by merging the dense data. In some exemplary embodiments, the dense data can be sparsified using a clustering or graph neural network method to obtain nodes of a low scale. Then, the nodes of the low scale can be further processed to obtain nodes of a high scale.

[0061] According to some embodiments, the nodes of at least one scale can be obtained by merging the nodes of a low scale, which can be obtained by sparsifying the dense data. In some exemplary embodiments, the multiple nodes of the low scale obtained by sparsifying the dense data can be clustered using a clustering or graph neural network method, or a subgraph containing the multiple nodes of the low scale can be input into a graph neural network to obtain nodes of a higher scale and / or attributes of the nodes.

[0062] The merging of the nodes can be based on the attributes of the scalar type of the nodes (e.g., location information), the attributes of the vector type of the nodes (e.g., feature vectors), or the attributes of the scalar, vector type of the adjacent edges in the graph representation (e.g., co-occurrence probability, correlation, etc. of the two connected nodes), without limitation.

[0063] The attributes of the nodes can be determined before, simultaneously with, or after the determination of the locations of the nodes at each scale, and the correspondence with other nodes at different scales or dense nodes.

[0064] According to some embodiments, the attributes of the nodes obtained by the sparsification can be determined according to the attributes of at least a portion of the dense nodes corresponding to the nodes. In some embodiments, the attributes of the nodes can be determined according to the attributes of the neighbor nodes within a certain range of the dense nodes corresponding to the locations of the nodes in the dense data. For example, the portion of the neighbor nodes can be input into a feature extraction network to obtain the feature vectors of the nodes, or the average of the feature vectors of the portion of the neighbor nodes can be determined as the feature vector of the node, or the weighted average of the feature vectors of the portion of the neighbor nodes based on the significance can be determined as the feature vector of the node. In an exemplary embodiment, the nodes are determined by target detection, and the feature vectors (attributes of the vector type) corresponding to the nodes can be extracted by inputting the dense nodes within the detection boxes corresponding to the nodes in the dense data into a feature extraction network. In another exemplary embodiment, the nodes are determined by merging, and the attributes of the nodes can be determined by clustering or graph neural networks, etc. according to the attributes of all the low-scale nodes used to obtain the nodes.

[0065] According to some embodiments, the nodes at at least one scale can be obtained by merging the nodes at another scale obtained by the sparsification, and the attributes of the nodes obtained by the merging can be determined according to the attributes of the nodes having a subordinate relationship with the nodes at the other scale.

[0066] In some embodiments, the attributes of at least a portion of the dense nodes corresponding to the nodes or the nodes having a subordinate relationship with the nodes can be further processed to obtain the attributes of the nodes. In an exemplary embodiment, the attributes of the nodes corresponding to the nodes can be processed by using a graph neural network to obtain the attributes of the nodes. In addition to the above methods, other ways can be used to determine various attributes of the nodes, without limitation.

[0067] When the relative relationship between the nodes is helpful for characterizing the data, the multi-scale graph can include adjacent edges. For example, the distance between two targets in an image, the interaction between two targets in an image, the correlation between the front and back words in a speech, and the interaction between different groups in a sequence.

[0068] According to some embodiments, the at least one adjacent edge can be determined according to the respective properties of the at least one node in the same scale. By analyzing the properties of the nodes, pairs of nodes with a correlation relationship can be determined in the single-scale graph representation to generate the corresponding adjacent edges.

[0069] In some embodiments, the adjacent edges can be generated based on rules. In some embodiments, the adjacent edges can be generated between pairs of nodes with a distance less than a preset threshold and / or the closest k pairs of nodes. In some embodiments, the adjacent edges can be generated only in a specific direction. It can be understood that those skilled in the art can set the corresponding adjacent edge generation rules according to prior knowledge and generate the adjacent edges according to the set rules, which are not limited herein.

[0070] In some embodiments, candidate adjacent edges can be generated first, and then the adjacent edges are selected from the candidate adjacent edges. According to some embodiments, the at least one adjacent edge is determined by performing the following steps: determining at least one candidate adjacent edge based on the at least one node in the same scale; determining the significance of each of the at least one candidate adjacent edge based on the respective properties of the at least one node in the same scale; and determining the adjacent edge with a significance satisfying a fourth preset condition from the at least one candidate adjacent edge as the at least one adjacent edge. By using the significance to generate the adjacent edges, the generation process of the adjacent edges can be optimized by training to improve the effectiveness of the generated adjacent edges. It can be understood that those skilled in the art can set the corresponding fourth preset condition according to the requirements. In an exemplary embodiment, the fourth preset condition can be that the significance is greater than a significance threshold and / or the top k with the highest significance.

[0071] According to some embodiments, the property of each of the at least one adjacent edge can be determined according to at least one of the respective properties of the two nodes connected by the adjacent edge and the relative relationship between the two nodes. In an exemplary embodiment, the position, length, angle, interaction size, etc. of the adjacent edge connecting the two nodes can be determined as the property of the adjacent edge according to the respective positions / properties of the two nodes. In some embodiments, the relative relationship between the two nodes can be determined based on rules using prior knowledge, and the property of the adjacent edge can be determined according to the relative relationship.

[0072] According to some embodiments, at least one of the dependency edges can be determined directly according to the dependency relationship between the nodes of two scales. In one example embodiment, the first nodes of the first scale are obtained by target detection on the dense data, the second nodes of the second scale are obtained by further target detection on the regions corresponding to the first nodes in the dense data, and the first nodes and the second nodes have a dependency relationship, and the dependency edges can be generated between the first nodes and the second nodes. In another example embodiment, the nodes of the second scale are obtained by clustering on the dense data, and the nodes of the first scale are obtained by merging the nodes of the second scale, and the nodes of the second scale used for merging to obtain the nodes of the first scale and the nodes of the first scale have a dependency relationship, and the dependency edges can be generated between the nodes of the second scale and the nodes of the first scale.

[0073] According to some embodiments, the attribute of the dependency edge can be determined according to the attributes of the two nodes connected by the dependency edge. As described above, the attribute of the dependency edge can be determined in various ways according to the vector-type attributes and / or scalar-type attributes of the two nodes connected by the dependency edge, which are not limited herein.

[0074] According to some embodiments, the graph representation of each of the first data and the second data in the first scale can be generated by using the first network, and / or the graph representation of each of the first data and the second data in the second scale can be generated by using the second network. In some embodiments, the generation of the nodes, the adjacent edges, and the dependency edges, and the determination of the attributes of the nodes, the adjacent edges, and the dependency edges can be performed by using the first network or the second network in whole or in part, or by using a rule-based method in whole or in part, or by using the first network or the second network in part and by using a rule-based method in part, which are not limited herein. When the generation of the nodes, the adjacent edges, and the dependency edges, and / or the determination of the attributes of the nodes, the adjacent edges, and the dependency edges are performed by using the network, a differentiable part can be added to the matching result, so that the generation process and / or the attribute determination can be optimized by training to further improve the expression ability of the graph representation.

[0075] After obtaining the multi-scale graph representations of the first data and the second data, the graph matching can be performed on the graph representations of different scales of the first data and the second data respectively to obtain the matching results corresponding to each scale, and then the multi-scale matching result can be determined according to the matching results.

[0076] According to some embodiments, as shown in Figure 3 The graph matching process of each of the first scale and the second scale can include:

[0077] Step 301, determining a candidate matching point pair according to at least one node included in the graph representation of the scale of the first data and at least one node included in the graph representation of the scale of the second data, wherein the candidate matching point pair comprises a first candidate matching node belonging to the graph representation of the scale of the first data and a second candidate matching node belonging to the graph representation of the scale of the second data;

[0078] Step 302, determining a matching result of the candidate matching point pair based on a feature vector of the first candidate matching node included in the candidate matching point pair and a feature vector of the second candidate matching node included in the candidate matching point pair.

[0079] Step 303, determining a matching result of the graph representation of the scale of the first data and the graph representation of the scale of the second data based on the matching result of the candidate matching point pair.

[0080] It can be understood that the matching result of the graph representation of the scale of the first data and the graph representation of the scale of the second data can be determined based on the matching results of multiple candidate matching point pairs, and when the graph representation comprises adjacent edges, the matching result of the graph representation of the scale of the first data and the graph representation of the scale of the second data can also be determined based on the matching results of multiple candidate matching edge pairs.

[0081] Thus, by matching in two dimensions of the graph structure constituted by the nodes in the graph representation and the attributes (e.g. feature vectors) of the nodes themselves, it is possible to make full use of the information contained in the data for matching, thereby improving the accuracy of the matching result and the result of subsequent tasks.

[0082] In step 301, the similarity information of the structure presented by the nodes (and optionally the adjacent edges between the nodes) in the graph representations of different data can be utilized to determine the matching relationship between the nodes in the graph representations of different data, so as to obtain the candidate matching point pair. The existing matching algorithm can be combined to perform node matching between the graph representations of different data, so as to obtain the candidate matching point pair.

[0083] In some embodiments, the candidate matching point pair can be quickly obtained using a point-by-point matching manner.

[0084] In an exemplary embodiment, when the matching result of the candidate matching point pair is determined to be matching in step 302, a new candidate matching point pair can be determined (e.g. according to the matching point pair A and B that have been confirmed to match, the nearest neighbor C of node A and the nearest neighbor D of node B are determined as a new candidate matching point pair), and step 302 is performed on the new candidate matching point pair until the new candidate matching point pair is not matching or a new candidate matching point pair cannot be determined. Further, in step 303, the matching result of the graph representation of the scale of the first data and the graph representation of the scale of the second data is determined according to the matching results of all the candidate matching point pairs in the graph representation of the scale.

[0085] In some example embodiments, steps 302 and 303 can be performed each time a new candidate matching point pair is obtained, and then it is determined whether to continue searching for more candidate matching point pairs based on the matching result of the current obtained graph representation. If the matching result of the graph representation can determine two data matches at this time (e.g., the matching score is greater than a preset threshold), the search can be stopped and the result is returned; otherwise, the search can be continued until no more candidate matching point pairs can be found.

[0086] In some embodiments, by combining the tree growing algorithm and the beam search, a branch can be grown on the tree of matched nodes at each step of recursion, and the scores of the new leaves (i.e., all possible grown branches) are calculated, and the best k leaves are selected as the branches for the next step to achieve point-wise matching. It can be understood that other methods can also be used to achieve point-wise matching, which are not limited herein.

[0087] In some embodiments, the candidate matching point pairs can be obtained using a global matching method (e.g., the Hungarian algorithm).

[0088] In some embodiments, the candidate matching point pairs can be obtained using a dynamic programming method. The dynamic programming method can obtain a globally optimal matching result. In an example embodiment, the matching result can include multiple candidate matching point pairs, and step 302 can be performed on each of the candidate matching point pairs to obtain the corresponding matching result, and step 303 can be used to determine the matching result of the graph representation based on the matching results of all candidate matching point pairs.

[0089] In step 302, the matching result of the first candidate matching node and the second candidate matching node can be determined based on the attributes of the two nodes using various methods.

[0090] In some embodiments, the matching result of the candidate matching point pair can be, for example, the similarity between the feature vector of the first candidate matching node and the feature vector of the second candidate matching node. In some embodiments, the matching result of the candidate matching point pair can also be the product of the saliency of the first candidate matching node, the saliency of the second candidate matching node, and the similarity between the feature vector of the first candidate matching node and the feature vector of the second candidate matching node. Such a numerical matching result can also be referred to as the matching score of the node.

[0091] In some embodiments, the scalar type attribute of the node can be used to determine the first point pair matching result, and then the first point pair matching result can be used to determine whether the vector type attribute of the node needs to be used to determine the second point pair matching result. For example, if the first point pair matching result is greater than a preset threshold, the vector type attribute of the node can be used to determine the second point pair matching result; otherwise, the vector type attribute of the node can not be used to determine the second point pair matching result. Figure 4As shown, the determining the matching result of the candidate matching point pair in step 302 can include: determining a first point pair matching result of the candidate matching point pair based on the attribute of the scalar type of the first candidate matching node included in the candidate matching point pair and the attribute of the scalar type of the second candidate matching node included in the candidate matching point pair in step 401; determining a second point pair matching result of the candidate matching point pair based on the feature vector of the first candidate matching node included in the candidate matching point pair and the feature vector of the second candidate matching node included in the candidate matching point pair in response to determining that the first point pair matching result of the candidate matching point pair satisfies a first preset condition in step 402; and determining the matching result of the candidate matching point pair based on the second point pair matching result in step 403. In this way, on the one hand, the prior knowledge can be utilized to determine the matching result based on the attribute of the scalar type, and on the other hand, the calculation amount can be reduced and the calculation speed of the matching result can be improved.

[0092] In step 401, for example, the consistency or correlation of the category attribute included in the attribute of the scalar type can be determined as the first point pair matching result, the difference, ratio or other calculation result of the numerical attribute included in the attribute of the scalar type can also be determined as the first point pair matching result, and the first point pair matching result can also be determined in other ways, which are not limited herein.

[0093] In step 402, the first preset condition can correspond to the first point pair matching result described above, for example, can be the consistency of the category attribute, or can be that the difference of the numerical attribute is less than a threshold. It can be understood that the first preset condition can be set by the person skilled in the art according to the needs, which is not limited herein. The determination manner of the second point pair matching result is similar to the manner of determining the matching result of the two nodes by using the feature vectors of the two nodes respectively, which is not described herein.

[0094] In step 403, the second point pair matching result can be directly determined as the matching result of the candidate matching point pair, or the matching result of the candidate matching point pair can be determined based on the first point pair matching result and the second point pair matching result. In an exemplary embodiment, the first point pair matching result is the ratio of the numerical attributes of the two nodes in the candidate matching point pair, and the second point pair matching result is the similarity of the feature vectors of the two nodes, and then the comprehensive calculation result of the ratio and the similarity can be determined as the matching result of the candidate matching point pair.

[0095] In some embodiments, the candidate matching point pair can also be screened by using the attribute of the scalar type of the node after obtaining the candidate matching point pair, so that part of the unmatched point pairs can be filtered to obtain more accurate graph representation matching results, and the calculation amount of the graph representation matching result calculation process can be reduced.

[0096] Back to Figure 3In some embodiments, the matching result of the two candidate matching nodes can also be determined by using the properties of the neighbor nodes and the neighbor adjacency edges of the first candidate matching node and the properties of the neighbor nodes and the neighbor adjacency edges of the second candidate matching node. It can be understood that when the neighbor nodes of the two nodes are similar and the edges connecting the two nodes are similar, the two nodes are more likely to match.

[0097] In step 303, the matching result of the graph representations of the first data and the second data in the scale can be, for example, the sum of the matching scores of all candidate matching node pairs. It can be understood that other ways can also be used to determine the matching result of the graph representations. In an embodiment, the comparison result of the sum of the matching scores and a preset threshold can be determined as the final matching result. In an embodiment, each candidate matching node pair can have a weight, and the final matching result can be, for example, the weighted sum of the matching scores of all candidate matching node pairs. In an embodiment, the matching result of a candidate matching node pair indicates whether the properties of the candidate matching node pair are consistent, and the matching result of the graph representations can be determined according to these binary judgment results.

[0098] When performing graph matching, the adjacency edges included in the graph representations can also be matched, and the matching result of the graph representations can be determined according to the matching result of the adjacency edges. In some embodiments, if only nodes are included in the graph representations, the nodes can be matched; if the graph representations include node adjacency edges, both can be used for matching.

[0099] According to some embodiments, as shown in FIG. 3, the graph matching process in each of the first scale and the second scale can further include: Figure 3

[0100] Step 304, determining a candidate matching edge pair according to at least one adjacency edge included in the graph representation of the first data in the scale and at least one adjacency edge included in the graph representation of the second data in the scale, wherein the candidate matching edge pair includes a first candidate matching adjacency edge belonging to the graph representation of the first data in the scale and a second candidate matching adjacency edge belonging to the graph representation of the second data in the scale;

[0101] Step 305, for the candidate matching edge pair, determining a matching result of the candidate matching edge pair based on the properties of the first candidate matching adjacency edge included in the candidate matching edge pair and the properties of the second candidate matching adjacency edge included in the candidate matching edge pair; and

[0102] Step 306, determining a matching result of the graph representation of the first data in the scale and the graph representation of the second data in the scale based on the matching results of the candidate matching edge pairs.

[0103] ​Therefore, by matching the graph structure formed by nodes and adjacent edges in the graph representation and the attributes included in the adjacent edges, the information contained in the data can be fully utilized for matching, thereby improving the accuracy of the matching results and the results of subsequent tasks.

[0104] In some embodiments, step 304 can be performed simultaneously with step 301. That is, the method described above can be used to obtain candidate matching point pairs and candidate matching edge pairs simultaneously. In some embodiments, candidate matching point pairs can be determined first, and then candidate matching edge pairs can be determined based on the adjacent edges between the points included in these candidate matching point pairs.

[0105] It is understandable that the method for determining the matching results of candidate edge pairs is similar to the method for determining the matching results of candidate vertex pairs. The method for determining the matching results of the graph representation based on the matching results of candidate edge pairs is similar to the method for determining the graph representation based on the matching results of candidate vertex pairs, and will not be elaborated here.

[0106] In step 305, the matching result of the two adjacent edges can be determined in a variety of ways based on the attributes of the first candidate matching adjacent edge and the attributes of the second candidate matching adjacent edge.

[0107] In some embodiments, the matching result of a candidate matching edge pair may be, for example, the similarity between the feature vectors of the first candidate matching adjacent edge and the feature vectors of the second candidate matching adjacent edge. In some embodiments, the matching result of a candidate matching edge pair may be the saliency of the first candidate matching adjacent edge, the saliency of the second candidate matching adjacent edge, and the product of the similarity between the feature vectors of the first candidate matching adjacent edge and the feature vectors of the second candidate matching adjacent edge.

[0108] In some embodiments, the matching result of the first edge pair can be determined first using the scalar type attribute of the adjacent edges, and then it can be determined whether the matching result of the second edge pair needs to be further determined using the vector type attribute of the adjacent edges. For example... Figure 5 As shown, step 305, determining the matching result of the candidate matching edge pair may include: step 501, determining the first edge pair matching result of the candidate matching edge pair based on the scalar type attribute of the first candidate matching adjacent edge included in the candidate matching edge pair and the scalar type attribute of the second candidate matching adjacent edge included in the candidate matching edge pair; step 502, in response to determining that the first edge pair matching result of the candidate matching edge pair satisfies the second preset condition, determining the second edge pair matching result of the candidate matching edge pair based on the feature vector of the first candidate matching adjacent edge included in the candidate matching edge pair and the feature vector of the second candidate matching adjacent edge included in the candidate matching edge pair; and step 503, determining the matching result of the candidate matching edge pair based on the second edge pair matching result.

[0109] It can be understood that the operations of steps 501-503 on the candidate matching edge pairs are similar to the operations of steps 401-403 on the candidate matching point pairs, and will not be described herein. A person skilled in the art can set the second preset condition according to the needs, which will not be limited herein.

[0110] Returning to Figure 3 In some embodiments, the matching results of the two candidate matching adjacent edges can also be determined by using the neighbor nodes of the first candidate matching adjacent edge and the neighbor nodes of the second candidate matching adjacent edge.

[0111] In step 303, the matching result of the graph representations of the first data and the second data at the scale can be the sum of the matching scores of all candidate matching point pairs and / or the matching scores of all candidate matching edge pairs, and can also be obtained by using other manners based on the matching results of the candidate matching point pairs and / or the matching results of the candidate matching edge pairs, which will not be limited herein.

[0112] In some embodiments, in addition to the matching score, the matching result can also be determined according to the node / edge pairing check result. For example, the node / edge pairing check includes the node / edge pairing check by solving the geometric relationship through the projection transformation, etc. It can be understood that the matching of the nodes in different graph representations can also correspond to the transformation relationship in the geometric space between the data, in which the explicit transformation can include the projection transformation in the scene matching, the isometric transformation in the fingerprint matching, and the implicit transformation can include the changes of the speaker and the environment in the speech-related task. The node / edge pairing check result can affect the matching result in two ways: first, in the process of node matching to obtain the candidate matching point pairs, a constraint condition can be added, and the point pairs / edge pairs that meet the constraint condition are taken as the candidate matching point pairs / edge pairs, so as to bring the prior knowledge into the matching process and accelerate the matching process. The second: after obtaining the initial graph matching result according to the matching results of the candidate matching point pairs / edge pairs, the node / edge pairing check result can be determined, and the final graph matching result can be determined according to the initial graph matching result and the check result. For example, the initial graph matching result shows that the matching degree is 80%, and the check result shows that the matching is not matched, and the final graph matching result can be obtained by weighting, for example, 70%.

[0113] In the process of graph matching of multi-scale graph representations, the graph matching of each scale can be independently performed, or the graph matching at a scale can be performed first, and then it is determined whether to perform the graph matching at other scales according to the matching result at the scale, or the graph matching strategy or the graph matching parameter at other scales is adjusted.

[0114] According to some embodiments, the step 105 of performing graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result can comprise: in response to determining that the first matching result is a successful matching, performing graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result. In this way, by first performing the first scale graph matching embodying the overall information (less information amount), and then determining whether to perform the second scale graph matching embodying the local information (greater information amount) according to the first scale graph matching result, the number of times of the second scale graph matching can be reduced, so as to reduce the overall time consumption of the matching process without affecting the matching result and the subsequent task processing result, and improve the task processing efficiency.

[0115] According to some embodiments, the step 105 of performing graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result can comprise: in response to determining that the first matching result is a successful matching, performing graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result. In this way, by first performing the first scale graph matching embodying the overall information (less information amount), and then determining whether to perform the second scale graph matching embodying the local information (greater information amount) according to the first scale graph matching result, the number of times of the second scale graph matching can be reduced, so as to reduce the overall time consumption of the matching process without affecting the matching result and the subsequent task processing result, and improve the task processing efficiency.

[0116] In this way, by first performing the first scale graph matching, and then performing matching on the subgraph of the node indicated by the first scale graph matching result as a successful matching, the part of the graph representation that is most likely not matched does not need to be matched, so as to reduce the number of nodes and / or adjacent edges that need to be calculated for the matching result, and further reduce the overall time consumption of the matching process without affecting the matching result and the subsequent task processing result, and improve the task processing efficiency.

[0117] According to some embodiments, the step 105 of performing graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result can comprise: determining the matching result of the current node based on the attribute of the current node, and whether the first scale node having a subordinate relationship with the current node is successfully matched, wherein the current node is a second scale node. By considering the vertical relationship in the low scale graph matching process, and taking the matching result of the high scale node having a subordinate relationship with the node as a reference element, the accuracy of the low scale graph matching result can be improved.

[0118] In step 106, the manner and logic of determining the multi-scale matching result based on the first matching result and the second matching result can be set according to requirements. In some embodiments, when both the first matching result and the second matching result are successful matching, the multi-scale matching result is determined as successful matching. In some embodiments, when the second matching result of the low scale is successful matching, the multi-scale matching result is determined as successful matching. In some embodiments, the first matching result and the second matching result can be, for example, the matching degrees of the graph representations under the first scale and the second scale, and the multi-scale matching result can be, for example, the average of the matching degrees of the graph representations under the two scales. It can be understood that the multi-scale matching result can also be determined in other manners, which are not limited herein.

[0119] According to some embodiments, the task is a matching task, and step 107 of determining the task processing result based on the multi-scale matching result can include: taking the multi-scale matching result as the result of the final task.

[0120] According to some embodiments, the second data can be obtained from a database. Step 107 of determining the task processing result based on the multi-scale matching result can include: determining at least one second data matched with the first data based on the multi-scale matching result of the first data and a plurality of second data in the database; and determining the task processing result based on the at least one second data. In this way, by the above manner, other types of tasks based on multi-scale graph representations can be converted into matching tasks based on multi-scale graph representations. In an exemplary embodiment, the final task can be a recognition task, a matching task, or a search task implemented by matching means, and the at least one second data matched can be directly taken as the search result. In an exemplary embodiment, the final task is a classification task, and the first data and the at least one second data matched can be all input into a model for the classification task, so that the model takes the at least one second data as a reference for classification to complete the classification of the first data. In an exemplary embodiment, the final task can be a generation task (e.g., a fill-in-the-blank of text or image), and the first data with a local blank and the at least one second data matched can be all input into a model for the generation task, so that the model takes the at least one second data as a reference for generation to complete the generation of the first data. In this way, the task can be completed by means of data similar to the first data, which provides more abundant information for the model by inputting only the first data into the model, and more accurate classification or generation task results can be obtained without increasing the complexity of the model.

[0121] Different types of data, multi-scale graph representations, graph matching, and task processing will be described below in conjunction with embodiments.

[0122] In one example embodiment, the first data and the second data can both be image data, the dense data can be a feature map obtained based on the corresponding image data, and the plurality of dense nodes in the dense data can be a plurality of pixels in the feature map. The plurality of second-scale nodes can be obtained by sparsifying (e.g., based on saliency) the first data. The attributes of these nodes can include the positions of the nodes in the first data, and the feature vectors corresponding to the nodes (e.g., the feature vector of a node can be determined according to the neighborhood of the node in the feature map, or the feature vector of a node can be determined according to the local image corresponding to the node in the first data, and the feature vector of a node can be used to describe the attributes of the neighborhood of the node, such as the direction field, if the first data is a palmprint data, the feature vector of a node can be used to describe the texture density of the neighborhood of the node, etc.). Similarly, the second-scale nodes of the second data can be obtained. The plurality of first-scale nodes can be obtained by merging the second-scale nodes of the first data. The attributes of these first nodes can also include the positions of the nodes in the first data, and the feature vectors corresponding to the nodes. Similarly, the first-scale nodes of the second data can be obtained. The second-scale can further include adjacency edges for establishing connections between the plurality of second-scale nodes having a dependency relationship with the same first-scale node, and the first-scale node and the second-scale node having the dependency relationship can further have a dependency edge. The attributes of the adjacency edge and the dependency edge can include, for example, the relative position, angle, length of the edge (e.g., used to describe the force of the connected nodes) between the edge and its corresponding nodes, etc.

[0123] The multi-scale graph structure of the image data can extract the geometric information of the target in the image (e.g., the positional relationship of a plurality of targets in the image, or the positional relationship between different parts of the same target), while retaining rich detailed information (e.g., the feature vector of a node). The graph representation of different scales takes into account the whole and the part, and is more robust and more interpretable for missing, deformation, perspective transformation, occlusion, attack samples, etc. By using such a multi-scale graph representation for graph matching, and using the graph matching method to solve downstream complex tasks (e.g., image matching, image search, image classification, image generation), more accurate and reliable results can be obtained. In addition, due to the multi-scale characteristic, when performing image data retrieval, comparison, etc., a preliminary screening can be performed according to the high-scale graph representation, and then a precise retrieval and comparison can be performed using the low-scale graph representation, while constraints (e.g., geometric constraints) are used based on prior knowledge to obtain accurate results.

[0124] In an example embodiment, the first data and the second data can both be text data, and the dense data can be a text passage, and the nodes in the dense data can be words in the text passage. It can be appreciated that the nodes in the dense data can also be text features corresponding to the words. By sparsifying the first data and the second data, nodes of the first scale and the second scale can be obtained, which can correspond to different scale text segments in the text passage, such as sentences, clauses, phrases, words, etc. The attributes of the nodes can include, for example, word embeddings of the corresponding text segments, and can also include their positions in the text passage. Adjacent edges between the nodes can be used to represent the relationship between different text segments, and the subordinate edges can be used to represent the subordinate relationship between text segments of different scales.

[0125] The multi-scale graph representation of the text data can extract the structural relationship and / or logical relationship between different scale text segments in the text passage, such as words, phrases, clauses, sentences, passages, etc., and can also retain the text feature vectors corresponding to the elements in the text, so that various natural language processing tasks can be better handled. The graph representation of different scales takes into account the whole and the part, and is more robust to incomplete sentences, missing sentences, sentence deformation, different languages, etc. The downstream task can be text translation, text continuation, automatic question answering, etc.

[0126] In an example embodiment, the first data and the second data can both be audio data, and the dense data can be a frequency spectrum graph of the audio data, and the nodes in the dense data can be pixels in the frequency spectrum graph. The nodes of the first scale can be, for example, a plurality of segment regions obtained by segmenting the frequency spectrum graph in the time direction, and the nodes of the second scale can be, for example, feature points extracted from the frequency spectrum graph. Adjacent edges can exist between the second nodes, which are used to connect adjacent feature points.

[0127] The multi-scale graph representation of the audio data can extract a plurality of segments in the time direction, a plurality of feature points in each segment, and the association relationship (e.g., time distance, frequency domain distance) between the feature points, and retain the feature vectors corresponding to the feature points, so that the problems caused by different voices, tones, speaking styles, and the randomness of the content can be solved when completing audio-related tasks, especially speech-related tasks. The graph representation of different scales takes into account the whole and the part, and is more robust to incomplete speech, noise, etc. The downstream task can be speech translation, etc.

[0128] In some example embodiments, the first data and the second data can also be various complex data such as molecules, genes, proteins, sequences, etc., and the nodes in the dense data can be the smallest units in the corresponding data types, for example, atoms, base pairs, amino acids, etc. The nodes of the graph representation can be consistent with the dense data, or can be units of a higher scale, for example, atomic groups, functional groups, fragments composed of multiple base pairs (for example, coding regions and non-coding regions, or enhancers, promoters, exons, introns, terminators, etc. at a lower scale), amino acid sequences in proteins, peptide chains, etc. The adjacency edges and the dependent edges between the nodes can be used to represent various relationships between units of the same scale (for example, chemical bonds, hydrogen bonds) and various relationships between units of different scales (for example, dependent relationships). In addition, the multi-scale graph representation can also represent the structure of the data at different scales, for example, the primary, secondary, tertiary, and quaternary structures of a protein. The downstream task can be property / structure prediction of molecular structure data, sequence data, etc.

[0129] The multi-scale graph representation of these complex data can represent the complex spatial structure and detailed information of the complex data, and can represent various relationships between units in the complex data. Therefore, using the multi-scale graph representation can make full use of the above information of the complex data to perform matching tasks or other downstream tasks.

[0130] In some embodiments, other types of data can be converted into image data first, and then a multi-scale graph representation is generated according to the converted image data. For example, other types of data such as audio data, text data, etc. can be converted into image data, and then a multi-scale graph representation is extracted according to the image data, and then various downstream tasks are completed according to the graph representation.

[0131] In some embodiments, graph matching can also be performed between graph representations of different types of data to complete specific cross-modal tasks.

[0132] Figure 6 A flowchart of a method 600 of training a neural network according to one embodiment of the present disclosure is shown, the method 600 comprising:

[0133] Step 601, obtaining first sample data and second sample data, the first sample data and the second sample data being one of image data, audio data, text data, molecular structure data, and sequence data;

[0134] Step 602, obtaining a multi-scale graph representation of each of the first sample data and the second sample data, wherein the multi-scale graph representation is determined by a graph representation extraction network, and the multi-scale graph representation includes a graph representation of a first scale and a graph representation of a second scale;

[0135] Step 603, performing graph matching on the graph representation of the first scale of the first sample data and the graph representation of the first scale of the second sample data to obtain a first current matching result representing a matching degree of the first scale;

[0136] Step 604, performing graph matching on the graph representation of the second scale of the first sample data and the graph representation of the second scale of the second sample data to obtain a second current matching result representing a matching degree of the second scale;

[0137] Step 605, obtaining a target matching result and / or a target task processing result of the first sample data and the second sample data;

[0138] Step 606, determining a loss value according to the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and

[0139] Step 607, training the graph representation extraction network according to the loss value.

[0140] According to the method of the embodiment, by training the graph representation extraction network by using the loss value determined according to the graph matching result and the target matching result and / or the target task processing result, accurate multi-scale graph representations suitable for downstream tasks can be obtained by using the graph representation extraction network in the inference stage, thereby helping the downstream tasks to obtain accurate task processing results.

[0141] It can be understood that the first sample data and the second sample data are similar to the first data and the second data described above, and the operations of obtaining the first sample data and the multi-scale graph representations thereof, obtaining the second sample data and the multi-scale graph representations thereof, and performing graph matching on the graph representations of different scales in steps 601-604 are similar to the operations of steps 101-105 in Figure 1 and will not be described here.

[0142] According to some embodiments, each of the graph representations of the multi-scale graph representations can include at least one node, the node can include attributes, and the attributes of the node can include scalar type attributes and vector type attributes. At least one of the graph representations of the multi-scale graph representations can include at least one adjacent edge, each of the at least one adjacent edge is used to represent a relative relationship between two nodes of the same scale, the adjacent edge has attributes, and the attributes of the adjacent edge include scalar type attributes and vector type attributes.

[0143] According to some embodiments, the attributes of the nodes of the scalar type can include the saliency, label, other attributes of the nodes, and the attributes of the nodes of the vector type include the feature vectors of the nodes; the attributes of the adjacent edges of the scalar type can include the saliency, label, other attributes of the adjacent edges, and the attributes of the adjacent edges of the vector type include the feature vectors of the adjacent edges.

[0144] The result of the graph matching can be the similarity of the two graph representations. The similarity of the two graph representations can be the sum of the similarity of each node / edge, the sum of the similarity of the saliency of each node / edge, and the similarity of the node / edge can be determined according to its attributes, for example. In this way, a supervised signal can be generated for each local feature (the attributes of the node / edge), and a certain local feature can be trained separately.

[0145] The target matching result can be a matching result of matching or not matching, or a result representing the degree of matching (e.g., 99% matching degree); and the target task can be a matching task, a retrieval task, a classification task, an identification task, a fill-in-the-blank task, and other various data analysis and processing related tasks. When the target task is a matching task, the result of the target task is the target matching result.

[0146] In one specific embodiment, the target matching result and / or the target task processing result can be determined according to the annotation of the sample data. For example, the annotation is that the two sample data are positive samples of each other, that is, the target matching result is annotated as "matching". For example, the target task is a classification task of classifying sample images, and the target task processing result can be annotated as class "1". In this way, the matching result and / or the final task result can be annotated, without annotating the graph representation extracted by the specific graph representation extraction network.

[0147] According to some embodiments, the target matching result and / or the target task processing result can be determined according to one of the following: based on artificial annotation, based on a teacher model and / or a pre-trained model, based on auxiliary constraint information, and based on a rule-based manner.

[0148] Specifically, the target matching result and / or the target task processing result can be artificially annotated. It can be understood that the artificial annotation can be in the data dimension rather than the scale dimension, for example, whether the first data and the second data match can be annotated, without annotating whether a certain scale in the first data and a certain scale in the second data match. In fact, if it is known whether the first and second data match, it is known whether each scale matches. In this way, the label of the scale dimension can be obtained according to the label of the data dimension, greatly increasing the number of supervised signals.

[0149] In another specific implementation, the target matching result and / or the target task processing result can be determined according to a teacher model and / or a pre-trained model. The teacher model and the pre-trained model can be models trained using a large amount of data and having certain reasoning capabilities, or knowledge distillation can be performed using such models to train the graph representation extraction network. For example, a first data and a multi-scale graph representation of a second scale are extracted using the teacher model / pre-trained model, the matching result and / or the task processing result are determined based on the multi-scale graph representation, and the target matching result and / or the target task processing result are determined according to the matching result and / or the task processing result determined based on the multi-scale graph representation (for example, a matching result or a task processing with high confidence is selected as the target matching result and / or the target task processing result).

[0150] In another specific implementation, the target matching result and / or the target task processing result can be determined based on rules. The rules in the rule-based manner can be determined according to prior knowledge. For example, a first data and a multi-scale graph representation of a second scale are extracted based on specific rules, the matching result and / or the task processing result are determined based on the multi-scale graph representation, and the matching result and / or the task processing result are determined as the target matching result and / or the target task processing result.

[0151] It can be understood that other manners can also be used to obtain the target matching result and / or the target task processing result, which are not limited herein.

[0152] According to some embodiments, the target matching result can be determined using the network trained for the Nth round, and the target matching result can be determined according to the network trained for the Nth round. As shown in FIG. 6, step 601 of obtaining first sample data and second sample data can include: Figure 7

[0153] Step 701, using the network trained for the Nth round to extract a multi-scale graph representation of each of the first unlabeled data and the second unlabeled data;

[0154] Step 702, performing graph matching on the graph representation of the first scale of the first unlabeled data and the graph representation of the first scale of the second unlabeled data to obtain a first unlabeled data matching result representing a matching degree of the first scale;

[0155] Step 703, performing graph matching on the graph representation of the second scale of the first unlabeled data and the graph representation of the second scale of the second unlabeled data to obtain a second unlabeled data matching result representing a matching degree of the second scale;

[0156] Step 704, determining an unlabeled data matching result according to the first unlabeled data matching result and / or the second unlabeled data matching result; ​

[0157] In step 705, in response to determining that the first unlabeled data and the second unlabeled data satisfy the first condition, the first unlabeled data and the second unlabeled data are determined as the first sample data and the second sample data that are positive samples of each other, wherein the first unlabeled data and the second unlabeled data satisfying the first condition includes the unlabeled data matching result satisfying the first matching condition, and the target matching result of the positive sample indicating that the corresponding first sample data and the second sample data match; and / or

[0158] In step 706, in response to determining that the first unlabeled data and the second unlabeled data satisfy the second condition, the first unlabeled data and the second unlabeled data are determined as the first sample data and the second sample data that are negative samples of each other, wherein the first unlabeled data and the second unlabeled data satisfying the second condition includes the unlabeled data matching result satisfying the second matching condition, and the target matching result of the negative sample indicating that the corresponding first sample data and the second sample data do not match.

[0159] The matching result of the unlabeled data can be a floating point number or an integer, for example, the first unlabeled data matching result is a similarity, which is a floating point number; and the matching result of the second unlabeled data is the number of matched nodes / edges, which is an integer.

[0160] The first condition, the second condition, the first matching condition and the second matching condition can be set by a user, for example, the first matching condition can be that the first unlabeled data matching result is greater than 80% and the second unlabeled data matching result is greater than 5 nodes / edges. It can be understood that the more stringent the first matching condition and the second matching condition are set, the more reliable the target matching result corresponding to the positive sample / negative sample generated from the unlabeled data is.

[0161] In addition to the requirement of the matching condition, when the first unlabeled data and the second unlabeled data are determined as sample data, the first condition and the second condition can also have auxiliary condition requirements. The auxiliary condition can be a time and place condition, an expert secondary confirmation condition, etc. For example, when the first unlabeled data and the second unlabeled data are image data, the spatiotemporal information when they are taken can be used as an auxiliary condition to determine whether they are positive samples / negative samples. For example, two images have a high similarity, a large number of matched nodes / edges, and close shooting time and place, so the probability that they contain the same object is greater, and the probability that they are positive samples of each other is greater.

[0162] Thus, by using the positive samples and / or negative samples generated by the Nth round of network in the above manner, the graph representation extraction network can use these samples for the (N+1)th round of training. Only a small amount of sample data needs to be labeled to obtain the model after the Nth round of training, and then more sample data can be used to further train the model after the Nth round of training, greatly reducing the requirement for the amount of labeled data in the model training process. Moreover, in the case of simultaneously generating positive samples and negative samples, the positive samples and negative samples can be used for contrastive learning to have the ability to extract accurate graph representations, while reducing the cost of obtaining samples.

[0163] In an example embodiment, in the multi-scale graph representations generated by the graph representation extraction network, the features on a certain node are less robust than other features (lack of robustness), and in the graph matching of the positive samples, the matching error will mainly come from the features of this node. Therefore, the supervision signal during training will focus on the features of this node to strengthen the robustness of the features.

[0164] It can be understood that, Figure 7 The operations of extracting the multi-scale graph representations of the first unlabeled data and the second unlabeled data in steps 701-703 in the method 700, and the operations of graph matching 1 of the different scale graph representations of the first unlabeled data and the second unlabeled data in steps 704-705 in the method 700 are similar to those in steps 101-105 in the method 1000, and will not be repeated here. Figure 1 The operations of steps 101-105 in the method 1000 are similar, and will not be repeated here.

[0165] In step 704, the unlabeled data matching result can be determined according to one or both of the first unlabeled data matching result and the second unlabeled data matching result. In a relatively strict data quality embodiment, in response to determining that the first unlabeled data matching result and the second unlabeled data matching result both indicate successful matching, the unlabeled data matching result is determined to be matching. In some embodiments, the first scale and the second scale can be set to satisfy a specific matching condition, and the unlabeled data matching result is determined to be matching. For example, if the first scale similarity is greater than 80% and 5 nodes are matched in the second scale, the unlabeled data matching result is determined to be matching. In some cases, cross-validation can be performed between different scales to generate more supervision signals. In some embodiments, since the matching result of the lower scale graph representation involves more detailed features, the credibility is higher than that of the macro features, and when the matching result of the lower scale graph representation indicates successful matching, the unlabeled data matching result can be determined to be matching. In some embodiments, when the multi-scale graph representation includes three or more scales, the unlabeled data matching result can be determined to be matching when the matching results of the highest and lowest scale graph representations indicate successful matching. In a more tolerant data embodiment, the unlabeled data matching result can be determined to be matching when the matching result of the higher scale graph representation indicates successful matching.

[0166] It can be understood that cross-validation between different scales can also be performed in other manners to generate a supervision signal, which is not limited herein.

[0167] It can be understood that "Nth training" means that the network has undergone at least one round of training and has certain inference ability, but is not intended to limit the specific number of training rounds of the network.

[0168] According to some embodiments, the loss value can include a matching loss value and / or a task loss value. As shown, step 606, determining the loss value according to the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result can include: step 801, determining a matching loss value according to the first current matching result and / or the second current matching result; and / or, step 802, determining a current task result according to the first current matching result and / or the second current matching result, and determining a task loss value according to the target task processing result and the current task result. Figure 8

[0169] In some embodiments, the target matching result of one or more scales can be directly obtained, or the target matching result between multi-scale graph representations, and then the corresponding matching loss value can be determined according to the target matching result and the first current matching result and / or the second current matching result, so as to generate a supervision signal to train the network.

[0170] In some embodiments, for example, in a fill-in-the-blank task, the corresponding target task processing result can be obtained, and then the current task processing result can be determined according to the first current matching result and / or the second current matching result, and further the corresponding task loss value can be determined according to the target task processing result and the current task processing result, so as to generate a corresponding supervision signal to train the network. For example, there are multiple second data, the current matching result of the first data and the second data is determined according to the first current matching result and / or the second current matching result, the first data and the second data that match the current matching result of the first data are input into the fill-in-the-blank network, and the task processing result is obtained. At this time, the target matching result can not be known, but the target task processing result is known, and then the supervision signal can be determined according to the target task processing result.

[0171] According to some embodiments, step 801, determining the matching loss value according to the first current matching result and / or the second current matching result can include: determining a current matching result according to the first current matching result and / or the second current matching result; and determining the matching loss value according to the current matching result and the target matching result.

[0172] ​In some embodiments, the target matching result can be directly obtained from the multi-scale graph representation, and the current matching result can be determined according to the first current matching result and / or the second current matching result, and then the corresponding matching loss value can be determined according to the current matching result and the target matching result, so as to generate a corresponding supervision signal to train the network.

[0173] According to some embodiments, the graph representation extraction network can include a first network for extracting a graph representation of a first scale. In some embodiments, the step 606 of determining a matching loss value according to the first current matching result and / or the second current matching result can include determining a first scale matching loss value according to the target matching result and the first current matching result. The step 607 of training the graph representation extraction network according to the loss value can include training the first network according to the first scale matching loss value.

[0174] According to some embodiments, the graph representation extraction network can include a second network for extracting a graph representation of a second scale. In some embodiments, the step 606 of determining a matching loss value according to the first current matching result and / or the second current matching result can include determining a second scale matching loss value according to the target matching result and the second current matching result. The step 607 of training the graph representation extraction network according to the loss value can include training the second network according to the second scale matching loss value. In this way, the loss values of the first scale and the second scale can be calculated respectively, and the corresponding network models can be trained respectively.

[0175] According to some embodiments, the graph representation extraction network can include at least one of: a network module for determining a scalar type attribute of a node; a network module for determining a vector type attribute of a node; a network module for determining a scalar type attribute of an adjacent edge; and a network module for determining a vector type attribute of an adjacent edge. It can be understood that the graph representation extraction network can also include a feature extraction network module for obtaining dense data from raw data. The loss value can act on the differentiable part corresponding to these network modules, so as to realize the training of these network modules.

[0176] According to some embodiments, the aforementioned sparse module for sparsifying the dense data to obtain the sparsified node, and the merging module for merging the low-scale node obtained by sparsification to obtain the high-scale node are also implemented by a neural network. Accordingly, the graph representation extraction network includes at least one of: a sparse module for sparsifying the dense data to obtain the sparsified node; and a merging module for merging the low-scale node obtained by sparsification to obtain the high-scale node.

[0177] According to some embodiments, the nodes are obtained according to a sparsification module, the nodes are connected to form adjacent edges, the adjacent edges with a significance greater than a threshold value are determined as reserved adjacent edges through a network module for determining the significance of the adjacent edges, and the attribute of the node / edge vector type is extracted according to a module for determining the node / edge vector type attribute.

[0178] According to some embodiments, the nodes and the edges each include a determination module and an attribute extraction module. The determination module is used to determine the nodes / edges, the node determination module can include a sparsification module (for example, a detection module or a significance module) or a merging module, and the edge determination module can include a significance module; the attribute extraction module can be a module for determining other attributes in addition to the significance. According to some embodiments, these modules are all network modules.

[0179] In one embodiment, the current matching result can be obtained based on the matching degree of the two graph representations. The matching degree of the graph representations can be represented as the sum of the matching degrees of all candidate matching node pairs and the matching degrees of all candidate matching adjacent edge pairs, where the matching degree of the candidate matching node pair is the product of the significance of the first candidate matching node, the significance of the second candidate matching node, and the similarity of the feature vector of the first candidate matching node and the feature vector of the second candidate matching node, and the matching degree of the candidate matching adjacent edge pair is the product of the significance of the first candidate matching adjacent edge, the significance of the second candidate matching adjacent edge, and the similarity of the feature vector of the first candidate matching adjacent edge and the feature vector of the second candidate matching adjacent edge. In this way, by the above-mentioned manner, the nodes / edges that do not match can be weakened, so that stable and reliable local features can be preserved at different scales.

[0180] According to some embodiments, the graph representation extraction network can include a rule module and a network module. The rule module can be, for example, a rule-based module using prior knowledge. Such a module can be used without training, but has poor accuracy compared to a trained network module, and has poor robustness, strong limitations, and is usually difficult to train or optimize. Although the trained network module can output accurate results, and has a wider range of adaptation and strong robustness, it is difficult to quickly converge when the training difficulty is high.

[0181] According to some embodiments, as shown in Figure 9 The training method 900 further includes at least one of the following steps: step 901, in response to determining that a fifth preset condition is met, replacing a first rule module in the rule module with a network module; and step 902, in response to determining that a sixth preset condition is met, adding a network module in the graph representation extraction network. Figure 9 The operations of steps 903-909 in Figure 6The operations of steps 601-607 in the method 600 are similar and are not limited herein. At step 909, training the graph representation extraction network according to the loss value can include training the network modules according to the loss value.

[0182] In some embodiments, in the initial stage of training, the rule modules can be used in part of the links in the graph representation extraction network, and the network modules can be used in another part of the links to train the network modules. After the part of the network modules converges, more network modules can be added or the rule modules can be replaced by the network modules and the training can be continued to improve the performance of the network. In this way, the prior knowledge can be fully utilized and the training speed and effect of the network can be prompted.

[0183] In some embodiments, the fifth preset condition and the sixth preset condition can be, for example, a specific number of training rounds, can be the current matching accuracy of the network, or can be other preset conditions such as convergence speed, trend, etc. It can be understood that the fifth preset condition and the sixth preset condition can be determined by the person skilled in the art according to the needs, and are not limited herein.

[0184] Figure 10A structural block diagram of a task processing apparatus 1000 is shown according to one embodiment of the present disclosure, the apparatus 1000 comprising: a first obtaining unit 1010 configured to obtain first data and second data, the first data and the second data being one of image data, audio data, text data, molecular structure data and sequence data respectively; a second obtaining unit 1020 configured to obtain a first scale graph representation of each of the first data and the second data, the first scale graph representation comprising at least one first scale node, wherein the first scale node has attributes, and the attributes of the first scale node comprise attributes of a vector type; a third obtaining unit 1030 configured to obtain a second scale graph representation of each of the first data and the second data, the second scale being lower than the first scale, the second scale graph representation comprising at least one second scale node, wherein the second scale node has attributes, and the attributes of the second scale node comprise attributes of a vector type, wherein the at least one scale node of each of the first data and the second data is obtained by sparsifying dense data corresponding to the data, and the at least one scale graph representation of each data comprises at least one adjacent edge, each of the at least one adjacent edge being used to represent a relative relationship between two nodes of the same scale, and the adjacent edge has attributes; a first graph matching unit 1040 configured to perform graph matching on the first scale graph representation of the first data and the first scale graph representation of the second data to obtain a first matching result; a second graph matching unit 1050 configured to perform graph matching on the second scale graph representation of the first data and the second scale graph representation of the second data to obtain a second matching result; a first determining unit 1060 configured to determine a multi-scale matching result based on the first matching result and the second matching result; and a second determining unit 1070 configured to determine a task processing result based on the multi-scale matching result.

[0185] It can be understood that the operations of the units 1010-1070 in the apparatus 1000 and the operations of the steps 101-107 in the method 100 are similar, and will not be repeated here.

[0186] Figure 11A structural block diagram of a training apparatus 1100 of a neural network according to one embodiment of the present disclosure is shown, the apparatus 1100 comprising: a fourth obtaining unit 1110 configured to obtain first sample data and second sample data, the first sample data and the second sample data being one of image data, audio data, text data, molecular structure data and sequence data respectively; a fifth obtaining unit 1120 configured to obtain a multi-scale graph representation of each of the first sample data and the second sample data, wherein the multi-scale graph representation is determined by a graph representation extraction network, and the multi-scale graph representation comprises a graph representation of a first scale and a graph representation of a second scale; a third graph matching unit 1130 configured to perform graph matching on the graph representation of the first scale of the first sample data and the graph representation of the first scale of the second sample data to obtain a first current matching result representing a matching degree of the first scale; a fourth graph matching unit 1140 configured to perform graph matching on the graph representation of the second scale of the first sample data and the graph representation of the second scale of the second sample data to obtain a second current matching result representing a matching degree of the second scale; a seventh obtaining unit 1150 configured to obtain a target matching result and / or a target task processing result of the first sample data and the second sample data; a third determining unit 1160 configured to determine a loss value according to the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and a training unit 1170 configured to train the graph representation extraction network according to the loss value.

[0187] It can be understood that the operations of the units 1111-1170 in the apparatus 1100 and the operations of the steps 601-607 in the method 600 are similar, and will not be described here in detail.

[0188] According to embodiments of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0189] In the following, the embodiments of the present disclosure will be described in conjunction with Figure 12 Illustrative examples of such electronic devices, non-transitory computer-readable storage mediums and computer program products are described below.

[0190] Figure 12 An example configuration of an electronic device 1200 that can be used to implement the methods described herein is shown. Each of the apparatuses 1000 and 1100 described above can also be implemented in whole or in part by the electronic device 1200 or a similar device or system.

[0191] The electronic device 1200 can be various different types of devices. Examples of the electronic device 1200 include, but are not limited to, a desktop computer, a server computer, a notebook or netbook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless phone (e.g., a smart phone), a notepad computer, a mobile station), a wearable device (e.g., eyewear, a watch), an entertainment device (e.g., an entertainment appliance, a set-top box communicatively coupled to a display device, a game console), a television or other display device, a car computer, and the like.

[0192] The electronic device 1200 can include at least one processor 1202, memory 1204, communication interface(s) 1206, a display device 1208, other input / output (I / O) devices 1210, and one or more mass storage devices 1212, which are capable of communicating with each other, such as via a system bus 1214 or other appropriate connection.

[0193] The processor 1202 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing units or multiple cores. The processor 1202 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 1202 can be configured to fetch and execute computer-readable instructions stored in the memory 1204, the mass storage device 1212, or any other computer-readable medium, such as program code for an operating system 1216, program code for applications 1218, program code for other programs 1220, and the like.

[0194] The memory 1204 and the mass storage device 1212 are examples of computer-readable storage media for storing instructions which are executed by the processor 1202 to implement the various functions described above. By way of example, the memory 1204 can generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the mass storage device 1212 can generally include a hard disk drive, a solid state drive, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD or DVD), storage arrays, network attached storage, storage area networks, and the like. Both the memory 1204 and the mass storage device 1212 can be collectively referred to herein as the memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by the processor 1202 as a particular machine configured to implement the operations and functions described in the examples herein.

[0195] A number of programs can be stored on the mass storage device 1212. These programs include an operating system 1216, one or more application programs 1218, other programs 1220, and program data 1222, and they can be loaded into the memory 1204 for execution. Examples of such application programs or program modules can include, for example, computer program logic (e.g., computer program code or instructions) for implementing the method 100, the method 600, and / or the method 900 (including any suitable steps of the method 100, the method 600, the method 900), and / or additional embodiments described herein.

[0196] Although illustrated in Figure 12 the memory 1204 of the electronic device 1200, the modules 1216, 1218, 1220, and 1222, or portions thereof, can be implemented using any form of computer-readable media that is accessible by the electronic device 1200. As used herein, "computer-readable media" includes both computer-readable storage media and communication media.

[0197] Computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by an electronic device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer-readable storage media does not include communication media.

[0198] One or more communication interfaces 1206 are used to exchange data with other devices, such as over a network, direct connection, or the like. Such communication interfaces can be one or more of: any type of network interface (e.g., network interface card (NIC)), wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, Worldwide Interoperability Microwave Access (Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth™ interface, near field communication (NFC) interface, or the like. The communication interfaces 1206 can facilitate communications within a variety of networks and protocol types, including wired networks (e.g., LAN, cable, or the like) and wireless networks (e.g., WLAN, cellular, satellite, or the like), the Internet, or the like. The communication interfaces 1206 can also provide for communication with external storage (not shown), such as in a storage array, network attached storage, storage area network, or the like.

[0199] In some examples, a display device 1208, such as a monitor, can be included for displaying information and images to a user. Other I / O devices 1210 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, or the like.

[0200] The technology described herein can be supported by these various configurations of the electronic device 1200 and is not limited to the specific examples described herein. For example, the functionality can also be implemented all or in part through use of a distributed system, such as over a "cloud." The cloud includes and / or comprises a platform of resources. The platform abstracts underlying functionality of hardware (e.g., servers) and software resources of the cloud. Resources can include applications and / or data that are stored on servers that are remote to the electronic device 1200, and that can be accessed by the electronic device 1200 through the Internet and / or through a subscriber network, such as a cellular or Wi-Fi network. The platform can abstract resources and functionality of the cloud to connect the electronic device 1200 with other electronic devices. Thus, the functionality described herein can be implemented all or in part within the cloud. For example, the functionality can be implemented in part on the electronic device 1200 as well as part by the platform that abstracts the functionality of the cloud.

[0201] While the disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description is to be considered illustrative or exemplary and not restrictive; the disclosure is not limited to the disclosed embodiments. Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, the word "a" or "an" does not exclude a plurality, the term "multiple" means two or more, and the term "based on" means "based at least in part on." The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A task processing method, comprising: Acquire first data and second data, wherein the first data and the second data are respectively one of image data, audio data, text data, molecular structure data and sequence data, and the sequence data is one of protein sequence data and gene sequence data; Obtain graph representations of the first data and the second data at a first scale, wherein the graph representation at the first scale includes at least one node at the first scale, and the node at the first scale has attributes, including attributes of vector type. Obtain graph representations of the first data and the second data at a second scale, where the second scale is lower than the first scale. The graph representation at the second scale includes at least one node at the second scale, wherein the nodes at the second scale have attributes, including vector-type attributes. In this context, at least one node of each of the first and second data is obtained by sparsifying the dense data corresponding to that data. The graph representation of at least one scale of each data includes at least one adjacent edge. Each of the at least one adjacent edge is used to characterize the relative relationship between two nodes of the same scale, and the adjacent edge has attributes. Graph matching is performed between the first-scale graph representation of the first data and the first-scale graph representation of the second data to obtain a first matching result; Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result; Based on the first matching result and the second matching result, determine the multi-scale matching result; and Based on the multi-scale matching results, the task processing result is determined. The multi-scale graph representation of each data point includes a graph representation of the data at a first scale and a graph representation of the data at a second scale. The multi-scale graph representation of the data includes at least one dependent edge. Each dependent edge is used to characterize the dependency relationship between two nodes at different scales. The dependent edge has attributes.

2. The method according to claim 1, wherein, The attributes of the dependent edge are determined based on the attributes of the two nodes connected to the dependent edge.

3. The method according to any one of claims 1-2, wherein, At least one of the first-scale and second-scale graphical representations of each data set satisfies at least one of the following: The attributes of nodes at this scale include scalar type attributes; The properties of adjacent edges at this scale include scalar type properties; The attributes of the dependent edges of this scale include scalar type attributes; The properties of adjacent edges at this scale include vector-type properties; and The attributes of the dependent edges at this scale include attributes of vector type.

4. The method according to any one of claims 1-2, wherein, The first-scale graph representations of the first data and the second data are generated using a first network and / or the second-scale graph representations of the first data and the second data are generated using a second network.

5. The method according to any one of claims 1-2, wherein, The vector-type attributes of a node include a feature vector, where the graph matching process at each scale includes: Candidate matching point pairs are determined based on at least one node included in the graph representation of the first data at that scale and at least one node included in the graph representation of the second data at that scale, wherein the candidate matching point pair includes a first candidate matching node belonging to the graph representation of the first data at that scale and a second candidate matching node belonging to the graph representation of the second data at that scale; For the candidate matching point pair, the matching result of the candidate matching point pair is determined based on the feature vector of the first candidate matching node included in the candidate matching point pair and the feature vector of the second candidate matching node included in the candidate matching point pair. Based on the matching results of the candidate matching point pairs, the matching results of the graph representation of the first data at this scale and the graph representation of the second data at this scale are determined; And / or, Candidate matching edge pairs are determined based on at least one adjacent edge included in the graph representation of the first data at that scale and at least one adjacent edge included in the graph representation of the second data at that scale, wherein the candidate matching edge pair includes a first candidate matching adjacent edge belonging to the graph representation of the first data at that scale and a second candidate matching adjacent edge belonging to the graph representation of the second data at that scale. For the candidate matching edge pair, based on the attributes of the first candidate matching adjacent edge included in the candidate matching edge pair and the attributes of the second candidate matching adjacent edge included in the candidate matching edge pair, the matching result of the candidate matching edge pair is determined; and Based on the matching results of the candidate matching edge pairs, the matching results of the graph representation of the first data at this scale and the graph representation of the second data at this scale are determined.

6. The method according to claim 5, wherein, The node's attributes also include scalar type attributes, wherein determining the matching result of the candidate matching point pair includes: Based on the scalar type attributes of the first candidate matching node included in the candidate matching point pair and the scalar type attributes of the second candidate matching node included in the candidate matching point pair, the first point pair matching result of the candidate matching point pair is determined. In response to determining that the first point pair matching result of the candidate matching point pair satisfies a first preset condition, the second point pair matching result of the candidate matching point pair is determined based on the feature vectors of the first candidate matching nodes included in the candidate matching point pair and the feature vectors of the second candidate matching nodes included in the candidate matching point pair; and Based on the matching result of the second point pair, the matching result of the candidate matching point pair is determined. and / or The attributes of adjacent edges include scalar type attributes and vector type attributes. The vector type attributes of adjacent edges include feature vectors. The matching result for determining the candidate matching edge pairs includes: Based on the scalar type attributes of the first candidate matching adjacent edge included in the candidate matching edge pair and the scalar type attributes of the second candidate matching adjacent edge included in the candidate matching edge pair, the first edge pair matching result of the candidate matching edge pair is determined; In response to determining that the first edge pair matching result of the candidate matching edge pair satisfies a second preset condition, the second edge pair matching result of the candidate matching edge pair is determined based on the feature vectors of the first candidate matching adjacent edges included in the candidate matching edge pair and the feature vectors of the second candidate matching adjacent edges included in the candidate matching edge pair; and Based on the matching result of the second edge pair, the matching result of the candidate matching edge pair is determined.

7. The method according to claim 5, wherein, The scalar type properties of a node include the saliency of the node, and / or the scalar type properties of adjacent edges include the saliency of the adjacent edges.

8. The method according to claim 7, wherein, Determining the matching result of the candidate matching point pair includes: The matching result of the candidate matching point pair is determined by multiplying the salience of the first candidate matching node included in the candidate matching point pair, the salience of the second candidate matching node included in the candidate matching point pair, and the similarity between the feature vector of the first candidate matching node and the feature vector of the second candidate matching result. And / or, The matching result for the candidate matching edge pair includes: The matching result of the candidate matching edge pair is determined by multiplying the salience of the first candidate matching adjacent edge included in the candidate matching edge pair, the salience of the second candidate matching adjacent edge included in the candidate matching edge pair, and the similarity between the feature vectors of the first candidate matching adjacent edge and the feature vectors of the second candidate matching adjacent edge.

9. The method according to any one of claims 1-2, wherein, Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, including: In response to determining that the first matching result is a successful match, a graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result.

10. The method according to any one of claims 1-2, wherein, Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, including: In response to determining that the first matching result is a successful match, the first subgraph at the second scale of the first data and the second subgraph at the second scale of the second data are matched, wherein the first matching result indicates that the first node in the graph representation at the first scale of the first data and the second node in the graph representation at the first scale of the second data are successfully matched, the first subgraph includes nodes in the graph representation at the second scale of the first data that have a subordinate relationship with the first node, and the second subgraph includes nodes in the graph representation at the second scale of the second data that have a subordinate relationship with the second node.

11. The method according to any one of claims 1-2, wherein, Graph matching is performed between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result, including: The matching result of the current node is determined based on the attributes of the current node and whether the nodes of the first scale that are subordinate to the current node have been successfully matched, wherein the current node is a node of the second scale.

12. The method according to any one of claims 1-2, wherein, The at least one node at the first scale and the at least one node at the second scale are obtained by sparsifying the same dense data respectively. And / or, The dense data includes multiple scales, and the at least one node at the first scale and the at least one node at the second scale are obtained by sparsifying two of the multiple scales of the dense data respectively.

13. The method according to any one of claims 1-2, wherein, At least one node at each scale of each data point is obtained by sparsifying the portion of the dense data that corresponds to the node position at another scale, wherein the node at the other scale is obtained by sparsifying the dense data.

14. The method according to any one of claims 1-2, wherein, At least one node of each data scale is obtained by merging nodes of another scale, which is obtained by sparsifying the dense data.

15. The method according to claim 12, wherein, The dense data includes multiple dense nodes, each with attributes. These attributes include scalar and vector type attributes. The scalar type attributes include saliency, and the vector type attributes include feature vectors. The saliency of the dense nodes is determined based on the feature vectors of the dense nodes, and the sparsification of the dense data includes determining the nodes whose saliency satisfies a third preset condition among at least a portion of the dense nodes as the sparsified nodes.

16. The method according to claim 12, wherein, The dense data includes multiple dense nodes, and each dense node has attributes. The attributes of the nodes obtained through sparsification are determined based on the attributes of at least a subset of the dense nodes corresponding to that node. And / or, In this process, a node at at least one scale is obtained by merging nodes at another scale, and the attributes of the merged node are determined based on the attributes of the nodes at the other scale that have a subordinate relationship with the merged node.

17. The method according to any one of claims 1-2, wherein, The at least one adjacent edge is determined based on the attributes of at least one node of the same scale, wherein the attribute of each of the at least one adjacent edge is determined based on at least one of the attributes of the two nodes connected by the adjacent edge and the relative relationship between the two nodes.

18. The method according to claim 17, wherein, The at least one adjacent edge is determined by performing the following steps: At least one candidate adjacent edge is determined based on at least one node of the same scale; Based on the attributes of at least one node at the same scale, determine the saliency of each of the at least one candidate adjacent edges; as well as The at least one candidate adjacent edge whose saliency satisfies the fourth preset condition is determined as the at least one adjacent edge.

19. The method according to any one of claims 1-2, wherein, The second data is obtained from the database, wherein determining the task processing result based on the multi-scale matching result includes: Based on the multi-scale matching results of the first data and multiple second data in the database, at least one second data matching the first data is determined; and The task processing result is determined based on the at least one second data.

20. The method according to any one of claims 1-2, wherein, Both the first data and the second data are image data. The dense data is a feature map obtained based on the corresponding image data, and the multiple dense nodes in the dense data are multiple pixels in the feature map.

21. A method for training a neural network, the method comprising: Acquire first sample data and second sample data, wherein the first sample data and the second sample data are respectively one of image data, audio data, text data, molecular structure data and sequence data, and the sequence data is one of protein sequence data and gene sequence data; Obtain multi-scale graph representations of the first sample data and the second sample data respectively, wherein the multi-scale graph representation is determined by a graph representation extraction network, the multi-scale graph representation includes a graph representation of the first scale and a graph representation of the second scale, and the multi-scale graph representation also includes at least one dependent edge, each of the at least one dependent edge being used to characterize the dependent relationship between two nodes at different scales, and the dependent edge having attributes; Graph matching is performed between the graph representation of the first sample data at the first scale and the graph representation of the second sample data at the first scale to obtain a first current matching result that represents the degree of matching at the first scale. Graph matching is performed between the graph representation of the first sample data at the second scale and the graph representation of the second sample data at the second scale to obtain a second current matching result that characterizes the degree of matching at the second scale; Obtain the target matching results and / or target task processing results of the first sample data and the second sample data; The loss value is determined based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and The graph representation extraction network is trained based on the loss value.

22. The method according to claim 21, wherein, The loss value includes the matching loss value and / or the task loss value. The determination of the loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result includes: The matching loss value is determined based on the first current matching result and / or the second current matching result and the target matching result; And / or, Determine the current task result based on the first current matching result and / or the second current matching result; and The task loss value is determined based on the target task processing result and the current task result.

23. The method of claim 22, wherein, Determining the matching loss value based on the first current matching result and / or the second current matching result includes: Determine the current matching result based on the first current matching result and / or the second current matching result; and Determine the matching loss value based on the current matching result and the target matching result.

24. The method according to claim 22, wherein, The graph representation extraction network includes a first network for extracting graph representations at a first scale, wherein determining the matching loss value based on the first current matching result and / or the second current matching result includes: Based on the target matching result and the first current matching result, determine the first scale matching loss value; Training the graph representation extraction network based on the loss value includes: The first network is trained based on the loss value matched by the first scale. And / or, The graph representation extraction network includes a second network for extracting graph representations at a second scale, wherein determining the matching loss value based on the first current matching result and / or the second current matching result includes: Based on the target matching result and the second current matching result, determine the second scale matching loss value; Training the graph representation extraction network based on the loss value includes: The second network is trained based on the loss value matched by the second scale.

25. The method according to any one of claims 21-24, wherein, The target matching result and / or the target task processing result are determined based on one of the following: Methods based on manual annotation, teacher-based models and / or pre-trained models, auxiliary constraint information, and rules.

26. The method according to any one of claims 21-24, wherein, The target matching result is determined using a network trained for the Nth round, wherein obtaining the first sample data and the second sample data includes: The network trained for the Nth round is used to extract multi-scale graph representations of the first and second unlabeled data, respectively. Graph matching is performed between the first scale graph representation of the first unlabeled data and the first scale graph representation of the second unlabeled data to obtain the first unlabeled data matching result that characterizes the degree of matching at the first scale; Graph matching is performed between the second-scale graph representation of the first unlabeled data and the second-scale graph representation of the second unlabeled data to obtain the second unlabeled data matching result that characterizes the degree of matching at the second scale; The unlabeled data matching result is determined based on the first unlabeled data matching result and / or the second unlabeled data matching result; In response to determining that the first unlabeled data and the second unlabeled data satisfy a first condition, the first unlabeled data and the second unlabeled data are determined as first sample data and second sample data as positive samples, wherein satisfying the first condition includes the unlabeled data matching result satisfying a first matching condition, and the target matching result of the positive samples indicates that the corresponding first sample data and second sample data match; and / or In response to determining that the first unlabeled data and the second unlabeled data satisfy the second condition, the first unlabeled data and the second unlabeled data are determined as the first sample data and the second sample data as negative samples, wherein the first unlabeled data and the second unlabeled data satisfying the second condition includes the unlabeled data matching result satisfying the second matching condition, and the target matching result of the negative sample indicates that the corresponding first sample data and the second sample data do not match.

27. The method according to claim 26, wherein, Based on the first unlabeled data matching result and / or the second unlabeled data matching result, the unlabeled data matching result is determined as follows: In response to determining that the second unlabeled data matching result indicates that the second-scale graph representation of the first unlabeled data and the second-scale graph representation of the second unlabeled data have successfully matched, the unlabeled data matching result is determined to be a match.

28. The method according to any one of claims 21-24, wherein, The graph represents the extraction network, which includes a rule module and a network module. The method further includes at least one of the following steps: In response to determining that the fifth preset condition is met, the first rule module in the rule module is replaced with a network module; and In response to the determination that the sixth preset condition is met, a network module is added to the extraction network represented in the figure. Training the graph representation extraction network based on the loss value includes: The network module is trained based on the loss value.

29. The method according to any one of claims 21-24, wherein, Each scale of the multi-scale graph representation includes at least one node, and each node includes attributes, including scalar type attributes and vector type attributes. In the multi-scale graph representation, at least one scale of the graph representation includes at least one adjacent edge. Each of the at least one adjacent edge is used to represent the relative relationship between two nodes at the same scale. The adjacent edge has attributes, including scalar type attributes and vector type attributes. The graph representation extraction network includes at least one of the following: A network module used to determine the scalar type attributes of nodes; A network module used to determine the vector-type attributes of nodes; A network module used to determine the scalar type attributes of adjacent edges; and A network module used to determine the vector type attributes of adjacent edges.

30. The method according to claim 28, wherein, The scalar type attributes of a node include the node's saliency, and / or the vector type attributes of a node include the node's eigenvector, and / or the scalar type attributes of adjacent edges include the adjacent edge's saliency, and / or the vector type attributes of adjacent edges include the adjacent edge's eigenvector.

31. A task processing apparatus, comprising: The first acquisition unit is configured to acquire first data and second data, wherein the first data and the second data are respectively one of image data, audio data, text data, molecular structure data and sequence data, and the sequence data is one of protein sequence data and gene sequence data. The second acquisition unit is configured to acquire graph representations of the first data and the second data at a first scale, wherein the graph representation at the first scale includes at least one node at the first scale, and the node at the first scale has attributes, including attributes of vector type. The third acquisition unit is configured to acquire graph representations of the first data and the second data at a second scale, wherein the second scale is lower than the first scale, and the graph representation at the second scale includes at least one node at the second scale, wherein the node at the second scale has attributes, including vector-type attributes. In this context, at least one node of each of the first and second data is obtained by sparsifying the dense data corresponding to that data. The graph representation of at least one scale of each data includes at least one adjacent edge. Each of the at least one adjacent edge is used to characterize the relative relationship between two nodes of the same scale, and the adjacent edge has attributes. The first graph matching unit is configured to perform graph matching between the graph representation of the first data at a first scale and the graph representation of the second data at a first scale to obtain a first matching result; The second graph matching unit is configured to perform graph matching between the second-scale graph representation of the first data and the second-scale graph representation of the second data to obtain a second matching result; The first determining unit is configured to determine a multi-scale matching result based on the first matching result and the second matching result; and The second determining unit is configured to determine the task processing result based on the multi-scale matching result. The multi-scale graph representation of each data point includes a graph representation of the data at a first scale and a graph representation of the data at a second scale. The multi-scale graph representation of the data includes at least one dependent edge. Each dependent edge is used to characterize the dependency relationship between two nodes at different scales. The dependent edge has attributes.

32. A training apparatus for a neural network, the method comprising: The fourth acquisition unit is configured to acquire first sample data and second sample data, wherein the first sample data and the second sample data are one of image data, audio data, text data, molecular structure data and sequence data, respectively, and the sequence data is one of protein sequence data and gene sequence data. The fifth acquisition unit is configured to acquire the multi-scale graph representations of the first sample data and the second sample data respectively, wherein the multi-scale graph representation is determined by a graph representation extraction network, the multi-scale graph representation includes a graph representation at the first scale and a graph representation at the second scale, and the multi-scale graph representation also includes at least one dependent edge, each of the at least one dependent edge being used to characterize the dependent relationship between two nodes at different scales, and the dependent edge having attributes; The third graph matching unit is configured to perform graph matching between the graph representation of the first sample data at a first scale and the graph representation of the second sample data at a first scale, so as to obtain a first current matching result that characterizes the degree of matching at the first scale; The fourth graph matching unit is configured to perform graph matching between the second-scale graph representation of the first sample data and the second-scale graph representation of the second sample data to obtain a second current matching result characterizing the degree of matching at the second scale; The seventh acquisition unit is configured to acquire the target matching result and / or target task processing result of the first sample data and the second sample data; The third determining unit is configured to determine a loss value based on the target matching result and / or the target task processing result, and the first current matching result and / or the second current matching result; and The training unit is configured to train the graph representation extraction network based on the loss value.

33. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-30.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-30.

35. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-30.

Images