Object recognition method, device, computer-readable storage medium, and computer equipment

Abstract

This application discloses an object recognition method, apparatus, computer-readable storage medium, and computer device. The method includes: generating text description data and graph structure data corresponding to an object to be recognized in a target scene. The graph structure data includes keywords in the object to be recognized and the relationships between these keywords. The text description data and graph structure data are then feature-encoded to obtain text-encoded features corresponding to the text description data and graph-encoded features corresponding to the graph structure data. The text-encoded features and graph-encoded features are then fused to determine the recognition result corresponding to the object to be recognized. This application can retain more original information in the object to be recognized, maximize the utilization of features in the object to be recognized, and achieve object recognition based on both text description information and graph structure data, thereby improving the accuracy of object recognition.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an object recognition method, device, computer-readable storage medium and computer equipment. Background Art

[0002] With the advancement of technology, more and more users are obtaining various required information, such as medical classification information, through interactive means on the Internet. At present, the recognition of corresponding data obtained through interactive means mostly uses text recognition. Specifically, the computer device first obtains data information, then extracts text features from the data information, and then classifies it according to the text features to obtain corresponding classification information. In most object recognition scenarios, it is found that identifying data through text recognition will make the data recognition results inaccurate, which reduces the user experience. Summary of the invention

[0003] The embodiments of the present application provide an object recognition method, an apparatus, a computer-readable storage medium, and a computer device, which can improve the accuracy of identifying an object to be identified.

[0004] The present application provides an object recognition method, including:

[0005] Obtain the object to be identified in the target scene;

[0006] Generate text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and associations between the keywords;

[0007] Performing feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data;

[0008] Performing feature coding processing on the graph structure data to obtain graph coding features corresponding to the graph structure data;

[0009] The text encoding feature and the image encoding feature are fused to determine the recognition result corresponding to the object to be recognized.

[0010] The present application also provides an object recognition device, including:

[0011] An acquisition module is used to acquire the object to be identified in the target scene;

[0012] A generating module, used to generate text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and associations between the keywords;

[0013] A first encoding module, used for performing feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data;

[0014] A second encoding module, used for performing feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data;

[0015] The fusion processing module is used to fuse the text encoding features and the image encoding features to determine the recognition result corresponding to the object to be recognized.

[0016] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the object recognition method described in any of the above embodiments.

[0017] An embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the object recognition method described in any of the above embodiments by calling the computer program stored in the memory.

[0018] The object recognition method, apparatus, computer-readable storage medium and computer equipment provided in the embodiments of the present application generate text description data and graph structure data corresponding to the object to be recognized in the target scene, the graph structure data includes keywords in the object to be recognized and the association between the keywords, and perform feature encoding processing on the text description data and the graph structure data respectively to obtain text encoding features corresponding to the text description data and graph encoding features corresponding to the graph structure data, and fuse the text encoding features and the graph encoding features to determine the recognition result corresponding to the object to be recognized. In the present application, in addition to generating text description data corresponding to the object to be identified, graph structure data corresponding to the object to be identified is also generated. Since the graph structure data includes keywords in the object to be identified and the association between the keywords, the text description information and graph structure information in the object to be identified can be retained, that is, more original information in the object to be identified can be retained, and the features in the object to be identified can be maximized. The text encoding features corresponding to the text description data and the graph encoding features corresponding to the graph structure data are fused to determine the recognition result corresponding to the object to be identified. In this way, the identification of the object to be identified is achieved based on the text description information and the graph structure data information, thereby improving the accuracy of identifying the object to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 The present invention provides a flowchart of an object recognition method according to an embodiment of the present application.

[0021] Figure 2 A schematic diagram of generating text electronic medical records and graph structured medical records provided in an embodiment of the present application.

[0022] Figure 3 A schematic diagram of a sub-process of an object recognition method provided in an embodiment of the present application.

[0023] Figure 4 A schematic diagram of another sub-process of the object recognition method provided in an embodiment of the present application.

[0024] Figure 5 A schematic diagram of multiple corresponding nodes and edges of a graph structured medical record provided in an embodiment of the present application.

[0025] Figure 6 A schematic diagram of another sub-process of the object recognition method provided in an embodiment of the present application.

[0026] Figure 7 A schematic diagram of another sub-process of the object recognition method provided in an embodiment of the present application.

[0027] Figure 8 A flowchart of an object recognition method provided in an embodiment of the present application.

[0028] Fig. 9 A schematic diagram of the training process of the object recognition method provided in an embodiment of the present application.

[0029] Fig.10 A schematic diagram of another training process of the object recognition method provided in an embodiment of the present application.

[0030] Fig.11 A schematic diagram of the structure of an object recognition device provided in an embodiment of the present application.

[0031] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0033] The embodiments of the present application provide an object recognition method, apparatus, computer-readable storage medium, and computer equipment. Specifically, the object recognition method of the embodiments of the present application can be executed by a computer device, wherein the computer device can be a terminal or a server or other device. The terminal can be a smart phone, a tablet computer, a laptop computer, a game console, a personal computer (PC, Personal Computer), an intelligent vehicle terminal, a robot, and other devices. The server can be an independent physical server, or a server cluster composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0034] The solution in this application can be applied to any target scenario, including interactive target scenarios and non-interactive target scenarios. The embodiment of this application is explained by applying to interactive target scenarios as an example, and the interactive target scenarios include more structured features. For example, in the field of intelligent question and answer, when an electronic device used by a user fails, the intelligent question and answer provided by the Internet allows the user to explain some fault phenomena corresponding to the failure of the electronic device, and finally realizes specific fault identification; for example, when a user is sick, the intelligent question and answer of the intelligent doctor provided by the Internet allows the user to explain some symptoms and signs corresponding to the illness, and finally realizes specific disease identification in the electronic medical record, etc.

[0035] In existing fault identification scenarios, intelligent question-answering is used to extract text description data such as the fault phenomenon and fault duration corresponding to the corresponding fault in the corresponding interactive dialogue information, and the final fault of the electronic device is identified based on the text description data. However, there may be multiple reasons for the same fault phenomenon in the electronic device, and there is a complex correlation between the fault phenomenon, fault duration and the fault, which cannot be understood through text description data; in addition, when the same fault phenomenon appears in different faults, its impact on the determination of the final fault is also different, and the importance of the same fault phenomenon in the current fault determination cannot be determined through text description data. This leads to inaccurate identification of the final fault of the electronic device.

[0036] In the existing disease identification scenarios of electronic medical records, intelligent questions and answers are used to extract the text description data (text electronic medical records) corresponding to the corresponding interactive dialogue information, and then the disease corresponding to the patient is determined based on the text description data. However, in the process of disease identification, there are complex correlations between symptoms and symptoms, symptoms and causes, and symptoms and diseases. One disease may cause multiple symptoms, and one symptom may be caused by multiple diseases or causes. There are also correlations between different symptoms in the same disease, which cannot be understood through text description data; in addition, when the same symptom appears in different diseases, its impact on the final determination of the disease is also different. The importance of the same symptom in the current disease determination cannot be determined through text description data. This leads to inaccurate identification of the final disease.

[0037] In order to solve the above method, the embodiments of the present application propose an object recognition method, device, computer-readable storage medium and computer equipment, which will be described in detail below. It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0038] Figure 1 1 is a flow chart of an object recognition method provided in an embodiment of the present application. The object recognition method is applied in a computer device and comprises the following steps.

[0039] 101, obtaining an object to be identified in a target scene.

[0040] The target scenario can be any scenario that requires data recognition / object recognition. For example, the fault recognition scenario of electronic equipment, the disease recognition scenario of electronic medical records, the character recognition scenario, and so on. The object to be recognized can include data information included in people or objects, and can also include other data information such as diseases. If in the field of intelligent question and answer, the object to be recognized can be interactive data information to be recognized. For example, in the fault recognition scenario, the interactive dialogue information between the user and the machine; in the disease recognition scenario / consultation scenario of electronic medical records, the interactive dialogue information between the patient and the doctor, etc., where the doctor can be either a virtual doctor or a real doctor. The interactive dialogue information can be text information, voice information, or even picture information, video information, etc. That is, the data information of the object to be recognized can include voice information and / or picture information and / or video information, etc.

[0041] The data information of the object to be identified may be sent by other computer devices through the network to the computer device that executes the object identification method in the embodiment of the present application, or it may be that an application related to intelligent question and answer is installed in the computer device, and the computer device obtains the interactive dialogue information corresponding to the intelligent question and answer in the application.

[0042] 102 , generating text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and associations between the keywords.

[0043] Among them, if the object to be identified is interactive dialogue information, the text description data corresponding to the object to be identified is not the entire interactive dialogue information, but the text information closely related to the target scene extracted from the interactive dialogue information, and the text information is used to describe the main content of the object to be identified (interactive dialogue information). For example, in the fault identification scenario, the text description data includes text information related to the fault, fault phenomenon, fault duration, etc. corresponding to the electronic device extracted from the interactive dialogue information. In the disease identification scenario of electronic medical records, the text description data includes text electronic medical records. Specifically, for example, the text electronic medical record includes: Chief complaint, head tremor for 3 years; Current medical history: head tremor occurred in a calm state 3 years ago, accompanied by anxiety and palpitations.

[0044] The graph structure data corresponding to the object to be identified is a kind of structured data. The graph structure data is data formed based on a graph structure. The graph structure data includes keywords in the object to be identified and the association relationship between the keywords.

[0045] In one embodiment, the step of generating graph structure data corresponding to the object to be identified includes: extracting multiple keywords from the object to be identified; identifying the association relationship between the keywords; taking each keyword as a node and the association relationship between the keywords as the edge corresponding to the node, thereby generating graph structure data corresponding to the object to be identified, wherein the graph structure data includes a node set corresponding to the keyword and a relationship set corresponding to the association relationship between the keywords. The keywords include primary keywords and secondary keywords, and the association relationship between the keywords includes at least one of the association relationship between primary keywords and primary keywords, the association relationship between primary keywords and secondary keywords, and the association relationship between secondary keywords and secondary keywords.

[0046] In the embodiment of the present application, the target scenario is the disease identification scenario / consultation scenario of the electronic medical record, and the object to be identified is the interactive dialogue information between the patient and the doctor as an example to illustrate how to generate text description data (text electronic medical record) and graph structure data (graph structure medical record). The following also uses the object to be identified in this scenario as an example for explanation, and will not be repeated later.

[0047] Figure 2 A schematic diagram of generating text electronic medical records and graph structured medical records provided in an embodiment of the present application.

[0048] The user inputs a colloquial symptom description text through the dialog box of the intelligent question-answering system or by voice. First, the key information extraction module extracts the main symptoms (main keywords) and their related symptom attributes (secondary keywords) from the colloquial text. For example, if the input is: left and right tremors in a static head state, the key information extraction module extracts: symptom: head tremor, cause: in a calm state. Among them, head tremor is the main keyword; in a calm state is the secondary keyword, corresponding to the symptom attribute; the cause is understood as the association between the main keyword and the secondary keyword.

[0049] After the key information extraction module obtains the key information (chief complaint symptoms and their related symptom attributes), the medical record template is filled in through the preset medical record template library, such as filling in symptom attributes such as onset time and frequency characteristics. Among them, the medical record template library includes a question template library for which questions to ask patients corresponding to the key information. The filled symptom attributes such as onset time, frequency characteristics, etc. are connected with the chief complaint symptoms to form a graph structure of the basic medical record.

[0050] Then, based on the basic medical records, symptoms are recommended and selected through multiple rounds of interaction with users. The symptom recommendation module is used to implement symptom recommendation and selection. The symptom recommendation module adopts a fusion strategy of resource recommendation and model recommendation. Among them, the resource recommendation strategy includes various pre-set questions. For example, if the main symptom is headache, then ask whether there is dizziness and other problems. The model recommendation strategy is learned based on the doctor's diagnosis data. When a headache occurs, what kind of questions the doctor generally asks, etc. The fusion strategy of resource recommendation and model recommendation is understood as combining resource strategy and model strategy to ensure the accuracy and breadth of the acquired symptoms (keywords). In this way, the data information of the object to be identified is obtained. Among them, the user's selected symptoms are connected with the known basic graph structure, and the relationship is the accompanying symptoms.

[0051] The main complaint and current medical history of the object to be identified are obtained to generate text description data (text electronic medical record) corresponding to the object to be identified. For example, the text electronic medical record is: main complaint: head tremor for 3 years; current medical history: head tremor occurred in a calm state 3 years ago, accompanied by anxiety and palpitations.

[0052] Extract the main complaint symptoms (main keywords), selected symptoms (secondary keywords), and symptom attributes (secondary keywords) from the data information of the object to be identified, identify the association between the main complaint symptoms, selected symptoms, and symptom attributes, and take each main complaint symptom, accompanying symptom, symptom attribute, etc. as a node, and take the association between the main complaint symptoms, accompanying symptoms, and symptom attributes as the edge corresponding to the node, so as to generate the graph structure data (graph structure medical record) corresponding to the object to be identified, as shown in the following example: Figure 2 shown.

[0053] The graph structure data finally obtained includes a node set corresponding to the keywords and a relationship set corresponding to the association relationship between the keywords, that is, a set of edges.

[0054] use Figure 2 The text description data and graph structure data extracted by the technical solution shown have high accuracy. For example, the graph structure data is 100% correct. Using high-accuracy input data for processing can improve the recognition results of the object to be recognized.

[0055] It should be noted that the text description data and graph structure data corresponding to the object to be identified can be generated dynamically according to the corresponding interactive dialogue information, or can be generated after obtaining the complete interactive dialogue information. At the same time, it should be reminded that the above-mentioned text description data and graph structure data corresponding to the object to be identified are only an example to facilitate understanding of the solution in the embodiment of the present application, and do not constitute a limitation.

[0056] In other target scenarios, due to the different corresponding basic data, the generated text description data and graph structure data may also be different, but can be generated using the same / similar method.

[0057] 103 , performing feature coding processing on the text description data to obtain text coding features corresponding to the text description data.

[0058] This step implements feature encoding of the text description data, and any scheme that can implement feature encoding of the text description data can be used. In the embodiment of the present application, a text feature extraction model / text feature encoding model is used to implement feature extraction / feature encoding of the text description data. Among them, the text feature extraction model can adopt a BERT (Bidirectional Encoder Representations from Transformers) model.

[0059] First, unsupervised pre-training of the BERT model (e.g., 12 layers) is performed based on a large amount of unlabeled medical data such as outpatient medical records, inpatient medical records, medical guidelines, and textbooks. Then, based on the pre-trained BERT model, task pre-training and fine-tuning of the consultation scenario are performed, that is, a large number of text electronic medical records (including chief complaints and current medical history) in the consultation scenario are collected for consultation task pre-training and parameter fine-tuning of the BERT model, so that the BERT model can adapt to the data of the consultation scenario to improve the effect of feature extraction / feature encoding.

[0060] The contents of the chief complaint and current medical history fields in the text description data (text electronic medical record) are concatenated, separated by a special delimiter "[SEP]", and the identifier [CLS] is added in front of the chief complaint. The specific input format is: [CLS] chief complaint [SEP] current medical history. According to the input format, the contents of the chief complaint and current medical history are input into the fine-tuned BERT model for feature encoding processing to obtain the output sequence of the BERT model, which uses the output sequence as the text encoding feature corresponding to the text description data. The obtained text encoding feature is a two-dimensional matrix with a dimension of (w,h), where w is the length of the text description data and h is the length of the hidden layer in the BERT model.

[0061] For example, the main complaint is head tremor for 3 years; the current medical history is head tremor in a calm state 3 years ago, accompanied by anxiety and palpitations. The input text description data is: [CLS] head tremor for 3 years [SEP] head tremor in a calm state 3 years ago, accompanied by anxiety and palpitations. The length of the corresponding text description data (each word and each punctuation mark) is 29. If the length of the hidden layer is 256, the final text encoding feature is 29*256. It can be understood that each text (each word) in the text description data corresponds to a 256-dimensional feature.

[0062] Among them, the pre-training BERT model, the fine-tuning BERT model, and the use of the BERT model to extract features / encode features of text description data can be found in the existing implementation scheme, which will not be repeated here.

[0063] 104 , performing feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data.

[0064] This step implements feature encoding processing of graph structure data. In the embodiment of the present application, a graph neural network is used to perform feature encoding processing on the graph structure data.

[0065] In recent years, the Graph Convolutional Network (GCN) has been widely used in modeling various non-Euclidean spatial data structures and has achieved rich results. It can effectively utilize the connection information and node information of the spatial graph structure to obtain richer node feature representation and whole graph feature representation. In the field of intelligent question and answer, the obtained graph structure data (graph structure medical record) includes node information and connection information, so GCN can be used to perform feature encoding processing on the graph structure data in the field of intelligent question and answer to obtain graph encoding features. It should be noted that the GCN graph neural network is only an example, and other graph neural networks can also be used to perform feature encoding processing on the graph structure information to obtain graph encoding features. Among them, there are differences in the specific encoding processing methods for adjusting the encoding processing of the text description data and adjusting the encoding processing of the graph structure data.

[0066] The graph structure data includes a set of nodes and a set of edges corresponding to the keywords. In one embodiment, the above step 104 includes: performing vector initialization on each node in the node set to obtain an initialized feature vector of each node; inputting the initialized feature vector into the graph neural network for feature encoding processing to obtain graph encoding features corresponding to the graph structure data.

[0067] Assume that the current graph structure data (graph structure medical record) is represented by G = (V, E), where V and E are the node set and edge set, also called the relationship set. In the initial state, each node needs to be assigned an initialization feature vector The initialization feature vector has an important influence on the feature expression of the final graph neural network. In order to improve the feature encoding effect of the graph neural network, a Word-to-Node Attention module is proposed in the embodiment of the present application to improve the node initialization effect of the initialization graph neural network.

[0068] The Word-to-Node Attention module implements vector initialization for each node in the node set according to the text encoding features to obtain the initialized feature vector of each node. In this way, the initialized feature vector obtained includes the information of the text encoding features, which improves the accuracy of the initialized feature vector and further improves the accuracy of the graph encoding features.

[0069] In one embodiment, the above step of performing vector initialization on each node in the node set according to the text encoding feature to obtain the initialized feature vector of each node includes: for each node, calculating the initial attention weight coefficient of each text in the node relative to the text encoding feature; determining the initialized feature vector of the node according to the initial attention weight coefficient and the text encoding feature corresponding to the node. Wherein, each text is understood as each word, one node corresponds to one keyword, and one keyword includes at least one word, generally multiple words.

[0070] In one embodiment, the step of calculating the initial attention weight coefficient of each text in the node relative to the text encoding feature includes: calculating the dot product of the text encoding feature and each text in the node; and using the ratio of the dot product of the corresponding text in the node to the sum of the dot products of each text in the node as the initial attention weight coefficient of the text relative to the text encoding feature.

[0071] The initial attention weight coefficient of each text in the node relative to the text encoding feature can be calculated as shown in the following formula (1).

[0072]

[0073] Assume that node v is represented as: {w start ,...,w j ,...,w end}, the text encoding feature corresponding to the node v is represented as {x start ,...,x j ,...,x end}, where x j is the text encoding feature of the jth text (i.e., the jth word) in the node, start represents the first text in node v, end represents the last text in node v, M is the text encoding feature corresponding to the text description data, Attn(.) is a single-layer neural network, and this neural network is used to implement M,x j The dot product operation, or to achieve M,x j The operations corresponding to the fully connected layer or similar calculations, etc. Represents x j For the initial attention weight coefficient of the entire node, that is, the degree of importance, the initial attention weight coefficient is a value between [0,1]. exp is the natural base e, and exp() is an exponential with base e.

[0074] In one embodiment, the step of determining the initialization feature vector of the node according to the initial attention weight coefficient and the text encoding feature corresponding to the node can be specifically calculated by the following formula (2), wherein: represents the initialization feature vector of the 0th layer node v, which is based on The vectors of different words in the node v are weighted and summed up, and the corresponding dimension is 1*256.

[0075]

[0076] In the above-mentioned embodiment, when calculating the initial attention weight coefficient, it is calculated according to the text coding features corresponding to the text description data. In some embodiments, the overall representation coding features of the text coding features can also be determined, and the initial attention weight coefficient is calculated according to the overall representation coding features. Usually, the text coding features corresponding to the text description data include text coding features corresponding to some accompanying symptoms, and such text coding features are less important than the text coding features of the main complaint.

[0077] Therefore, in one embodiment, the text encoding feature corresponding to the [CLS] main complaint is used as the overall representation encoding feature of the text encoding feature. In other embodiments, the text encoding corresponding to the main complaint refers to the main feature obtained by extracting and identifying the data information of the object to be identified using key information.

[0078] Correspondingly, if Figure 3 As shown, the above step of performing vector initialization on each node in the node set according to the text encoding feature to obtain the initialized feature vector of each node includes the following steps 201 to 203.

[0079] 201, obtaining text coding features corresponding to the main complaint in the text coding features as overall representation coding features of the text coding features.

[0080] 202, for each node, the initial attention weight coefficient of each text in the node relative to the overall representation encoding feature is calculated.

[0081] In one embodiment, the above-mentioned step 202 includes: calculating the dot product of the overall representation coding feature and each text in the node; and using the ratio of the dot product of the corresponding text in the node to the sum of the dot products of each text in the node as the initial attention weight coefficient of the text relative to the overall representation coding feature.

[0082] The initial attention weight coefficient of each text in the node relative to the overall representation encoding feature can be calculated according to the following formula (3), where m M Represents the overall representation encoding features of text description data.

[0083]

[0084] 203, determine the initialization feature vector of the node according to the initial attention weight coefficient and the text encoding feature corresponding to the node. For example, the initialization feature vector of the node can be determined according to formula (2).

[0085] The attention mechanism of the above-mentioned Word-to-Node Attention module ensures that each node not only captures the meaning of its components, but also captures the semantics of the entire text description data (text electronic medical record). It is an attention mechanism that includes the local to the whole. In this way, the initialization feature vector of each node includes the information of the text encoding features corresponding to the entire text description data, which improves the accuracy of the initialization feature vector and further improves the accuracy of the graph encoding features.

[0086] After determining the initialization feature vector of each node, the initialization feature vector is input into the graph neural network for feature encoding processing to obtain the graph encoding feature corresponding to the graph structure data. The graph neural network can be a GCN neural network, etc.

[0087] It should be noted that in the medical field, graph structure data (graph structure medical records) belongs to a complex heterogeneous graph structure with multiple relationships (the nodes and connection relationships of the graph can be symptoms, onset time, inducement, etc.), and ordinary GCN cannot effectively encode it. Therefore, in the medical field or other complex heterogeneous graph structures, a multi-relational gated graph neural network (GGNN) can be used to perform feature encoding processing on the graph structure information to obtain graph encoding features.

[0088] Considering that different symptoms and their attributes in the same electronic medical record have different effects on the final disease identification, the embodiment of the present application improves GGNN and uses the attention mechanism to dynamically determine the weights of adjacent nodes in message transmission on the basis of GGNN, thus obtaining a new attention-based gated graph neural network (Att-GGNN). The following uses the attention-based gated graph neural network as an example to illustrate how to perform feature encoding processing on graph structure data to obtain the graph encoding features corresponding to the graph structure data.

[0089] In one embodiment, if Figure 4 As shown, the above step of inputting the initialized feature vector into the graph neural network for feature encoding processing to obtain the graph encoding features corresponding to the graph structure data includes the following steps 301 to 303.

[0090] 301 , for each node, determine the total neighbor nodes, in-degree neighbor nodes, and out-degree neighbor nodes associated with the node.

[0091] by Figure 2The graph structure data (graph structure medical record) in is used as an example to illustrate. The corresponding graph structure medical record is converted into Figure 5 The data shown. Among them, node 1 (head tremor), node 2 (anxiety), node 3 (palpitations), node 4 (3 years), node 5 (calm state). For node 1, its corresponding total neighbor nodes are node 2, node 3, node 4 and node 5, and their corresponding out-degree neighbor nodes; for node 2, its corresponding total neighbor nodes are node 1 and node 4, the in-degree neighbor node is node 1, and the out-degree neighbor node is node 4; for node 3, its corresponding total neighbor nodes are node 1 and node 5, the in-degree neighbor node is node 1, and the out-degree neighbor node is node 5; and so on, the total neighbor nodes, in-degree neighbor nodes and out-degree neighbor nodes associated with each node can be obtained.

[0092] 302. According to the initialized feature vector, the total neighbor nodes, in-degree neighbor nodes and out-degree neighbor nodes of the node, combined with the attention mechanism, determine the node feature vector of the node in the first layer of the graph neural network, and use the node feature vectors of all nodes in the first layer of the graph neural network as the output feature vector of the first layer.

[0093] Assume that in the initial state, the initialization feature vector of each node is The constructed Att-GGNN model has K layers, and each layer will combine the attention mechanism to obtain the vector representation of the nodes in each layer, that is, the node feature vector of each layer node, recorded as In the calculation of each layer, from the current node v i The total neighbor nodes are aggregated to obtain the vector representation of the current node (node ​​feature vector). Since the graph structure medical record is a heterogeneous multi-relation graph with two different out-degree and in-degree relations, a multi-relation feature aggregation algorithm is used to aggregate and calculate its surrounding neighbor nodes (total neighbor nodes).

[0094] Take the node v in the first layer i Take an example to illustrate how to determine the vector representation of the node, that is, how to calculate the node feature vector of the node.

[0095] In one embodiment, if Figure 6 As shown, the above steps of determining the node feature vector of the node in the first layer of the graph neural network based on the initialized feature vector, the total neighbor nodes, in-degree neighbor nodes and out-degree neighbor nodes of the node, combined with the attention mechanism, include the following steps 401 to 403.

[0096] 401, based on the attention mechanism, determine the attention weight coefficient corresponding to each neighbor node to the node in the total neighbor nodes according to the initialized feature vector.

[0097] Calculate the attention weight coefficient corresponding to each neighbor node in the total neighbor nodes to this node, that is, the importance of each neighbor node to this node.

[0098] In one embodiment, the above step 401 includes: calculating the dot product of the initialized feature vector of each neighbor node in the total neighbor nodes and the initialized feature vector of the node; and taking the ratio of the dot product corresponding to each neighbor node to the sum of the dot products corresponding to the total neighbor nodes as the attention weight coefficient corresponding to each neighbor node to the node. The attention weight coefficient is a value between [0,1].

[0099] Among them, Represents the neighbor node v j For node v i The attention weight coefficient of node v j For node v i The degree of influence The k in represents the kth layer. If the current calculation is the corresponding data of the 1st layer, then k = 1. j For node v i The attention weight coefficient can be determined by the following formula (4).

[0100]

[0101] in, represents the i-th node v in the k-th layer i The feature representation (node ​​feature vector) of is, if k = 1, then it corresponds to the i-th node v i The initialization feature vector of , the corresponding dimension is 1*256, represents the jth node v in the kth layer j The feature representation (node ​​feature vector) of is, if k = 1, then it corresponds to the jth node v j The initialization feature vector of , the corresponding dimension is 1*256. It should be noted that, in some embodiments, the first layer of the graph neural network is represented by 0, then when k=0, represents the initialization feature vector of the corresponding node.

[0102] N i Represents node v i The total set of neighbor nodes, including out-degree neighbor nodes and in-degree neighbor nodes. Attn(.,.) is a single-layer neural network (a neural network combined with an attention mechanism) used to calculate and The correlation of , Attn(.,.) is calculated as follows: where a T and W Ais a learnable parameter, and the semicolon; indicates that the corresponding two vectors are superimposed, and the dimension of the superimposed vector remains unchanged.

[0103] 402 , determining, according to the attention weight coefficient, an in-degree feature vector corresponding to the in-degree neighbor node of the node and an out-degree feature vector corresponding to the out-degree neighbor node of the node.

[0104] Among them, use N← ( i )and N→ ( i ) represent the nodes v i The set of in-degree neighbor nodes and the set of out-degree neighbor nodes, represents the sum of the eigenvectors corresponding to all in-degree neighbor nodes, that is, the in-degree eigenvectors corresponding to all in-degree neighbor nodes. It represents the sum of the eigenvectors corresponding to all out-degree neighbor nodes, that is, the out-degree eigenvectors corresponding to all out-degree neighbor nodes. The in-degree eigenvector and out-degree eigenvector can be calculated according to the following formula (5) and formula (6).

[0105]

[0106]

[0107] in, Represents the relationship from node i to node j. Represents the weight parameters learned by this relationship, and the corresponding dimension is F*256, where F is adjusted according to the training process. and are learnable parameters, representing the i To node v j Node relationships and node v j To node v i Relationship The relationship weight matrix.

[0108] 403. Determine the node feature vector of the first layer of the graph neural network according to the initialization feature vector, the in-degree feature vector and the out-degree feature vector corresponding to the node.

[0109] It can be understood that the output of each node in the graph structure data is related to the node feature vector, in-degree feature vector and out-degree feature vector corresponding to the node. If it is the first layer, the output corresponding to the node is related to the initialization feature vector, in-degree feature vector and out-degree feature vector corresponding to the node.

[0110] In one embodiment, determining the node v iThe node feature vector in the first layer of the graph neural network, that is, the node v i The output feature vector of the first layer of the graph neural network can be determined by the following formula (7).

[0111]

[0112] Among them, k represents the k-th layer of the graph neural network. If it is the first layer, k=1. It means to concatenate / superimpose the out-degree feature vector and the in-degree feature vector. is the node v of the k-th layer graph neural network i The input feature vector of itself. If k=1, it is the node v i Initialize the feature vector of node v i Its own input feature vector and node v i The out-degree feature vector and the in-degree feature vector are superimposed as the input of the GRU function, and the final output node v i In the output feature vector of the kth layer, if k = 1, then the output node v i The output feature vector of the kth layer has a dimension of 1*256. The output feature vector is used as the node v i Input feature vector at layer k+1.

[0113] The node v that can be obtained from this i The output feature vector (node ​​feature vector) of the kth layer not only includes its own information, but also dynamically aggregates the feature representations of all neighboring nodes through the attention mechanism, reflecting the correlation between symptoms and symptoms, and symptoms and symptom attributes in the medical field, so that the corresponding feature vector includes the features corresponding to the structured data, thereby improving the accuracy of identifying the objects to be identified.

[0114] It should be noted that although steps 401 to 403 describe how to calculate the output feature vector (node ​​feature vector) of each node in the first layer of the graph neural network, the formulas provided are general formulas, and the output feature vector of the first layer can be obtained by these general formulas.

[0115] For each node, steps 401 to 403 are performed to obtain the output feature vectors (node ​​feature vectors) of all nodes in the first layer of the graph neural network, and the output feature vectors (node ​​feature vectors) of all nodes in the first layer of the graph neural network are used as the output feature vectors of the first layer. For example, the dimension of the output feature vector of the first layer is n*256, where n is the number of nodes.

[0116] 303. Use the output feature vector as the input feature vector of the next layer of the graph neural network, and calculate the output feature vector of each layer of the graph neural network in the same way as the first layer of the graph neural network until the output feature vector of the last layer of the graph neural network is obtained, and use the output feature vector of the last layer as the graph coding feature corresponding to the graph structure data.

[0117] In this way, each layer of Att-GGNN is calculated according to the calculation method corresponding to the first layer above to obtain the graph coding feature corresponding to the graph structure data, and the dimension of the graph coding feature is n*256. It should be noted that the Att-GGNN model in the embodiment of the present application can improve the effectiveness of the feature representation of the graph structure data.

[0118] Since the graph coding feature includes the feature vector of the node's own information and is also combined with the feature vector of the neighboring nodes, the obtained graph coding feature can reflect the graph coding feature corresponding to the corresponding association relationship in the graph structure data, that is, the graph coding feature can reflect the nodes in the graph structure data and the association relationship between the nodes, retain the structured information in the object to be identified, and improve the accuracy of identifying the object to be identified.

[0119] 105, fusing the text encoding features and the image encoding features to determine the recognition result corresponding to the object to be recognized.

[0120] After obtaining the text encoding features and image encoding features of the object to be identified, in order to prevent the obtained encoding features from being lost, the text encoding features and image encoding features are fused to improve the accuracy of identifying the object to be identified.

[0121] In one embodiment, the above step 105 includes: based on the text encoding features, the text encoding features and the image encoding features are correlated and fused to obtain new text encoding features including the image encoding features; the new text encoding features and the image encoding features are aggregated to determine the recognition result corresponding to the object to be identified.

[0122] In one embodiment, the above-mentioned step of fusing the text encoding features with the graph encoding features based on the text encoding features to obtain new text encoding features including the graph encoding features includes: calculating the correlation between the text encoding features and the graph encoding features; fusing the text encoding features according to the correlation to obtain new text encoding features including the graph encoding features.

[0123] Since the new text encoding features are obtained by performing correlation fusion processing on the text encoding features and the graph encoding features, the obtained new text encoding features include the corresponding graph encoding features. At the same time, the fusion processing is performed according to the correlation, which strengthens the key information and weakens the redundant information. The expression of the key information in the new text encoding features is enhanced, while the expression of the redundant information is weakened.

[0124] The correlation between the text encoding feature and the image encoding feature can be calculated in a variety of ways. In one embodiment, Figure 7 As shown, calculating the correlation between the text encoding feature and the image encoding feature can be completed through the following steps 501 to 503.

[0125] 501, calculating the similarity value between the text encoding feature and the image encoding feature.

[0126] The matrix corresponding to the text encoding feature and the matrix corresponding to the image encoding feature may be multiplied to calculate a similarity value between the text encoding feature and the image encoding feature, and the similarity value corresponds to a matrix.

[0127] 502, normalize the similarity value to obtain a normalized similarity value.

[0128] The matrix corresponding to the similarity value is normalized to obtain a normalized similarity value.

[0129] 503, weighting the graph encoding features using the normalized similarity value to determine the correlation between the text encoding features and the graph encoding features.

[0130] The correlation between text encoding features and graph encoding features can be determined by the following formula (8).

[0131]

[0132] The text encoding feature is represented by M, and the corresponding dimension is w*h, where w is the length of the text corresponding to the text description data, and h is the dimension of the output feature vector, such as 256. The graph encoding feature is represented by G, and the corresponding dimension is n*h, where n is the number of nodes and h is the output feature vector of each node. Q , W K , W V are learnable parameters, Represents the feature normalization operation, and H is the dimension of the corresponding feature vector. Q ×GW KThe dimension of the obtained feature vector is w*n, and the dimension of the feature vector obtained after normalization is also w*n. The normalized feature vector is processed by softmax and then weighted with the graph encoding feature to obtain the correlation between the text encoding feature and the graph encoding feature. The correlation is represented by the matrix MG, and the dimension of MG is w*h. The correlation represents the vector representation of the graph structure data from the perspective of text description data.

[0133] After determining the correlation between the text encoding feature and the image encoding feature, step 504 is further performed.

[0134] 504 , fusing the text encoding features according to the correlation to obtain new text encoding features including the graph encoding features.

[0135] Specifically, the text encoding feature M is concatenated with the matrix MG corresponding to the correlation to obtain a new text encoding feature including the graph encoding feature. Specifically, it can be determined according to formula (9).

[0136] M=concat(M,MG) (9)

[0137] The new text encoding feature is also represented by M, and its corresponding dimension is w*2h. The new text encoding feature contains part of the information of the graph nodes, integrates the graph encoding features, and performs fusion processing based on correlation, which enhances the expression of key information in the new text encoding feature and weakens the expression of redundant information.

[0138] The above process of realizing the correlation MG between text encoding features and graph encoding features can be completed through the CrossAttention module. Figure 8 shown.

[0139] In one embodiment, if Figure 7 As shown, the above step of aggregating the new text encoding features and the image encoding features to determine the recognition result corresponding to the object to be recognized includes steps 505 to 507.

[0140] 505, performing a first feature aggregation process on the new text encoding features to obtain a text description data vector.

[0141] Specifically, the new text encoding features are subjected to the first feature aggregation process through the self-attention network layer corresponding to the Self Attention mechanism to obtain a text description data vector. The text description data vector can be obtained by the following formula (10).

[0142] V M =Sum(softmax(W M ×M)×M) (10)

[0143] Among them, W M is a learnable parameter, M is the new text encoding feature, the dimension is w*2h, and the softmax function implements W M The output of ×M is mapped to values ​​of (0,1), and the sum of these values ​​is 1. Sum is the summation function, and the dimension of the final text description data vector is 1*h.

[0144] 506 , performing a second feature aggregation process on the graph encoding features to obtain a structured data vector.

[0145] Specifically, the graph encoding features are subjected to the second feature aggregation process through the self-attention network layer corresponding to the Self Attention mechanism to obtain a structured data vector. The structured data vector can be obtained by the following formula (11).

[0146] V G =Sum(softmax(W G ×G)×G) (11)

[0147] Among them, W G is a learnable parameter, G is the graph encoding feature with a dimension of n*h, and the softmax function implements W G The output of ×G is mapped to values ​​of (0,1), and the sum of these values ​​is 1. Sum is the summation function, and the dimension of the final structured data vector is 1*h.

[0148] 507 , determining a recognition result corresponding to the object to be recognized according to the text description data vector and the structured data vector.

[0149] In one embodiment, the above step 507 includes: performing weighted processing on the text description data vector and the structured data vector to obtain a recognition vector corresponding to the object to be recognized; and determining a recognition result corresponding to the object to be recognized according to the recognition vector. Specifically, based on a gating mechanism, weighted processing is performed on the text description data vector and the structured data vector.

[0150] In one embodiment, the processing layer Gate_Layer corresponding to the gating mechanism performs weighted processing on the text description data vector and the structured data vector through the following formula (12) to obtain a recognition vector V, and the dimension of the recognition vector V is 1*h.

[0151] V=W a *V M +(1-W a )*V G (12)

[0152] After obtaining the identification vector V, the fault probability or disease probability corresponding to the object to be identified is calculated according to the identification vector V. The identification result can be obtained according to formula (13).

[0153]

[0154] The softmax function maps V to values ​​of (0,1), and the sum of these values ​​is 1 to satisfy the properties of probability. is the probability value corresponding to the i-th candidate classification. For example, the probability value of the failure probability corresponding to the i-th fault, or the probability value of the failure probability corresponding to the i-th disease, etc.

[0155] The category corresponding to the largest probability in the probability value is determined as the target category, and the target category is used as the recognition result. For example, the fault corresponding to the largest probability in the fault probability is determined as the target fault, and the disease corresponding to the largest probability in the disease probability is determined as the target disease, so as to obtain the recognition result of the object to be recognized.

[0156] The above-mentioned entire object recognition method can be combined with reference to Figure 8 The schematic diagram is shown. Finally, the recognition result corresponding to the object to be recognized is obtained. For example, the final failure of the electronic device can be obtained, and the final determined disease information can be obtained.

[0157] The above embodiment realizes the recognition of the object to be recognized based on the text description data and the graph structure data, thereby improving the recognition accuracy of the object to be recognized.

[0158] Fig. 9 and Fig.10 This is a flow chart of the object recognition method provided in the embodiment of the present application. The object recognition method involves a training process. Fig.10 For reference, the object recognition method includes the following steps.

[0159] 601, obtaining a training object set in a training target scene.

[0160] 602 , generating training text description data and training graph structure data corresponding to each training to-be-recognized object in the training object set.

[0161] 603 , performing feature encoding processing on the training text description data to obtain training text encoding features corresponding to the training text description data.

[0162] In one embodiment, a BERT model is used to perform feature encoding processing on the training text description data to obtain training text encoding features corresponding to the training text description data.

[0163] 604 , performing feature encoding processing on the training graph structure data to obtain training graph encoding features corresponding to the training graph structure data.

[0164] In one embodiment, the Att-GGNN model is used to perform feature encoding processing on the training graph structure data to obtain corresponding training graph encoding features.

[0165] 605 , fusing the training text encoding features and the training graph encoding features to determine a training recognition vector corresponding to the training object to be recognized.

[0166] The training recognition vector is a recognition vector corresponding to the training object to be recognized obtained by weighting the training text description data vector and the structured data vector. Specifically, the vector obtained according to the above formula (13) is the training recognition vector. For the process of obtaining the training recognition vector, please refer to the corresponding process of obtaining the recognition vector above, which will not be repeated here.

[0167] 606, calculating a loss function according to the training recognition vector, and adjusting the training parameters according to the loss function.

[0168] In one embodiment, the loss function of the present application is a joint loss function of Focal Loss and GHM Loss. The idea of ​​the Focal Loss loss function is to reduce the weight of easy-to-distinguish samples (i.e., samples with high confidence), increase the weight of difficult-to-distinguish samples, and force the model to focus mainly on those difficult-to-distinguish samples. In the medical field, especially in the consultation scenario, due to the lack of information and the inability of the disease labeling of the medical record to be unique, there are many difficult-to-distinguish samples and a large number of cases of labeling errors and confusion. Focal Loss will affect the stability and optimization direction of the model in this case, so GHM Loss is introduced to balance the model's attention to difficult-to-distinguish samples. GHM Loss will dynamically balance the weight of difficult-to-distinguish samples, and too difficult samples will not be given more attention because they are often confused or wrongly labeled samples. In summary, the final loss function Loss is obtained by weighted balancing Focal Loss and GHM Loss. Specifically, as shown in formula (14).

[0169] Loss = L FL +β*L GHM (14)

[0170] Where β is a hyperparameter configuration used to balance L GHM The proportion is specified according to the distribution of the data set.

[0171] According to the loss function, the training parameters are adjusted in reverse order until the loss function converges, or until the number of training rounds reaches the preset number of rounds, or other termination conditions are met, and the training is stopped. The training parameters involved can refer to all the learnable parameters involved in formulas (1) to (13). It should be noted that all the learnable parameters involved in formulas (1) to (13) are only part of the training parameters, and there are other training parameters not reflected in the formula.

[0172] It should be noted that the processing steps of the training process are the same as those of the application process. Please refer to the above process for details. The details will not be repeated here. However, the word "training" is added before all nouns in the training process to distinguish them.

[0173] In the above embodiment, for the object to be identified, in addition to constructing text description data, graph structure data is also constructed to maximize the use of features in the object to be identified under the premise that the data information corresponding to the object to be identified is limited; a gated graph neural network integrating an attention mechanism is proposed to perform feature encoding processing on the graph structure data, making full use of the association between keywords; in the Word-to-Node Attention module, the dynamic weighted initialization feature vector of each node in the graph structure data of the text encoding features corresponding to the text description data is used to initialize the initialization feature vector, paying attention to both global information and local information; in the Cross Attention module, the interactive attention mechanism between the text encoding features and the graph structure encoding features is used to strengthen key information and prevent the loss of original information. Finally, the two interactive features are integrated to improve the accuracy of identifying the object to be identified.

[0174] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0175] In order to better implement the object recognition method of the embodiment of the present application, the embodiment of the present application also provides an object recognition device. Please refer to 11, Fig.11 The object recognition device 700 may include an acquisition module 701 , a generation module 702 , a first encoding module 703 , a second encoding module 704 and a fusion processing module 705 .

[0176] The acquisition module 701 is used to acquire the object to be identified in the target scene.

[0177] The generation module 702 is used to generate text description data and graph structure data corresponding to the object to be identified. The graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and the association relationship between the keywords.

[0178] In one embodiment, when the generation module 702 generates the generated graph structure data corresponding to the object to be identified, it specifically performs the following steps: extracting multiple keywords from the object to be identified; identifying the association relationship between the keywords; taking each keyword as a node and the association relationship between the keywords as the edge corresponding to the node, so as to generate the graph structure data corresponding to the object to be identified, wherein the graph structure data includes a node set corresponding to the keywords and a relationship set corresponding to the association relationship between the keywords.

[0179] The first encoding module 703 is used to perform feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data.

[0180] The second encoding module 704 is used to perform feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data.

[0181] The fusion processing module 705 is used to fuse the text encoding feature and the image encoding feature to determine the recognition result corresponding to the object to be recognized.

[0182] In one embodiment, the fusion processing module 705 includes a correlation fusion unit and an aggregation processing unit. The correlation fusion unit is used to perform correlation fusion processing on the text encoding feature and the image encoding feature based on the text encoding feature to obtain a new text encoding feature including the image encoding feature. The aggregation processing unit is used to perform aggregation processing on the new text encoding feature and the image encoding feature to determine the recognition result corresponding to the object to be recognized.

[0183] In one embodiment, the correlation fusion unit is specifically used to calculate the correlation between the text encoding feature and the image encoding feature based on the text encoding feature; and fuse the text encoding feature according to the correlation to obtain a new text encoding feature including the image encoding feature.

[0184] In one embodiment, the correlation fusion unit, when executing the step of calculating the correlation between the text encoding feature and the image encoding feature based on the text encoding feature, specifically performs: calculating the similarity value between the text encoding feature and the image encoding feature; normalizing the similarity value to obtain a normalized similarity value; and weighting the image encoding feature using the normalized similarity value to determine the correlation between the text encoding feature and the image encoding feature.

[0185] In one embodiment, the correlation fusion unit, when executing the step of fusing the text encoding features according to the correlation to obtain new text encoding features including graph encoding features, specifically performs: concatenating the correlation matrix corresponding to the correlation and the matrix corresponding to the text encoding features to obtain new text encoding features including graph encoding features.

[0186] In one embodiment, the aggregation processing unit is specifically used to perform a first feature aggregation processing on the new text encoding feature to obtain a text description data vector; perform a second feature aggregation processing on the graph encoding feature to obtain a structured data vector; and determine the recognition result corresponding to the object to be identified based on the text description data vector and the structured data vector.

[0187] In one embodiment, the aggregation processing unit, when executing the step of determining the recognition result corresponding to the object to be recognized based on the text description data vector and the structured data vector, specifically performs: weighted processing on the text description data vector and the structured data vector to obtain the recognition vector corresponding to the object to be recognized; and determining the recognition result corresponding to the object to be recognized based on the recognition vector.

[0188] In one embodiment, the graph structure data includes a node set corresponding to a keyword, each node corresponds to a keyword, and the second encoding module 704 includes an initialization unit and a graph feature encoding unit. The initialization unit is used to initialize the vector of each node in the node set according to the text encoding feature to obtain an initialized feature vector of each node. The graph feature encoding unit is used to input the initialized feature vector into the graph neural network for feature encoding processing to obtain the graph encoding feature corresponding to the graph structure data.

[0189] In one embodiment, the initialization unit is specifically used to calculate, for each node, an initial attention weight coefficient of each text in the node relative to the text encoding feature; and determine the initialization feature vector of the node based on the initial attention weight coefficient and the text encoding feature corresponding to the node.

[0190] In one embodiment, the text encoding features include text encoding features corresponding to the main complaint, and the initialization unit, when executing the step of calculating, for each node, an initial attention weight coefficient of each text in the node relative to the text encoding features, specifically performs: obtaining the text encoding features corresponding to the main complaint in the text encoding features as the overall representation encoding features of the text encoding features; for each node, calculating the initial attention weight coefficient of each text in the node relative to the overall representation encoding features.

[0191] In one embodiment, a graph feature encoding unit is specifically used to determine, for each node, the total neighbor nodes, in-degree neighbor nodes, and out-degree neighbor nodes that are associated with the node; determine the node feature vector of the node in the first layer of the graph neural network based on the initialization feature vector, the total neighbor nodes, in-degree neighbor nodes, and out-degree neighbor nodes of the node, in combination with the attention mechanism, and use the node feature vectors of all nodes in the first layer of the graph neural network as the output feature vector of the first layer; use the output feature vector as the input feature vector of the next layer of the graph neural network, and calculate the output feature vector of each layer of the graph neural network in the manner of the first layer of the graph neural network until the output feature vector of the last layer of the graph neural network is obtained, and use the output feature vector of the last layer as the graph encoding feature corresponding to the graph structure data.

[0192] In one embodiment, the graph feature encoding unit, when executing the step of determining the node feature vector of the node in the first layer of the graph neural network according to the initialization feature vector, the total neighbor nodes, in-degree neighbor nodes and out-degree neighbor nodes of the node in combination with the attention mechanism, specifically performs: based on the attention mechanism, determining the attention weight coefficient corresponding to each neighbor node in the total neighbor nodes to the node according to the initialization feature vector; determining the in-degree feature vector corresponding to the in-degree neighbor nodes of the node and the out-degree feature vector corresponding to the out-degree neighbor nodes of the node according to the attention weight coefficient; determining the node feature vector of the node in the first layer of the graph neural network according to the initialization feature vector, the in-degree feature vector and the out-degree feature vector corresponding to the node.

[0193] In one embodiment, the object recognition device 700 may further include a training module 706. The training module 706 is specifically used to obtain a training object set in a target scene; generate training text description data and training graph structure data corresponding to each training object to be recognized in the training object set; perform feature encoding processing on the training text description data to obtain training text encoding features corresponding to the training text description data; perform feature encoding processing on the training graph structure data to obtain training graph encoding features corresponding to the training graph structure data; perform fusion processing on the training text encoding features and the training graph encoding features to determine a training recognition vector corresponding to the training object to be recognized; calculate a loss function based on the training recognition vector, and adjust training parameters based on the loss function.

[0194] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0195] Accordingly, the embodiment of the present application also provides a computer device, which may be a terminal or a server. Fig.12 As shown, Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 800 includes a processor 801 having one or more processing cores, a memory 802 having one or more computer-readable storage media, and a computer program stored in the memory 802 and executable on the processor. The processor 801 is electrically connected to the memory 802. It will be understood by those skilled in the art that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0196] The processor 801 is the control center of the computer device 800. It uses various interfaces and lines to connect the various parts of the entire computer device 800, and executes various functions of the computer device 800 and processes data by running or loading software programs (computer programs) and / or modules stored in the memory 802, and calling data stored in the memory 802, thereby monitoring the computer device 800 as a whole.

[0197] In the embodiment of the present application, the processor 801 in the computer device 800 will load instructions corresponding to the processes of one or more application programs into the memory 802 according to the following steps, and the processor 801 will run the application programs stored in the memory 802 to implement various functions:

[0198] Acquire an object to be identified in a target scene; generate text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and the association relationship between the keywords; perform feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data; perform feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data; and fuse the text encoding features and the graph encoding features to determine a recognition result corresponding to the object to be identified.

[0199] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0200] Optional, such as Fig.12As shown, the computer device 800 further includes: a touch screen 803, a radio frequency circuit 804, an audio circuit 805, an input unit 806, and a power supply 807. The processor 801 is electrically connected to the touch screen 803, the radio frequency circuit 804, the audio circuit 805, the input unit 806, and the power supply 807, respectively. Those skilled in the art can understand that Fig.12 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0201] The touch display screen 803 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 803 can include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the computer device, and these graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel can be used to collect the user's touch operation on or near it (such as the user uses any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. The touch panel can cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor 801 to determine the type of the touch event, and then the processor 801 provides corresponding visual output on the display panel according to the type of the touch event. In the embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 803 to realize the input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 803 can also be used as a part of the input unit 806 to realize the input function.

[0202] In the embodiment of the present application, the touch display screen 803 is used to present a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface.

[0203] The radio frequency circuit 804 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other computer device through wireless communication, and to send and receive signals between the network device or other computer device.

[0204] The audio circuit 805 can be used to provide an audio interface between the user and the computer device through a speaker and a microphone. The audio circuit 805 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 805 and converted into audio data, and then the audio data is output to the processor 801 for processing, and then sent to another computer device through the radio frequency circuit 804, or the audio data is output to the memory 802 for further processing. The audio circuit 805 may also include an earphone jack to provide communication between an external headset and the computer device.

[0205] The input unit 806 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0206] The power supply 807 is used to supply power to various components of the computer device 800. Optionally, the power supply 807 can be logically connected to the processor 801 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 807 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0207] although Fig.12 Not shown, the computer device 800 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0208] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0209] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0210] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any object recognition method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0211] Acquire an object to be identified in a target scene; generate text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and the association relationship between the keywords; perform feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data; perform feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data; and fuse the text encoding features and the graph encoding features to determine a recognition result corresponding to the object to be identified.

[0212] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0213] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0214] Since the computer program stored in the storage medium can execute the steps in any object recognition method provided in the embodiments of the present application, the beneficial effects that can be achieved by any object recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0215] The above is a detailed introduction to an object recognition method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An object recognition method, characterized in that: include: Obtain the object to be identified in the target scene; The object to be identified includes at least one or more of text information, voice information, picture information and video information; Generate text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and associations between the keywords; Performing feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data; Performing feature coding processing on the graph structure data to obtain graph coding features corresponding to the graph structure data; Calculating the similarity value between the text encoding feature and the image encoding feature; Normalizing the similarity value to obtain a normalized similarity value; The graph encoding feature is weighted by using a normalized similarity value to determine the correlation between the text encoding feature and the graph encoding feature; a correlation matrix corresponding to the correlation is calculated based on the text encoding feature, the graph encoding feature, a learnable parameter, and a feature vector dimension; Concatenating the correlation matrix corresponding to the correlation and the matrix corresponding to the text encoding feature to obtain a new text encoding feature including the graph encoding feature; The new text encoding feature and the image encoding feature are aggregated to determine a recognition result corresponding to the object to be recognized.

2. The object recognition method according to claim 1, characterized in that: The step of aggregating the new text encoding feature and the image encoding feature to determine the recognition result corresponding to the object to be recognized includes: Performing a first feature aggregation process on the new text encoding feature to obtain a text description data vector; Performing a second feature aggregation process on the graph encoding features to obtain a structured data vector; A recognition result corresponding to the object to be recognized is determined according to the text description data vector and the structured data vector.

3. The object recognition method according to claim 2, characterized in that: The step of determining the recognition result corresponding to the object to be recognized according to the text description data vector and the structured data vector comprises: Performing weighted processing on the text description data vector and the structured data vector to obtain a recognition vector corresponding to the object to be recognized; A recognition result corresponding to the object to be recognized is determined according to the recognition vector.

4. The object recognition method according to claim 1, characterized in that: The graph structure data includes a set of nodes corresponding to keywords, each node corresponds to a keyword, and the step of performing feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data includes: Initializing a vector of each node in the node set according to the text encoding feature to obtain an initialized feature vector of each node; The initialized feature vector is input into the graph neural network for feature encoding processing to obtain the graph encoding feature corresponding to the graph structure data.

5. The object recognition method according to claim 4, characterized in that: The step of performing vector initialization on each node in the node set according to the text encoding feature to obtain an initialized feature vector of each node comprises: For each node, calculating an initial attention weight coefficient of each text in the node relative to the text encoding feature; An initialization feature vector of the node is determined according to the initial attention weight coefficient and the text encoding feature corresponding to the node.

6. The object recognition method according to claim 5, characterized in that: The text encoding feature includes a text encoding feature corresponding to the main complaint, and the step of calculating, for each node, an initial attention weight coefficient of each text in the node relative to the text encoding feature includes: Obtaining the text coding features corresponding to the main complaint in the text coding features as the overall representation coding features of the text coding features; For each node, the initial attention weight coefficient of each text in the node relative to the overall representation encoding feature is calculated.

7. The object recognition method according to claim 6, characterized in that: The step of inputting the initialized feature vector into the graph neural network for feature encoding processing to obtain the graph encoding feature corresponding to the graph structure data includes: For each node, determine the total neighbor nodes, in-degree neighbor nodes, and out-degree neighbor nodes associated with the node; According to the initialized feature vector, the total neighbor nodes, in-degree neighbor nodes and out-degree neighbor nodes of the node, in combination with the attention mechanism, determine the node feature vector of the node in the first layer of the graph neural network, and use the node feature vectors of all nodes in the first layer of the graph neural network as the output feature vector of the first layer; The output feature vector is used as the input feature vector of the next layer of the graph neural network, and the output feature vector of each layer of the graph neural network is calculated in the same way as the first layer of the graph neural network until the output feature vector of the last layer of the graph neural network is obtained, and the output feature vector of the last layer is used as the graph coding feature corresponding to the graph structure data.

8. The object recognition method according to claim 7, characterized in that: The step of determining the node feature vector of the node in the first layer of the graph neural network according to the initialized feature vector, the total neighbor nodes, in-degree neighbor nodes and out-degree neighbor nodes of the node, in combination with the attention mechanism, comprises: Based on the attention mechanism, determining the attention weight coefficient corresponding to each neighbor node in the total neighbor nodes to the node according to the initialized feature vector; According to the attention weight coefficient, determining an in-degree feature vector corresponding to an in-degree neighbor node of the node and an out-degree feature vector corresponding to an out-degree neighbor node of the node; The node feature vector of the node in the first layer of the graph neural network is determined according to the initialization feature vector, the in-degree feature vector and the out-degree feature vector corresponding to the node.

9. The object recognition method according to claim 1, characterized in that: The step of generating graph structure data corresponding to the object to be identified includes: Extracting multiple keywords from the object to be identified; Identify the relationship between keywords; Each keyword is taken as a node, and the association relationship between the keywords is taken as the edge corresponding to the node, so as to generate graph structure data corresponding to the object to be identified, wherein the graph structure data includes a node set corresponding to the keyword and a relationship set corresponding to the association relationship between the keywords.

10. The object recognition method according to claim 1, characterized in that: Also includes: Obtain a set of training objects in the target scene; Generate training text description data and training graph structure data corresponding to each training to-be-recognized object in the training object set; Performing feature encoding processing on the training text description data to obtain training text encoding features corresponding to the training text description data; Performing feature encoding processing on the training graph structure data to obtain training graph encoding features corresponding to the training graph structure data; The training text encoding feature and the training image encoding feature are fused to determine the training recognition vector corresponding to the training object to be recognized; A loss function is calculated based on the training recognition vector, and training parameters are adjusted based on the loss function.

11. An object recognition device, characterized in that: include: An acquisition module is used to acquire the object to be identified in the target scene; The object to be identified includes at least one or more of text information, voice information, picture information and video information; A generating module, used to generate text description data and graph structure data corresponding to the object to be identified, wherein the graph structure data is data formed based on a graph structure, and the graph structure data includes keywords in the object to be identified and associations between the keywords; A first encoding module, used for performing feature encoding processing on the text description data to obtain text encoding features corresponding to the text description data; A second encoding module, used for performing feature encoding processing on the graph structure data to obtain graph encoding features corresponding to the graph structure data; A fusion processing module, used for calculating the similarity value between the text encoding feature and the image encoding feature; normalizing the similarity value to obtain a normalized similarity value; weighting the image encoding feature using the normalized similarity value to determine the correlation between the text encoding feature and the image encoding feature; the correlation matrix corresponding to the correlation is calculated based on the text encoding feature, the image encoding feature, the learnable parameter and the feature vector dimension; splicing the correlation matrix corresponding to the correlation and the matrix corresponding to the text encoding feature to obtain a new text encoding feature including the image encoding feature; The new text encoding feature and the image encoding feature are aggregated to determine a recognition result corresponding to the object to be recognized.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the object recognition method according to any one of claims 1 to 10.

13. A computer device, characterized in that: The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps in the object recognition method according to any one of claims 1 to 10 by calling the computer program stored in the memory.

Images