Chinese relation extraction method and system based on multi-modal semantic fusion

Abstract

The application provides a Chinese relation extraction method and system based on multi-modal semantic fusion, and relates to the technical field of information extraction. The method comprises: obtaining a Chinese sentence and entities corresponding to the Chinese sentence; extracting text semantics, shape semantics and structure semantics of each Chinese character in the Chinese sentence; constructing a multi-modal semantic fusion model through an improved Transformer network, encoding the shape semantics and the structure semantics respectively, splicing the encoded semantic features to obtain auxiliary features, taking the text semantics as main features, optimizing the feature distribution of the main features according to the correlation coefficient between the main features and the auxiliary features, and then obtaining the fused multi-modal semantic features; and determining the Chinese relation between the entities according to the multi-modal semantic features. In this way, the shape semantics and the structure semantics of the Chinese characters are used to enrich the context information of the Chinese sentence, which can reduce the influence of Chinese ambiguity in Chinese relation extraction and improve the Chinese relation extraction effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information extraction, and particularly relates to a Chinese relation extraction method and system based on multi-modal semantic fusion. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art known to those skilled in the art.

[0003] With the development of mobile Internet, smart phones and social networks, there are massive Chinese text information. It is a time-consuming and laborious task to extract useful knowledge from massive Chinese text, so an effective Chinese relation extraction method is particularly important.

[0004] The traditional Chinese relation extraction adopts a pattern matching method, which is not efficient and time-consuming. Then, relation extraction methods represented by machine learning and deep learning technology appear, such as support vector machine, convolutional neural network, recurrent neural network, graph neural network and attention mechanism. However, due to the complexity of Chinese structure and ambiguity, the same entity has different relationship information in different contexts, so the existing Chinese relation extraction method is greatly affected by Chinese ambiguity, and it is difficult to accurately extract the relationship between entities. In particular, Chinese characters evolve from pictographic characters, and the pictographic structure itself contains part of the semantic information, and the existing relation extraction method does not fully use the pictographic semantic of Chinese characters to assist in extracting the relationship between entities. SUMMARY

[0005] In order to solve the above problems, the present application provides a Chinese relation extraction method and system based on multi-modal semantic fusion, which extracts the shape semantics and structure semantics of Chinese sentences, and fuses them with the text semantics for Chinese relation extraction, to enrich the context information of Chinese sentences and improve the Chinese relation extraction effect.

[0006] In order to achieve the above purpose, the present application mainly includes the following aspects:

[0007] In a first aspect, the present application provides a Chinese relation extraction method based on multi-modal semantic fusion, comprising:

[0008] obtaining a Chinese sentence and an entity corresponding to the Chinese sentence;

[0009] extracting the text semantics, shape semantics and structure semantics of each Chinese character in the Chinese sentence;

[0010] The multi-modal semantic fusion model is constructed by an improved Transformer network, shape semantics and structure semantics are encoded respectively, the encoded semantic features are spliced to obtain auxiliary features, and text semantics are taken as main features, the feature distribution of the main features is optimized according to the correlation coefficient between the main features and the auxiliary features, and then the fused multi-modal semantic features are obtained.

[0011] According to the multi-modal semantic features, the Chinese relationship between entities is determined.

[0012] In a possible implementation, the text semantics of each Chinese character in the Chinese sentence are extracted by: processing the Chinese sentence to obtain a Chinese character vector; calculating the relative distance between each Chinese character and an entity to obtain a position vector; and splicing the Chinese character vector and the position vector and inputting them into an MG-Lattice model to extract the text semantics.

[0013] In a possible implementation, when extracting the text semantics, the MG-Lattice model also takes the Chinese character as a keyword to search for a related word, and obtains a word vector of the related word by searching a pre-trained word vector table; and the word vector and the Chinese character vector are input into the MG-Lattice model to extract the text semantics.

[0014] In a possible implementation, a plurality of font images of each Chinese character in the Chinese sentence are obtained, and the shape semantics and the structure semantics of each Chinese character in the Chinese sentence are extracted according to the plurality of font images.

[0015] In a possible implementation, each font image of a Chinese character is regarded as a channel, each Chinese character has a multi-channel image representation, image features are extracted through a convolutional neural network, and the image features are input into a bidirectional long short-term memory model to extract shape semantics.

[0016] In a possible implementation, for each Chinese character, each font image is input into a main neighborhood aggregation network to read out a font single vector, the font single vectors are spliced to obtain a single vector representation corresponding to the Chinese character, and the single vector representations are input into a bidirectional long short-term memory model in the order of Chinese characters in the Chinese sentence to extract structure semantics.

[0017] In a possible implementation, before the shape semantics and the structure semantics are encoded, the following steps are further included: calculating a correlation probability of the shape semantics, and encoding the shape semantics when the correlation probability of the shape semantics is not less than a preset threshold; or calculating a correlation probability of the structure semantics, and encoding the structure semantics when the correlation probability of the structure semantics is not less than a preset threshold.

[0018] In a second aspect, the embodiments of the present application provide a Chinese relation extraction system based on multi-modal semantic fusion, comprising:

[0019] An acquisition module is configured to acquire a Chinese sentence and entities corresponding to the Chinese sentence.

[0020] A semantic extraction module is configured to extract text semantics, shape semantics and structure semantics of each Chinese character in the Chinese sentence.

[0021] A feature fusion module is configured to construct a multi-modal semantic fusion model through an improved Transformer network, encode the shape semantics and the structure semantics respectively, splice the encoded semantic features to obtain auxiliary features, take the text semantics as main features, optimize the feature distribution of the main features according to the correlation coefficient between the main features and the auxiliary features, and then obtain fused multi-modal semantic features.

[0022] A Chinese relation determination module is configured to determine Chinese relations between the entities according to the multi-modal semantic features.

[0023] In a third aspect, the embodiments of the present application provide a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the Chinese relation extraction method based on multi-modal semantic fusion as described in the first aspect and any possible implementation manner of the first aspect.

[0024] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the Chinese relation extraction method based on multi-modal semantic fusion as described in the first aspect and any possible implementation manner of the first aspect.

[0025] This application provides a method and system for extracting Chinese relations based on multimodal semantic fusion. The method extracts the textual semantics, shape semantics, and structural semantics of each Chinese character in a Chinese sentence. A multimodal semantic fusion model is constructed using an improved Transformer network. The shape and structural semantics are encoded separately, and the encoded shape and structural semantics are concatenated to obtain auxiliary features. Textual semantics are used as the primary feature. Based on the correlation coefficient between the primary and auxiliary features, the feature distribution of the primary feature is optimized, resulting in fused multimodal semantic features. Based on these multimodal semantic features, the Chinese relations between entities are determined. Thus, by utilizing Chinese pictographic semantics (shape and structural semantics) to enrich the contextual information of Chinese sentences, the influence of Chinese ambiguity can be reduced in Chinese relation extraction, improving the extraction efficiency. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is one of the flowcharts illustrating the Chinese relation extraction method based on multimodal semantic fusion provided in the embodiments of this application;

[0028] Figure 2 This is the second flowchart of the Chinese relation extraction method based on multimodal semantic fusion provided in the embodiments of the present invention;

[0029] Figure 3 This is a structural diagram of the main and auxiliary fusion module provided in the embodiments of this application;

[0030] Figure 4 This is an example diagram of Chinese character processing provided in the embodiments of this application;

[0031] Figure 5 This is a structural diagram of the shape semantic extraction module provided in the embodiments of this application;

[0032] Figure 6 This is a structural diagram of the structural semantic extraction module provided in the embodiments of this application;

[0033] Figure 7 This is a schematic diagram of the structure of the Chinese relation extraction system based on multimodal semantic fusion provided in the embodiments of this application;

[0034] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application pertains.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] The current Chinese relation extraction methods are greatly affected by Chinese ambiguity, and the existing Chinese relation extraction methods ignore the utilization of Chinese pictographic semantics. Secondly, there is a lack of effective multimodal semantic fusion methods to fuse pictographic semantics and text semantics. This embodiment provides a Chinese relation extraction method and system based on multimodal semantic fusion, which enriches the context information of Chinese sentences and improves the Chinese relation extraction effect by extracting the shape semantics and structural semantics of Chinese sentences and fusing them with text semantics for Chinese relation extraction.

[0039] Please refer to Figure 1 and Figure 2 , as Figure 1 shown in

[0040] S101: Obtain a Chinese sentence and the entities corresponding to the Chinese sentence.

[0041] In a specific implementation, given a Chinese sentence Sen = {sen1,..., sen n}(sen i represents a Chinese character) and the corresponding entities, where n is the number of Chinese characters. For example, for the Chinese sentence "Beijing belongs to China", the corresponding entities are "Beijing" and "China", and "belongs to" is the relationship between the two entities.

[0042] S102: Extract the text semantics, shape semantics, and structural semantics of each Chinese character in the Chinese sentence.

[0043] In a specific implementation, the text semantics, shape semantics, and structural semantics corresponding to each Chinese character are respectively extracted from the Chinese sentence. As Figure 2As shown, the MG-Lattice model is used to extract the textual semantics of Chinese sentences, the shape semantics extraction module is used to extract the shape semantics of Chinese sentences, and the structure semantics extraction module is used to extract the structure semantics of Chinese sentences.

[0044] S103: A multimodal semantic fusion model is constructed by improving the Transformer network. Shape semantics and structural semantics are encoded separately. The encoded semantic features are concatenated to obtain auxiliary features. Text semantics are used as the main features. Based on the correlation coefficient between the main features and auxiliary features, the feature distribution of the main features is optimized, and thus the fused multimodal semantic features are obtained.

[0045] In practical implementation, based on heuristic thinking, text semantics considers pre-training information during semantic extraction, containing more semantic features compared to shape and structure semantics. Therefore, text semantics is used as the primary feature, with shape and structure semantics as auxiliary features. During fusion, the feature distribution of text semantics is optimized using the auxiliary features. Currently, Transformer networks have demonstrated powerful capabilities in learning semantic distributions across many tasks. In this embodiment, the Transformer network is introduced and improved to achieve multimodal semantic feature fusion. For example... Figure 3 As shown, shape semantics and structural semantics are encoded separately. The encoded semantic features are concatenated to obtain auxiliary features, and text semantics are used as the main features. Based on the correlation coefficient between the main features and auxiliary features, the feature distribution of the main features is optimized, and thus the fused multimodal semantic features are obtained.

[0046] S104: Determine the Chinese relationships between entities based on the multimodal semantic features.

[0047] In practice, the Chinese relationships between entities are determined based on the fused multimodal semantic features. This approach, utilizing Chinese pictographic semantics (shape semantics and structural semantics) to enrich the contextual information of Chinese sentences, can reduce the impact of Chinese ambiguity in relation extraction and improve its effectiveness.

[0048] In this embodiment of the application, as an optional implementation, the textual semantics of each Chinese character in a Chinese sentence are extracted in the following way: the Chinese sentence is processed to obtain a Chinese character vector; the relative distance between each Chinese character and the entity is calculated to obtain a position vector; the Chinese character vector and the position vector are concatenated and input into the MG-Lattice model to extract the textual semantics.

[0049] In a specific implementation, in order to obtain multi-granularity word embedding, the embodiment adopts both character set embedding and word embedding. Firstly, pre-trained word vectors are read, and then word2vec is used to obtain Chinese character vectors mapped by each Chinese character. In addition, the position information between Chinese characters and entities is also considered, so as to better represent the semantic representation of Chinese. For example, given two entities e1 and e2. The relative distance between e1 and e2 is obtained by the following formula and

[0050]

[0051] wherein, and are the start position and end position of entity e1. The method for calculating is the same as that for calculating . Then, a layer of multi-layer perception (MLP) is used to map the position and into a d p -dimensional vector. The Chinese character vector and the position vector are spliced and input into the MG-Lattice model to extract text semantic features. In order to reduce the influence of Chinese ambiguity on Chinese relation extraction, the MG-Lattice model also uses Chinese characters as keywords when extracting text semantics, searches all related words in HowNet, and then obtains word vectors by searching a pre-trained word vector table. Then, the multi-granularity word vector and the Chinese character vector are input into the MG-Lattice model to extract character-level semantic information. In addition, the MG-Lattice model also uses an attention mechanism to aggregate the semantics of Chinese ambiguous words during information propagation. Here, the following formula is used to define the semantic extraction process of the text semantic extraction module:

[0052] T = TextSemanticsExtrationModule (Sen);

[0053] wherein, the text semantic

[0054] After thousands of years of evolution, Chinese characters have evolved from pure pictographic characters to simplified characters, and each change in Chinese characters inevitably loses some pictographic information to some extent. The pictographic semantics contained in different fonts of Chinese characters are not the same, especially the fonts similar to oracle bone script, which contain more specific pictographic semantics. Therefore, as an optional implementation, multiple font images of each Chinese character in the Chinese sentence are obtained, and the shape semantics and structure semantics of each Chinese character in the Chinese sentence are extracted according to the multiple font images.

[0055] In a specific implementation, six Chinese character fonts (Jinwen, Lishu, Zhubu, Traditional Chinese, Simplified Chinese and Caoshu) are used to generate font images, and the generated font images are as shown in Figure 4 part (a).

[0056] Next, these font images are input into a shape semantic extraction module to extract shape semantics. Each font image of a Chinese character is regarded as a channel, and each Chinese character has a multi-channel image representation. Image features are extracted through a convolutional neural network and input into a bidirectional long short-term memory model to extract shape semantics.

[0057] In a specific implementation, as shown in Figure 5 , according to a given sentence Sen, each font image of each Chinese character obtained is regarded as a channel, so each Chinese character has a six-channel image representation. Next, image information of Chinese is extracted using CNN, and a max-pooling layer with a convolution kernel size of 2 is used to select image features. Then, two fully connected layers are used and their outputs are input into a bidirectional long short-term memory model (BiLSTM) to extract Chinese shape semantic information. The following formula defines the extraction process of this module:

[0058] I = ShapeSemanticsExtrationModule (Sen)

[0059] wherein the shape semantics

[0060] Compared with shape semantics, the pictographic information of Chinese characters exists more in their topological structures (such as the components, radicals and character formation structures of Chinese). Moreover, topological structure semantics can be understood as a kind of global semantic feature, which can better reflect the overall structural semantics of Chinese characters. However, manually marking the topological structures of Chinese characters will consume huge human and time costs. Therefore, it is very challenging to extract the structural information of Chinese characters.

[0061] In order to automatically generate the topological structure diagram of Chinese characters, a superpixel segmentation algorithm and a k-Nearest Neighbor (k-NN) classification algorithm are used to extract the structure of Chinese characters. In order to discard useless superpixel points (to prevent the connection of useless points established by edges from introducing structure noise of Chinese characters), only superpixel points with a superpixel value greater than or equal to 0.5 are retained, and the superpixel image of Chinese characters is as shown in Figure 4 part (b). Finally, according to the distance between superpixel points, k-NN is used to cluster k-neighbor superpixel points and establish connection edges, and when the weight of the generated connection edge is greater than or equal to 0.5, the connection edge is retained, and the generated structure of Chinese characters is as shown in Figure 4 part (c).

[0062] Different Chinese characters have certain similarities in topological structure, that is, part of the topological structure contains certain semantic information. For structural semantic extraction, a graph neural network is used to extract the structural semantics of Chinese characters. However, the traditional graph neural network has the disadvantage of over-smoothing, which leads to poor ability to distinguish the extracted Chinese structure. Therefore, the PNA, which is currently good at solving the problem of over-smoothing, is introduced to extract the structural features of Chinese characters. The PNA (Principal Neighbourhood Aggregation) network uses four aggregators (average aggregator, maximum aggregator, minimum aggregator and standard deviation aggregator) combined with a degree-based scaler, which can solve the problem of over-smoothing. Let d be the degree of the node, and γ be a variable parameter. The mathematical form of the degree-based scaler is defined as follows:

[0063]

[0064] In the formula, ψ(d, γ) is a one-to-one function under the condition of d > 0. The method of aggregating adjacent node information is shown in the right part of Figure 6 .

[0065] The PNA aggregates adjacent node information as follows:

[0066]

[0067] In the formula, I s represents no scaling, defines the tensor multiplication operation, defines the tensor multiplication operation. Further, the process of PNA extracting Chinese structural information is defined as follows:

[0068]

[0069] In the formula, l is the current layer of PNA, is a layer of MLP. As shown in the left part of Figure 6 , the structural differences of different Chinese characters are large. Therefore, 6 PNA are used to process 6 Chinese character font graphs respectively, and a single vector of one font is read out through the readout operation. Then, 6 readout font single vectors are spliced to obtain a single vector representation of the structural information of each Chinese character. Finally, according to the order of Chinese characters in the sentence, these single vectors are input into the BiLSTM to extract Chinese structural semantics. The extraction process of this module is defined as follows:

[0070] S = StructureSemanticsExtractionModule (Sen)

[0071] In the formula, the structure semantics

[0072] The multi-modal semantic fusion model specifically fuses the multi-modal semantic features in the following manner:

[0073] N I The shape semantics is encoded by the image encoder, and the formal definition is as shown in the following formula:

[0074]

[0075] In the formula, N I = 6, is the hidden layer output of the image encoder. Similar to the image encoder, the structure diagram encoder can be defined as follows:

[0076]

[0077] In the formula, N G = 6, is the hidden layer output of the structure diagram encoder. Then, the two kinds of features are spliced to obtain auxiliary features Here, the vector a1 is obtained by splicing and Similarly, the vector a n is obtained by splicing and .

[0078] As an optional implementation, before encoding the shape semantics and the structure semantics, the following steps are further included: calculating the correlation probability of the shape semantics, and encoding the shape semantics when the correlation probability of the shape semantics is not less than a preset threshold; or, calculating the correlation probability of the structure semantics, and encoding the structure semantics when the correlation probability of the structure semantics is not less than a preset threshold.

[0079] In specific implementation, unlike the text semantics containing rich semantic information, the shape semantics and the structure semantics contain less semantic information. Therefore, the embodiment dynamically selects the shape semantics and the structure semantics useful for subsequent fusion. In addition, in different Chinese sentences, the influence of the character shape on the relationship mainly exists in a few Chinese characters, and the pictographic information of all Chinese characters in the sentence does not promote the relationship. Therefore, it is proposed to dynamically select part of the useful pictographic semantic information as auxiliary information to optimize the distribution of text semantic information during the training of the model, and a corresponding dynamic semantic information selector is designed, as shown in the following formula:

[0080] P(c) = Sigmoid(Wx glyph +b glyph );

[0081] where x glyph represents the structural semantic vector in structural semantics and the shape semantic vector in shape semantics; b glyph represents the corresponding bias in the corresponding structural semantics and shape semantics; and W represents the weight.

[0082] The relevance probability of each Chinese character shape and the relevance probability of the structural semantics are calculated by one layer of MLP and sigmoid function. When the relevance probability is greater than or equal to 0.5, the structural or shape semantics is input to the fusion module, otherwise, it is discarded.

[0083] The decoder in the traditional Transformer network is not suitable for main-aid feature fusion. Therefore, a new decoder is designed by the main-aid attention mechanism to fuse the three modal features. First, the text semantic features are input into the decoder, and the main features Q that can be fused are obtained through multi-head self-attention and normalization processing. In order to further optimize the feature distribution of Q, fine-grained attention mechanism is adopted and fine-tuned by fusing auxiliary features. The attention matrix M is defined as follows: ij which represents the correlation degree between the i-th feature vector in Q and the j-th feature vector in A, and M ij is calculated as shown in the following formula:

[0084] M ij = QA T ;

[0085] Then, the Softmax function is used to normalize each row in M, as shown in the following formula:

[0086]

[0087] Further, the output of fine-grained attention is obtained by the following formula:

[0088] F g = MA;

[0089] wherein Next, a single-layer MLP is used to optimize the main feature distribution, and the calculation method is as follows:

[0090] F i = Tanh((F gi + Q i )W f + b f );

[0091] wherein W f and b fare trainable parameters. Finally, the fused semantic features are obtained by normalization and feed-forward network

[0092] In this way, the Transformer network based on primary and auxiliary attention can better fuse the semantic features of Chinese text and pictograph, and the design of the encoder-decoder enhances the interaction of different modal semantic features, so that the model can further extract relational semantics.

[0093] In the fusion process, the text semantic feature is regarded as the primary feature and the pictograph semantic feature is regarded as the auxiliary feature. First, two independent encoders are used to extract shape and structure semantic information respectively. Then, the Chinese pictograph semantic feature is obtained by splicing. In order to effectively fuse the text semantic feature and the pictograph semantic feature of Chinese, a primary and auxiliary attention mechanism is used in the decoder to fuse the two modal semantic features. Based on the primary and auxiliary attention mechanism, a fine-grained attention matrix is obtained through the pictograph semantic feature, and then the feature distribution of the text semantic feature is optimized through the fine-grained pictograph semantic feature, so as to improve its expression ability.

[0094] After learning the fused feature F D , the sentence context information is aggregated through the attention mechanism, and its aggregation formula is shown in the following two formulas:

[0095] α=Softmax(w T Tanh(F D ));

[0096] f D * =F D α T ;

[0097] where w is a trainable parameter and α is an attention coefficient. Finally, we perform relationship classification through an MLP and softmax, and the calculation method is defined as follows:

[0098]

[0099] where W r and b r are trainable parameters, and y is the predicted probability of each relationship type. Here, there are 9 relationship types in the data set used by the embodiment, including "Located", "Part-Whole", "Family", "Social", "Use", "Create", "General-Special", "Ownership", "Near" and "Other". The Chinese relationship between entities is determined by calculating the probability value.

[0100] In order to optimize the model parameters, the loss is calculated using the cross-entropy loss function, and the model parameters are optimized by back propagation using the Adam optimizer.

[0101] Referring to Figure 7 The embodiment of the present application also provides a Chinese relation extraction system based on multi-modal semantic fusion, the Chinese relation extraction system 700 comprises:

[0102] An acquisition module 710 is configured to acquire a Chinese sentence and entities corresponding to the Chinese sentence;

[0103] A semantic extraction module 720 is configured to extract text semantics, shape semantics and structure semantics of each Chinese character in the Chinese sentence;

[0104] A feature fusion module 730 is configured to construct a multi-modal semantic fusion model through an improved Transformer network, encode the shape semantics and the structure semantics respectively, splice the encoded semantic features to obtain auxiliary features, take the text semantics as main features, optimize the feature distribution of the main features according to the correlation coefficient between the main features and the auxiliary features, and then obtain fused multi-modal semantic features.

[0105] A Chinese relation determination module 740 is configured to determine Chinese relations between entities according to the multi-modal semantic features.

[0106] The Chinese relation extraction system based on multi-modal semantic fusion provided in the embodiment is used to implement the Chinese relation extraction method based on multi-modal semantic fusion, and therefore the specific implementation of the Chinese relation extraction system based on multi-modal semantic fusion can be seen in the embodiment part of the Chinese relation extraction method based on multi-modal semantic fusion in the foregoing, and will not be described here again.

[0107] Referring to Figure 8 , Figure 8 is a schematic diagram of a computer device according to an embodiment of the present application. As shown in Figure 8 , the computer device 800 comprises a processor 810, a memory 820 and a bus 830.

[0108] The memory 820 stores machine readable instructions executable by the processor 810, when the computer device 800 is running, the processor 810 and the memory 820 communicate through the bus 830, and the machine readable instructions are executed by the processor 810, which can execute the steps of the Chinese relation extraction method based on multi-modal semantic fusion in the method embodiment as described above. Figure 1 The specific implementation can be referred to the method embodiment, and will not be described here again.

[0109] Based on the same inventive concept, the application further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the Chinese relation extraction method based on multi-modal semantic fusion are executed.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0111] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A Chinese relation extraction method based on multi-modal semantic fusion, characterized in that, The method comprises the following steps: obtaining a Chinese sentence and entities corresponding to the Chinese sentence; extracting text semantics, shape semantics and structure semantics of each Chinese character in the Chinese sentence; building a multi-modal semantic fusion model through an improved Transformer network, encoding the shape semantics and the structure semantics respectively, splicing the encoded semantic features to obtain auxiliary features, taking the text semantics as main features, optimizing the feature distribution of the main features according to the correlation coefficient between the main features and the auxiliary features, and then obtaining fused multi-modal semantic features; determining Chinese relationships between the entities according to the multi-modal semantic features; The text semantics of each Chinese character in the Chinese sentence is extracted by processing the Chinese sentence to obtain a Chinese character vector, calculating the relative distance between each Chinese character and the entity to obtain a position vector, and splicing the Chinese character vector and the position vector and inputting them into an MG-Lattice model to extract the text semantics; When extracting the text semantics, the MG-Lattice model also takes the Chinese character as a keyword to search for related words, obtains the word vector of the related words by searching a pre-trained word vector table, and inputs the word vector and the Chinese character vector into the MG-Lattice model to extract the text semantics; obtaining multiple font images of each Chinese character in the Chinese sentence, and extracting shape semantics and structure semantics of each Chinese character in the Chinese sentence according to the multiple font images; Before encoding the shape semantics and the structure semantics, the method further comprises the following steps: calculating the correlation probability of the shape semantics, and encoding the shape semantics when the correlation probability of the shape semantics is not less than a preset threshold; Or, calculating the correlation probability of the structure semantics, and encoding the structure semantics when the correlation probability of the structure semantics is not less than a preset threshold. 2.The Chinese relation extraction method based on multi-modal semantic fusion of claim 1, wherein, Each font image of a Chinese character is regarded as a channel, each Chinese character has a multi-channel image representation, image features are extracted through a convolutional neural network, and the image features are input into a bidirectional long short-term memory model to extract shape semantics. 3.The Chinese relation extraction method based on multi-modal semantic fusion of claim 1, wherein, For each Chinese character, each font image is input into a main neighborhood aggregation network to read out a font single vector, the font single vectors are spliced to obtain a single vector representation corresponding to the Chinese character, and the single vector representation is input into a bidirectional long short-term memory model in the order of Chinese characters in the Chinese sentence to extract structure semantics.

4. A Chinese relation extraction system based on multi-modal semantic fusion, characterized in that, The method comprises the following steps: an obtaining module is configured to obtain a Chinese sentence and entities corresponding to the Chinese sentence; a semantic extraction module is configured to extract text semantics, shape semantics and structure semantics of each Chinese character in the Chinese sentence; a feature fusion module is configured to build a multi-modal semantic fusion model through an improved Transformer network, encode the shape semantics and the structure semantics respectively, splice the encoded semantic features to obtain auxiliary features, take the text semantics as main features, optimize the feature distribution of the main features according to the correlation coefficient between the main features and the auxiliary features, and then obtain fused multi-modal semantic features; A Chinese relation determining module is configured to determine Chinese relations between entities according to the multi-modal semantic features. The text semantics of each Chinese character in the Chinese sentence is extracted by: processing the Chinese sentence to obtain a Chinese character vector; calculating the relative distance between each Chinese character and an entity to obtain a position vector; and concatenating the Chinese character vector and the position vector and inputting them into an MG-Lattice model to extract the text semantics. When extracting the text semantics, the MG-Lattice model also takes the Chinese character as a keyword to search for a related word, and obtains a word vector of the related word by searching a pre-trained word vector table; and inputs the word vector and the Chinese character vector into the MG-Lattice model to extract the text semantics. A plurality of font images of each Chinese character in the Chinese sentence are obtained, and shape semantics and structure semantics of each Chinese character in the Chinese sentence are extracted according to the plurality of font images. Before encoding the shape semantics and the structure semantics, the method further includes: calculating a correlation probability of the shape semantics, and encoding the shape semantics when the correlation probability of the shape semantics is not less than a preset threshold. Alternatively, a correlation probability of the structure semantics is calculated, and the structure semantics is encoded when the correlation probability of the structure semantics is not less than a preset threshold.

5. A computer device, comprising: The method comprises: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the Chinese relation extraction method based on multi-modal semantic fusion as claimed in any one of claims 1 to 3.

6. A computer readable storage medium characterized by, The computer readable storage medium stores a computer program, the computer program is executed by the processor to execute the steps of the Chinese relation extraction method based on multi-modal semantic fusion as claimed in any one of claims 1 to 3.

Images