Entity relationship extraction method based on multi-mode administrative division data
The administrative division data is processed through the multimodal feature extraction model, which solves the problems of complexity, limited efficiency and accuracy of multimodal data processing in the existing technology, and realizes efficient and accurate entity relationship extraction.
Patent Information
- Application Number
- CN202411706253.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-11
AI Technical Summary
The existing administrative division entity identification and relationship extraction methods rely on a single data source, and are complex when processing multimodal data, with limited efficiency and accuracy.
A multimodal feature extraction model is adopted, including image feature conversion structure, semantic feature conversion structure and feature decoder, and the entity relationship extraction of administrative division multimodal data is achieved through the learning and mining of image features and semantic features.
The processing process of administrative division data is simplified, the efficiency and accuracy of multimodal data entity relationship extraction is improved, and the end-to-end extraction process is realized.
Smart Images
Figure CN119597899A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an entity relationship extraction method based on multimodal administrative division data, belonging to the technical field of artificial intelligence. Background Art
[0002] Administrative division entities usually include a large amount of geographic information and administrative division data, such as administrative division code, government seat, government website, superior administrative division, subordinate administrative division, telephone area code, postal code, administrative division level unit, position in the superior administrative region, adjacent administrative division, total area, water pattern landform, climate type, main rivers, main transportation network, total population, ethnic composition, major scenic spots and historical sites, important historical events, major industries, etc. Therefore, the data source of administrative division entities is composed of multimodal data, and administrative division information includes administrative division remote sensing images, map images, administrative division vector graphics, administrative division information text, and administrative division broadcasts or audio podcasts.
[0003] At present, the methods of entity recognition and relationship extraction of administrative divisions mainly rely on a single data source. When the data is multimodal, it needs to be manually collected and edited one by one to identify the characteristics of each type in the administrative division, and then the required structured information is extracted based on the original data and the above recognition results through the boundary positioning module, named entity classification module, error semantic correction module, etc. This relationship extraction method requires the use of multiple independent modules to extract structured information from images, texts or audios, and cannot extract structured information end-to-end. In addition, the overall architecture for extracting structured information is complex and not suitable for training, which limits the efficiency and accuracy of extraction. Summary of the invention
[0004] The purpose of the present invention is to provide an entity relationship extraction method based on multimodal administrative division data to solve the problems that the current entity relationship extraction method is complex and has limited efficiency and accuracy when extracting entity relationships from multimodal data.
[0005] In order to solve the above technical problems, the present invention provides an entity relationship extraction method based on multimodal administrative division data, and the extraction method comprises the following steps:
[0006] 1) Obtain multimodal data of the target administrative division, including administrative division image data and administrative division text data;
[0007] 2) using the constructed multimodal feature extraction model to process the acquired administrative division image data and administrative division text data to extract the corresponding entity relationship; the multimodal feature extraction model includes an image feature conversion structure, a semantic feature conversion structure and a feature decoder;
[0008] The image feature conversion structure is used to convert the acquired administrative division image data into corresponding image features, and the semantic feature conversion structure is used to convert the acquired administrative division text data into corresponding semantic features; the feature decoder is used to take the acquired image features and semantics as input, and obtain the target administrative division entity relationship information by learning and mining the image features and semantic features.
[0009] Furthermore, the feature decoder adopts a Transformer Decoder, and the Transformer Decoder adopts an autoregressive decoding operation.
[0010] Furthermore, the autoregressive decoding operation includes: when the number of decoding times is equal to 1, the previous semantic unit is the starting symbol of the feature decoding structure; based on the image features, semantic features, and the previous semantic unit in the target administrative division entity information, the next semantic unit in the target administrative division entity information is determined.
[0011] Furthermore, the semantic features input to the feature decoder also include semantic features obtained by converting cogo results identified from administrative division image data using the semantic feature conversion structure.
[0012] Furthermore, the training process adopted by the constructed multimodal feature extraction model is:
[0013] Acquire a training sample including a training image, training information text and annotated entity information; the training information text is used to indicate the training type of entity information to be extracted from the training image, and the annotated entity information is entity information of the training type in the training image;
[0014] According to the training image and its conversion result text, the target entity information is determined and the multimodal feature extraction model is trained.
[0015] Furthermore, the Transformer Decoder is trained using Teacher Forcing.
[0016] Furthermore, the process of the image feature conversion structure converting the acquired administrative division image data into corresponding image features includes:
[0017] The image feature conversion structure processes the administrative division image data to obtain feature representations corresponding to the vector graphics in the administrative division image;
[0018] A vector feature sequence is constructed based on feature representations corresponding to each of the vector graphics in the administrative division image, and the vector feature sequence is used as a relational feature corresponding to the vector graphics in the administrative division image.
[0019] Furthermore, the multimodal data acquired in step 1) further includes administrative division broadcast audio data, and the administrative division image data includes a scanned map image. The method further includes:
[0020] Obtain the optical character recognition (OCR) results of administrative division information in the scanned map image, and obtain the audio speech recognition (ASR) results of administrative division broadcast audio data;
[0021] The OCR result and the ASR result are converted into corresponding administrative division semantic features through the semantic feature conversion structure in the multimodal feature extraction model;
[0022] The target administrative division entity information in the administrative division image is determined according to the image features and the semantic features through the feature decoding structure in the multimodal feature extraction model.
[0023] The beneficial effects of the present invention are as follows: as an improved invention, the present invention adopts a multimodal feature extraction model including an image feature conversion structure, a semantic feature conversion structure and a feature decoder, wherein the image feature conversion structure can be used to convert the acquired administrative division image data into corresponding image features, the semantic feature conversion structure can be used to convert the acquired administrative division text data into corresponding semantic features, and the feature decoder can be used to take the acquired image features and semantics as input, and the image features and semantic features are studied and mined to obtain the target administrative division entity relationship information. Therefore, the multimodal feature extraction model constructed by the present invention can process administrative division image data and administrative division text data at the same time, thereby realizing the extraction of multimodal data entity relationships, and the model structure adopted by the method is simple, so as to realize both efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a flow chart of the entity relationship extraction method based on multimodal administrative division data of the present invention;
[0025] Figure 2 is a flow chart of preprocessing the acquired administrative division image data in an embodiment of the present invention;
[0026] Figure 3 It is a schematic diagram of an application scenario of the entity relationship extraction method of multimodal administrative division data according to an embodiment of the present invention;
[0027] Figure 4 is a schematic diagram of an autoregressive decoding operation performed by a feature decoder in an embodiment of the present invention;
[0028] Figure 5 Schematic diagram of the working principle of the multimodal feature extraction model in an embodiment of the present invention;
[0029] Figure 6 is a schematic diagram of a relationship extraction process provided in an embodiment of the present invention;
[0030] Figure 7 It is a schematic diagram of another relationship extraction process provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0031] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.
[0032] The entity relationship extraction method based on multimodal administrative division data of the present invention first obtains multimodal data of the target administrative division, including administrative division image data and administrative division text data; then uses the constructed multimodal feature extraction model to process the acquired administrative division image data and administrative division text data to extract the corresponding entity relationships; the multimodal feature extraction model includes an image feature conversion structure, a semantic feature conversion structure and a feature decoder; the image feature conversion structure is used to convert the acquired administrative division image data into corresponding image features, and the semantic feature conversion structure is used to convert the acquired administrative division text data into corresponding semantic features; the feature decoder is used to use the acquired image features and semantics as input, and learn and mine the image features and semantic features to obtain the target administrative division entity relationship information. The implementation process of this method is as follows: Figure 1 As shown, the following is a detailed description with reference to specific examples.
[0033] 1. Obtain the target image and information text and preprocess them.
[0034] The target image refers to an image containing information required by the user. As an example, if the user needs geographic location information, it is necessary to identify the remote sensing image containing administrative division objects and obtain an image containing administrative division information as the target image. The information text is used to indicate the target type of entity information that needs to be extracted from the target image.
[0035] Entity information refers to the information that the user needs to obtain from the target image. For example, if the target image is an administrative division map of Henan Province, the entity information can be "167,000KM 2” (administrative area of Henan Province); for example, if the target image is the administrative division map of Henan Province and the administrative division map of Hubei Province, then the entity information can be "Hubei Province" (the adjacent administrative division of Henan Province) and "Shicaogou Village" (Henan provincial enclave). The target type means the relationship information corresponding to the entity information. As an example, if the target text is the administrative division information text of Henan Province, and the entity information is "98.15 million people" (the permanent population of the province at the end of 2023), then the target type can be "2023 permanent population"; for example, if the target image is an administrative division map, the entity information can be "167,000KM 2 ” (administrative area of Henan Province), then the target type can be “total area”.
[0036] The information text is used to indicate the target type of entity information that needs to be extracted from the target image. As an example, suppose the information in the target image is "The total area of Henan Province is 167,000 square kilometers" and the entity information is "167,000KM 2 ", the target type is "total area", then the information text can become the question corresponding to the entity information "What is the total area of Henan Province", "How many square kilometers is the administrative area of Henan Province", etc., and the information text is used to indicate the target type corresponding to the entity information. Among them, the information text can indicate one target type or multiple target types. For example, if the target type is "total population", "total area", "adjacent administrative divisions", or "provincial enclaves", the information text can be "What is the total area and total population of Henan Province in 2020", "Where is the provincial enclave in Henan Province", "What is the entity information adjacent to the administrative division image of Henan Province", etc., and no specific limitation is made here.
[0037] The multimodal data acquired in this embodiment includes administrative division image data, administrative division text data and administrative division broadcast audio data, wherein the administrative division image can be an administrative division scan map, administrative division vector graphics, landmark photos, traffic signs, etc.; the administrative division text data can be official administrative division policy documents, administrative division news reports, administrative division historical documents, etc.; the administrative division audio can be administrative division introduction audio, administrative division broadcast, administrative division podcast, etc. Figure 3 As shown, the above multimodal data can be obtained through portable computers, desktop computers, smart phones, smart platforms and scanning devices.
[0038] This embodiment preprocesses each modality data according to the modality type of the multimodal data. The preprocessing of the vector graphics in the administrative division image data is as follows: Figure 2As shown, it includes coordinate conversion, map merging, cogo orientation recognition, etc.; the scanned map image is a raster image, and the present invention does not process the raster image, but can extract text information from the map image, such as annotations, altitudes, legends and other information. Therefore, the preprocessing of the scanned map in the administrative division image data includes resolution adjustment, denoising, optical character recognition text conversion, etc.; the preprocessing of the administrative division text data includes removal of stop words, word segmentation, part-of-speech tagging, etc.; the preprocessing of the administrative division audio data includes noise reduction, speech recognition text conversion, etc.
[0039] 2. Build a multimodal feature extraction model and train it.
[0040] Since the multimodal data in the present invention contains images, texts and audios, the multimodal feature extraction model used is as follows: Figure 5 As shown, it includes an image feature conversion structure, a semantic feature conversion structure and a feature decoding structure. The multimodal feature extraction model in this embodiment adopts the Transformer model, such as Figure 5 As shown, the feature decoding structure is Transformer Decoder.
[0041] The image feature conversion structure is used to convert the administrative division image into the corresponding image feature structure. The image feature conversion structure can adopt a deep learning model, such as the Transformer model of the convolutional neural network CNN. The semantic feature conversion structure is used to convert the target administrative division text into the corresponding semantic features. The target administrative division scanned image can obtain the optical character recognition OCR result after preprocessing, and the administrative division audio data can obtain the ASR result after preprocessing. Therefore, the semantic feature conversion structure is also used to convert the OCR result and the ASR result into the corresponding semantic features. The semantic feature conversion structure can adopt the BERT pre-training model. Its main model structure is composed of a pre-training framework and a large model fine-tuning framework for specific tasks.
[0042] Image features are used to characterize the basic information of the target image and to identify and mark the image content. For example, in deep learning, taking the classification network as an example, the image is input into the classification network to obtain the category of the image. At this time, if the input of the last layer of the classifier is extracted (generally the output of the second to last layer), the output obtained at this time is the feature obtained by the classifier, or the embedded layer feature. Semantic features are feature words extracted from the information text through the semantic feature conversion structure, and the features of the text information are represented by quantization. Therefore, the embodiment of the present invention converts the target image into the corresponding image features so that the multimodal feature extraction model no longer completely relies on manual recognition, OCR recognition and ASR recognition results, avoids errors in the extraction of entity information caused by errors in manual recognition, OCR recognition and ASR recognition results, and improves the accuracy of relationship extraction.
[0043] The feature decoding structure is a structure for decoding image feature information and semantic feature information to obtain target entity information. Specifically, the structure of the Transformer Decoder model can be used as the feature decoding structure. The embodiment of the present invention converts the target image into image features through the image feature conversion structure in the multimodal feature extraction model, converts the information text into semantic features through the semantic feature conversion structure in the multimodal feature extraction model, and determines the target entity information according to the image features and semantic features through the feature decoding structure in the multimodal feature extraction model, so that the relationship extraction model can directly extract the answer to the query question from the image based on the input image and query question in actual applications, thereby realizing end-to-end relationship extraction.
[0044] This embodiment uses the image feature conversion structure in the multimodal feature extraction model to convert the target image into corresponding image features. The specific steps include:
[0045] A1: The target image is processed through the image feature conversion structure to obtain the feature representation corresponding to each vector in the target image.
[0046] It should be noted that, as an example, a CNN (Convolutnonal Neural Network) model, a Transformer model such as BERT, etc. can be used as an image feature conversion structure to process each vector in the target image and obtain a feature representation corresponding to each vector.
[0047] A2: Based on the feature representation corresponding to each vector in the target image, a vector feature sequence is constructed, and the vector feature sequence is used as the image feature corresponding to the target image.
[0048] It should be understood that in this embodiment, by taking the vector feature sequence as the image features corresponding to the target image, the multimodal feature extraction model performs relationship extraction based on the image features of the original target image, rather than relying entirely on manual recognition, OCR recognition, and ASR recognition results, so as to improve the accuracy of the extracted entity information.
[0049] In order to further improve the accuracy of the target entity information, the cogo orientation recognition result of the target image can be introduced in the process of relationship extraction, and the cogo orientation recognition result of the target image is used to assist in extracting the entity information of the target type. In a possible implementation, the method further includes:
[0050] B1: Get the cogo recognition orientation result of the target image.
[0051] It should be noted that the cogo position recognition result can be obtained by identifying the target image through the cogo recognition technology in the relevant technology, and the specific acquisition method is not specifically limited.
[0052] B2: The cogo results are converted into corresponding semantic features through the semantic feature conversion structure in the multimodal feature extraction model.
[0053] The implementation method of converting the cogo results into corresponding semantic features through the semantic feature conversion structure in the multimodal feature extraction model is similar to the implementation method of converting the information text into corresponding semantic features through the semantic feature conversion structure, and will not be repeated here.
[0054] Correspondingly, the feature decoding structure in the multimodal feature extraction model can determine the target entity information of the target type in the target image based on the image features, the semantic features corresponding to the cogo results, and the semantic features corresponding to the information text.
[0055] It should be understood that when the multimodal feature extraction model extracts target entity information of the target type, it simultaneously refers to the image features of the target image, the semantic features corresponding to the cogo orientation recognition results, and the semantic features corresponding to the information text. It can better learn the text feature information and layout information in the target image at the same time, thereby improving the accuracy of the extraction results of the multimodal feature extraction model.
[0056] Based on the relationship extraction method provided in the above embodiment, in order to further illustrate the process of determining the target entity information, this step may also include: performing multiple autoregressive decoding operations through the feature decoding structure until a termination symbol is obtained through decoding.
[0057] The autoregressive decoding operation may specifically include: determining the next text unit in the target entity information according to the image features, semantic features, and the previous text unit in the target entity information through the feature decoding structure; the number of decodings is an integer greater than or equal to 1, and when the number of decodings is equal to 1, the previous text unit is the starting symbol. It should be understood that the autoregressive decoding operation refers to adding the output of the decoder's own current step to the input of the next step, and the decoder fuses all the vectors that have been output to output the next vector. Therefore, the output in the future will take into account more inputs. Reasoning and self-targeting can be performed to a certain extent.
[0058] An autoregressive decoder provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, as an example, when the decoded output is marked as Zhengzhou, its historical input is marked as <s> The capital of Henan Province, in other words, "Zhengzhou" is the label of the decoded output. According to this rule, the autoregressive decoding is cyclical. When the decoded output label is< / s> , that is, stop decoding. In this embodiment, through the feature decoding structure, multiple autoregressive decoding operations are performed until the decoding obtains the termination symbol, and the text unit can be inferred and the target, that is, the target entity information is determined through the autoregressive decoding operation, which can improve the accuracy of the generated target entity information.
[0059] For example, taking the semantic features of the previous text unit as a historical mark can be understood as having been decoded through the feature decoding structure to obtain a decoding result. When the semantic features of the current text unit need to be decoded, the semantic features of the current text unit can be used as the current mark. The current mark can see the historical mark, as well as related image features and semantic features (corresponding to cogo results and information text), but cannot see the mark information after the current mark, so as to ensure the correlation between the current mark and the historical mark, so as to achieve the result target for the current mark.
[0060] Based on the relationship extraction method provided in the above embodiment, the multimodal feature extraction model is trained in the following manner:
[0061] C1: Get training samples.
[0062] The training samples may include training images, training information text and labeled entity information. The training information text is used to indicate the training type of entity information to be extracted from the training images. The labeled entity information is entity information of the training type in the training images.
[0063] The training samples may be obtained through a training database or through real-time acquisition, which is not limited here; the labeled entity information refers to entity information that actually corresponds to the training type in the training image.
[0064] C2: Determine the target entity information based on the training images and training information text through the multimodal feature extraction model to be trained.
[0065] In step C2, the training image is mainly converted into training image features through the image feature conversion structure in the multimodal feature extraction model to be trained, and the training information text is converted into corresponding training semantic features through the semantic feature conversion structure in the multimodal feature extraction model to be trained; then, through the feature decoding structure in the multimodal feature extraction model, the semantic features corresponding to the target entity information are determined according to the training image features and the training semantic features, and the target entity information is mapped accordingly.
[0066] It should be understood that during the training process, since the length of the training information text is different, the length of the corresponding semantic features is also different. In order to better train the multimodal feature extraction model to be trained, the semantic features corresponding to the training information text can be padded. That is, after the training information text is converted into the corresponding semantic features through the semantic feature conversion structure in the multimodal feature extraction model to be trained, the semantic features are padded to a preset length, and the padded semantic features are used as training semantic features.
[0067] C3: Based on the differences between the semantic features corresponding to the target entity information and the annotated entity information, a multi-task loss function is constructed to combine the losses of different tasks.
[0068] The deep semantic information of the entity is captured by using the pre-trained deep learning model BERT or its variants. A matching strategy is defined to correspond the features of the predicted entity with the features of the actual labeled entity. This involves determining the best one-to-one correspondence or many-to-one relationship.
[0069] For each pair of matching features, calculate the difference between them. This can be achieved through various methods such as Euclidean distance, cosine similarity, or more advanced perceptual loss. In some cases, it may be necessary to give different weights to different types of feature differences to reflect their importance in the overall entity recognition task.
[0070] C4: Train a multimodal feature extraction model.
[0071] By training the multimodal feature extraction model to be trained with training images, training information texts and annotated entity information, the multimodal feature extraction model to be trained is not completely dependent on the semantic recognition results, thus avoiding entity information extraction errors caused by semantic recognition errors and improving the accuracy of entity relationship extraction.
[0072] In the embodiment, the training image is mainly converted into training image features through the image feature conversion structure in the multimodal feature extraction model to be trained, and the training information text and the training semantic results are respectively converted into corresponding training semantic features through the semantic feature conversion structure in the multimodal feature extraction model to be trained. Then, the target entity information is determined through the feature decoding structure in the multimodal feature extraction model to be trained according to the training image features, the training semantic features corresponding to the training character recognition and audio recognition results, and the training semantic features corresponding to the training information text.
[0073] It should be understood that by training the semantic results, the information of recognition results such as cogo can be integrated, so that the multimodal feature extraction model to be trained can learn the same type of feature information and layout information in the target image through the feature features corresponding to the feature recognition results, thereby improving the accuracy of the extraction results of the multimodal feature extraction model.
[0074] The training of Transformer Decoder can be carried out using Teacher Forcing, which is based on the next tag of the historical tag target. Therefore, the input can include an image tag sequence, an OCR recognition sequence, extracted query information, and a tag sequence to be extracted, and the output tag is the tag sequence to be extracted.
[0075] At the same time, during the training process of Transformer Decoder, the design of the mask matrix also needs to be considered. Assume that the tag sequence of the training image, the src sequence of the training semantic results, and the pro sequence of the training information text together constitute the training sequence Q. The previous sequence t is the answer, recorded as sequence A, which is the training tag sequence. During the training process, the tags in sequence Q are visible to each other, that is, the current tag can see all other tags in Q. A causal mask is used inside sequence A (the current tag can only see historical tags, not future tags). All tags in sequence A can see all tags in sequence Q.
[0076] As a possible implementation method, Figure 6 As shown, the Transformer Decoder in the multimodal feature extraction model can extract entity relationships from the input sequence. The {capital Beijing} in the input sequence is the semantic recognition result, and the semantic recognition result can include text tag information {t1, t2, …, tN} corresponding to the semantic recognition result. <s> is the starting symbol of the entire sequence,< / s> is the terminator of the entire sequence, <nm> is the start symbol of the image marker sequence,< / nm>is the terminator of the image tag sequence, the middle of the two is the image tag sequence nm, and {where is the capital} is the text tag information corresponding to the information text, which can also be other similar questions, such as {the capital is}, {where is the capital}, etc. Inputting the input sequence into the Transformer Decoder in the multimodal feature extraction model, the target entity information can be obtained.
[0077] A token is a smallest unit in a text. Usually, a token can be a word, a punctuation mark, a number, a symbol, etc. In the application of multimodal feature extraction models, the multimodal feature extraction model receives a string of tokens s as input. For many models, the tokens s can also be converted into vector representations through embeddnng operations for processing in neural networks.
[0078] Another example of an entity relationship extraction process provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, this embodiment is for extracting single entity information. In another possible implementation, the Transformer Decoder in the multimodal feature extraction model can also support the extraction of multiple or all entity information. Figure 7 As shown in the figure, {capital and its postal code are} is the text tag information corresponding to the information text, that is, {government location} and {postal code} need to be queried at the same time, {Beijing} and {100000} are the query results, that is, the target entity information, where, <sep>is the entity separator. If you need to query all entity information, the information text is similar to the entity information design. For example, the information text can be {extract key entity information in the graph}, and the target entity information can be {the capital is Beijing <sep>The postal code is 100000 <sep>The area is NUM...}.
[0079] As another implementation, the entity relationship extraction method of multimodal administrative division data of the present invention can also be implemented through corresponding software modules to form an entity relationship extraction tool for multimodal administrative division data.
[0080] The embodiment of the present invention converts the target image into image features through the image feature conversion structure in the multimodal feature extraction model, converts the information text into semantic features through the semantic feature conversion structure in the multimodal feature extraction model, and determines the target entity information according to the image features and semantic features through the feature decoding structure in the multimodal feature extraction model.< / sep> < / sep> < / sep>
Claims
1. A method for extracting entity relations based on multimodal administrative division data, characterized in that: The extraction method includes the following steps: 1) Obtain multimodal data of the target administrative division, including administrative division image data and administrative division text data; 2) using the constructed multimodal feature extraction model to process the acquired administrative division image data and administrative division text data to extract the corresponding entity relationship; the multimodal feature extraction model includes an image feature conversion structure, a semantic feature conversion structure and a feature decoder; The image feature conversion structure is used to convert the acquired administrative division image data into corresponding image features, and the semantic feature conversion structure is used to convert the acquired administrative division text data into corresponding semantic features; the feature decoder is used to take the acquired image features and semantics as input, and obtain the target administrative division entity relationship information by learning and mining the image features and semantic features.
2. The entity relationship extraction method based on multimodal administrative division data according to claim 1 is characterized in that: The feature decoder adopts Transformer Decoder, and the Transformer Decoder adopts autoregressive decoding operation.
3. The entity relationship extraction method based on multimodal administrative division data according to claim 2 is characterized in that: The autoregressive decoding operation includes: when the number of decoding times is equal to 1, the previous semantic unit is the starting symbol of the feature decoding structure; based on the image features, semantic features, and the previous semantic unit in the target administrative division entity information, the next semantic unit in the target administrative division entity information is determined.
4. The entity relationship extraction method based on multimodal administrative division data according to claim 1 is characterized in that: The semantic features input to the feature decoder also include semantic features obtained by converting the cogo results identified from the administrative division image data using the semantic feature conversion structure.
5. The entity relationship extraction method based on multimodal administrative division data according to any one of claims 1 to 4, characterized in that: The training process used in the constructed multimodal feature extraction model is: Obtain training samples including training images, training information text and labeled entity information; The training information text is used to indicate the training type of entity information that needs to be extracted from the training image, and the labeled entity information is the entity information of the training type in the training image; According to the training image and its conversion result text, the target entity information is determined and the multimodal feature extraction model is trained.
6. The entity relationship extraction method based on multimodal administrative division data according to claim 2 is characterized in that: The Transformer Decoder is trained using Teacher Forcing.
7. The entity relationship extraction method based on multimodal administrative division data according to claim 1 is characterized in that: The process of converting the acquired administrative division image data into corresponding image features by the image feature conversion structure includes: The image feature conversion structure processes the administrative division image data to obtain feature representations corresponding to the vector graphics in the administrative division image; A vector feature sequence is constructed based on feature representations corresponding to each of the vector graphics in the administrative division image, and the vector feature sequence is used as a relational feature corresponding to the vector graphics in the administrative division image.
8. The entity relationship extraction method based on multimodal administrative division data according to claim 1 is characterized in that: The multimodal data acquired in step 1) further includes administrative division broadcast audio data, and the administrative division image data includes a scanned map image. The method further includes: Obtain the optical character recognition (OCR) results of administrative division information in the scanned map image, and obtain the audio speech recognition (ASR) results of administrative division broadcast audio data; The OCR result and the ASR result are converted into corresponding administrative division semantic features through the semantic feature conversion structure in the multimodal feature extraction model; The target administrative division entity information in the administrative division image is determined according to the image features and the semantic features through the feature decoding structure in the multimodal feature extraction model.