Government affair atlas implementation method and system based on polymorphic data storage

Abstract

The invention discloses a method and system for realizing a government affair map based on polymorphic data storage, and the method comprises the steps: preprocessing source data, the preprocessing comprises wrong data cleaning, repeated data cleaning and missing data completion, and the source data comprises structured data, semi-structured data and non-structured data; performing information extraction on the cleaned data to obtain knowledge and form a triple for constructing a knowledge graph; multi-modal data are fused in multiple fusion modes, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, and shared space representation is established to describe entities more completely from different angles. According to the method, government affair polymorphic data are fused, all incidence relations of entities are extracted for analysis and reasoning, and the information quality and accuracy of a traditional relation graph are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular to a method and system for implementing a government affairs graph based on polymorphic data storage. Background Art

[0002] Knowledge graph technology is a technology that describes concepts and their relationships in the physical world in symbolic form. It is essentially a large-scale semantic network graph with entities as nodes and semantic relationships between entities as edges. It has been widely used in government affairs, medical care, finance and other fields. However, most of the existing knowledge graphs are traditional text knowledge graphs, which greatly affects the ability of machines to understand and display the real world, and the semantic information that people can obtain from knowledge graphs is also limited. Therefore, it is necessary to combine text with modalities such as images, videos and sounds. This requires us to pay attention to unstructured data, such as visual data, and multimodal knowledge graphs have also emerged. Multimodal knowledge graphs are knowledge graphs that contain multiple modal data (such as visual modalities, sound modalities, etc.). Multimodal knowledge graphs are not only richer in the display of knowledge, but also more and more widely used in many other fields. For example, in relationship extraction tasks, information in image modalities will greatly improve the accuracy of some text retrieval.

[0003] In recent years, with the continuous improvement of government digitalization, departments at all levels have accumulated a large amount of data in their daily work. On the one hand, these valuable data resources help improve the operating efficiency of various functional agencies and promote social and economic development; on the other hand, due to the professionalism and complexity of government data, it has become particularly difficult to manually extract data from massive information. Knowledge graphs (KGs) use mathematical graph structures to describe knowledge and model the relationship between all things in the world. They are the main tools for achieving "knowledge-based, personalized, and intelligent services" in various fields. In recent years, they have been widely used in industry and academia. Knowledge graph construction technology can mine information from structured, semi-structured, and even unstructured data sources, integrate information into knowledge graphs represented by graphs, and realize hyper-relational data modeling and rapid knowledge reasoning. Therefore, the demand for knowledge graphs in government has also been gradually awakened. At present, in the field of government affairs, research on the construction and application of knowledge graphs is still in its initial stage. Most of the existing construction methods directly search for the corresponding image data through text to obtain the corresponding image entity, but this method does not use the semantic information of the image and text, that is, if the image is only searched through the text, then it is the image entity corresponding to the text entity, rather than because the information described by the image is the most consistent with the information described by the text entity, so it is the image entity corresponding to the text entity. Finally, how to select the most representative image from a group of related images as the corresponding image entity is also an important issue.

[0004] As the basis for knowledge graph construction, entity relationship extraction has attracted more and more attention from researchers. Entity relationship extraction can automatically and accurately obtain knowledge from a large amount of data, and represent and store it in a structured form. Therefore, the correctness of entity relationship extraction directly affects the accuracy of knowledge graph construction and the subsequent application effect of knowledge graph. However, for different research hotspots such as complex structure, open domain, multilingual, multimodal, small sample data and entity relationship joint extraction, the existing entity relationship extraction methods still have some limitations. Based on the current research hotspots of entity relationship extraction, entity relationship extraction is divided into six aspects: complex structure research field, open domain, multilingual research field, multimodal research field, small sample data research field and entity relationship joint extraction.

[0005] The current forms of information presentation have become richer and more diverse, including text, images, videos, voice and other modalities. These data contain huge value, which will become the basic resource for data fusion and intelligent decision-making. Information between different modalities complements each other and can provide people with more comprehensive and specific information, thus helping people better understand and solve complex tasks. Therefore, multimodal information research has become a hot area that has attracted much attention.

[0006] Traditional knowledge graphs. With "people" as the central entity (ontology), the relationship between people and "telephone numbers", the relationship between "people" and "cars", and the relationship between "people" and "people" are established. At the same time, the travel events between "people" and "trains" and the accommodation events between "people" and "hotels" are established. All the associations are structured information, which makes it difficult to deeply mine and explore the information. A large amount of structured and unstructured data processing in government affairs is idle. Summary of the invention

[0007] In order to solve the above-mentioned technical problems existing in the prior art, the present invention proposes an implementation method and system of a government affairs graph based on polymorphic data storage to solve the above-mentioned technical problems.

[0008] According to a first aspect of the present invention, a method for implementing a government affairs graph based on polymorphic data storage is proposed, comprising:

[0009] S1: Preprocess the source data, including cleaning of erroneous data, cleaning of duplicate data, and filling in missing data. The source data includes structured data, semi-structured data, and unstructured data;

[0010] S2: Extract information from the cleaned data, obtain knowledge and form triples to build the knowledge graph;

[0011] S3: Multiple fusion methods are used to fuse multimodal data, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, to establish a shared spatial representation to more completely describe entities from different perspectives.

[0012] In some specific embodiments, information extraction in S2 includes information extraction of semi-structured data and unstructured data, and information extraction includes entity extraction, relationship extraction, event extraction and reference disambiguation.

[0013] In some specific embodiments, entity extraction is to identify entity information from text and classify it into specified categories; relationship extraction is to query the potential relationship between entities from massive data on the premise of obtaining entities; relationship extraction is to obtain entities from multi-source heterogeneous knowledge and simultaneously establish the relationship between entities; reference disambiguation is used to determine the reference relationship of entities.

[0014] In some specific embodiments, multimodal data includes text, images, video and audio, which are processed and characterized according to the characteristics of each modality and represented as single modal data. Based on the single modal data representation, a comprehensive representation of each modality is performed, and the comprehensive features are fused using a deep learning model.

[0015] In some specific embodiments, for unimodal data of text, extraction and analysis are performed to extract character strings from text types, and text preprocessing is performed, including word segmentation, deletion of stop words and filtering of low-frequency words, vectorization modeling is performed and feature extraction training is performed; for unimodal data of images, image denoising is performed to remove noise that affects image quality, image enhancement is performed to improve image quality and highlight key information, image compression is performed to reduce space occupancy, and feature extraction training is performed; for unimodal data of video, feature extraction training is performed after video editing; for unimodal data of audio, feature extraction training is performed after filtering, windowing, calculation of energy and zero crossing points, and integration.

[0016] In some specific embodiments, the stage fusion in S3 is to decompose the overall goal into several independent stage goals, and use different data sets in different stages, so that multiple data sets serve the same overall goal; direct fusion is data layer fusion, which directly connects data from different sources in series or uses a linear weighted method to splice them into one data set; feature fusion is the process of classifying, aggregating and integrating features after extracting features from data from various sources; decision fusion is to use different models for data from different sources, and then fuse the decision results after making a decision; hybrid fusion is a data fusion method that combines feature fusion and decision fusion; model fusion is to fuse multiple models to meet the needs of data fusion.

[0017] In some specific embodiments, S2 extracts entities and their relationships through an end-to-end neural network model. The end-to-end neural network model performs joint extraction of entity mentions and relationships without accessing the dependency tree, converts the joint extraction task into labeling problem processing, and uses the end-to-end model to directly extract entities and their relationships.

[0018] In some specific embodiments, it also includes information fused through structured data and semi-structured data, associating to generate a first association relationship node, the first association relationship nodes are again associated with each other to generate a second association relationship node, and the weighted trust scores of the first association relationship node and the second association relationship node are weighted. When searching for entity activity information, the entity activity information is scored according to the shortest path, weight, and label feature weighted value, and new entity association relationship information is introduced through a big data model to form a closed loop.

[0019] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0020] According to a second aspect of the present invention, a system for implementing a government affairs graph based on polymorphic data storage is proposed, comprising:

[0021] A preprocessing unit is configured to preprocess the source data, the preprocessing includes cleaning of erroneous data, cleaning of duplicate data, and completion of missing data, and the source data includes structured data, semi-structured data, and unstructured data;

[0022] An information extraction unit, configured to extract information from the cleaned data, acquire knowledge and form triples for constructing a knowledge graph;

[0023] The fusion unit is configured to fuse the multimodal data using a variety of fusion methods, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, to establish a shared space representation to more completely describe the entity from different perspectives.

[0024] In some specific embodiments, information extraction in the information extraction unit includes information extraction of semi-structured data and unstructured data, and information extraction includes entity extraction, relationship extraction, event extraction and reference disambiguation; entity extraction is to identify entity information from text and classify it into specified categories; relationship extraction is to query the potential relationship between entities from massive data on the premise of obtaining entities; relationship extraction is to obtain entities from multi-source heterogeneous knowledge and simultaneously establish the relationship between entities; reference disambiguation is used to determine the reference relationship of entities.

[0025] In some specific embodiments, multimodal data includes text, images, videos and audios, which are processed and characterized according to the characteristics of each modality and represented as single-modal data. On the basis of the single-modal data representation, a comprehensive representation of each modality is performed, and the comprehensive features are fused using a deep learning model. For single-modal data of text, extraction and analysis are performed to extract character strings from text types, and text preprocessing is performed, including word segmentation, deletion of stop words and filtering of low-frequency words, vectorization modeling and feature extraction training are performed. For single-modal data of images, image denoising is performed to remove noise that affects image quality, image enhancement is performed to improve image quality and highlight key information, image compression is performed to reduce space usage, and feature extraction training is performed. For single-modal data of videos, feature extraction training is performed after video editing. For single-modal data of audio, feature extraction training is performed after filtering, windowing, calculation of energy and zero crossing points, and integration.

[0026] In some specific embodiments, stage fusion in the fusion unit is to decompose the overall goal into several independent stage goals, and use different data sets in different stages, so that multiple data sets serve the same overall goal; direct fusion is data layer fusion, which directly connects data from different sources in series or uses a linear weighted method to splice them into one data set; feature fusion is the process of classifying, aggregating and integrating features after extracting features from data from various sources; decision fusion is to use different models for data from different sources, and then fuse the decision results after making a decision; hybrid fusion is a data fusion method that combines feature fusion and decision fusion; model fusion is to fuse multiple models to meet the needs of data fusion.

[0027] In some specific embodiments, entities and their relationships are extracted in the information extraction unit through an end-to-end neural network model. The end-to-end neural network model performs joint extraction of entity mentions and relationships without accessing the dependency tree, converts the joint extraction task into labeling problem processing, and uses the end-to-end model to directly extract entities and their relationships.

[0028] The present invention proposes a method and system for realizing a government affairs graph based on polymorphic data storage, performs topic modeling, constructs a regional government affairs knowledge graph based on topic division, and visualizes the graph for hot spot analysis and retrieval. It aims to solve the problems of data disorder and fragmentation in "one-stop service" and improve the efficiency of government services. It performs entity recognition, relationship extraction and entity linking on unstructured and semi-structured government affairs data and integrates them with structured government affairs data. It helps to improve the efficiency and accuracy of the entire judgment when judging government affairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 It is a flowchart of a method for implementing a government affairs graph based on polymorphic data storage according to an embodiment of the present application;

[0032] Figure 3 This is a schematic diagram of the process of constructing a knowledge graph of a specific embodiment of the present application;

[0033] Figure 4 It is a schematic diagram of a processing flow of multimodal data fusion of a specific embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a polymorphic knowledge graph of a specific embodiment of the present application;

[0035] Figure 6 This is a system architecture diagram for implementing a government affairs graph based on polymorphic data storage in one embodiment of the present application;

[0036] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] Figure 1 An exemplary system architecture 100 is shown to which an implementation method of a government affairs graph based on polymorphic data storage according to an embodiment of the present application can be applied.

[0040] like Figure 1As shown, system architecture 100 may include data server 101, network 102 and main server 103. Network 102 is used to provide a medium for a communication link between data server 101 and main server 103. Network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0041] The main server 103 may be a server that provides various services, such as a data processing server that processes information uploaded by the data server 101 .

[0042] It should be noted that the implementation method of the government affairs graph based on polymorphic data storage provided in the embodiment of the present application is generally executed by the main server 103. Accordingly, the implementation system for the government affairs graph based on polymorphic data storage is generally set in the main server 103.

[0043] It should be noted that the data server and the main server can be hardware or software. When it is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When it is software, it can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it can be implemented as a single software or software module.

[0044] It should be understood that Figure 1 The numbers of data servers, networks and main servers in the embodiment are only for illustration purposes. Any number of terminal devices, networks and servers may be provided as required.

[0045] Figure 2 FIG. 1 shows a flow chart of a method for implementing a government affairs graph based on polymorphic data storage according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0046] S201: Preprocess the source data. The preprocessing includes cleaning of erroneous data, cleaning of duplicate data, and completion of missing data. The source data includes structured data, semi-structured data, and unstructured data.

[0047] S202: Extract information from the cleaned data to obtain knowledge and form triplets for constructing a knowledge graph.

[0048] In a specific embodiment, information extraction includes information extraction from semi-structured data and unstructured data, and information extraction includes entity extraction, relationship extraction, event extraction and reference disambiguation. Entity extraction is to identify entity information from text and classify it into specified categories; relationship extraction is to query the potential relationship between entities from massive data on the premise of obtaining entities; relationship extraction is to obtain entities from multi-source heterogeneous knowledge and establish the relationship between entities at the same time; reference disambiguation is used to determine the reference relationship of entities.

[0049] In a specific embodiment, entities and their relationships are extracted through an end-to-end neural network model. The end-to-end neural network model performs joint extraction of entity mentions and relationships without accessing the dependency tree, converting the joint extraction task into labeling problem processing, and using the end-to-end model to directly extract entities and their relationships.

[0050] S203: Multiple fusion methods are used to fuse multimodal data, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, to establish a shared space representation to more completely describe entities from different perspectives.

[0051] In a specific embodiment, the modal data includes text, image, video and audio, and the model is processed and characterized according to the characteristics of each modality and represented as single modal data. On the basis of the single modal data representation, the comprehensive representation of each modality is performed, and the comprehensive features are fused using a deep learning model. For the single modal data of text, extraction and analysis are performed to extract the string from the text type, and text preprocessing is performed, including word segmentation, deletion of stop words and filtering of low-frequency words, vectorization modeling and feature extraction training are performed; for the single modal data of image, image denoising is performed to remove the noise that affects the image quality, image enhancement is performed to improve the image quality and highlight the key information, image compression is performed to reduce the space occupancy, and feature extraction training is performed; for the single modal data of video, feature extraction training is performed after video editing processing; for the single modal data of audio, feature extraction training is performed after filtering, windowing, calculating energy and zero crossing points, and integration.

[0052] In a specific embodiment, stage fusion is to decompose the overall goal into several independent stage goals, and use different data sets in different stages, so that multiple data sets serve the same overall goal; direct fusion is data layer fusion, which directly connects data from different sources in series or uses a linear weighted method to splice them into one data set; feature fusion is the process of classifying, aggregating and integrating features after extracting features from data from various sources; decision fusion is to use different models for data from different sources, and then fuse the decision results after making a decision; hybrid fusion is a data fusion method that combines feature fusion and decision fusion; model fusion is to fuse multiple models to meet the needs of data fusion.

[0053] Figure 3 A schematic diagram of the process of constructing a knowledge graph of a specific embodiment of the present application is shown as follows: Figure 3 As shown, this application is for the fusion of polymorphic government data (structured, semi-structured, and unstructured data), extraction of all entity relationships for analysis and reasoning, and the use of shortest path weighting, feature labeling, scoring and sorting, and knowledge completion to further improve the information quality and accuracy of traditional relationship graphs.

[0054] In a specific embodiment, information extraction serves as the basis for constructing a knowledge graph, and its correctness directly affects the quality of the knowledge graph and the subsequent application effect of the knowledge graph. Information extraction, as a key technology for acquiring knowledge, includes four subtasks: entity extraction (EE), relationship extraction (RE), event extraction (EE), and coreference resolution (CR). Entity extraction and relationship extraction are two important subtasks in the field of information extraction. Entity extraction is to identify entity information from text and divide it into specified categories. Common entity categories include people, cars, cases, addresses, organizations, etc.; relationship extraction is to query the potential relationship between entities from massive data on the premise of obtaining entities. Entity relationship extraction is to obtain entities from multi-source heterogeneous knowledge and establish relationships between entities at the same time, thereby forming triples necessary for building a knowledge graph.<Ent ity1,Re l at ion,Ent ity2> , where Entity1 and Entity2 are entities, and Relation describes the relationship between entities. This article will continue to use the term "entity relationship extraction" to represent entity extraction and relationship extraction and review them.

[0055] In a specific embodiment, the present application proposes a new end-to-end neural model to extract entities and their relations. The recursive neural network-based model captures word sequence and dependency tree substructure information by superimposing a bidirectional tree structure on a bidirectional sequence. An attention-based recursive neural network is proposed to convert the joint extraction task of entity mentions and relations without accessing the dependency tree into a labeling problem. Then, the end-to-end model is used to directly extract entities and their relations without separately identifying entities and relations.

[0056] In a specific embodiment, before extracting entity relationship information, it is necessary to clean the source data for erroneous data, duplicate data, and missing data, that is, the process of data cleaning. The cleaned data is fused. In real life, humans experience the objective world through vision, hearing, taste, and touch, and then describe an entity from various perspectives. At present, artificial intelligence technology is changing with each passing day. People hope that computers can also perform comprehensive processing of data of various modalities like humans. Therefore, multimodal data fusion has become a research hotspot and technical difficulty in the current field of data fusion. Different forms of information can be called a modality, such as text, pictures, videos, audio, etc. Although multimodal data is heterogeneous in form, multimodal data of the same entity is semantically consistent. Through multimodal data fusion, the complementarity and communication of each modal information can be achieved, thereby more comprehensively presenting the objective entity. According to the characteristics of each modal, appropriate preprocessing methods and characterization models are selected, that is, single-modal data representation; then, based on the single-modal data representation, comprehensive representation of each modality is performed, that is, multi-modal data representation; finally, the most popular deep learning model is used to fuse these comprehensive features to obtain the fusion result.

[0057] In a specific embodiment, Figure 4 A schematic diagram of a processing flow of multimodal data fusion of a specific embodiment of the present application is shown as follows: Figure 4 As shown in the figure, for unimodal data of text, firstly, extraction and analysis are performed to extract strings from text types, and then text preprocessing is performed, including word segmentation, deletion of stop words and filtering of low-frequency words, and then vectorization modeling (Word2vec, Doc2vec) and feature extraction training (including RNN, LSTM or GRU) are performed; for unimodal data of image, firstly, image denoising is performed to remove noise that affects image quality, and then image enhancement is performed to improve image quality and highlight key information, and then image compression is performed to reduce space usage, and finally feature extraction training is performed (AlexNet, VGG, GoogLeNet, ResNet can be used); for unimodal data of video, in video editing processing (Adobe Premiere, Moviemaker, After Effects, Boris Effects), which can be specifically performed through a single-channel convolutional neural network, a dual-channel convolutional neural network, or a hybrid network. For the unimodal audio data, feature extraction training is performed after filtering (removing noise signals), windowing (dividing into shorter frames), calculating energy and zero-crossing points (determining blank frames), and integrating (removing blank segments and integrating complete outputs). Specifically, it can be performed through the Librosa audio processing library, openSMILE open source software, Wav2vec, and wav2vec 2.0.

[0058] In a specific embodiment, unimodal data representation provides the basis and premise for multimodal data representation. Although multimodal data is heterogeneous in its underlying form, multimodal data describing the same subject are highly consistent in semantics. Multimodal representation can narrow the heterogeneity gap between modalities while maintaining the integrity of the specific semantics of each modality, thereby describing the entity more completely from different angles. The purpose of multimodal data representation is to establish a shared space representation. Fusion methods include:

[0059] 1. Stage fusion: decompose the overall goal into several independent stage goals, use different data sets in different stages, and make multiple data sets serve the same overall goal.

[0060] 2. Direct fusion, that is, data layer fusion, is to directly connect data from different sources into one data set, and sometimes linear weighted methods are used for splicing.

[0061] 3. Feature fusion: first extract features from data from various sources, and then classify, aggregate and integrate these features.

[0062] 4. Decision fusion: Use different models for data from different sources, and then fuse the decision results after making a decision.

[0063] 5. Hybrid fusion, a data fusion method that combines feature fusion and decision fusion.

[0064] 6. Model fusion: Combined with the special requirements of data fusion, the existing models are processed and modified, and multiple models are integrated to meet the needs of data fusion.

[0065] The fusion of polymorphic data sources enriches data semantic information, and can combine reasoning to obtain implicit information to provide better services for users. The knowledge graph converts the massive multi-source heterogeneous data of relevant departments into entities based on elements such as "people, things, places, objects, organizations, and virtual identities", defines and mines various relationships between entities, and is conducive to early analysis of events. The database tables, text records, pictures, and surveillance videos that describe the same subject are supplemented and verified, and further processed to form a knowledge graph.

[0066] Figure 5 A schematic diagram of a polymorphic knowledge graph of a specific embodiment of the present application is shown, such as Figure 5As shown, this application adds and expands pictures, audio, and video content on the basis of the text on the basis of the simple traditional knowledge graph. The dark green in the circle in the figure represents entities mainly composed of elements such as "people, things, places, objects, organizations, and virtual identities". Bright green represents the attributes of the entity-related person, which can be a directly described place, activity, certificate information, or associated pictures, audio, and video information. The purple outside the circle indicates that the associated hotel, flight, passport, picture, audio, video and other information are related. By fusing information through structured and semi-structured data, new purple associations are generated (different from the traditional single-mode graph, which is the association between structured information). The purple nodes are associated with each other again to generate additional black nodes to hide information. Such as the same place, the same activity, the same address, the same itinerary and other information.

[0067] In a specific embodiment, in addition to weighting the weighted credibility score of the new node, polymorphic data can also confirm that Zhang San has appeared in a certain place (accommodation information, picture information, call audio, etc.). At the same time, the characteristic information in the polymorphic data can also generate more additional associated nodes. For example, A and B appear in the picture place, and A and C also appear in the picture place. The potential relationship between B and C can be inferred. When such a graph appears repeatedly in different places, the association weights of B and C can be weighted. When looking for a certain entity activity information, the entity activity information is scored based on the shortest path as the weight and the label feature weighted value. Through the big data model, new labels and scores are continuously run out, and new entity association relationship information is deduced to form a closed loop.

[0068] In a specific embodiment, the intent is first used to identify the input keywords and sentences in the display layer. After identifying and splitting the input keyword sentences, a comprehensive search is performed on the shortest path, scoring weight and feature tags. It can be that the entities with high scores are ranked in front, the entities with sensitive tags are ranked in front, or the recommended information is actively predicted (search keyword prediction, active reminder of related relationships, hot news information recommendation, etc.). For example, "Where did Zhang San go today?" "Zhang San" is identified as a person's name, "today" is time information, and "where to go" is geographic location and place information. The Zhang San entity is found through the local government household information, and then all the entity's associated point information today is filtered according to time, and the information such as the checkpoints, the base station, etc. can be captured. It can be an explicit discovery of the place (possible check-in information, structured information), or it can be based on the shortest path and score, implicit discovery of entertainment, leisure and other places (combination of structured and semi-structured information).

[0069] Figure 6 The following is a diagram showing the system architecture of an implementation of a government affairs graph based on polymorphic data storage according to an embodiment of the present application. Figure 6As shown, the system includes a preprocessing unit 601, an information extraction unit 602 and a fusion unit 603, wherein the preprocessing unit 601 is configured to preprocess the source data, the preprocessing includes cleaning of erroneous data, cleaning of duplicate data, and completion of missing data, and the source data includes structured data, semi-structured data and unstructured data; the information extraction unit 602 is configured to extract information from the cleaned data, acquire knowledge and form triples for constructing a knowledge graph; the fusion unit 603 is configured to fuse multimodal data using a variety of fusion methods, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, to establish a shared space representation to more completely describe entities from different angles.

[0070] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0071] like Figure 7 As shown, the computer system includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0072] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0073] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0074] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0075] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0077] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: preprocesses the source data, and the preprocessing includes cleaning of error data, cleaning of duplicate data, and completion of missing data, and the source data includes structured data, semi-structured data, and unstructured data; extracts information from the cleaned data, obtains knowledge and forms triples for constructing a knowledge graph; uses a variety of fusion methods to fuse multimodal data, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion, and model fusion, and establishes a shared space representation to more completely describe entities from different angles.

[0078] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for implementing a government affairs graph based on polymorphic data storage, characterized in that: include: S1: Preprocessing the source data, wherein the preprocessing includes cleaning of erroneous data, cleaning of duplicate data, and filling of missing data. The source data includes structured data, semi-structured data, and unstructured data; S2: Extract information from the cleaned data, obtain knowledge and form triples to build the knowledge graph; S3: Multiple fusion methods are used to fuse multimodal data, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, to establish a shared spatial representation to more completely describe entities from different perspectives.

2. The method for implementing the government affairs graph based on polymorphic data storage according to claim 1 is characterized in that: The information extraction in S2 includes extracting information from the semi-structured data and the unstructured data, and the information extraction includes entity extraction, relationship extraction, event extraction and reference disambiguation.

3. The method for implementing the government affairs graph based on polymorphic data storage according to claim 2 is characterized in that: The entity extraction is to identify entity information from the text and classify it into specified categories; the relationship extraction is to query the potential relationship between entities from massive data on the premise of obtaining entities; The relationship extraction is to obtain entities from multi-source heterogeneous knowledge and simultaneously establish entity-entity relationships; the reference disambiguation is used to determine the reference relationship of entities.

4. The method for implementing the government affairs graph based on polymorphic data storage according to claim 1 is characterized in that: The multimodal data includes text, images, video and audio, which are processed and characterized according to the characteristics of each modality and represented as single modal data. Based on the single modal data representation, a comprehensive representation of each modality is performed, and the comprehensive features are fused using a deep learning model.

5. The method for implementing the government affairs graph based on polymorphic data storage according to claim 4 is characterized in that: For the unimodal data of the text, extract and parse to extract character strings from the text type, perform text preprocessing, including word segmentation, stop word deletion and low-frequency word filtering, perform vectorized modeling and feature extraction training; For the unimodal data of the image, image denoising is performed to remove noise that affects the image quality, image enhancement is performed to improve the image quality and highlight key information, image compression is performed to reduce the space occupancy, and feature extraction training is performed; for the unimodal data of the video, feature extraction training is performed after video editing; for the unimodal data of the audio, feature extraction training is performed after filtering, windowing, calculating energy and zero crossing points, and integration.

6. The method for implementing the government affairs graph based on polymorphic data storage according to claim 1 is characterized in that: The stage fusion in S3 is to decompose the overall goal into several independent stage goals, and use different data sets in different stages, so that multiple data sets serve the same overall goal; the direct fusion is data layer fusion, which directly connects data from different sources in series or uses a linear weighted method to splice them into one data set; the feature fusion is the process of classifying, aggregating and integrating features after extracting features from data from various sources; the decision fusion is to use different models for data from different sources, and then fuse the decision results after making a decision; the hybrid fusion is a data fusion method that combines the feature fusion and the decision fusion; the model fusion is to fuse multiple models to meet the needs of data fusion.

7. The method for implementing the government affairs graph based on polymorphic data storage according to claim 1 is characterized in that: In S2, entities and their relationships are extracted through an end-to-end neural network model. The end-to-end neural network model performs joint extraction of entity mentions and relationships without accessing a dependency tree, converts the task of the joint extraction into a labeling problem, and uses an end-to-end model to directly extract entities and their relationships.

8. The method for implementing the government affairs graph based on polymorphic data storage according to claim 1 is characterized in that: It also includes associating the information fused by the structured data and the semi-structured data to generate a first association relationship node, the first association relationship nodes are again associated with each other to generate a second association relationship node, and the weighted trust scores of the first association relationship node and the second association relationship node are weighted. When searching for entity activity information, the entity activity information is scored according to the shortest path, weight, and label feature weighted value, and new entity association relationship information is introduced through a big data model to form a closed loop.

9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 8 is implemented.

10. A system for implementing a government affairs graph based on polymorphic data storage, characterized in that: include: A preprocessing unit configured to preprocess the source data, wherein the preprocessing includes cleaning of erroneous data, cleaning of duplicate data, and completion of missing data, and the source data includes structured data, semi-structured data, and unstructured data; An information extraction unit, configured to extract information from the cleaned data, acquire knowledge and form triples for constructing a knowledge graph; The fusion unit is configured to fuse the multimodal data using a variety of fusion methods, including stage fusion, direct fusion, feature fusion, decision fusion, hybrid fusion and model fusion, to establish a shared space representation to more completely describe the entity from different perspectives.

11. The system for implementing the government affairs graph based on polymorphic data storage according to claim 10 is characterized in that: The information extraction in the information extraction unit includes extracting information from the semi-structured data and the unstructured data, and the information extraction includes entity extraction, relationship extraction, event extraction and reference disambiguation; the entity extraction is to identify entity information from the text and classify it into a specified category; the relationship extraction is to query the potential relationship between entities from the massive data on the premise of obtaining the entity; The relationship extraction is to obtain entities from multi-source heterogeneous knowledge and simultaneously establish entity-entity relationships; the reference disambiguation is used to determine the reference relationship of entities.

12. The system for implementing the government affairs graph based on polymorphic data storage according to claim 10 is characterized in that: The multimodal data includes text, images, videos and audios, which are processed and characterized according to the characteristics of each modality and represented as single-modal data. Based on the single-modal data representation, a comprehensive representation of each modality is performed, and the comprehensive features are fused using a deep learning model; for the single-modal data of the text, extraction and analysis are performed to extract character strings from the text type, and text preprocessing is performed, including word segmentation, deletion of stop words and filtering of low-frequency words, vectorization modeling and feature extraction training are performed; For the unimodal data of the image, image denoising is performed to remove noise that affects the image quality, image enhancement is performed to improve the image quality and highlight key information, image compression is performed to reduce the space occupancy, and feature extraction training is performed; for the unimodal data of the video, feature extraction training is performed after video editing; for the unimodal data of the audio, feature extraction training is performed after filtering, windowing, calculating energy and zero crossing points, and integration.

13. The system for implementing the government affairs graph based on polymorphic data storage according to claim 10 is characterized in that: The stage fusion in the fusion unit is to decompose the overall goal into several independent stage goals, and use different data sets in different stages, so that multiple data sets serve the same overall goal; the direct fusion is data layer fusion, which directly connects data from different sources in series or uses a linear weighted method to splice them into one data set; the feature fusion is the process of classifying, aggregating and integrating features after extracting features from data from various sources; the decision fusion is to use different models for data from different sources, and then fuse the decision results after making a decision; the hybrid fusion is a data fusion method that combines the feature fusion and the decision fusion; the model fusion is to fuse multiple models to meet the needs of data fusion.

14. The system for implementing the government affairs graph based on polymorphic data storage according to claim 10 is characterized in that: The information extraction unit extracts entities and their relationships through an end-to-end neural network model. The end-to-end neural network model performs joint extraction of entity mentions and relationships without accessing a dependency tree, converts the joint extraction task into a labeling problem, and uses an end-to-end model to directly extract entities and their relationships.

Images