A Chinese text semantic analysis and reconstruction system

Abstract

The invention discloses a Chinese text semantic analysis and reconstruction system, comprising: a text preprocessing module for generating text vectors, a font query module for processing concept relevance query and concept judgment of word items to support the needs of query semantic expansion and concept retrieval, a semantic resource module for packaging synonym word forest resources and responding to query requests for synonyms, a query module and a semantic search module interacting with the semantic resource module, a query module for realizing concept expansion and keyword expansion of non-conceptual content, a semantic search module for calculating the relevance of query and related texts and sorting related texts according to the size of the relevance in response to the query request of the user, and an interaction module for interacting with the user. The invention supports the unified text representation and unified processing of semi-structured text by the retrieval system, making up for the deficiency of the traditional retrieval system ignoring text structure information.

Description

Technical Field

[0001] The present invention relates to the field of text analysis, and in particular to a Chinese text semantic analysis and reconstruction system. Background Art

[0002] In recent years, the field of semantic parsing has become a research hotspot in natural language processing. At the same time, neural network technology has also been used in various aspects of natural language processing. Therefore, semantic parsing research combined with the efficient expression capabilities of neural network technology is very important.

[0003] At present, the application of intelligent robot technology in Internet products is relatively concentrated, but the lack of explicit pattern definition has brought difficulties and uncertainties to its retrieval. Many retrieval systems only treat the text as a simple word sequence, ignoring the structural information of the text, and severing the relationship between the logical structure of semi-structured text and its semantic connotation. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a Chinese text semantic analysis and reconstruction system, which can realize automatic navigation planning and intelligent navigation.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A Chinese text semantic analysis and reconstruction system comprises: a text preprocessing module, a font query module, a semantic resource module, a query module, a semantic search module and an interaction module, wherein the text preprocessing module is used to generate text vectors, the font query module is used to process concept relevance queries and concept judgments of terms to support the needs of query semantic expansion and concept retrieval, the semantic resource module is used to package synonymous word forest resources and respond to query requests for synonyms, the query module and the semantic search module interact with the semantic resource module, the query module is used to realize concept expansion and keyword expansion of non-conceptual content, the semantic search module is used to calculate the relevance of queries and related texts according to the query requests of users and sort the related texts according to the relevance, and the interaction module is used to interact with users.

[0007] Optionally, the text preprocessing module includes a word segmentation module, a term setting module, and an indexing module.

[0008] Optionally, the word segmentation module is used to perform statistical calculations on the keywords obtained by text segmentation to obtain the weight of each word item so as to create an index for the text.

[0009] Optionally, the term setting module is used to obtain a final text vector representation for a unified XML description text, wherein the XML description text used as input in the module is obtained by using information extraction technology on network text.

[0010] Optionally, the index module is implemented by an inverted index based on scalable hashing, which makes each bucket occupy at most one storage block, thereby shortening the query time.

[0011] Optionally, the font query module is used to determine whether a word is an ontology concept, return a candidate extended concept set for a concept, that is, a set of concepts whose relevance to the concept is greater than a certain threshold, and obtain the correlation between each concept in a query candidate extended concept and the query concept vector.

[0012] Optionally, the query module is used to perform concept expansion on the conceptual part of the query and perform synonym expansion on the non-concept part.

[0013] Optionally, the semantic search module is used to target the expanded query vector. The semantic search module first obtains the relevant text and the text query related concept set and the text query related keyword set corresponding to the relevant text through indexing, and then calculates the correlation between each relevant text and the query concept vector and the query keyword vector, and finally merges the calculated query concept correlation and query keyword correlation to finally obtain the correlation score between the relevant text and the query and sort the relevant text according to the correlation score.

[0014] Optionally, the semantic resource module uses the extended version of the synonym dictionary to provide query operations related to synonyms. In the query keyword expansion operation, the synonym set of the non-concept keyword is obtained. In the process of generating the text query concept set in the semantic search module, the weights of the synonymous concepts of a query concept are proportionally superimposed together to form a reasonable query text concept vector.

[0015] Optionally, the interaction module is used to accept the user's query statement, forward it to the query module to generate an expanded query vector, and then pass the expanded query vector to the semantic search module, and finally present the relevant text generated in the semantic search module and sorted from large to small in terms of relevance to the user.

[0016] Compared with the prior art, the present invention has the following technical advances:

[0017] The present invention supports the unified text representation and unified processing of semi-structured text by the retrieval system, making up for the deficiency of the traditional retrieval system ignoring text structure information. The query semantic expansion integrates concept expansion and keyword expansion, which can make up for the deficiency caused by the ambiguity of user queries to a certain extent, reduce the pressure of users to construct precise queries, and improve the recall rate of the retrieval system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0019] In the attached picture:

[0020] Figure 1 It is a structural principle diagram of the present invention.

[0021] Figure 2 It is a division diagram of the word segmentation module of the present invention. DETAILED DESCRIPTION

[0022] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings. In the description of the following embodiments, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.

[0023] Embodiment 1

[0024] like Figure 1 As shown, the present invention discloses a Chinese text semantic parsing and reconstruction system, including a text preprocessing module, a font query module, a semantic resource module, a query module, a semantic search module and an interaction module. The text preprocessing module first converts the captured network text into XML description text, and then performs word segmentation and density diffusion on the XML description text to obtain a reasonable weight for each text term, and finally represents the text as a text vector, and stores the text vector in an index.

[0025] The font query module is responsible for processing concept relevance queries and concept judgment of terms to support the related needs of query semantic expansion and concept retrieval.

[0026] The semantic resource module packages the synonym dictionary resources and responds to query requests about synonyms. Both the query module and the semantic search module need to interact with the semantic resource module.

[0027] The query module is an important consideration of the system in terms of semantics and intelligence. It enables the query to be elevated from the keyword level to the concept level, and realizes concept expansion and keyword expansion of non-conceptual content, thereby solving the semantic gap problem between users and the retrieval system.

[0028] The semantic search module calculates the relevance between the query and related text based on the user's query request and sorts the related text according to the relevance. This module is the core component of the system.

[0029] The interaction module is the communication interface between the system and the user. It receives the user's query request and transfers it to the query module, and presents the query results returned by the semantic search module to the user.

[0030] Embodiment 2

[0031] Specifically, the text preprocessing module includes the word segmentation module, the word item setting module, and the indexing module. The word segmentation module is the basis of natural language processing. The system performs statistical calculations on the keywords obtained by text segmentation, obtains the weight of each word item, and then creates an index for the text. In addition, the first step in query vector construction is to segment the query statement. Figure 2 As shown in the figure, word segmentation adopts a method combining statistics and rules, takes the cascade hidden Markov model as the theoretical framework, and uses a 5-layer hidden Markov model to unify word segmentation, part-of-speech tagging, segmentation and disambiguation, and unregistered word recognition. According to the use requirements of word segmentation, a large module division is obtained, which is mainly divided into: simple text processing, text segmentation, stop word removal, and keyword extraction. The word segmentation module provides the use class CResult to the outside world. The CResult class defines the following public interfaces and data members.

[0032] Member functions:

[0033] bool fileProcessing(string sFileName), file segmentation interface;

[0034] bool paraProcessing(string paragraph), sentence segmentation interface;

[0035] bool paraProcessing(string paragra,vector <string>&key), sentence segmentation interface, provides the result string of keywords and solves the cyclic segmentation of sentences.

[0036] Member variables:

[0037] map<string,double> keyFreq, records the frequency of keywords;

[0038] map<string,int> keyNum, records the number of keywords;

[0039] vector <string>wordSegResult, word segmentation result string;

[0040] vector <string>keySegResult, keyword result string;

[0041] unsigned int wordCount, the number of word segments obtained;

[0042] Embodiment 3

[0043] The term setting module is part of the text preprocessing. It obtains the final text vector representation for the unified XML description text. The XML description text used as input in the module is obtained by using information extraction technology on network text, and the output of the module is the text vector stored in the index. This module processes the XML description text generated by the information extraction scheme based on the structural characteristics of semi-structured text. The functions of this module are provided to the outside through the class ATfIdf, which is implemented using the open source DOM-based XML parsing library TinyXML.

[0044] Embodiment 4

[0045] Index modules are generally faced with a large number of user retrieval requirements. General database systems cannot respond to large-scale user requests for industrial-grade retrieval systems. In order to support the normal operation of the system, it is necessary to design an efficient index module to ensure query response speed and index creation speed, process massive data and support dynamic update of indexes.

[0046] The sub-modules of the index module include:

[0047] ehash module: the core of the entire index organization structure. This module implements the core algorithm of extensible hashing and its management, mainly including the organization of hash structure, management of directory table, dynamic growth of hash, writing of memory image and index loading, etc.

[0048] Bucket module: implements the bucket of the hash table, responsible for the storage form of blocks on disk, the organization form after loading into memory, bucket splitting, insertion, deletion and search of data in buckets, etc. The bucket module defines a series of functions to complete convenient operations based on buckets.

[0049] Freemap module: manages the space allocation of disk files and the mapping of file names and file descriptors. The entire disk file is considered as a storage space, and freemap is responsible for managing this space. You can "apply for space" from the freemap module. If successful, a pointer to this space is returned. The form of this "pointer" is (fd, offset, size), that is, <file descriptor, offset, space size>. It is similar to malloc applying for memory space from the system.

[0050] Cache module: completes cache management. Disk files are mapped to memory page by page, and the size of each page is the block size of an I / O performed by the operating system. The index module manages the cache itself instead of letting the operating system manage it, because within the scope of the program, it knows more about which pages need to be cached, while the cache management of the operating system is global and based on the information of the entire system.

[0051] vbyte module: responsible for the compression of postings chains.

[0052] tmpindex module: responsible for index updates.

[0053] merge module: used to merge temporary indexes with the main index, and to merge multiple runs when creating an index.

[0054] Index module: It is a package for the previous modules. It provides a packaged index interface and shields the internal implementation details. The index data structure is a good package for the index module. The outside world does not need to understand the internal implementation, but only needs to call it through some function interfaces.

[0055] Ehash is the center of the entire module. It manages each bucket through an extensible hash algorithm. The allocation operation of a bucket is actually to apply for disk space by calling the freemap module. The mapping of these disk spaces to memory is managed by cache, so ehash depends on the freemap and cache modules, while freemap and cache, bucket and cache are indirect relationships. These modules are closely linked and together form the core of the index structure. For batch updates, the index system maintains a temporary index, namely the tmpindex module. The merge module merges the main index and the temporary index at a certain time. The entire index system is finally packaged by an index submodule to provide a simple interface to the outside world.

[0056] Embodiment 5

[0057] The font query module has the following query operations for the ontology: determine whether a word is an ontology concept; return a candidate extended concept set for a concept, that is, a set of concepts whose relevance to the concept is greater than a certain threshold; obtain the correlation between each concept in a query candidate extended concept and the query concept vector. In the semantic search module, it is necessary to return a query-related concept set for a concept, that is, a set of concepts whose relevance to the concept is greater than a certain threshold. This module provides two classes, namely ConceptSet and SimMatrix.

[0058] The query module performs conceptual expansion on the conceptual part of the query. In the specific implementation, the final result of the query semantic expansion scheme based on concept expansion and keyword expansion is the expanded query vector. Therefore, the query expansion is integrated into the query vector generation process. The query vector is represented in the program as the class QueryVector.

[0059] The QueryVector class encapsulates query concept expansion and query keyword expansion, so the code for obtaining an extended vector of a query statement externally is very concise. Assuming that query is a query string of string type, the statement QueryVectorqueryVector(query) generates an expanded query vector queryVector object. The conceptVector member of the queryVector object stores the expanded query concept vector, and the keywordVector member stores the expanded query keyword vector.

[0060] For the expanded query vector, the semantic search module first obtains the relevant texts and the text query related concept set and text query related keyword set corresponding to the relevant texts through indexing, that is, generates a set of triples in the form of <text ID, concept, weight> and <text ID, keyword, weight>; then, for each relevant text, calculates its correlation with the query concept vector and the query keyword vector; finally, the calculated query concept correlation and query keyword correlation are merged to finally obtain the correlation score between the relevant text and the query and sort the relevant texts according to the correlation score.

[0061] This module needs to interact with the query module to obtain the expanded query vector; it needs to interact with the index module to obtain the query text related concept set and the query text related keyword set, as well as the weights of the terms in these two sets in the related text; it needs to interact with the font query module, using the concept relevance query function of the font query module and the function of returning a concept set with a relevance greater than a certain threshold to support the calculation of the relevance between the related text and the query.

[0062] The semantic resource module uses the extended version of the synonym dictionary to provide query operations related to synonyms. In the query keyword expansion operation, it is necessary to obtain the synonym set for the non-conceptual keyword; in the process of generating the text query concept set in the semantic search module, it is necessary to superimpose the weights of the synonyms of a query concept in proportion to form a reasonable query text concept vector, so there is a need to obtain the synonym set of the concept. The interactive module is responsible for accepting the user's query statement, transferring it to the query module to generate an expanded query vector, and then passing the expanded query vector to the semantic search module, and finally presenting the relevant text generated in the semantic search module, which is sorted from large to small in terms of relevance, to the user.

[0063] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0064] In various embodiments, the hardware implementation of the technology can directly adopt existing intelligent devices, including but not limited to industrial computers, PCs, smart phones, handheld stand-alone computers, floor-standing stand-alone computers, etc. Its input device preferably adopts a screen keyboard, its data storage and calculation module adopts existing memory, calculator, controller, its internal communication module adopts existing communication ports and protocols, and its remote communication adopts existing GPRS network, World Wide Internet, etc.

[0065] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0066] Each functional unit in each embodiment of the present invention may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and may also instruct the relevant hardware to complete it through a computer program. The computer program may be stored in a computer-readable storage medium, and the computer program may be executed by the processor to implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunications signals.

[0067] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0068] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0069] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the claims of the present invention.< / string> < / string> < / string>

Claims

1. A Chinese text semantic analysis and reconstruction system, characterized in that: include: Text preprocessing module, font query module, semantic resource module, query module, semantic search module and interaction module; The text preprocessing module is used to generate text vectors. First, the captured network texts are uniformly converted into XML description texts, and then the XML description texts are segmented and density diffusion processed to obtain a reasonable weight for each text term. Finally, the texts are represented as text vectors and the text vectors are stored in an index. The XML description texts are generated based on an information extraction scheme of the structural characteristics of semi-structured texts. The font query module is used to process concept relevance queries and concept judgment of terms to support the needs of query semantic expansion and concept retrieval; The query module is used to perform concept expansion on the concept part of the query to obtain the expanded query vector, and perform synonym expansion on the non-concept part; The semantic resource module is used to package the synonym word forest resources and respond to the query request about the synonym. The query module and the semantic search module interact with the semantic resource module. The query module is used to realize the keyword expansion of concept extension and non-concept content. The semantic search module is used to calculate the correlation between the query and the related text according to the query request of the user and sort the related text according to the correlation. The interaction module is used to interact with the user; The semantic search module is used for the expanded query vector. The semantic search module first obtains the relevant text and the text query related concept set and the text query related keyword set corresponding to the relevant text through indexing, and then calculates the correlation between each relevant text and the query concept vector and the query keyword vector, and finally merges the calculated query concept correlation and query keyword correlation, finally obtains the correlation score between the relevant text and the query and sorts the relevant text according to the correlation score. The semantic resource module uses the extended version of the synonym dictionary to provide query operations related to synonyms. In the query keyword expansion operation, a synonym set is obtained for a non-concept keyword. In the process of generating a text query concept set in the semantic search module, the weights of synonymous concepts of a query concept are proportionally superimposed together to form a reasonable query text concept vector.

2. The Chinese text semantic analysis and reconstruction system according to claim 1, characterized in that: The text preprocessing module includes a word segmentation module, a term setting module, and an indexing module.

3. The Chinese text semantic analysis and reconstruction system according to claim 2, characterized in that: The word segmentation module is used to perform statistical calculations on the keywords obtained by text segmentation to obtain the weight of each word item so as to create an index for the text.

4. The Chinese text semantic analysis and reconstruction system according to claim 3 is characterized by: The term setting module is used to obtain the final text vector representation for the unified XML description text, wherein the XML description text used as input in the module is obtained by using information extraction technology on network text.

5. The Chinese text semantic analysis and reconstruction system according to claim 4, characterized in that: The index module is implemented by an inverted index based on scalable hashing, which makes each bucket occupy at most one storage block, thereby shortening the query time.

6. The Chinese text semantic analysis and reconstruction system according to claim 5, characterized in that: The font query module is used to determine whether a word is an ontology concept, return a candidate extended concept set for a concept, that is, a concept set whose relevance to the concept is greater than a certain threshold, and obtain the relevance between each concept in a query candidate extended concept and the query concept vector.

7. The Chinese text semantic analysis and reconstruction system according to claim 1, characterized in that: The interactive module is used to accept the user's query statement, transfer it to the query module to generate an expanded query vector, and then pass the expanded query vector to the semantic search module, and finally present the relevant text generated in the semantic search module and sorted from large to small according to the relevance size to the user.

Images