Statistical analysis engine device of unstructured file based on AI semantic understanding

Abstract

The invention discloses an unstructured file statistical analysis engine device based on AI semantic understanding. An uploaded file is processed through a knowledge base management module; the rule setting module is used for setting a reasoning triggering rule and controlling a reasoning output result; the knowledge reasoning module records and reasones all files and problems in the knowledge base one by one according to rules in the rule setting module and records results; the knowledge analysis module automatically analyzes the returned data and removes duplication; the knowledge storage module automatically creates a corresponding data service table according to format requirements; the statistical analysis module performs structured analysis on the data and generates a corresponding statistical list or content; and the knowledge display module displays the content generated by statistics according to different display modes and traces back to the original text file, and the unstructured text is creatively converted into the structured text file through combined use of the modules.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a statistical analysis engine device for unstructured files based on AI semantic understanding. Background Art

[0002] In the existing technology, questions and answers can be conducted on the large model display library, but accurate answers cannot be obtained. Due to the limitations of the large model, it is impossible to accurately and detailedly analyze and summarize the information in the model; it is impossible to automatically obtain the specific content desired, and it is impossible to accurately hit the query content in an intuitive way. Summary of the invention

[0003] The purpose of the present invention is to provide a statistical analysis engine device for unstructured files based on AI semantic understanding to solve one or more problems raised in the above-mentioned background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a statistical analysis engine device for unstructured files based on AI semantic understanding: comprising a knowledge base management module, a rule setting module, a knowledge reasoning module, a knowledge parsing module, a knowledge storage module, a statistical analysis module, and a knowledge display module. The knowledge base management module is used to classify and process the uploaded files; The rule setting module is used to set the rules for triggering reasoning and control the output results of reasoning; The knowledge reasoning module is used to record the reasoning and results of all the files and questions in the knowledge base one by one according to the rules in the rule setting module, and verify the reasoning results. If the verification is incorrect, the polling reasoning is performed again, and the rule module is adjusted in the opposite direction; The knowledge parsing module is used to automatically parse and remove duplicates from the returned data; The knowledge storage module is used to analyze the format requirements set in the question-answer rule module, automatically create the corresponding data business table, and name it according to the rules; The statistical analysis module is used to perform structured analysis on the data and generate corresponding statistical lists or contents; The knowledge display module is used to display through different display modes, and the displayed content can be traced back to the original document.

[0005] Preferably, the knowledge base management module connects to the business system or manually uploaded file information according to different standard interfaces to classify and process the files; the file classification method is to process according to different file forms and formats, and convert various types of files into OFD files or other file formats; the processing methods include OCR recognition, audio and video text recognition, various waveform recognition, and document content batch supplement processing, wherein the document content batch processing method refers to the content in the file or the existence of certain agreed relationships between users between files. Such relationships are known to users and do not need to be described in the file. Due to the establishment of the knowledge base, files from various dimensions and directions are stored in the library, and the original readers are changed from users to robots, then the original agreed relationship will not be understood by the semantic robot, and may even be misunderstood. In order to prevent such problems from occurring, one-time or repeated annotations are performed on key parts of the file when the file is stored, and such behavior is document content batch supplement processing.

[0006] Preferably, the rule setting module triggers reasoning behavior in the form of data entry, question and answer, API call, and scheduled task by setting a set of rules for triggering reasoning, and controls the results of knowledge reasoning output. The restriction methods include limiting the content type, content length, and content format of the output answer; if the answer output by the knowledge reasoning does not meet the requirements, multiple rounds of question and answer are set; for complex knowledge matching, a set of question and answer rules can be set for it, and according to different answer replies, the output answer can be queried again to obtain results that meet the format requirements, and the results are output in a semi-structured format. The output formats include xml and json.

[0007] Preferably, in the knowledge reasoning module, the rule model in the rule setting module is input into the model, and then all uploaded files and questions are reasoned one by one, and the results are recorded one by one; the recorded results include the content of the reasoning, the result of the reasoning, the reference file of the reasoning, and the verification result of the reasoning record result; if the output answer is not output according to the format, the question is asked again N times, and the reason is recorded if the number of times exceeds the limit, and various thresholds and rules are adjusted in reverse, and the rule conditions in the setting, addition, and expansion module are set.

[0008] Preferably, the knowledge parsing module automatically performs json or xml parsing on the returned data according to the format for the reasoning results in the knowledge reasoning module, and records the source of the reasoning content. For multiple source references for the same problem, the reverse reasoning confirmation method is used to confirm whether the source of the knowledge is accurate. If not, the source is discarded. Finally, the content of the reasoning, the result of the reasoning, the reference file of the reasoning, and the specific output source are organized in an array form to form a structured reasoning result information.

[0009] Preferably, the knowledge storage module automatically creates the corresponding data business table and names it according to the format requirements set in the knowledge reasoning module; the naming method is Chinese annotation, with the first letter of the pinyin as the English name, and if there is a repetition, a timestamp is added; the output content field in json or xml is used as a field, and the data is written into the data landing system, and the entered data is matched according to the rules and the data is cleaned.

[0010] Preferably, when displaying data, the statistical analysis module uses a graphical drag and drop and configuration method to perform interface analysis and configuration on the structure and data in the data table to generate a corresponding statistical list or statistical interface.

[0011] Preferably, the knowledge display module is configured with different display modes, including list, relational model, and timeline; if the display mode is a list, an Excel table or report is exported; if a relational model is used to display the relationship between people, enterprises, and objects in the knowledge base, the relationship forms include progressive, causal, inclusion and exclusion, comparison, association, and parallel relationship; if the displayed content relationship is a parallel relationship, the exported form includes x-mind and mind map; if the displayed content relationship is progressive, causal, inclusion and exclusion, comparison, and association, the exported text format is word, wps, true, html, pdf, ofd; various chart displays are supported at the same time; the timeline refers to a tool for representing time, which is displayed in text or graphics; during the display process, all, part, or related parts can be selected for display; and the results can be supplemented and corrected.

[0012] Preferably, each content of the reasoning displayed in the knowledge display module can be traced back to the original text by clicking. When tracing back, the file is displayed in the form of a bound volume; and multiple sources are summarized in a guided manner, and each source can be switched and clicked to jump to read; at the same time, a list of similar questions is displayed in association, and different questions are switched for comparative display and reading. The reading can revise and refine the questions, results, and reference content; among them, if the file is a scan or a picture, it can jump to the OFD file after OCR for precise matching and tracing; the content of the knowledge display module can be summarized according to the preset template to form a corresponding streaming or layout or web page file, which can be forwarded offline or backed up and stored.

[0013] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention makes full use of AI semantic understanding technology and creatively realizes the conversion of unstructured text into structured text files and the conversion of unstructured content into structured content; (2) The present invention makes full use of AI semantic understanding technology to enable users to arbitrarily extract the content of unstructured knowledge bases according to actual business needs, and generate various structured reports, timelines, relationships, relationship maps of things (conditions) comparisons, mind maps, etc. that they need, greatly improving the user's operating experience; (3) The present invention makes full use of the technology of virtual robots, which can traverse the entire knowledge base file by file, and achieve the maximum extent possible without blind spots or omissions in knowledge reasoning; (4) The present invention supports direct export of various file formats, allowing users to quickly obtain the desired streaming and layout files from a pile of knowledge base files. DETAILED DESCRIPTION

[0014] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Example

[0015] The present invention provides a statistical analysis engine device for unstructured files based on AI semantic understanding, including a knowledge base management module, a rule setting module, a knowledge reasoning module, a knowledge parsing module, a knowledge storage module, a statistical analysis module, and a knowledge display module. The knowledge base management module connects to the business system or manually uploaded file information according to different standard interfaces, and classifies and processes the uploaded files. When classifying files, it processes them according to different file forms and formats, and classifies various files such as doc, docx, wps, xls, xlsx, ppt, pptx, txt, bmp, jpg, png, tif, gif, wmv, rm, mov, mp4, avi, mp3, html, true Converted into OFD file or PDF or layout data stream file; the processing method can be OCR recognition, audio and video text recognition, various waveform recognition such as audio waveform recognition, document content batch supplement processing and so on. Among them, the document content batch processing method refers to the content in the file or the existence of certain agreed relationships between users between files. Such agreed relationships are known to users and do not need to be described in the file. Due to the establishment of the knowledge base, files from various dimensions and directions are stored in the library. The original reader is changed from the user to the robot, so the original agreed relationship will not be understood by the semantic robot, or even misunderstood. In order to prevent such problems from occurring, one-time or repeated annotations are made on the key parts of the file when the file is stored in the library. Such behavior is document content batch supplement processing; for example, the common way of question and answer in judicial records is one question and one answer, and the user is the subject who understands this one question and one answer. However, after such a file is stored in the library, the robot has no way to understand the specific question and answer subject. The document content can be batch supplemented to allow the robot to understand who is asking and who is answering; or there is a causal relationship between files, so it is necessary to mark between files.

[0016] The rule setting module triggers reasoning behavior by setting a set of rules for triggering reasoning, and controls the results of knowledge reasoning output by means of data entry, question and answer, API call, and scheduled tasks. For example, when using the question and answer method to control the results of reasoning output, the control method includes controlling the content type, content length, and content format of the output answer. For example, the information of the suspect's crime is obtained in the format of [{'name': '','means of crime':'','location of crime':'','time of crime':''}] and output in JSON format; if the answer output by the knowledge question and answer does not meet the requirements, multiple rounds of question and answer can be set to obtain a result that meets the format requirements; according to different answer replies, the result is queried twice, in the format of [{'name': '','gender': '','ID number': '','Place of registered residence':'','Current address':''}] to obtain the suspect's basic information and output it in JSON format. If the current person being questioned is a criminal suspect, if so, output true, if not, output false. For example, if it is for complex knowledge matching, you can set a set of question-and-answer rules for it, and reply based on different answers, ask the output answer again to obtain a result that meets the format requirements, and output the result in a semi-structured format. The output format can be xml or json.

[0017] In the knowledge reasoning module, the rule model in the rule setting module is input into the model, and then all uploaded files and questions are reasoned one by one, and the results are recorded one by one; the recorded results include the content of the reasoning, the result of the reasoning, the reference file of the reasoning, and the verification result of the reasoning record result; if the output answer is not output according to the format, the question and answer is repeated, and the question and answer can be repeated several times. If the number of times exceeds the limit, the reason is recorded, and various thresholds are adjusted in reverse. For example, in the actual operation process, only 80% of the similarity can be matched from the knowledge base, or 0.9 is taken according to the title and 0.8 is taken according to the content, and then the similarity value is taken to finally determine the rule, and the rule conditions in the knowledge reasoning module can be set, added, and expanded, so that the rule setting module can be flexibly adjusted according to the user's use.

[0018] For the inference results in the knowledge reasoning module, the knowledge parsing module automatically parses the returned data in json or xml format according to the inference results, and records the source of the inference content. For multiple source references of the same problem, reverse reasoning can be used to confirm whether the source of the knowledge of the inference result is accurate. If not, the source is discarded. Finally, the inference content, reasoning results, reasoning reference files, and specific output sources are organized in array form to form a structured reasoning result information for easy viewing.

[0019] The knowledge storage module automatically creates the corresponding data business table and names it according to the format requirements set in the knowledge reasoning module; the naming method is Chinese annotation, with the first letter of the pinyin as the English name. If there is a duplication, a timestamp is added; the output content field in json or xml is used as a field, and the data is written into the data landing system, and the entered data is matched and cleaned according to the rules.

[0020] The statistical analysis module performs structured analysis on the data in the knowledge storage module and generates corresponding statistical lists or contents. When displaying data, the statistical analysis module uses graphical drag and drop and configuration methods to perform interface analysis and configuration on the structure and data in the data table and generate corresponding statistical lists or statistical interfaces. The knowledge display module can be set to different display modes, which can be lists, relationship models, timelines, etc.; if the display mode is a list, an Excel table or report will be exported; when using a relationship model to display the relationship between people, companies, and objects in the knowledge base, lists, progressions (such as timelines, event axes, increasing and decreasing), causality (such as conditional relationships), inclusion and exclusion, comparisons (such as contradictory relationships), associations, and parallel relationships can be used; if the displayed content relationship is a parallel relationship, the exported forms include x-mind and mind maps; if the displayed content relationship is other than a parallel relationship, the exported text format is word, wps, true, html, pdf, ofd; it can also support various chart displays; the timeline refers to a tool for expressing time, which is displayed in text or graphics; during the display process, users can choose to display all, part, or related parts; they can also supplement and correct the results; for example, if there is no specific time description in the text, the user can supplement the time based on their understanding of the event; if the same thing has different names, it may be It will be understood as multiple things, and the names of things can be unified; in the knowledge display module, the various contents of the reasoning displayed by the knowledge display module can be traced back to the original text by clicking. When tracing back, the file is displayed in the form of a bound volume; and multiple sources are summarized in a guided manner, and each source can be switched and clicked to jump to read; at the same time, a list of similar questions is displayed in association, and different questions are switched for comparative display and reading. Reading can revise and refine the questions, results, and reference content; among them, if the file is a scan or a picture, it can jump to the OFD file after OCR for precise matching and tracing; the content of the knowledge display module can be summarized according to the preset template to form a corresponding streaming or layout or web page file, which can be forwarded offline or backed up and stored.

[0021] Therefore, through the interaction of the knowledge base management module, rule setting module, knowledge reasoning module, knowledge parsing module, knowledge storage module, statistical analysis module, and knowledge display module, the AI ​​semantic understanding technology is fully utilized to enable users to extract the content of the unstructured knowledge base at will according to actual business needs, and generate various structured reports, timelines, relationships, relationship diagrams of things (conditions) comparisons, mind maps, etc. that they need, which greatly improves the user's operating experience.

[0022] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments are to be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A statistical analysis engine device for unstructured files based on AI semantic understanding, characterized by: It includes knowledge base management module, rule setting module, knowledge reasoning module, knowledge parsing module, knowledge storage module, statistical analysis module and knowledge display module. The knowledge base management module is used to classify and process the uploaded files; The rule setting module is used to set the rules for triggering reasoning and control the output results of reasoning; The knowledge reasoning module is used to record the reasoning and results of all the files and questions in the knowledge base one by one according to the rules in the rule setting module, and verify the reasoning results. If the verification is incorrect, the polling reasoning is performed again, and the rule module is adjusted in the opposite direction; The knowledge parsing module is used to automatically parse and remove duplicates from the returned data; The knowledge storage module is used to analyze the format requirements set in the knowledge reasoning module, automatically create the corresponding data business table, and name it according to the rules; The statistical analysis module is used to perform structured analysis on the data and generate corresponding statistical lists or contents; The knowledge display module is used to display through different display modes, and the displayed content can be traced back to the original document.

2. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: The knowledge base management module connects to the business system or manually uploaded file information according to different standard interfaces to classify and process the files; the file classification method is to process according to different file forms and formats, and convert various types of files into OFD files or other file formats; the processing methods include OCR recognition, audio and video text recognition, waveform recognition, and document content batch supplement processing, wherein the document content batch processing method refers to the content in the file or the existence of certain agreed relationships between users between files. Such relationships are known to users and do not need to be described in the file. Due to the establishment of the knowledge base, files from various dimensions and directions are stored in the library, and the original readers are changed from users to robots, then the original agreed relationships will not be understood by the semantic robots, and may even be misunderstood. In order to prevent such problems from occurring, one-time or repeated annotations are made to the key parts of the files when they are stored in the library, and such behavior is document content batch supplement processing.

3. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: The rule setting module triggers reasoning behavior by setting a set of rules for triggering reasoning, such as data entry, question and answer, API call, and scheduled task, and controls the results of knowledge reasoning output. The restrictions include limiting the content type, content length, and content format of the output answer; if the answer output by the knowledge reasoning does not meet the requirements, multiple rounds of question and answer are set; for complex knowledge matching, a set of question and answer rules are set for it, and based on different answers, the output answers are queried again to obtain results that meet the format requirements, and the results are output in a semi-structured format, and the output formats include xml and json.

4. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: In the knowledge reasoning module, the rule model in the rule setting module is input into the model, and then all uploaded files and questions are reasoned one by one, and the results are recorded one by one; the recorded results include the content of reasoning, the result of reasoning, the reference file of reasoning, and the verification result of the reasoning record result; If the output answer is not in accordance with the format, the question will be asked again N times. If the number of times exceeds the limit, the reason will be recorded, and various thresholds and rules will be adjusted in reverse, and the rule conditions in the module will be set, added, and expanded.

5. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: The knowledge parsing module automatically parses the returned data in json or xml format according to the reasoning results in the knowledge reasoning module, and records the source of the reasoning content. For multiple source references of the same problem, the reverse reasoning confirmation method is used to confirm whether the source of the knowledge is accurate. If not, the source is discarded. Finally, the content of the reasoning, the result of the reasoning, the reference file of the reasoning, and the specific output source are organized in array form to form a structured reasoning result information.

6. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: The knowledge storage module automatically creates the corresponding data business table and names it according to the format requirements set in the knowledge reasoning module; the naming method is Chinese annotation, with the first letter of the pinyin as the English name, and if there is a duplication, a timestamp is added; The output content field in json or xml is used as a field, and the data is written into the data landing system, and the entered data is matched and cleaned according to the rules.

7. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: When displaying data, the statistical analysis module uses a graphical drag and drop and configuration method to perform interface analysis and configuration on the structure and data in the data table to generate a corresponding statistical list or statistical interface.

8. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: The knowledge display module is provided with different display modes, including list, relationship mode, and timeline; If the display method is a list, an Excel table or report is exported; if a relational model is used to display the relationship between people, companies, and objects in the knowledge base, the forms of the relationship include progressive, causal, inclusion and exclusion, comparison, association, and parallel relationships; if the displayed content relationship is a parallel relationship, the exported form includes x-mind and mind map; if the displayed content relationship is progressive, causal, inclusion and exclusion, comparison, and association, the exported text format is word, wps, true, html, pdf, ofd; various chart displays are supported at the same time; the timeline refers to a tool for representing time, which is displayed in text or graphics; in the process of display, all, part, or related parts can be selected for display; and the results can be supplemented and corrected.

9. The statistical analysis engine device for unstructured files based on AI semantic understanding according to claim 1, characterized in that: In the knowledge display module, each content of the reasoning displayed in the knowledge display module can be traced back to the original text by clicking. When tracing back, the file is displayed in the form of a bound volume; and multiple sources are summarized in a guided navigation manner, and each source can be switched and clicked to jump to read; At the same time, a list of similar questions can be displayed in association, and different questions can be switched for comparative display and reading. Questions, results, and references can be revised and processed in detail during reading. If the file is a scan or a picture, it can jump to the OFD file after OCR for accurate matching and tracing. The content of the knowledge display module can be summarized according to a preset template to form a corresponding stream or layout or web page file, and forwarded offline or backed up and stored.