Knowledge base analysis method based on MAAS platform

Abstract

The invention discloses a knowledge base analysis method based on an MAAS platform. The analysis method comprises the following steps: establishing a knowledge base platform; acquiring external data according to the question and answer condition, and performing data processing to obtain processed data; and carrying out data storage on the processed data. According to the method, the ability to understand and analyze questions is improved, the context relationship can be understood, the answer accuracy is improved, and the method has better learning and understanding ability to files of picture, audio and video types.

Description

Technical Field

[0001] The present invention relates to the field of enterprise-level software development, and in particular to a knowledge base parsing method based on a MAAS platform. Background Art

[0002] The introduction of the MAAS platform solves multiple pain points in question and answering based on NLP natural language processing knowledge base.

[0003] First, NPL natural language processing has limited semantic understanding, which may lead to the situation where the question is not answered.

[0004] Second, due to the limitations of NPL natural language processing dialogue analysis, it is impossible to parse contextual relationships, and when similar questions are asked, there will be two different or completely unrelated answers.

[0005] Third, when asking questions based on NPL natural language processing that are beyond the scope of the knowledge base, there will be questions that cannot be answered.

[0006] The introduction of the MAAS platform improves the ability to understand and parse questions, understand contextual relationships, improve the accuracy of answers, and has a better ability to learn and understand image, audio, and video files. Summary of the invention

[0007] In view of the above problems, the present invention is proposed to provide a MAAS platform knowledge base parsing method that overcomes the above problems or at least partially solves the above problems.

[0008] According to one aspect of the present invention, a method for parsing a knowledge base based on a MAAS platform is provided, the method comprising:

[0009] Establish a knowledge base platform;

[0010] Obtain external data based on the question and answer situation, and process the data to obtain processed data;

[0011] The processed data is stored in data.

[0012] Optionally, the knowledge base platform establishment specifically includes: QA question answering system, NLP natural language processing and MAAS platform.

[0013] Optionally, the QA question-answering system is developed using Spring Cloud architecture and implemented using HTTP+VUE front-end interaction.

[0014] Optionally, the NLP natural language processing improves the knowledge analysis capability in the financial field according to ERNIE-Finance;

[0015] Search ERNIE-Search's neural retriever based on a pre-trained language model improves open-domain question answering QA with better feedback capabilities; dialogue PLATO dialogue text understanding.

[0016] Optionally, the MAAS platform uses document images, product graphics, multi-task vision, and self-supervised visual representation learning models to improve question-answering capabilities and multi-scenario response capabilities. More learning models will be connected to the MAAS platform in the future.

[0017] Optionally, the acquisition of external data according to the question and answer situation and data processing specifically include: message queue, data collector, file synchronization, data labeling and processing;

[0018] The data collector pulls the company's intranet / extranet data materials based on Python scripts;

[0019] The file synchronization relies on rsync+inotify to synchronize the file data for the data material of the document video.

[0020] Optionally, the message queue relies on Kafka to send and receive messages to smooth out peak traffic;

[0021] The data annotation and processing are based on Java, using OpenCSV / Apache POI to process document data and penCV / Ffmpeg to process pictures and videos, and marking the source of data types.

[0022] Optionally, the data storage includes application data storage and file data storage, relies on the JuiceFS open source distributed file system, and uses POSIX and HDFS protocols to manage files.

[0023] Optionally, the application data storage includes: recording application running status, question and answer content, and part of the model training data; the data is stored in MySQL;

[0024] The file data stores text data and binary data files.

[0025] The present invention provides a method for parsing a knowledge base based on a MAAS platform, the method comprising: establishing a knowledge base platform; acquiring external data according to the question and answer situation, and performing data processing to obtain processed data; and storing the processed data. The method improves the ability to understand and parse questions, understands contextual relationships, improves the accuracy of answers, and has a better ability to learn and understand files of picture, audio, and video types.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1 A flowchart of a MAAS platform knowledge base parsing method provided in an embodiment of the present invention;

[0029] Figure 2 An architecture diagram of a system corresponding to a MAAS platform knowledge base parsing method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0032] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0033] Example 1

[0034] A method for improving the parsing capability of a MAAS platform knowledge base includes:

[0035] Knowledge base platform, data processing, data storage.

[0036] Knowledge base platform, including QA question-answering system, NLP natural language processing, MAAS platform. QA question-answering system is developed using Spring Cloud architecture and implemented with HTTP+VUE front-end interaction.

[0037] NLP natural language processing relies on ERNIE-Finance to improve the ability to analyze financial knowledge; ERNIE-Search uses a neural searcher based on a pre-trained language model to improve open-domain question-answering QA with better feedback; and PLATO dialogues understand dialogue text. MAAS platform uses document images, product images, multi-task vision, and self-supervised visual representation learning models to improve question-answering capabilities and multi-scenario response capabilities. More learning models can be connected to the MAAS platform in the future.

[0038] Data processing, including message queues, data collectors, file synchronization, data annotation and processing. The data collector pulls data materials from the company's intranet / extranet based on Python scripts. File synchronization relies on rsync+inotify for file data synchronization for data materials such as documents and videos. Message queues rely on kafka for message sending and receiving to smooth out peaks and fill valleys.

[0039] Data annotation and processing, using OpenCSV / Apache POI based on java to process document data and penCV / Ffmpeg to process images and videos, and annotate the source of data types.

[0040] Data storage includes application data storage and file data storage. Application data storage records application running status, Q&A content, some model training data, etc. The data is stored in MySQL. File data stores text data and binary data files. It relies on JuiceFS open source distributed file system and uses POSIX and HDFS protocols to manage files.

[0041] Based on the Nacos registration center, communication and access between various systems are realized.

[0042] Developed based on Spring Cloud architecture, based on Http communication protocol + Json message format.

[0043] Unified log printing based on Log4j2 framework.

[0044] Implement server-side code optimization based on IntelliJ IDEAFormatter.

[0045] Front-end code optimization based on Prettier.

[0046] Implement the repository layer code based on MyBatisPuls.

[0047] Acquire company intranet data based on Python.

[0048] Realize company intranet file synchronization based on rsync+inotify.

[0049] Data synchronization is achieved based on the kafak message queue.

[0050] Rely on NLP natural language processing to answer questions.

[0051] Rely on the MAAS platform to expand the question-answering model.

[0052] Relies on OpenCSV / Apache POI to process text data.

[0053] Rely on OpenCV / FFmpeg to process video and image data.

[0054] Account authentication is achieved by using spring security.

[0055] By using Spring Data Redis+Redis to cache hot data and distributed session cache.

[0056] Example 2

[0057] like Figure 1 As shown, an overall framework diagram for building a system to improve the knowledge base parsing capability based on the MAAS platform should include a knowledge base platform (QA question and answer system, NLP, MAAS), data processing (message queue, data collector, file synchronization, data labeling and processing), and data storage (application data storage, file data storage).

[0058] like Figure 2 As shown in the figure, a system architecture diagram for improving the knowledge base parsing capability of the MAAS platform is implemented. Each system is developed based on the Spring Cloud architecture and uses the Http communication protocol + Json message format for message transmission;

[0059] Each system uses Nacos as a service discovery and registration center to achieve access and scheduling between systems, health checks, dynamic configuration, and grayscale releases;

[0060] The knowledge base platform includes QA question-answering system, NLP natural language processing, and MAAS platform. The QA question-answering system is developed using the Spring Cloud architecture and implemented with HTTP+VUE front-end interaction. NLP natural language processing relies on ERNIE-Finance to improve the ability to analyze knowledge in the financial field; ERNIE-Search improves the open domain QA with better feedback capabilities; and PLATO supports dialogue text understanding. The MAAS platform supports multi-model expansion. The current system uses document images, product images, multi-task vision, and self-supervised visual representation learning models to improve question-answering capabilities and multi-scenario response capabilities.

[0061] Data processing, external data needs to be obtained according to the question and answer situation. Including message queues, data collectors, file synchronization, data labeling and processing. Message queues rely on kafka to send and receive external information, smooth out peaks and fill valleys, and decouple applications. The data collector pulls data materials from the company's intranet / extranet based on Python scripts. File synchronization, for data materials such as documents and videos, rsync+inotify is relied upon for file data synchronization. Data labeling and processing, document data is processed using OpenCSV / Apache POI based on java, and images and videos are processed using penCV / Ffmpeg, and the source of data types are labeled, etc.;

[0062] Data storage, storing application data and raw data for training. Including application data storage and file data storage. Application data storage, recording application running status, Q&A content, some model training data, etc., data is stored in MySQL. File data, storing text data and binary data files. Relying on JuiceFS open source distributed file system, using POSIX and HDFS protocols to manage files.

[0063] Beneficial effects:

[0064] Improve the ability to understand and parse questions, understand contextual relationships, improve answer accuracy, and have a better ability to learn and understand image, audio, and video files.

[0065] While satisfying customer experience, we can also understand what customers are asking and what they are concerned about, and continuously enrich the content of the knowledge base to make the knowledge base more valuable.

[0066] The knowledge base based on NPL natural language processing is optimized and the MAAS platform is introduced to improve the ability to understand and parse problems, understand contextual relationships, improve answer accuracy, and have better learning and understanding capabilities for image, audio, and video type files.

[0067] By introducing the MAAS platform and representation learning models such as documents and images, the breadth of the knowledge base is increased, the original NPL is supplemented and optimized, the accuracy of the knowledge base question and answer scenarios is improved, and the platform's ability to query data online is improved through data collection, labeling and processing.

[0068] The question-answering model uses NPL natural language processing to improve the system's understanding of the questioner's content and improve the accuracy of questions and answers. It uses the MAAS platform model and services to improve the application's learning and expansion capabilities and reduce development and operating costs.

[0069] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A knowledge base parsing method based on the MAAS platform, characterized in that: The analysis method comprises: Establish a knowledge base platform; Obtain external data based on the question and answer situation, and process the data to obtain processed data; The processed data is stored in data.

2. A method for parsing a knowledge base based on a MAAS platform according to claim 1, characterized in that: The establishment of the knowledge base platform specifically includes: QA question-answering system, NLP natural language processing and MAAS platform.

3. A method for parsing a knowledge base based on a MAAS platform according to claim 2, characterized in that: The QA question-and-answer system is developed using the Spring Cloud architecture and implemented using HTTP+VUE front-end interaction.

4. A method for parsing a knowledge base based on a MAAS platform according to claim 2, characterized in that: The NLP natural language processing improves the knowledge analysis capability in the financial field according to ERNIE-Finance; Search ERNIE-Search's neural retriever based on a pre-trained language model improves open-domain question answering QA with better feedback capabilities; dialogue PLATO dialogue text understanding.

5. A method for parsing a knowledge base based on a MAAS platform according to claim 2, characterized in that: The MAAS platform uses document images, product graphics, multi-task vision, and self-supervised visual representation learning models to improve question-answering capabilities and multi-scenario response capabilities. More learning models will be connected to the MAAS platform in the future.

6. A method for parsing a knowledge base based on a MAAS platform according to claim 1, characterized in that: The acquisition of external data according to the question and answer situation and data processing specifically include: message queue, data collector, file synchronization, data labeling and processing; The data collector pulls the company's intranet / extranet data materials based on Python scripts; The file synchronization relies on rsync+inotify to synchronize the file data for the data material of the document video.

7. A method for parsing a knowledge base based on a MAAS platform according to claim 6, characterized in that: The message queue relies on Kafka to send and receive messages, shaving peaks and filling valleys; The data annotation and processing uses OpenCSV / Apache POI based on java to process document data and penCV / Ffmpeg to process pictures and videos, and annotate the data type source.

8. The method for parsing a knowledge base based on a MAAS platform according to claim 1, characterized in that: The data storage includes application data storage and file data storage, relies on the JuiceFS open source distributed file system, and uses POSIX and HDFS protocols to manage files.

9. A method for parsing a knowledge base based on a MAAS platform according to claim 8, characterized in that: The application data storage includes: recording application running status, question and answer content, and part of the model training data; the data is stored in MySQL; The file data stores text data and binary data files.

Images