Information query system and method based on security compliance knowledge base

Abstract

The invention is suitable for the technical field of Internet, and provides an information query system and method based on a security compliance knowledge base. In the embodiment of the invention, the security compliance file is acquired through the data acquisition module so as to construct the security compliance database, and the latest regulations are quickly mastered through regular acquisition, so that the system is ensured to be capable of mastering the latest regulations in real time. And the security compliance file is processed and archived through the data conversion module and the archiving storage module, so that the system can enhance the context understanding capability and improve the answer accuracy. The efficient and accurate security compliance consultation service is provided for the user.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an information query system and method based on a securities compliance knowledge base. Background Art

[0002] With the rapid development of financial technology, higher requirements have been placed on the application of intelligent conversational big models in the field of securities compliance. As an advanced intelligent conversational big model, ChatGLM has powerful language processing capabilities, but it still has shortcomings in securities compliance knowledge processing. Summary of the invention

[0003] In view of this, the present application provides an information query system and method based on a securities compliance knowledge base to achieve rapid updating of the securities compliance knowledge base, ensure that the ChatGLM model can grasp the latest regulations in real time; strengthen the model's contextual understanding ability and improve the accuracy of answers; at the same time, enhance knowledge integration and screening capabilities and reduce interference from irrelevant information; finally, optimize the model processing speed and enhance real-time response capabilities to better meet the compliance needs of the modern securities market.

[0004] In a first aspect, the present application provides an information query system based on a securities compliance knowledge base, the system comprising a data acquisition module, an intermediate scanning module, a data conversion module, an archiving storage module and a data calling module;

[0005] The data collection module is used to collect the current stock of securities compliance files, convert the securities compliance files into target files in a preset format and save them to the data collection space, and record the file information of the target files, the file information including the file source and the update channel source;

[0006] The intermediate scanning module is used to obtain the target file from the data acquisition space and send the target file to the data conversion module;

[0007] The data conversion module is used to receive the target file sent by the intermediate scanning module, process and convert the target file according to the file type and file content of the target file, generate target data and send it to the archiving storage module;

[0008] The archiving storage module is used to store the received target data to achieve the creation and update of the securities compliance knowledge base;

[0009] The data calling module is used to, when receiving a query request, perform a query in the securities compliance knowledge base according to the query information in the query request, and output the target data with the highest matching degree with the query information.

[0010] Optionally, the data collection module also includes a timer, which is used to set the securities compliance document collection time. When the collection time is reached, the data collection module automatically collects the securities compliance document from the target website or target channel, and updates the collection time of the timer after the collection.

[0011] Optionally, the data conversion module performs processing and conversion according to the file type and file content of the target file, including:

[0012] Determine the file type of the target file, determine a tool to read the target file according to the file type, and read the target file;

[0013] The target file is segmented into paragraphs according to a preset character limit and file content, and special characters in the target file are processed to generate the target data.

[0014] Optionally, the data conversion module performs processing and conversion according to the file type and file content of the target file, including:

[0015] When it is determined that the target file is a pure data file, the target file is stored in groups and batches;

[0016] When it is determined that the target file is not a pure data file, the target file is divided into a word group and a pdf group according to the file type;

[0017] The target files in the word group are grouped into a word document set, each document in the word document set is traversed, and after being read into the memory to obtain a word document object, the data structure is split according to a specific format or paragraph, the split data content is assembled into data rows, and the data rows are grouped and stored in batches;

[0018] Convert each target file in the PDF group into word format, and group the converted target files into a word document set, traverse each document in the word document set, read it into memory to obtain a word document object, split the data structure according to a specific format or paragraph, assemble the split data content into data rows, and store the data rows in groups and batches.

[0019] Optionally, the outputting of the target data having the highest matching degree with the query information includes:

[0020] The target data that hit the query information are determined as initial target data, each initial target data is scored and sorted according to the number of times it hits the query information, and the initial target data with the highest score is determined as the target data with the highest matching degree with the query information and output.

[0021] The second aspect of the present application provides an information query method based on a securities compliance knowledge base, the method comprising:

[0022] Collect the current stock of securities compliance documents, convert the securities compliance documents into target files in a preset format and save them in the data collection space, and record the file information of the target files, including the file source and the update channel source;

[0023] The intermediate scanner scans the target file to be processed in the data collection space, and sends the target file to the data converter;

[0024] After receiving the target file, the data converter processes and converts the target file according to the file type and file content of the target file, generates target data and files it into the securities compliance knowledge base;

[0025] When a query request is received, a query is performed in the securities compliance knowledge base according to the query information in the query request, and target data with the highest matching degree with the query information is output.

[0026] Optionally, the processing and conversion according to the file type and file content of the target file includes:

[0027] Determine the file type of the target file, determine a tool to read the target file according to the file type, and read the target file;

[0028] The target file is segmented into paragraphs according to a preset character limit and file content, and special characters in the target file are processed to generate the target data.

[0029] Optionally, the processing and conversion according to the file type and file content of the target file includes:

[0030] When it is determined that the target file is a pure data file, the target file is stored in groups and batches;

[0031] When it is determined that the target file is not a pure data file, the target file is divided into a word group and a pdf group according to the file type;

[0032] The target files in the word group are grouped into a word document set, each document in the word document set is traversed, and after being read into the memory to obtain a word document object, the data structure is split according to a specific format or paragraph, the split data content is assembled into data rows, and the data rows are grouped and stored in batches;

[0033] Convert each target file in the PDF group into word format, and group the converted target files into a word document set, traverse each document in the word document set, read it into memory to obtain a word document object, split the data structure according to a specific format or paragraph, assemble the split data content into data rows, and store the data rows in groups and batches.

[0034] Optionally, the outputting of the target data having the highest matching degree with the query information includes:

[0035] The target data that hit the query information are determined as initial target data, each initial target data is scored and sorted according to the number of times it hits the query information, and the initial target data with the highest score is determined as the target data with the highest matching degree with the query information and output.

[0036] In the embodiment provided by the present application, securities compliance documents are collected through the data collection module to build a securities compliance database, and the latest regulations are quickly mastered through regular collection, ensuring that the system can master the latest regulations in real time. And through the data conversion module, the archiving storage module processes and archives the securities compliance documents, so that the system can enhance the context understanding ability and improve the accuracy of the answer. Provide users with efficient and accurate securities compliance consulting services. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A system module diagram provided for an embodiment of the present application;

[0038] Figure 2 A flowchart of file processing provided for an embodiment of the present application;

[0039] Figure 3 A flow chart of the method provided in the embodiment of the present application;

[0040] Figure 4 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0042] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] This application provides an information query system based on a securities compliance knowledge base to provide efficient and accurate securities compliance questions and answers.

[0045] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0046] like Figure 1 As shown, this is a module diagram of an information query system based on a securities compliance knowledge base provided by the present application. The system includes a data acquisition module, an intermediate scanning module, a data conversion module, an archiving storage module and a data calling module.

[0047] The functions of each module are described below:

[0048] 1. The data collection module is used to convert the current existing securities compliance files into a target file of a preset format and save it to the data collection space after collecting the current existing securities compliance files, and record the file information of the target file, which includes the file source and the update channel source. In this module, after downloading securities compliance files from authoritative websites and related channels, such as the official website of the China Securities Regulatory Commission (CSRC) and the official website of the Shanghai Stock Exchange (SSE), they can be converted into PDF / WORD format and saved to the data collection space.

[0049] In another embodiment, this module can also set a timer for the collection time. When the collection time set on the timer is reached, this module automatically collects securities compliance documents from authoritative websites and related channels to achieve automatic update of compliance documents. Moreover, after the compliance file is updated, the collection time of the timer is automatically updated, thereby achieving uninterrupted automatic update.

[0050] 2. The intermediate scanning module is used to obtain the target file from the data acquisition space and send the target file to the data conversion module. This module can also realize timed acquisition and sending through a timer.

[0051] 3. The data conversion module is used to receive the target file sent by the intermediate scanner, process and convert the target file according to the file type and file content of the target file, generate target data and send it to the archiving storage module.

[0052] Exemplarily, after determining the file type of the target file, a tool for reading the target file is determined according to the file type and the file is read, such as reading a file in word format through office software or wps, and reading a file in PDF format through PyPDF2;

[0053] After reading the file content, determine the paragraph separator based on the language conventions of the file content. In English, paragraphs are usually separated by two line breaks \n\n; Chinese may use one line break \n.

[0054] Use string splitting methods, such as str.split() in Python, to split the text based on the delimiter, and determine the maximum number of characters per paragraph during the splitting process. If the paragraph exceeds this limit, further splitting is required. At the same time, identify and replace or remove unnecessary line breaks, such as replacing \r\n with \n. For extra whitespace characters, you can use str.strip() to remove whitespace at the beginning and end of the string, and use str.replace(",") to remove multiple consecutive whitespace in a string.

[0055] Through the above processing, the target file is converted into target data for storage, thereby improving the retrieval efficiency.

[0056] In another embodiment, the data conversion module performs processing and conversion according to the file type and file content of the target file, including:

[0057] When it is determined that the target file is a pure data file, the target file is stored in groups and batches;

[0058] When it is determined that the target file is not a pure data file, the target file is divided into a word group and a pdf group according to the file type;

[0059] The target files in the word group are grouped into a word document set, each document in the word document set is traversed, and after being read into the memory to obtain a word document object, the data structure is split according to a specific format or paragraph, the split data content is assembled into data rows, and the data rows are grouped and stored in batches;

[0060] For the target files in the PDF group, you can first convert them into word format, and then use the processing method in the word group to group and store them in batches, such as Figure 2 shown.

[0061] 4. The archiving storage module is used to persist the target data in an orderly manner to a database such as (mysql, oracle, milvus) after receiving the target data, and provide data interface query services. It can also support data fuzzy query functions on the application side and provide data support to the data caller.

[0062] 5. When the data call module receives a query request, it searches the securities compliance knowledge base according to the query information in the query request, and outputs the target data with the highest matching degree with the query information. For example, when the query information received is a keyword, a keyword search is performed in the above securities compliance knowledge base to obtain a list of keywords, paragraphs, files and other data after the search, and scores (weighted) according to the number of keyword hits, and selects the 5-10 items with the highest scores as the basis for the promo context data for the ChatGLM model to answer. The large model professionally summarizes the provided promo data list and questions, makes in-depth and professional answers, and marks the name of the compliance file, terms and file source on which the answer depends.

[0063] So far, completed Figure 1 System module description shown.

[0064] In the embodiment of the present application, securities compliance documents are collected through the data collection module to build a securities compliance database, and the latest regulations are quickly mastered through regular collection, ensuring that the system can master the latest regulations in real time. And through the data conversion module, the archiving storage module processes and archives the securities compliance documents, so that the system can enhance the context understanding ability and improve the accuracy of the answer. Provide users with efficient and accurate securities compliance consulting services.

[0065] like Figure 3 As shown, the present application also provides an information query method based on a securities compliance knowledge base, the method comprising:

[0066] S301, collecting the current stock of securities compliance files, converting the securities compliance files into target files in a preset format and saving them to a data collection space, while recording file information of the target files, the file information including the file source and the update channel source;

[0067] S302, the intermediate scanner scans the target file to be processed in the data collection space, and sends the target file to the data converter;

[0068] S303, after receiving the target file, the data converter processes and converts the target file according to the file type and file content of the target file, generates target data and files it into the securities compliance knowledge base;

[0069] S304: When a query request is received, a query is performed in the securities compliance knowledge base according to the query information in the query request, and target data with the highest matching degree with the query information is output.

[0070] In another embodiment, the processing and conversion according to the file type and file content of the target file includes:

[0071] Determine the file type of the target file, determine a tool to read the target file according to the file type, and read the target file;

[0072] The target file is segmented into paragraphs according to a preset character limit and file content, and special characters in the target file are processed to generate the target data.

[0073] In another embodiment, the processing and conversion according to the file type and file content of the target file includes:

[0074] When it is determined that the target file is a pure data file, the target file is stored in groups and batches;

[0075] When it is determined that the target file is not a pure data file, the target file is divided into a word group and a pdf group according to the file type;

[0076] The target files in the word group are grouped into a word document set, each document in the word document set is traversed, and after being read into the memory to obtain a word document object, the data structure is split according to a specific format or paragraph, the split data content is assembled into data rows, and the data rows are grouped and stored in batches;

[0077] Convert each target file in the PDF group into word format, and group the converted target files into a word document set, traverse each document in the word document set, read it into memory to obtain a word document object, split the data structure according to a specific format or paragraph, assemble the split data content into data rows, and store the data rows in groups and batches.

[0078] In another embodiment, the outputting of the target data having the highest matching degree with the query information includes:

[0079] The target data that hit the query information are determined as initial target data, each initial target data is scored and sorted according to the number of times it hits the query information, and the initial target data with the highest score is determined as the target data with the highest matching degree with the query information and output.

[0080] The above-mentioned embodiments of the present invention provide an information query system based on a securities compliance knowledge base, and based on this system provide an information query method based on a securities compliance knowledge base. Through the above-mentioned system and method, it is possible to provide users with efficient and accurate securities compliance consulting services.

[0081] This embodiment also discloses a computer device, such as Figure 4 As shown, the computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement any of the above-mentioned information query systems based on the securities compliance knowledge base.

[0082] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An information query system based on a securities compliance knowledge base, characterized in that: The system includes a data acquisition module, an intermediate scanning module, a data conversion module, an archiving storage module and a data calling module; The data collection module is used to collect the current stock of securities compliance files, convert the securities compliance files into target files in a preset format and save them to the data collection space, and record the file information of the target files, the file information including the file source and the update channel source; The intermediate scanning module is used to obtain the target file from the data acquisition space and send the target file to the data conversion module; The data conversion module is used to receive the target file sent by the intermediate scanning module, split the target file by a string segmentation method according to the file type and file content of the target file, generate target data and send it to the archiving storage module; The archiving storage module is used to store the received target data to achieve the creation and update of the securities compliance knowledge base; The data calling module is used to query the securities compliance knowledge base according to the query information in the query request when receiving a query request, score the query information according to the number of times it hits the query information, and use the query results with the highest scores as the basis for the promote context data for the ChatGLM model to answer, so that the ChatGLM model summarizes the query results and outputs summary information.

2. The system according to claim 1, characterized in that The data collection module also includes a timer, which is used to set the securities compliance document collection time. When the collection time is reached, the data collection module automatically collects the securities compliance document from the target website or target channel, and updates the collection time of the timer after the collection.

3. The system according to claim 1, characterized in that The data conversion module performs processing and conversion according to the file type and file content of the target file, including: Determine the file type of the target file, determine a tool to read the target file according to the file type, and read the target file; The target file is segmented into paragraphs according to a preset character limit and file content, and special characters in the target file are processed to generate the target data.

4. The system according to claim 1, characterized in that The data conversion module performs processing and conversion according to the file type and file content of the target file, including: When it is determined that the target file is a pure data file, the target file is stored in groups and batches; When it is determined that the target file is not a pure data file, the target file is divided into a word group and a pdf group according to the file type; The target files in the word group are grouped into a word document set, each document in the word document set is traversed, and after being read into the memory to obtain a word document object, the data structure is split according to a specific format or paragraph, the split data content is assembled into data rows, and the data rows are grouped and stored in batches; Convert each target file in the PDF group into word format, and group the converted target files into a word document set, traverse each document in the word document set, read it into memory to obtain a word document object, split the data structure according to a specific format or paragraph, assemble the split data content into data rows, and store the data rows in groups and batches.

5. The system according to claim 1, characterized in that The output of the target data having the highest matching degree with the query information includes: The target data that hit the query information are determined as initial target data, each initial target data is scored and sorted according to the number of times it hits the query information, and the initial target data with the highest score is determined as the target data with the highest matching degree with the query information and output.

6. An information query method based on a securities compliance knowledge base, characterized in that: The method adopts an information query system based on a securities compliance knowledge base as described in any one of claims 1 to 5 to provide securities compliance questions and answers.

7. The method according to claim 6, characterized in that The method comprises: Collect the current stock of securities compliance documents, convert the securities compliance documents into target files in a preset format and save them to the data collection space, and record the file information of the target files, including the file source and the update channel source; The intermediate scanner scans the target file to be processed in the data collection space, and sends the target file to the data converter; After receiving the target file, the data converter splits the target file by a string segmentation method according to the file type and file content of the target file, generates target data and files it into the securities compliance knowledge base; When a query request is received, a query is performed in the securities compliance knowledge base according to the query information in the query request, and a score is given according to the number of times the query information is hit, and the query results with the highest scores are used as the basis for the promote context data for the ChatGLM model to answer, so that the ChatGLM model summarizes the query results and outputs summary information.

8. The method according to claim 7, characterized in that The processing and conversion according to the file type and file content of the target file includes: Determine the file type of the target file, determine a tool to read the target file according to the file type, and read the target file; The target file is segmented into paragraphs according to a preset character limit and file content, and special characters in the target file are processed to generate the target data.

9. The method according to claim 7, characterized in that: The processing and conversion according to the file type and file content of the target file includes: When it is determined that the target file is a pure data file, the target file is stored in groups and batches; When it is determined that the target file is not a pure data file, the target file is divided into a word group and a pdf group according to the file type; The target files in the word group are grouped into a word document set, each document in the word document set is traversed, and after being read into the memory to obtain a word document object, the data structure is split according to a specific format or paragraph, the split data content is assembled into data rows, and the data rows are grouped and stored in batches; Convert each target file in the PDF group into word format, and group the converted target files into a word document set, traverse each document in the word document set, read it into memory to obtain a word document object, split the data structure according to a specific format or paragraph, assemble the split data content into data rows, and store the data rows in groups and batches.

10. The method according to claim 6, characterized in that The output of the target data having the highest matching degree with the query information includes: The target data that hit the query information are determined as initial target data, each initial target data is scored and sorted according to the number of times it hits the query information, and the initial target data with the highest score is determined as the target data with the highest matching degree with the query information and output.

Images