A kibana-based account following data preparation method and system

Abstract

The application relates to a Kibana-based account following data preparation method and system, which comprises the following steps: acquiring first log information and second log information, setting a crawler rule, screening specific sub-database transaction flow information according to the crawler rule and the second log information, and then acquiring the first log information according to the sub-database transaction flow information, to obtain request messages and response messages of transactions, wherein the request messages are mainly used for test input of account following, and the response messages are used for comparison with response messages obtained through account following test, so as to verify the correctness of new system functions. The method and system adopt the method and system, realize collection of various Kibana platform-based financial transaction test messages, improve the collection efficiency of account following test data, and guarantee high coverage of account following test.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a Kibana-based account tracking data preparation method and system. Background Art

[0002] Kibana is an open-source analysis and visualization platform designed for use with Elasticsearch. It allows for easy querying of system log data stored in Elasticsearch indexes. Kibana interacts with Elasticsearch through various charts, tables, and maps, intuitively displaying data for advanced data analysis and visualization. Currently, Kibana is primarily used for system log analysis and visualization.

[0003] A crawler, also known as a web spider, is a program or script that automatically crawls the World Wide Web according to certain rules. It can replace humans in automatically collecting and organizing data on the Internet.

[0004] Account tracking refers to the process of recreating transactions from the old system in some way after a simulated launch of a new system, and then comparing the results with those of the old system. This typically involves preparing tracking data, submitting transaction messages, and comparing databases.

[0005] Traditional core financial accounting systems often require comprehensive functional testing to verify the correctness of the new system when migrating from an old one. Therefore, these systems often incorporate account tracking testing to fully verify the functionality of the new system. However, traditional account tracking test data preparation typically requires manual generation of test data through simulated business transactions. This process often consumes significant labor costs and can be incomplete due to the human factor involved in ensuring the comprehensiveness of the account tracking test data. Therefore, a fully automated account tracking test data preparation method is urgently needed. Summary of the Invention

[0006] This patent proposes a Kibana-based account tracking data preparation method and system. By using crawler technology combined with the characteristics of the Kibana system, transaction messages of financial core accounting systems with sharded databases and tables are automatically acquired. The crawled transaction messages can provide input data for account tracking tests and output data for comparison in the account tracking process, thereby realizing the automated acquisition of account tracking test data.

[0007] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0008] A method for preparing account tracking data based on Kibana, characterized by comprising:

[0009] S1. Obtain first and second log information from Kibana, where the first log information is transaction performance information and the second log information is transaction content information.

[0010] S2. Set crawler rules. The crawler rules include the transaction information and quantity thresholds for crawling transaction messages. The transaction information includes the transaction code range, sub-database information, and transaction processing timestamp range.

[0011] S3. Search and match in the first log information according to the crawler rules;

[0012] S4. Obtain the serial number of the transaction information crawled from the first log information, and use the serial number as a query condition to search for the second log information;

[0013] S5. Storing the transaction message in the found second log information as the request message and response message required for the account tracking test in the database;

[0014] S6: Determine whether the number of found transaction messages reaches the threshold.

[0015] If the number of found transaction messages reaches the threshold, the search stops;

[0016] If the number of found transaction messages does not reach the threshold, S3 to S5 are repeated.

[0017] Furthermore, the performance information includes transaction serial number, database processing time, inbound transaction processing time and sub-database information to which the transaction processing unit belongs.

[0018] Furthermore, the content information includes a transaction serial number, a transaction message, a transaction processing timestamp, a transaction processing unit and a transaction code.

[0019] Furthermore, the first log information and the second log information are matched with transaction information fields by a script program embedded in the system and stored in a real-time search engine.

[0020] Furthermore, the request message in S5 is the input of the account tracking test, and the response message is used to compare with the response message obtained from the account tracking test.

[0021] Furthermore, the database includes an Oracle database and a Mysql database.

[0022] The present invention also relates to a Kibana-based account tracking data preparation system, which is characterized by comprising:

[0023] The acquisition module is used to obtain the first log information, second log information and crawled transaction information from Kibana;

[0024] Crawler rule management module, used to set crawler rules;

[0025] The processing module is used to crawl transaction information using crawler rules;

[0026] A storage module, configured to store the first log information, the second log information, crawler rules, and crawled transaction information;

[0027] An information search module, configured to use the serial number as a query condition to search for second log information;

[0028] The judgment module is used to judge whether the number of found transaction messages reaches a threshold.

[0029] The present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0030] The present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0031] The memory is used to store the first log information, the second log information, the crawler rules and the crawled transaction information;

[0032] The processor is used to execute the above method according to the log information and crawler rules stored in the memory.

[0033] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.

[0034] The beneficial effects of the present invention are:

[0035] By adopting the Kibana-based account tracking data preparation method and system described in the present invention, transaction messages of financial core accounting systems with sharded databases and tables are automatically acquired by using crawler technology in combination with the characteristics of the Kibana system. The crawled transaction messages can provide input data for account tracking tests and output data for comparison in the account tracking process, thereby realizing the collection of various financial transaction test messages based on the Kibana platform. At the same time, the automated acquisition of account tracking test data also improves the collection efficiency of account tracking test data and ensures high coverage of account tracking tests. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of the account tracking data preparation method based on Kibana of the present invention.

[0037] Figure 2 This is a schematic diagram of the Kibana-based account data preparation system structure of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] The first aspect of the present invention relates to a process of steps as follows Figure 1 The Kibana-based account tracking data preparation method shown includes:

[0040] Kibana is a browser-based front-end display tool for Elasticsearch. It is also an open source and free tool. Kibana can provide a log analysis-friendly web interface for Logstash and ElasticSearch, which can help aggregate, analyze and search important data logs. ElasticSearch is an open source distributed search server based on Lucene. Its features include: distribution, zero configuration, automatic discovery, automatic index sharding, index replication mechanism, RESTful style interface, multiple data sources, automatic search load, etc. It provides a distributed multi-user full-text search engine based on a RESTful web interface. Elasticsearch is developed in Java and released as open source under the terms of the Apache license. It is the second most popular enterprise search engine. Designed for cloud computing, it can achieve real-time search and is stable, reliable, fast, and easy to install and use.

[0041] Specifically, in the financial sector, the Kibana platform can search transaction logs for core financial accounts, facilitating system operations and maintenance personnel's monitoring and analysis of the financial system's transaction processing capabilities and system performance. Two key log types can be searched: log_perftrace and log_message. Specifically, after obtaining log information from ElasticSearch, Kibana is used to perform visual retrieval and processing of this log information.

[0042] The first log information (log_perftrace) mainly records the detailed performance of each transaction, including the financial transaction serial number EVT_TRACE_ID, database processing time DB_COSTS, inbound call transaction processing time INNERCALL_COSTS, and the sub-library group SPUNAME to which the transaction processing unit belongs;

[0043] The second log information (log_message) mainly records the transaction request and response messages, transaction flow and other information, including the financial transaction flow number EVT_TRACE_ID, transaction message UserMessage, transaction processing timestamp timestamp, transaction processing unit UnitName, and transaction code TX_CODE information.

[0044] The first log information (log_perftrace) and the second log information (log_message) are both stored in elasticsearch in a structured manner using a system-embedded script program to match regular expressions with keywords such as serial numbers and transaction messages.

[0045] Since the second log message (log_message) doesn't contain information about sharded databases and tables, and account tracking test data requires preparing test data based on sharded databases and tables, it's necessary to first filter out the specific transaction flow information for each sharded database and table using the first log message (log_perftrace). Then, based on this flow information, retrieve the transaction request and response messages. The request message is primarily used as input for account tracking tests, and the response message is compared with the response message from the account tracking test transaction to verify the correctness of the new system's functionality.

[0046] First, test transaction data based on the financial transaction log types of the first log information (log_perftrace) and the second log information (log_message) are collected on the Kibana platform. Then, the search results of the first log information (log_perftrace) and the second log information (log_message) are parsed and matched to filter out the account tracking test data.

[0047] Specifically, step S1: obtain the first log information (log_perftrace) and the second log information (log_message) of Kibana. Specifically, after obtaining the log information from the ElasticSearch tool, use the Kibana tool to perform visual retrieval processing on the log information. The first log information (log_perftrace) is the performance information of the transaction, specifically including the financial transaction serial number EVT_TRACE_ID, database processing time DB_COSTS, inbound transaction processing time INNERCALL_COSTS, and the sub-library group SPUNAME to which the transaction processing unit belongs; the second log information (log_message) is the content information of the transaction, specifically including the financial transaction serial number EVT_TRACE_ID, transaction message UserMessage, transaction processing timestamp timestamp, transaction processing unit UnitName, and transaction code TX_CODE information; the first log information (log_perftrace) and the second log information (log_message) are matched against the transaction information field by the system's embedded script program and stored in the real-time search engine.

[0048] Step S2: Setting crawler rules. The crawler rules include the transaction information and quantity thresholds of the transaction messages to be crawled. The transaction information includes the transaction code TX_CODE range, sub-database information SPUNAME, and transaction processing timestamp range.

[0049] Step S3: Search and match in the first log information (log_perftrace) according to the crawler rule;

[0050] Step S4: using the serial number EVT_TRACE_ID of the transaction information crawled in the first log information (log_perftrace) in step S3 as a query condition to search for the second log information (log_message);

[0051] Step S5: The transaction message UserMessage in the second log information (log_message) found in step S4 is stored in the database as the request message and response message required for the account tracking test, wherein the transaction message UserMessage includes a request message and a response message.

[0052] Step S6: Determine whether the number of transaction messages UserMessage found reaches the threshold. If so, stop searching. If not, repeat steps S3 to S5 until the number of transaction messages UserMessage reaches the threshold. At this point, data preparation for the account tracking test is completed.

[0053] In the account tracking test, the request message in the second log information (log_message) found in step S4 is used as the input of the account tracking test, and the response message obtained by the account tracking test is compared with the response message in the second log information (log_message) found in step S4 to verify the correctness of the new system function.

[0054] By applying the above method, it is possible to collect various financial transaction test messages based on the Kibana platform. At the same time, it improves the collection efficiency of account tracking test data and ensures high coverage of account tracking tests.

[0055] Another aspect of the present invention relates to a Kibana-based account data preparation system, the structure of which is as follows: Figure 2 Shown, including:

[0056] The acquisition module is used to obtain Kibana's first log information (log_perftrace), second log information (log_message), and transaction information crawled according to crawler rules;

[0057] Crawler rule management module, used to set crawler rules;

[0058] The processing module is used to crawl transaction information using crawler rules;

[0059] A storage module, used to store the first log information (log_perftrace), the second log information (log_message), crawler rules, and crawled transaction information;

[0060] An information search module, configured to use the serial number as a query condition to search for second log information;

[0061] The judgment module is used to judge whether the number of found transaction messages reaches a threshold.

[0062] By using this system, the above-mentioned calculation and processing method can be executed and the corresponding technical effects can be achieved.

[0063] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all steps of the Kibana-based account tracking data preparation method in the above embodiment. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the Kibana-based account tracking data preparation method in the above embodiment.

[0064] An embodiment of the present invention also provides an electronic device for executing the above-mentioned method. As an implementation device of the method, the electronic device has at least a processor and a memory. In particular, the memory stores the data and related computer programs required for executing the method, such as log information, crawler rules, etc., and the processor calls the data and programs in the memory to execute all the steps of the implementation method and obtains the corresponding technical effects.

[0065] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and, therefore, will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter can be the same component, namely a transceiver, which provides a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory can be used to store data used by the processor when performing operations.

[0066] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) employed may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device. The memory may be one or more of a cache, flash memory, a hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, and may store the aforementioned data and information. It may also store programs for executing the relevant information, and the processor may execute the programs stored in the memory to implement information storage or processing. The input unit is used to provide input to the processor, and may, for example, be a keypad or touch input device. The power supply is used to provide power to the electronic device. The display is used to display objects such as images and text, and may, for example, be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, similar to the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone, thereby implementing common telecommunications functions. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processing unit, enabling local recording via the microphone and playback of stored audio via the speaker.

[0067] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.

[0069] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system that is implemented in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The present invention is described in detail below. ...

[0071] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for preparing account tracking data based on Kibana, characterized in that: include: S1. Obtain first and second log information from Kibana, where the first log information is transaction performance information and the second log information is transaction content information. S2. Setting crawler rules, wherein the crawler rules include transaction information and quantity thresholds for transaction messages to be crawled; S3. Search and match in the first log information according to the crawler rules; S4. Obtain the serial number of the transaction information crawled from the first log information, and use the serial number as a query condition to search for the second log information; S5. Storing the transaction message in the found second log information as the request message and response message required for the account tracking test in the database; S6: Determine whether the number of found transaction messages reaches the threshold. If the number of found transaction messages reaches the threshold, the search stops; If the number of found transaction messages does not reach the threshold, repeat S3 to S5; The performance information includes transaction serial number, database processing time, inbound transaction processing time, and information on the sub-database to which the transaction processing unit belongs; The content information includes transaction serial number, transaction message, transaction processing timestamp, transaction processing unit and transaction code.

2. The method according to claim 1, wherein The transaction information includes the transaction code range, sub-database information and transaction processing timestamp range.

3. The method according to claim 2, wherein The first log information and the second log information are matched against the transaction information fields by a script program embedded in the system and stored in a real-time search engine.

4. The method according to claim 1, wherein The request message in S5 is the input of the account tracking test, and the response message is used to compare with the response message obtained from the account tracking test.

5. The method according to claim 1, wherein The database includes an Oracle database and a Mysql database.

6. A Kibana-based account tracking data preparation system, characterized by: include: An acquisition module is configured to obtain first and second log information and crawled transaction information from Kibana. The first log information is transaction performance information, and the second log information is transaction content information. The performance information includes the transaction serial number, database processing time, inbound transaction processing time, and transaction processing unit database information. The content information includes the transaction serial number, transaction message, transaction processing timestamp, transaction processing unit, and transaction code. A crawler rule management module, used to set crawler rules, wherein the crawler rules include transaction information and quantity thresholds for transaction messages that need to be crawled; A processing module, configured to crawl transaction information using crawler rules and obtain a transaction serial number crawled in the first log information; A storage module, configured to store the first log information, the second log information, crawler rules, and crawled transaction information; An information search module, configured to use the transaction serial number as a query condition to search for second log information; The judgment module is used to judge whether the number of transaction messages found reaches a threshold. If the number of transaction messages found reaches the threshold, the search is stopped; if the number of transaction messages found does not reach the threshold, the above search steps are repeated.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which implements the method according to any one of claims 1 to 5 when executed by a processor.

8. An electronic device, characterized in that: including processor and memory; The memory is used to store the first log information, the second log information, the crawler rules and the crawled transaction information; The processor is configured to execute the method according to any one of claims 1 to 5 based on the log information and crawler rules stored in the memory.

9. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images