Message checking method and system, electronic equipment and computer program product

Abstract

The invention discloses a message checking method and system, electronic equipment and a computer program product, and relates to the field of data processing, and the method comprises the steps: collecting a transaction message of a transaction system, and transmitting the transaction message to a real-time message queue; reading a message domain of at least one transaction message in the real-time message queue, and determining a transaction code of the transaction message; searching a field name corresponding to the transaction code under each to-be-verified field type in a preset mapping relation table according to the transaction code; extracting a value of a field name corresponding to the transaction code from the message domain, and transmitting the value as to-be-verified data to a real-time data lake for storage; and performing data quality checking on the to-be-checked data in the real-time data lake. According to the application, the transaction message is sent to the real-time message queue, the transaction message is conveniently processed one by one in batches, the processing time is reduced, the to-be-checked data in the transaction message can be accurately determined according to the preset mapping relation table, the data which does not need to be checked is prevented from being extracted, and the processing time is further reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a message checking method, system, electronic device and computer program product. Background Art

[0002] In recent years, some regulatory agencies have increased their requirements for system data quality. Therefore, it is necessary to check the system data quality from time to time.

[0003] In the traditional data quality verification process, after collecting transaction messages, the data quality verification tool needs to consume a large amount of computing resources and a long time to extract the data for verification from massive transaction messages. Some of the extracted data often do not need to be verified, which makes the data quality verification time-consuming and the verification efficiency low. Summary of the invention

[0004] In view of the above problems, the present application provides a message checking method, system, electronic device and computer program product to achieve the purpose of improving the efficiency of data quality checking. The specific scheme is as follows:

[0005] A first aspect of the present application provides a message checking method, the message checking method comprising:

[0006] Collect transaction messages from the transaction system and send the transaction messages to the real-time message queue;

[0007] Reading a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message;

[0008] According to the transaction code, searching in a preset mapping relationship table for a field name corresponding to the transaction code under each field type to be verified, wherein the preset mapping relationship table stores a correspondence between a transaction code under at least one field type to be verified and a field name of a field of data to be verified;

[0009] Extracting the value of the field name corresponding to the transaction code from the message domain, and transmitting it to the real-time data lake for storage as the data to be verified;

[0010] Perform data quality check on the data to be verified in the real-time data lake.

[0011] In a possible implementation, the collecting transaction messages of the transaction system includes:

[0012] The mirror message data of each transaction system is collected through the bypass device as the transaction message of each transaction system.

[0013] In a possible implementation, the data storage format in the data lake is in a table format, each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under a field type to be verified.

[0014] In a possible implementation, performing data quality check on the to-be-verified data in the real-time data lake includes:

[0015] Obtaining at least one of the to-be-verified data from the data lake through a data quality verification tool, and performing automated verification;

[0016] The data quality checking tool includes corresponding checking scripts formulated according to the business rules of each transaction system.

[0017] A second aspect of the present application provides a message checking system, the message checking system comprising:

[0018] A collection unit, used for collecting transaction messages of the transaction system and sending the transaction messages to a real-time message queue;

[0019] A reading unit, configured to read a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message;

[0020] a query unit, configured to search, according to the transaction code, in a preset mapping relationship table for a field name corresponding to the transaction code under each field type to be verified, wherein the preset mapping relationship table stores a correspondence between a transaction code under at least one field type to be verified and a field name of a field of data to be verified;

[0021] An extraction unit, used to extract the value of the field name corresponding to the transaction code from the message domain, and transmit it to the real-time data lake for storage as the data to be verified;

[0022] A checking unit is used to perform data quality checking on the data to be checked in the real-time data lake.

[0023] In a possible implementation, the acquisition unit is specifically configured as follows:

[0024] The mirror message data of each transaction system is collected through the bypass device as the transaction message of each transaction system.

[0025] In a possible implementation, the data storage format in the data lake is in a table format, each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under a field type to be verified.

[0026] In a possible implementation, the checking unit is specifically configured as follows:

[0027] Obtaining at least one of the to-be-verified data from the data lake through a data quality verification tool, and performing automated verification;

[0028] The data quality checking tool includes corresponding checking scripts formulated according to the business rules of each transaction system.

[0029] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0030] The memory is used to store computer programs;

[0031] The processor is used to execute the computer program so that the electronic device can implement the message checking method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0032] A fourth aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the message checking method of the first aspect or any implementation of the first aspect.

[0033] By means of the above technical scheme, the present application provides a message verification method, system, electronic device and computer program product. After collecting the transaction message of the transaction system, the method sends it to the real-time message queue, which is convenient for batch processing of the transaction message, reducing the computing resources consumed by each transaction message processing, reducing the processing time, and querying the transaction code in the message field of the transaction message in the preset mapping relationship table, the field name of the data to be verified can be determined, and the data to be verified can be accurately intercepted from the transaction message, avoiding the extraction of data that does not need to be verified, so as to further reduce the processing time. Therefore, the present method can effectively reduce the time required for data quality verification and improve the verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0035] Figure 1 A flowchart of a message verification method provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a transaction link provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a real-time message queue provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of a corresponding relationship in a preset mapping relationship table provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of a message checking system provided in an embodiment of the present application;

[0040] Figure 6 A hardware structure block diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0042] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0043] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0044] In recent years, some regulatory agencies have raised their requirements for system data quality. However, for key regulatory scenarios, there is a lack of technical verification of data irregularities such as transaction messages in trading systems, resulting in delayed problem discovery. It is necessary to improve the quasi-real-time monitoring capability of data quality to meet regulatory data quality and other related requirements.

[0045] After identifying a transaction system with a lot of problematic data, the traditional data quality verification method collects transaction messages from the transaction system by turning on the transaction log switch, determines the transaction message domain that causes data quality problems, analyzes the business scenario, continues to trace the source and collect transaction messages, and loads the collected transaction messages into the database through data transmission. After performing data processing such as data cleaning on the transaction messages, the message data (message domain) that needs to be verified is extracted for data verification.

[0046] After collecting transaction messages, a large amount of computing resources are needed to convert data formats, clean data, and integrate data, and to extract data for verification from massive transaction messages. However, some of the extracted data often does not need to be verified, which increases the verification time and makes the verification efficiency low.

[0047] In order to solve the above problems, the embodiment of the present application provides a message verification method. After collecting transaction messages, the method will put them into a real-time message queue, so that the transaction messages can be processed in batches, without having to process a large number of transaction messages at one time, reducing the processing time, and the method uses a preset mapping relationship table to directly determine the data used for verification in the transaction message, avoiding the extraction of data that does not need to be verified, further reducing the processing time. The message verification method of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0048] Reference Figure 1 , Figure 1 A flow chart of a message checking method provided in an embodiment of the present application is as follows: Figure 1 As shown, a message checking method provided in an embodiment of the present application may include steps S10 to S14, and these steps are described in detail below.

[0049] S10, collecting transaction messages from the transaction system and sending the transaction messages to the real-time message queue;

[0050] The transaction message may be business data generated when the transaction system conducts production transactions, and may record the specific content of each transaction. Specifically, in this embodiment, the transaction message of the transaction system may be collected through a bypass device. The bypass device is a device using network bypass technology, which is a data collection technology that can copy data from the bypass (i.e., non-main transmission path) of the system without interfering with or affecting the normal operation of the collected system, thereby realizing data collection.

[0051] When tracing the source of problematic data, the traditional data quality verification method collects transaction messages by turning on the transaction log switch, which requires the coordinated changes of each transaction system, thereby affecting the production transactions of each transaction system. Therefore, it is difficult to obtain transaction messages without affecting the production transactions of each transaction system, and thus it is impossible to collect transaction messages in real time. Therefore, this embodiment collects the mirror message data of each transaction system through a bypass device as the transaction message of each transaction system. Since the bypass device can collect transaction messages without affecting the production transactions of the transaction system, it does not require the coordinated transformation of each transaction system. Therefore, this embodiment can collect transaction messages without affecting the production transactions of the transaction system. Compared with the collection of transaction messages by transaction logs, it greatly reduces the deployment difficulty and reduces the peak delay of the transaction system caused by the message collection of the transaction log, and improves the timeliness of data quality verification.

[0052] In this embodiment, the specific process of determining the collection object of the bypass device can be: perform a data check, after the data check is completed, cluster the problem data according to the transaction initiator (security node) and transaction code corresponding to the problem data, and obtain the transaction link by calling the transaction monitoring component, so as to locate the upstream transaction system and transaction code involved in the entire transaction link of the problem data, and the transaction system with the largest amount of problem data can be the collection object of the bypass device in this embodiment. Among them, the transaction link can be as follows Figure 2 As shown in the figure, access represents access switch, aggregation represents aggregation switch, and core represents core switch. If system A needs to call the transaction of system B, the transaction link can be: system A, access 1, access 2, system B; if system A needs to call the transaction of system C, the transaction link can be: system A, access 1, aggregation 1, aggregation 2, access 3, system C; if system A needs to call the transaction of system D, the transaction link can be: system A, access 1, aggregation 1, core 1, core 2, aggregation 3, access 4, system D.

[0053] The bypass device in this embodiment can filter transaction messages according to the dimensions of AP host, IP address and transaction code through the optical splitter configuration, accurately locate the relevant transaction messages in the transaction link, and convert the filtered transaction messages into data format through the network bypass probe, and then send them to the real-time message queue. The bypass device can automatically filter invalid messages to avoid data redundancy and reduce data burden.

[0054] Furthermore, due to the high-availability deployment mode, transaction messages can be randomly forwarded to any device in the corresponding functional cluster, and different devices may be located in different switches. Therefore, when collecting data from bypass devices, it is necessary to obtain the location of the access switch and obtain the access switch deployment area of ​​the transaction caller and the transaction service provider through the platform tool.

[0055] A real-time message queue is a queue system used to store and forward real-time messages. It allows producers (message senders) to send messages to the queue, and consumers (message receivers) can obtain these messages from the queue in real time. The real-time message queue can realize asynchronous transmission of data producers and consumers. Therefore, in this embodiment, the bypass device quickly sends the collected transaction messages (mirror message data) to the real-time message queue, and the real-time data integration tool can continuously obtain transaction messages from the real-time message queue and process them. Figure 3 As shown, the bypass device stores the collected messages D, C, B, and A in the real-time message queue in sequence, and the real-time data integration tool reads and processes messages D, C, B, and A in sequence, without the need for both ends to wait online at the same time.

[0056] The real-time message queue can cope with sudden traffic peaks. For example, during a big promotion, a large number of transaction messages are generated along with orders. The transaction messages can be temporarily placed in the real-time message queue and processed in batches to avoid the high operating pressure of the real-time data integration tool and slow processing. Therefore, the real-time message queue is used in this embodiment, which can not only avoid the problem of sudden increase in transaction messages caused by traffic peaks during the data quality verification process, but also ensure the reliable delivery of transaction messages. Furthermore, if the real-time data integration tool fails to successfully process the transaction message, the transaction message can remain in the real-time message queue.

[0057] S11, reading the message field of at least one transaction message in the real-time message queue, and determining the transaction code of the transaction message;

[0058] S12. Searching for a field name corresponding to the transaction code under each field type to be verified in a preset mapping relationship table according to the transaction code, wherein the preset mapping relationship table stores a correspondence between a transaction code under at least one field type to be verified and a field name of a field of data to be verified;

[0059] S13. Extract the value of the field name corresponding to the transaction code from the message domain and transmit it to the real-time data lake for storage as the data to be verified;

[0060] Among them, the message domain of the transaction message can be an important component of the transaction message. The transaction message can be composed of multiple message domains with a specific order and format, which are used to store and transmit various information of the transaction. After the bypass device transmits the transaction message to the real-time message queue, this embodiment can read the transaction message from the real-time message queue through the data integration tool, and decode the message domain of the transaction message, so as to restore the data in the message domain to the original transaction data, thereby obtaining the transaction code of the transaction message and the detailed transaction data in the transaction message. Of course, it is also possible to only decode the message domain corresponding to the transaction code to determine the transaction code of the transaction message. Among them, the transaction code is the identifier of the transaction message, which can indicate the type of the transaction message.

[0061] The preset mapping relationship table may be a mapping relationship table between field names, which may be prepared in advance by the operator according to the business rules of the transaction system (the business rules may include fixed requirements for transaction data). The preset mapping relationship table may include multiple verification field types, where the verification field type is the target field for data quality verification, each verification field type may correspond to multiple transaction codes, each transaction code may correspond to at least one field name, and the field name corresponding to the transaction code may be the field name of the field of the data to be verified in the transaction message.

[0062] The corresponding relationship in the preset mapping relationship table can be as follows Figure 4 As shown, the verification field type can include amount and time. Amount can correspond to transaction code 1 and transaction code 2. Transaction code 1 corresponds to payment amount and transaction code 2 corresponds to transfer amount. Time can correspond to transaction code 1 and transaction code 2. Transaction code 1 corresponds to payment time and transaction code 2 corresponds to transfer time. Of course, Figure 4 This is just an example to simply explain the corresponding relationship in the preset mapping relationship table. In actual situations, there may be more types of check fields, and the corresponding transaction codes may also be more. The transaction codes corresponding to each type of check field may be the same or different, and the number of corresponding transaction codes may also be different. There may also be multiple field names corresponding to the transaction code. This embodiment can automatically realize data integration through the preset mapping relationship table, and can also add new types of check fields in real time by modifying the preset mapping relationship table, which is more convenient.

[0063] After the data integration tool reads the transaction message and determines the transaction code of the transaction message, the data integration tool can perform data integration on the transaction message according to the preset mapping relationship table. Of course, the data integration tool can also clean and filter the transaction message before performing data integration. Specifically, after the data integration tool reads the transaction message, it queries the preset mapping relationship table according to the transaction code of the transaction message, determines at least one verification field type involving the transaction code, and determines the field name of the field of the data to be verified in the transaction message corresponding to the verification field type according to the field name corresponding to the transaction code recorded in the preset mapping relationship table, extracts the field value corresponding to the field name of the field in the transaction message, and uses it as the data to be verified in the transaction message under the verification field type.

[0064] For example, message A records the information that user B transferred 500 yuan to user C today. According to the transaction code of message A, the preset mapping relationship table is searched to determine that the transaction code of message A is involved in multiple transaction codes corresponding to a verification field type (amount), and the field name corresponding to the transaction code is recorded as the transfer amount. Therefore, it can be determined that the verification field type (amount) has a corresponding relationship with the transfer amount field in message A. The value of the transfer amount field in message A is the data to be verified of message A, and the field value (500) is extracted as the data to be verified of message A under the verification field type (amount).

[0065] Furthermore, when the data integration tool processes multiple transaction messages at the same time, it queries the preset mapping relationship table according to the transaction code of the transaction message to determine the field names of multiple fields of the data to be checked corresponding to the check field type. The check field type can then correspond to field values ​​of multiple different transaction messages.

[0066] After the data integration tool completes the data integration of the transaction message, it converts the extracted field values ​​into structured report data and transmits them to the data lake for storage. The data lake is an intermediate storage space that stores data from upstream systems or components for use by downstream systems or components.

[0067] The data storage format in the data lake is in table form. Each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of a transaction message under a field type to be verified. Optionally, the data to be verified can be marked with a transaction code. For example, the field type to be verified represented by the first row is the amount, the first column is the data to be verified of message A (the field value of the transfer amount field), the second column is the data to be verified of message B (the field value of the payment amount field), and so on. Among them, the correspondence between the transfer amount field and the field type to be verified (amount) and the correspondence between the payment amount field and the field type to be verified (amount) are determined by the data integration tool querying the preset mapping relationship table.

[0068] S14. Perform data quality check on the data to be verified in the real-time data lake.

[0069] In this embodiment, the real-time monitoring of the data to be verified in the real-time data lake can be realized through the real-time data quality monitoring tool. Specifically, at least one data to be verified is obtained from the data lake through the data quality verification tool, and automatic verification is performed at a fixed frequency. The data quality verification tool contains corresponding verification scripts formulated according to the business rules of each transaction system.

[0070] Among them, this embodiment can be configured for each trading system. After the business rules of the trading link of the trading system are converted into executable technical rules, an automated verification script is formulated. The verification scripts of multiple trading systems can be included in the data quality verification tool. When it is necessary to perform data quality verification on the transaction message of a certain trading system, the verification script can automatically read the data to be verified in the data lake for quality verification. If problematic data is found during the verification, it will be stored in the database in real time for query by the data management department.

[0071] The embodiment of the present application provides a message verification method. After the transaction message of the transaction system is collected, the method sends it to the real-time message queue, which is convenient for batch processing of the transaction message, reducing the computing resources consumed by each transaction message processing, reducing the processing time, and querying the transaction code in the message field of the transaction message in the preset mapping relationship table, the field name of the data to be verified can be determined, and the data to be verified can be accurately intercepted from the transaction message, avoiding the extraction of data that does not need to be verified, so as to further reduce the processing time. Therefore, the method can effectively reduce the time required for data quality verification and improve the verification efficiency.

[0072] Furthermore, the bypass device automatically collects transaction messages and stores them in the real-time message queue. The data integration tool automatically reads the transaction messages in the real-time message queue, and automatically extracts the data to be checked according to the preset mapping relationship, and transmits it to the data lake for storage. It is then monitored in real time by the data quality check tool, thus realizing the automation of the data quality check process. Compared with traditional data quality check methods, this method has stronger technical scalability, higher timeliness, and better automation capabilities.

[0073] A message checking method provided in an embodiment of the present application is introduced above, and a system applying the message checking method is introduced below.

[0074] See also Figure 5 , Figure 5 A schematic diagram of the structure of a message checking system provided in an embodiment of the present application. Figure 5 As shown, the message checking system may include:

[0075] The collection unit 100 is used to collect transaction messages of the transaction system and send the transaction messages to the real-time message queue;

[0076] The reading unit 110 is used to read the message field of at least one transaction message in the real-time message queue to determine the transaction code of the transaction message;

[0077] The query unit 120 is used to search the preset mapping relationship table for the field name corresponding to the transaction code under each field type to be verified according to the transaction code, and the preset mapping relationship table stores the correspondence between the transaction code under at least one field type to be verified and the field name of the field of the data to be verified;

[0078] The extraction unit 130 is used to extract the value of the field name corresponding to the transaction code from the message domain, and transmit it to the real-time data lake for storage as the data to be verified;

[0079] The checking unit 140 is used to perform data quality checking on the data to be checked in the real-time data lake.

[0080] In a possible implementation, the acquisition unit 100 is specifically configured as follows:

[0081] The mirror message data of each transaction system is collected through the bypass device as the transaction message of each transaction system.

[0082] In a possible implementation, the data storage format in the data lake is in table form, each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under the field type to be verified.

[0083] In a possible implementation, the checking unit 140 is specifically configured as follows:

[0084] Use data quality verification tools to obtain at least one piece of data to be verified from the data lake and perform automated verification.

[0085] Among them, the data quality verification tool includes corresponding verification scripts formulated according to the business rules of each transaction system.

[0086] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0087] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0089] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any message checking method provided in the embodiment of the present application.

[0090] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any message verification method provided in the embodiment of the present application.

[0091] It should also be noted that the system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0092] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0094] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0095] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0096] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0097] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A message checking method, characterized in that: The message checking method comprises: Collect transaction messages from the transaction system and send the transaction messages to the real-time message queue; Reading a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message; According to the transaction code, searching in a preset mapping relationship table for a field name corresponding to the transaction code under each field type to be verified, wherein the preset mapping relationship table stores a correspondence between a transaction code under at least one field type to be verified and a field name of a field of data to be verified; Extracting the value of the field name corresponding to the transaction code from the message domain, and transmitting it to the real-time data lake for storage as the data to be verified; Perform data quality check on the data to be verified in the real-time data lake.

2. The message checking method according to claim 1, characterized in that: The transaction message of the acquisition transaction system includes: The mirror message data of each transaction system is collected through the bypass device as the transaction message of each transaction system.

3. The message checking method according to claim 1, characterized in that: The data storage format in the data lake is in a table format, where each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under a field type to be verified.

4. The message checking method according to claim 1, characterized in that: The performing data quality check on the data to be verified in the real-time data lake includes: Obtaining at least one of the to-be-verified data from the data lake through a data quality verification tool, and performing automated verification; The data quality checking tool includes corresponding checking scripts formulated according to the business rules of each transaction system.

5. A message checking system, characterized in that: The message checking system comprises: A collection unit, used for collecting transaction messages of the transaction system and sending the transaction messages to a real-time message queue; A reading unit, configured to read a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message; a query unit, configured to search, according to the transaction code, in a preset mapping relationship table for a field name corresponding to the transaction code under each field type to be verified, wherein the preset mapping relationship table stores a correspondence between a transaction code under at least one field type to be verified and a field name of a field of data to be verified; An extraction unit, used to extract the value of the field name corresponding to the transaction code from the message domain, and transmit it to the real-time data lake for storage as the data to be verified; A checking unit is used to perform data quality checking on the data to be checked in the real-time data lake.

6. The message checking system according to claim 5, characterized in that: The acquisition unit is specifically configured as follows: The mirror message data of each transaction system is collected through the bypass device as the transaction message of each transaction system.

7. The message checking system according to claim 5, characterized in that: The data storage format in the data lake is in a table format, where each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under a field type to be verified.

8. The message checking system according to claim 5, characterized in that: The specific configuration of the verification unit is: Obtaining at least one of the to-be-verified data from the data lake through a data quality verification tool, and performing automated verification; The data quality checking tool includes corresponding checking scripts formulated according to the business rules of each transaction system.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the message checking method as described in any one of claims 1 to 4.

10. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the message checking method as claimed in any one of claims 1 to 4.

Images