A message checking method and system, electronic equipment and computer program product

Abstract

The application discloses a message checking method and system, electronic equipment and computer program product, relates to the field of data processing, and comprises the following steps: collecting transaction messages of a transaction system, and sending the transaction messages into a real-time message queue; reading a message field of at least one transaction message in the real-time message queue, and determining a transaction code of the transaction message; searching for a field name corresponding to the transaction code under each to-be-checked field type in a preset mapping relationship table according to the transaction code; extracting a value of the field name corresponding to the transaction code from the message field, and taking the value as to-be-checked data to be transmitted to a real-time data lake for storage; and performing data quality checking on the to-be-checked data in the real-time data lake. The application sends the transaction messages into the real-time message queue, facilitates processing the transaction messages in batches one by one, reduces processing time, and can accurately determine to-be-checked data in the transaction messages according to the preset mapping relationship table, so that data that does not need to be checked is avoided to be extracted, and processing time is further reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a message checking method and system, an electronic device, and a computer program product. BACKGROUND

[0002] In recent years, some regulatory agencies have increased their requirements for system data quality. Therefore, the data quality of the system needs to be checked from time to time.

[0003] In the traditional data quality checking process, after collecting transaction messages, the data quality checking tool needs to consume a large amount of computing resources and consume a lot of time to extract data for checking from a large amount of transaction messages. The extracted part of the data often does not need to be checked, which makes the data quality checking time-consuming and inefficient. SUMMARY

[0004] In view of the above problems, the present application provides a message checking method, system, electronic device and computer program product to achieve the purpose of improving the data quality checking efficiency. The specific scheme is as follows:

[0005] The first aspect of the present application provides a message checking method, which comprises:

[0006] Collecting transaction messages of a transaction system and sending the transaction messages to a real-time message queue;

[0007] Reading a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message;

[0008] According to the transaction code, finding a field name corresponding to the transaction code under each to-be-verified field type in a preset mapping relationship table, wherein the preset mapping relationship table stores a corresponding relationship between the transaction code and the field name of the to-be-verified data field under at least one to-be-verified field type;

[0009] Extracting the value of the field name corresponding to the transaction code from the message field and delivering the to-be-verified data to a real-time data lake for storage;

[0010] Performing data quality checking on the to-be-verified data in the real-time data lake.

[0011] In one possible implementation, the collecting of the transaction messages of the transaction system comprises:

[0012] Collecting mirror message data of each transaction system as transaction messages of each transaction system through a bypass device.

[0013] In a possible implementation, the data storage format in the data lake is in a table form, each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under the field type to be verified.

[0014] In a possible implementation, the data quality check on the data to be verified in the real-time data lake comprises:

[0015] acquiring at least one piece of the data to be verified from the data lake by using a data quality check tool, and performing automatic check;

[0016] The data quality check tool comprises a corresponding check script formulated according to the business rules of each transaction system.

[0017] The second aspect of the present application provides a message check system, which comprises:

[0018] a collection unit configured to collect transaction messages of a transaction system and send the transaction messages to a real-time message queue;

[0019] a reading unit configured to read a message field of at least one transaction message in the real-time message queue and determine a transaction code of the transaction message;

[0020] a query unit configured to find a field name corresponding to the transaction code under each field type to be verified in a preset mapping relationship table according to the transaction code, the preset mapping relationship table storing a corresponding relationship between the transaction code and the field name of the field to be verified under at least one field type to be verified;

[0021] an extraction unit configured to extract a value of the field name corresponding to the transaction code from the message field and deliver the value as the data to be verified to a real-time data lake for storage;

[0022] a check unit configured to perform data quality check on the data to be verified in the real-time data lake.

[0023] In a possible implementation, the collection unit is specifically configured to:

[0024] collect mirror message data of each transaction system by using a bypass device as the transaction messages of each transaction system.

[0025] In a possible implementation, the data storage format in the data lake is in a table form, each row of the table represents a field type to be verified, and each column of the table corresponds to the data to be verified of the transaction message under the field type to be verified.

[0026] In a possible implementation, the check unit is specifically configured to:

[0027] acquire at least one of the to-be-verified data from the data lake through a data quality verification tool and perform automatic verification;

[0028] The data quality verification tool comprises a corresponding verification script formulated according to the business rules of each transaction system.

[0029] The third aspect of the present application provides an electronic device comprising at least one processor and a memory connected to the processor, wherein:

[0030] The memory is configured to store a computer program.

[0031] The processor is configured to execute the computer program to enable the electronic device to implement the message verification method of the first aspect or any implementation manner of the first aspect.

[0032] The fourth aspect of the present application provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, enable the electronic device to implement the message verification method of the first aspect or any implementation manner of the first aspect.

[0033] By means of the above technical solution, the present application provides a message verification method, system, electronic device and computer program product. After the transaction message of the transaction system is collected, it is sent to the real-time message queue, which facilitates the batch-by-batch processing of the transaction message, reduces the computing resources consumed by each transaction message processing, reduces the processing time, and queries the transaction code in the message field of the transaction message in the preset mapping relationship table, which can determine the field name of the to-be-verified data, accurately extract the to-be-verified data from the transaction message, avoid extracting data that does not need to be verified, and further reduce the processing time. Therefore, the present method can effectively reduce the time required for data quality verification and improve the verification efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0034] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0035] Figure 1 A flowchart of a message verification method provided by an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a transaction link provided by an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a real-time message queue provided by an embodiment of the present application;

[0038] Figure 4 A corresponding relationship diagram in a preset mapping relationship table provided by an embodiment of the present application;

[0039] Figure 5 A structure diagram of a message checking system provided by an embodiment of the present application;

[0040] Figure 6 A hardware structure block diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0041] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0042] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those skilled in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0043] The terms "first", "second", and the like in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attributes. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0044] In recent years, some regulatory agencies have increased their requirements for system data quality. However, for key regulatory scenarios, there is a lack of technical checking for non-standard data such as transaction messages of the transaction system, resulting in a lag in problem discovery and the need to improve the real-time monitoring capability of data quality to meet the relevant requirements of regulatory data quality.

[0045] The traditional data quality checking method identifies the transaction system with more problem data, collects the transaction messages of the transaction system by opening the transaction log switch, determines the transaction message field causing the data quality problem, analyzes the business scenario, continues to trace and collect transaction messages, loads the collected transaction messages into the database through data transmission, processes the transaction messages through data processing such as data cleaning, and extracts the message data (message field) that needs to be checked for data checking.

[0046] After collecting transaction messages, a large amount of computing resources are required for data format conversion, data cleaning and data integration, and data for verification is extracted from the massive amount of transaction messages. However, some of the extracted data often does not need to be verified, which increases the verification time and makes the verification efficiency low.

[0047] To address the aforementioned issues, this application provides a message verification method. After collecting transaction messages, this method places them into a real-time message queue, enabling batch processing of transaction messages. This avoids processing massive amounts of transaction messages at once, reducing processing time. Furthermore, this method uses a preset mapping table to directly determine the data used for verification within the transaction messages, avoiding the extraction of data that does not require verification, further reducing processing time. The message verification method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0048] Reference Figure 1 , Figure 1 This is a flowchart illustrating a message verification method provided in an embodiment of this application, such as... Figure 1 As shown in the embodiment of this application, a message verification method may include steps S10 to S14, which are described in detail below.

[0049] S10. Collect transaction messages from the transaction system and send the transaction messages to the real-time message queue;

[0050] The transaction messages can be business data generated during production transactions in the transaction system, recording the specific content of each transaction. Specifically, in this embodiment, transaction messages from the transaction system can be collected through a bypass device. The bypass device is a device that uses network bypass technology, a data acquisition technology that can copy data from the system's bypass (i.e., non-primary transmission path) without interfering with or affecting the normal operation of the system being collected, thereby achieving data acquisition.

[0051] Traditional data quality verification methods, when tracing the source of problematic data, rely on collecting transaction messages by turning on the transaction log switch. This requires coordinated changes across all transaction systems, impacting their operational transactions. Therefore, it's difficult to acquire transaction messages without disrupting their operation, making real-time transaction message collection impossible. This embodiment addresses this by using a bypass device to collect mirrored message data from each transaction system. Since the bypass device can collect transaction messages without affecting operational transactions and doesn't require coordinated modifications across systems, this embodiment allows for transaction message collection without disrupting operational transactions. Compared to collecting transaction messages from the transaction log, this significantly reduces deployment complexity and minimizes peak-hour delays caused by transaction log message collection, improving the timeliness of data quality verification.

[0052] In this embodiment, the specific process for determining the collection target of the bypass device can be as follows: A data check is performed. After the data check is completed, the problematic data is clustered according to the transaction initiator (security node) and transaction code corresponding to the problematic data. The transaction link is obtained by calling the transaction monitoring component, thereby locating the upstream transaction system and transaction code involved in the entire transaction link of the problematic data. The transaction system with the largest amount of problematic data can be the collection target of the bypass device in this embodiment. The transaction link can be as follows: Figure 2 As shown, "access" represents the access switch, "aggregation" represents the aggregation switch, and "core" represents the core switch. If system A needs to call a transaction of system B, the transaction link can be: System A, Access 1, Access 2, System B; if system A needs to call a transaction of system C, the transaction link can be: System A, Access 1, Aggregation 1, Aggregation 2, Access 3, System C; if system A needs to call a transaction of system D, the transaction link can be: System A, Access 1, Aggregation 1, Core 1, Core 2, Aggregation 3, Access 4, System D.

[0053] In this embodiment, the bypass device can filter transaction packets according to the dimensions of AP host, IP address, and transaction code through the optical splitter configuration. It can accurately locate relevant transaction packets in the transaction link, and convert the data format of the filtered transaction packets through the network bypass probe before sending them to the real-time message queue. The bypass device can automatically filter invalid packets to avoid data redundancy and reduce data burden.

[0054] Furthermore, due to the high-availability deployment mode, transaction packets can be randomly forwarded to any device in the corresponding functional cluster, and different devices may be located on different switches. Therefore, when collecting data from bypass devices, it is necessary to obtain the location of the access switch and use platform tools to obtain the deployment areas of the access switches of the transaction caller and the transaction service provider.

[0055] The real-time message queue is a queue system for storing and forwarding real-time messages, which can allow producers (message senders) to send messages to the queue, and consumers (message receivers) can obtain the messages from the queue in real time. The real-time message queue can realize asynchronous transmission of data producers and consumers. Therefore, in the embodiment, the bypass device sends the collected transaction messages (mirrored message data) to the real-time message queue quickly, and the real-time data integration tool can continuously obtain the transaction messages from the real-time message queue and process them. As shown in FIG. 12, the bypass device sequentially stores the collected message D, message C, message B and message A in the real-time message queue, and the real-time data integration tool sequentially reads and processes the message D, message C, message B and message A. Both ends do not need to wait online at the same time. Figure 3

[0056] The real-time message queue can cope with sudden traffic peaks. For example, during a large promotion, a large number of transaction messages are accompanied by orders. The transaction messages can be temporarily placed in the real-time message queue first and processed in batches to avoid high running pressure of the real-time data integration tool and slow processing. Therefore, the real-time message queue used in the embodiment can not only avoid the problem of sudden increase of transaction messages caused by traffic peaks in the data quality check process, but also ensure reliable transmission of transaction messages. Further, if the real-time data integration tool fails to successfully process the transaction message, the transaction message can exist in the real-time message queue.

[0057] S11, reading a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message;

[0058] S12, searching for a field name corresponding to the transaction code under each to-be-verified field type in a preset mapping relationship table, the preset mapping relationship table storing a corresponding relationship between the transaction code and the field name of the to-be-verified data under at least one to-be-verified field type;

[0059] S13, extracting a value of the field name corresponding to the transaction code from the message field and inputting the value to the real-time data lake as to-be-verified data for storage;

[0060] ​The message field of the transaction message can be an important component in the transaction message. The transaction message can be composed of multiple message fields with specific order and format, used to store and transmit various information of the transaction. After the bypass device delivers the transaction message to the real-time message queue, the embodiment can read the transaction message from the real-time message queue through the data integration tool, and decode the message field of the transaction message, so as to restore the data in the message field to the original transaction data, thereby obtaining the transaction code of the transaction message and the detailed transaction data in the transaction message. Of course, only the message field corresponding to the transaction code can be decoded to determine the transaction code of the transaction message. The transaction code is an identifier of the transaction message, which can represent the type of the transaction message.

[0061] The preset mapping relationship table can be a mapping relationship table between field names, which can be formulated in advance by the operator according to the business rules of the transaction system (the business rules can include fixed requirements of the transaction data). The preset mapping relationship table can include multiple check field types, which are target fields of data quality check. Each check field type can correspond to multiple transaction codes, and each transaction code can correspond to at least one field name. The field name corresponding to the transaction code can be the field name of the field to be checked in the transaction message.

[0062] The corresponding relationship in the preset mapping relationship table can be as shown in Figure 4 The check field type can include amount and time. The amount can correspond to transaction code 1 and transaction code 2. The transaction code 1 corresponds to the payment amount, and the transaction code 2 corresponds to the transfer amount. The time can correspond to transaction code 1 and transaction code 2. The transaction code 1 corresponds to the payment time, and the transaction code 2 corresponds to the transfer time. Of course, Figure 4 This is only an example to simply illustrate the corresponding relationship in the preset mapping relationship table. In actual situations, the check field type can be more, the corresponding transaction code can be more, and the transaction code corresponding to each check field type can be the same or different. The number of corresponding transaction codes can also have differences, and the field name corresponding to the transaction code can also have multiple. The embodiment can automatically realize data integration through the preset mapping relationship table, and can also add check field types in real time by modifying the preset mapping relationship table, which is more convenient.

[0063] When the data integration tool reads the transaction message and determines the transaction code of the transaction message, the transaction message can be data integrated according to the preset mapping relationship table. Of course, the data integration tool can also filter the transaction message before data integration. Specifically, after the data integration tool reads the transaction message, the transaction code of the transaction message is queried in the preset mapping relationship table to determine at least one check field type related to the transaction code, and the field name corresponding to the check field type in the transaction message is determined according to the field name recorded in the preset mapping relationship table corresponding to the transaction code, so as to determine the field name of the field of the to-be-checked data in the transaction message corresponding to the check field type. The field value corresponding to the field name in the transaction message is extracted as the to-be-checked data of the transaction message under the check field type.

[0064] For example, message A records that user B transferred 500 yuan to user C today. According to the transaction code of message A, it is determined that the transaction code of message A is involved in a plurality of transaction codes corresponding to a check field type (amount), and the field name corresponding to the transaction code is recorded as the transfer amount. Therefore, it can be determined that the check field type (amount) has a corresponding relationship with the transfer amount field in message A, and the field value (500) of the transfer amount field in message A is extracted as the to-be-checked data of message A under the check field type (amount).

[0065] Further, when the data integration tool processes a plurality of transaction messages at the same time, the preset mapping relationship table is queried according to the transaction code of each transaction message to determine the field name of the field of a plurality of to-be-checked data corresponding to the check field type. Then, the field value of a plurality of different transaction messages under the check field type can be corresponded.

[0066] After the data integration tool completes the data integration of the transaction message, the extracted field value is converted into structured report data and then transmitted to the data lake for storage. The data lake is an intermediate storage space that stores data from upstream systems or components for use by downstream systems or components.

[0067] The data storage format in the data lake is in the form of a table. Each row of the table represents a to-be-verified field type, and each column of the table corresponds to to-be-verified data of a transaction message under a to-be-verified field type. Optionally, the to-be-checked data can be identified by a transaction code. For example, the to-be-verified field type represented by the first row is amount, the first column is the to-be-checked data of message A (the field value of the transfer amount field), the second column is the to-be-checked data of message B (the field value of the payment amount field), and so on. The corresponding relationship between the transfer amount field and the to-be-verified field type (amount) and the corresponding relationship between the payment amount field and the to-be-verified field type (amount) are determined by the data integration tool querying the preset mapping relationship table.

[0068] S14, data quality inspection is performed on the to-be-inspected data in the real-time data lake.

[0069] In the embodiment, the data quality real-time monitoring tool is used to realize real-time monitoring of the to-be-inspected data in the real-time data lake. Specifically, the data quality inspection tool is used to acquire at least one to-be-inspected data from the data lake, and the automatic inspection is performed at a fixed frequency. The data quality inspection tool includes corresponding inspection scripts formulated according to the business rules of each transaction system.

[0070] In the embodiment, the data quality real-time monitoring tool is used to realize real-time monitoring of the to-be-inspected data in the real-time data lake. Specifically, the data quality inspection tool is used to acquire at least one to-be-inspected data from the data lake, and the automatic inspection is performed at a fixed frequency. The data quality inspection tool includes corresponding inspection scripts formulated according to the business rules of each transaction system.

[0071] The message inspection method provided in the embodiment can effectively reduce the time required for data quality inspection and improve the inspection efficiency.

[0072] Further, the bypass device automatically acquires the transaction message and stores it in the real-time message queue. The data integration tool automatically reads the transaction message in the real-time message queue, automatically extracts the to-be-inspected data according to the preset mapping relationship, and transmits the to-be-inspected data to the data lake for storage. The data quality inspection tool is used for real-time monitoring, thereby realizing the automation of the data quality inspection process. Compared with the traditional data quality inspection method, the method has stronger technical expandability, higher timeliness, and better automation capability.

[0073] The message inspection method provided in the embodiment can effectively reduce the time required for data quality inspection and improve the inspection efficiency.

[0074] Please refer to Figure 5 , Figure 5 FIG. 1 is a structural schematic diagram of a message inspection system provided in the embodiment. Figure 5 As shown in FIG. 1, the message inspection system can include:

[0075] The collection unit 100 is configured to collect transaction messages of a transaction system and send the transaction messages to a real-time message queue.

[0076] The reading unit 110 is configured to read a message field of at least one transaction message in the real-time message queue, and determine a transaction code of the transaction message.

[0077] The querying unit 120 is configured to query a field name corresponding to the transaction code under each to-be-verified field type in a preset mapping relationship table, and the preset mapping relationship table stores a corresponding relationship between the transaction code and the field name of the to-be-verified data under at least one to-be-verified field type.

[0078] The extracting unit 130 is configured to extract a value of the field name corresponding to the transaction code from the message field, and deliver the value as to-be-verified data to a real-time data lake for storage.

[0079] The checking unit 140 is configured to perform data quality checking on the to-be-verified data in the real-time data lake.

[0080] In a possible implementation, the collection unit 100 is specifically configured to:

[0081] The mirror message data of each transaction system is collected by the bypass device as the transaction message of each transaction system.

[0082] In a possible implementation, the data storage format in the data lake is in a table form, each row of the table represents a to-be-verified field type, and each column of the table corresponds to to-be-verified data of the transaction message under the to-be-verified field type.

[0083] In a possible implementation, the checking unit 140 is specifically configured to:

[0084] At least one to-be-verified data is obtained from the data lake by using a data quality checking tool, and automatic checking is performed.

[0085] The data quality checking tool includes corresponding checking scripts formulated according to business rules of each transaction system.

[0086] The embodiment of the application further provides an electronic device. Figure 6 As shown in FIG. 1, a structure schematic diagram of an electronic device suitable for implementing the electronic device in the embodiment of the application is shown. The electronic device in the embodiment of the application can include but is not limited to a fixed terminal such as a mobile phone, a notebook computer, a PDA (personal digital assistant), a PAD (tablet computer), a desktop computer, and the like. Figure 6 The electronic device shown in the figure is only an example, and should not bring any limitation to the functions and use range of the embodiment of the application.

[0087] As shown in FIG. 1, a structure schematic diagram of an electronic device suitable for implementing the electronic device in the embodiment of the application is shown. The electronic device in the embodiment of the application can include but is not limited to a fixed terminal such as a mobile phone, a notebook computer, a PDA (personal digital assistant), a PAD (tablet computer), a desktop computer, and the like. Figure 6As shown, the electronic device can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded from a storage device 608 into a random access memory (RAM) 603. In a state in which the electronic device is powered on, various programs and data required for operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a memory card, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device having various devices is shown, but it is understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can alternatively be implemented or possessed.

[0089] The embodiments of the present application also provide a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the packet checking methods provided by the embodiments of the present application.

[0090] The embodiments of the present application also provide a computer readable storage medium carrying one or more computer programs, which, when executed by an electronic device, can cause the electronic device to implement any of the packet checking methods provided by the embodiments of the present application.

[0091] In addition, it should be noted that the system embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the system embodiments provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0092] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components, etc. Generally, any function completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special-purpose circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0093] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product.

[0094] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0095] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0096] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0097] The above only shows the embodiments of the present application and is not used to limit the present application. The present application can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A message screening method, characterized by, The message checking method comprises: Collecting transaction messages of a transaction system and sending the transaction messages to a real-time message queue; Reading a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message; According to the transaction code, finding a field name corresponding to the transaction code under each field type to be checked in a preset mapping relationship table, and the preset mapping relationship table stores a corresponding relationship between the transaction code and the field name of the field of the data to be checked under at least one field type to be checked; Extracting the value of the field name corresponding to the transaction code from the message field and delivering the data to be checked to a real-time data lake for storage; Performing data quality checking on the data to be checked in the real-time data lake.

2. The message check method of claim 1, wherein, The transaction messages of the transaction system are collected, comprising: Collecting mirror message data of each transaction system through a bypass device as transaction messages of each transaction system.

3. The message review method of claim 1, wherein, The data storage format in the data lake is in table form, each row of the table represents a field type to be checked, and each column of the table corresponds to the data to be checked of the transaction message under a field type to be checked.

4. The message review method of claim 1, wherein, The data quality checking on the data to be checked in the real-time data lake comprises: Obtaining at least one data to be checked from the data lake through a data quality checking tool and performing automatic checking; The data quality checking tool contains corresponding checking scripts formulated according to the business rules of each transaction system.

5. A message screening system, characterized by, The message checking system comprises: A collection unit for collecting transaction messages of a transaction system and sending the transaction messages to a real-time message queue; A reading unit for reading a message field of at least one transaction message in the real-time message queue to determine a transaction code of the transaction message; A query unit for finding a field name corresponding to the transaction code under each field type to be checked in a preset mapping relationship table according to the transaction code, and the preset mapping relationship table stores a corresponding relationship between the transaction code and the field name of the field of the data to be checked under at least one field type to be checked; An extraction unit for extracting the value of the field name corresponding to the transaction code from the message field and delivering the data to be checked to a real-time data lake for storage; A checking unit for performing data quality checking on the data to be checked in the real-time data lake.

6. The message review system of claim 5, wherein, The collection unit is specifically configured to: Collect mirror message data of each transaction system through a bypass device as transaction messages of each transaction system.

7. The message review system of claim 5, wherein, The data storage format in the data lake is in table form, each row of the table represents a field type to be checked, and each column of the table corresponds to the data to be checked of the transaction message under a field type to be checked.

8. The message review system of claim 5, wherein, The checking unit is specifically configured to: Obtain at least one data to be checked from the data lake through a data quality checking tool and perform automatic checking; The data quality checking tool contains corresponding checking scripts formulated according to the business rules of each transaction system.

9. An electronic device, comprising: It comprises at least one processor and a memory connected with the processor, wherein: The memory is used to store a computer program; The processor is configured to execute the computer program to enable the electronic device to implement the message checking method according to any one of claims 1 to 4.

10. A computer program product, characterised in that, The computer readable instructions, when run on an electronic device, cause the electronic device to implement the message checking method according to any one of claims 1 to 4.

Images