General T+0 data real-time transmission method

Abstract

The invention provides a universal T + 0 data real-time transmission method, which comprises the following steps of: designing a database type and a table structure, and generating a corresponding interface document and an SM4 key; each service system sends the real-time data encrypted by the SM4 to the server side through HTTPS / TLS; the server performs authority verification, data decryption and verification, and records a verification failure result; sending the data passing the verification to a message queue; the consumption monitoring program analyzes the message, performs corresponding operation on the database according to the operation type, and records an execution result and error information; and the server provides an interface query execution result and a detailed error log. The method has the beneficial effects that the flexibility of data processing and the data transmission efficiency are improved, the development cost is reduced, the safety and the reliability in the data transmission process are guaranteed, and the current data execution result is quickly obtained while the memory is reduced.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a method for real-time transmission of universal T+0 data. Background Art

[0002] With the acceleration of my country's digitalization process and the deepening of dataization trends, more and more data need to be collected and transmitted to data centers in real time for analysis and processing. However, data centers are usually isolated from the outside world, and the database types used for storage are not unified, which poses a great challenge to real-time data transmission.

[0003] At present, most transmission methods are based on offline and scheduled transmission, which cannot meet the growing real-time requirements. At the same time, there are problems such as data security, inconsistent data formats, and inability to guarantee data verification quality. To address these problems, the present invention provides a universal T+0 data real-time transmission method, which can ensure the real-time, security and quality requirements of data for relational databases, has broad application prospects, and can also be used to expand and adapt non-relational databases based on this method. Summary of the invention

[0004] In view of this, the present invention aims to propose a universal T+0 data real-time transmission method, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] A method for real-time transmission of universal T+0 data, comprising the following steps:

[0007] Design database type and table structure, and generate corresponding interface documents and SM4 keys;

[0008] Each business system sends real-time data encrypted by SM4 to the server via HTTPS / TLS;

[0009] The server performs permission verification, data decryption and verification, and records the results of verification failures;

[0010] Send the verified data to the message queue;

[0011] The consumer listener parses the message and performs corresponding operations on the database according to the operation type, and records the execution results and error information;

[0012] The server provides interface query execution results and detailed error logs.

[0013] Furthermore, the process of designing the database type and table structure and providing the corresponding interface document and SM4 key includes:

[0014] Determine the database type, database name, table name, generate the interface document for the data table fields and the key required for SM4 encryption, and define the message format of the REST interface;

[0015] The interface document of the data table word includes interface description, request method, request URL, and request parameters;

[0016] The request parameters include: database type, database name, table name, and real-time data encrypted by SM4.

[0017] Furthermore, the process of each business system sending the real-time data encrypted by SM4 to the server via HTTPS / TLS includes:

[0018] Each business system prepares the real-time data to be sent according to the agreed format and encrypts the data using the provided SM4 key. After encryption, it constructs an HTTP POST request and sends a JSON object containing the encrypted data to the server interface, processing the result according to the response status of the server.

[0019] Furthermore, the server performs permission verification, data decryption and verification, and records the result of verification failure, including:

[0020] The server verifies the permissions of the business system;

[0021] Decrypt the received SM4 encrypted data and check whether the decryption process is successful;

[0022] Perform dictionary item and optional field validation and record validation results;

[0023] If the verification passes, the data is sent to the message queue for processing. If the verification fails, the relevant information is recorded in the bitmap cache and error log.

[0024] Furthermore, the process of the consumer listening program parsing the message and performing corresponding operations on the database according to the operation type and recording the execution result and error information includes:

[0025] The consumer monitoring program dynamically monitors the messages in the message queue, parses the message content and extracts information such as business library type, business library name, table name and operation type;

[0026] Obtain the corresponding database connection from the database connection pool and perform add, update or delete operations according to the operation type;

[0027] Record the execution results to the bitmap cache and log the error information to the error log if an error occurs.

[0028] Compared with the prior art, the method for real-time transmission of universal T+0 data described in the present invention has the following beneficial effects:

[0029] It can unify the data centers of different types of databases and provide a common data transmission interface to the outside world, which improves the flexibility of data processing and data transmission efficiency and reduces development costs. The data center only needs to agree on the design of table structure and dictionary items to realize common real-time data transmission and storage. Data encryption, dictionary verification and interface permission control are used to ensure the security and reliability of the data transmission process. The message queue is used to reduce the pressure of database concurrency. The bitmap cache can quickly obtain the current data execution results while reducing memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 The present invention is a flowchart of a method for real-time transmission of universal T+0 data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0033] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0034] A method for real-time transmission of universal T+0 data, comprising the following steps:

[0035] S1. Design the database type and table structure, and generate the corresponding interface document and SM4 key.

[0036] The specific working process of step S1 is: the process of designing the database type and table structure, and providing the corresponding interface document and SM4 key includes:

[0037] Determine the database type, database name, table name, generate the interface document for the data table fields and the key required for SM4 encryption, and define the message format of the REST interface;

[0038] The interface document of the data table word includes interface description, request method, request URL, and request parameters;

[0039] The request parameters include: database type, database name, table name, and real-time data encrypted by SM4.

[0040] Preferably, a specific execution process of step S1 is as follows:

[0041] S11. Determine the database type to be used according to business requirements, such as SQL Server, MySQL, PostgreSQL, etc., and name the database according to the business name;

[0042] S12. When determining the table name, it is also necessary to determine the table fields, which include field types and constraints (such as primary key, index, and timestamp);

[0043] S13. Establish a structure document, wherein the interface document includes:

[0044] 1. Interface description used to describe the purpose and function of the structure;

[0045] 2. The request method used to determine the HTTP method used (such as POST);

[0046] 3. Used to define the specific URL of the request, such as the request URL of / api / data;

[0047] 4. Request parameters used to specify the format of the request body and the meaning of each field;

[0048] The request parameters include:

[0049] dbType: database type

[0050] dbName: database name

[0051] tableName: table name

[0052] data: real-time data after SM4 encryption

[0053] S14. Generate a secure SM4 encryption key, ensure that the key is complex and difficult to crack, and store the key securely in an environment variable or key management service to ensure that only authorized services can access it. Finally, explain in the interface document how to use the SM4 key for data encryption and decryption.

[0054] S15. Define the REST interface message format to ensure that all business systems follow the same message format to facilitate server parsing. At the same time, define the server's response format, including successful and failed response examples, to ensure that the business system can correctly handle the response.

[0055] S2. Each business system sends real-time data encrypted by SM4 to the server via HTTPS / TLS.

[0056] The specific working process of step S2 is: the process of each business system sending the real-time data encrypted by SM4 to the server through HTTPS / TLS includes:

[0057] Each business system prepares the real-time data to be sent according to the agreed format and encrypts the data using the provided SM4 key. After encryption, it constructs an HTTP POST request and sends a JSON object containing the encrypted data to the server interface, processing the result according to the response status of the server.

[0058] Preferably, a specific execution process of step S2 is as follows:

[0059] S21. Each business system prepares the data to be sent according to the format agreed upon in the first step, including:

[0060] Business database type (such as sqlserver);

[0061] Business library name (for example, jc_ryk);

[0062] Business table name (for example, test_table);

[0063] Data content (real-time data that needs to be encrypted by SM4, including data unique identifier, operation type, etc.);

[0064] S22, perform SM4 encryption on the prepared data, the specific steps include:

[0065] Encrypt the data using the provided SM4 key, where SM4 is a symmetric encryption algorithm used to ensure the confidentiality of the data during transmission, and convert the encrypted data into a format suitable for transmission, usually a string format (such as hexadecimal or Base64 encoding);

[0066] S23. Use an appropriate HTTP library (such as requests in Python) to construct an HTTP request, set the request method to POST, set necessary parameters in the request header (such as ContentType to application / json), and send the JSON object containing the encrypted data as the request body;

[0067] S24. When sending a request, make sure to use the HTTPS protocol. This will encrypt the entire transmission process through the TLS layer, protect the data from man-in-the-middle attacks, and ensure that the connection with the server is secure by setting up SSL / TLS certificates and trust chains.

[0068] S25. Execute the HTTP request, send the encrypted data to the server interface, monitor the response status of the request, ensure that the request is sent successfully, and process the returned result as needed.

[0069] S26. Perform corresponding processing according to the response returned by the server (such as a success or failure status code). If the request is successful, a log can be recorded for subsequent tracking; if the request fails, an error message is recorded and a retry or an alarm is issued.

[0070] S3. The server performs permission verification, data decryption and verification, and records the verification failure results;

[0071] The specific working process of step S3 is: the server performs authority verification, data decryption and verification, and the process of recording the result of verification failure includes:

[0072] The server verifies the permissions of the business system;

[0073] Decrypt the received SM4 encrypted data and check whether the decryption process is successful;

[0074] Perform dictionary item and optional field validation and record validation results;

[0075] If the verification passes, the data is sent to the message queue for processing. If the verification fails, the relevant information is recorded in the bitmap cache and error log.

[0076] Preferably, a specific execution process of step S3 is as follows:

[0077] The server receives a request from the business system, extracts user identity information (such as user ID, role, etc.) and related data in the request (such as database type, database name, table name, etc.), queries the permission configuration table, and confirms whether the requesting user has the permission to operate the specified database and table. If the permission is insufficient, an error response (such as 403Forbidden) is returned and a log is recorded;

[0078] S31. Obtain the corresponding SM4 key from the secure storage, and use the SM4 algorithm to decrypt the encrypted data in the request, convert the decrypted data into a usable format (such as a JSON object). If an error occurs during the decryption process (such as an incorrect key, inconsistent data format, etc.), record an error log and return an appropriate error response.

[0079] S32. Extract the fields that need to be verified from the decrypted data, and verify whether the field values ​​conform to the predefined dictionary items (such as status code, type, etc.) to ensure the validity and consistency of the data. If the verification fails, record the failed record to the bitmap cache, record the error log, and return the corresponding error response;

[0080] S33. Query the database according to the table name to obtain the corresponding table structure information (field type, constraint conditions, etc.), ensure that the decrypted data field value conforms to the field definition of the database table (such as data type, whether it is NULL, etc.), and for fields that do not meet the conditions, record the failure information to the bitmap cache and error log, and return an error response;

[0081] S34. If all checks pass, prepare to send the data to the message queue for subsequent processing. If the check fails, record all the check failure information, including field name, failure reason, etc., to facilitate subsequent auditing and troubleshooting.

[0082] S4. Send the verified data to the message queue.

[0083] S5. The consumer listener parses the message and performs corresponding operations on the database according to the operation type, and records the execution results and error information.

[0084] The specific working process of step S5 is: the consumer listening program parses the message and performs corresponding operations on the database according to the operation type, and the process of recording the execution result and error information includes:

[0085] The consumer monitoring program dynamically monitors the messages in the message queue, parses the message content and extracts information such as business library type, business library name, table name and operation type;

[0086] Obtain the corresponding database connection from the database connection pool and perform add, update or delete operations according to the operation type;

[0087] Record the execution results to the bitmap cache and log the error information to the error log if an error occurs.

[0088] Preferably, a specific execution process of step S5 is as follows:

[0089] S51. The consumer monitoring program is started, connected to the message queue, and dynamically monitors all scheduled message topics. The API of the message queue (such as Kafka, RabbitMQ, etc.) can be used to realize message consumption;

[0090] S52. When a new message arrives, the consumer program reads the message content, parses the message content, and extracts necessary information, such as: business library type, business library name, table name, data unique identifier, operation type (add, update, delete) and actual data fields.

[0091] S53. According to the parsed business library type and business library name, a corresponding database connection is obtained from the database connection pool to ensure that the connection is valid and to handle possible connection errors.

[0092] S54. Execute corresponding database operations according to the operation type (add, update or delete), including:

[0093] New: Use the INSERT statement to insert new data into the target table;

[0094] Update: Use the UPDATE statement to update existing records. You need to locate the specific record based on the data unique identifier.

[0095] Delete: Use the DELETE statement to delete the specified record, which also relies on the unique identifier of the data.

[0096] S55. After the operation is executed, the result is recorded in the bitmap cache for subsequent query. If the operation is successful, the success status is recorded; if the operation fails, the reason and status of the failure are recorded.

[0097] S56. During the operation, if an error occurs (such as database connection failure, SQL execution error, etc.), it is necessary to handle the exception and record the error information in the error log, including time, error type, reason for failure and other detailed information. A retry mechanism may also be required as needed, depending on the specific business needs.

[0098] S57. After the operation is completed, close the database connection, release resources, and ensure that the connections in the connection pool are used reasonably.

[0099] S6. The server provides interface query execution results and detailed error logs.

[0100] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time transmission of universal T+0 data, characterized in that: The steps include: Design database type and table structure, and generate corresponding interface documents and SM4 keys; Each business system sends real-time data encrypted by SM4 to the server via HTTPS / TLS; The server performs permission verification, data decryption and verification, and records the results of verification failures; Send the verified data to the message queue; The consumer listener parses the message and performs corresponding operations on the database according to the operation type, and records the execution results and error information; The server provides interface query execution results and detailed error logs.

2. A method for universal T+0 data real-time transmission according to claim 1, characterized in that: The process of designing the database type and table structure and providing the corresponding interface document and SM4 key includes: Determine the database type, database name, table name, generate the interface document for the data table fields and the key required for SM4 encryption, and define the message format of the REST interface; The interface document of the data table word includes interface description, request method, request URL, and request parameters; The request parameters include: database type, database name, table name, and real-time data encrypted by SM4.

3. A method for universal T+0 data real-time transmission according to claim 1, characterized in that: The process of each business system sending the real-time data encrypted by SM4 to the server through HTTPS / TLS includes: Each business system prepares the real-time data to be sent according to the agreed format and encrypts the data using the provided SM4 key. After encryption, it constructs an HTTP POST request and sends a JSON object containing the encrypted data to the server interface, processing the result according to the response status of the server.

4. A method for universal T+0 data real-time transmission according to claim 1, characterized in that: The process of the server performing permission verification, data decryption and verification, and recording the result of verification failure includes: The server verifies the permissions of the business system; Decrypt the received SM4 encrypted data and check whether the decryption process is successful; Perform dictionary item and optional field validation and record validation results; If the verification passes, the data is sent to the message queue for processing. If the verification fails, the relevant information is recorded in the bitmap cache and error log.

5. A method for universal T+0 data real-time transmission according to claim 1, characterized in that: The process of the consumer listening program parsing the message and performing corresponding operations on the database according to the operation type and recording the execution result and error information includes: The consumer monitoring program dynamically monitors the messages in the message queue, parses the message content and extracts information such as business library type, business library name, table name and operation type; Obtain the corresponding database connection from the database connection pool and perform add, update or delete operations according to the operation type; Record the execution results to the bitmap cache and log the error information to the error log if an error occurs.

Images