Financial information submission method and device, computer equipment and readable storage medium

Abstract

The invention discloses a financial information submission method and device, computer equipment and a readable storage medium, and realizes dynamic, low-delay and high-reliability data transmission by constructing a long connection data transmission channel and a heartbeat service mechanism based on an MCWebSocket protocol. The method comprises the following steps: a server starts a creation process of a long connection data transmission channel, calls an MCWebSocket communication protocol to construct the long connection data transmission channel, and establishes a heartbeat service mechanism to detect the state of the long connection data transmission channel; the server side obtains data to be submitted, divides the data to be submitted into a message queue according to data types, and converts static structured messages in the message queue into dynamic data streams to be submitted based on a streaming computation engine; and the API service encrypts the to-be-submitted data stream, and transmits the to-be-submitted data stream to the client through the exclusive channel and / or the shared channel.

Description

Technical Field

[0001] This application relates to the fields of computer technology and fintech, and particularly to a financial information reporting method, apparatus, computer device, and readable storage medium. Background Art

[0002] In related technologies, the transmission mode of financial information mostly uses non-real-time protocols such as HTTP / 1.1 and FTP, and batch data updates are performed at fixed time intervals (such as daily, hourly, etc.). However, batch data updates at fixed time intervals result in poor timeliness of financial information and are difficult to meet the ever-changing needs of the financial market. Therefore, there is an urgent need for a reporting apparatus and method that can significantly improve the timeliness of financial information. Summary of the Invention

[0003] In view of this, this application provides a financial information reporting method, apparatus, computer device, and readable storage medium, mainly aiming to solve the problem that batch data updates at fixed time intervals result in poor timeliness of financial information.

[0004] According to a first aspect of this application, a financial information reporting method is provided. The method includes: The API service forwards the data transmission request to the server in response to the data transmission request; The server starts the creation process of the long connection data transmission channel to enable the API service to notify the client to create a data receiving node. When receiving the node number returned by the client, the server calls the MCWebSocket communication protocol to construct a long connection data transmission channel for the node number and establishes a heartbeat service mechanism to detect the state of the long connection data transmission channel. Among them, the long connection data transmission channel includes an exclusive channel and a shared channel. The exclusive channel is used to transmit metadata, and the shared channel is used to transmit service data, task data, and process data; Based on the data transmission request, the server obtains the data to be reported, standardizes the data to be reported into a structured message, divides the data to be reported into Kafka message queues according to the data type, and based on the streaming computing engine Apache Flink, converts the static structured messages in the Kafka message queues into dynamic data streams to be reported, and passes the data streams to be reported to the API service; The API service encrypts the data stream to be reported. When there is a metadata data stream in the data stream to be reported, the encrypted metadata data stream is transmitted to the client through the exclusive channel, and the data reception progress synchronized by the client is received. When the data reception progress indicates that the client has completed data reception, the other data streams to be reported except the encrypted metadata data stream are transmitted to the client through the shared channel, and the data reception progress synchronized by the client is received; The server and the API service collect the process data in the financial information reporting process and transmit it to the management end for visual display.

[0005] According to the second aspect of the present application, a financial information reporting device is provided, and the device includes: an API service, a server, and a management end; The API service, in response to a data transmission request, forwards the data transmission request to the server, and when detecting that the server starts the long connection data transmission channel creation process, notifies the client to create a data reception node, and encrypts the data stream to be reported. When there is a metadata data stream in the data stream to be reported, the encrypted metadata data stream is transmitted to the client through the exclusive channel, and the data reception progress synchronized by the client is received. When the data reception progress indicates that the client has completed data reception, the other data streams to be reported except the encrypted metadata data stream are transmitted to the client through the shared channel, and the data reception progress synchronized by the client is received, and the process data in the financial information reporting process is collected and transmitted to the management end for visual display; The server starts the long connection data transmission channel creation process, and when receiving the node number returned by the client, for the node number, calls the MCWebSocket communication protocol to build a long connection data transmission channel, and establishes a heartbeat service mechanism to detect the state of the long connection data transmission channel, and based on the data transmission request, obtains the data to be reported, standardizes the data to be reported into a structured message, and divides the data to be reported into Kafka message queues according to the data type, and based on the streaming computing engine Apache Flink, converts the static structured messages in the Kafka message queues into dynamic data streams to be reported, transmits the data streams to be reported to the API service, and collects the process data in the financial information reporting process and transmits it to the management end for visual display, wherein the long connection data transmission channel includes an exclusive channel and a shared channel, and the exclusive channel is used to transmit metadata, and the shared channel is used to transmit service data, task data, and process data.

[0006] According to a third aspect of the present application, there is provided a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method according to any one of the above first aspects are implemented.

[0007] According to a fourth aspect of the present application, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to any one of the above first aspects are implemented.

[0008] By means of the above technical solutions, the present application provides a financial information reporting method, device, computer device and readable storage medium. In the embodiments of the present application, by constructing a long connection data transmission channel and a heartbeat service mechanism based on the MCWebSocket protocol, the dynamic, low-latency and high-reliability of data transmission are realized. Moreover, through the intelligent division strategy of exclusive channels and shared channels, metadata and business data are transmitted separately, which improves the utilization rate of channel resources while reducing the risk of network congestion caused by mixed data transmission. And through the collaborative processing of the Apache Flink streaming computing engine and the Kafka message queue, the real-time streaming conversion of structured messages is realized, improving the data flow efficiency. In addition, the embodiments of the present application also improve the security protection ability of financial data transmission through multi-level encrypted transmission and progress synchronization mechanism, and eliminate the process black box through the visualization monitoring system of the management end, realizing the visualization tracking of the full-link transmission status.

[0009] The above description is only an overview of the technical solutions of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the following specific embodiments of the present application are specifically exemplified. Brief Description of the Drawings

[0010] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. And throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 It shows a schematic flowchart of a financial information reporting method provided by an embodiment of the present application; Figure 2 It shows a schematic architecture diagram of a financial information reporting device provided by an embodiment of the present application; Figure 3 It shows a schematic flowchart of a financial information reporting method provided by an embodiment of the present application; Figure 4The figure shows a schematic diagram of module interaction of a financial information reporting method provided by an embodiment of the present application; Figure 5 The figure shows a schematic diagram of the device structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0011] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described by referring to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation to the present application.

[0012] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0013] Those skilled in the art of the present technology can understand that unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the field to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood as having a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as here.

[0014] Those skilled in the art can understand that the "terminal" used herein includes both a device with a wireless signal receiver that only has the ability to receive and no ability to transmit, and a device with receiving and transmitting hardware that has the receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices, which may have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; conventional laptop and / or palm-top computers or other devices, which are conventional laptop and / or palm-top computers or other devices with and / or including a radio frequency receiver. The "terminal" used herein may be portable, transportable, installed in a vehicle (air, sea, and / or land), or suitable for and / or configured to operate locally, and / or operate in a distributed manner at any other location on the earth and / or in space. The "terminal" used herein may also be a communication terminal, an Internet access terminal, a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback functions, or may also be a smart TV, a set-top box, and other devices.

[0015] As Figure 1 shown, the embodiments of the present application provide a financial information reporting method. Further, as a specific implementation of the financial information reporting method, the embodiments of the present application also provide a financial information reporting device. First, the carrier of the financial information reporting method, that is, the financial information reporting device, will be introduced below.

[0016] The financial information reporting device includes three parts: an API service, a server, and a management terminal. Among them, the server can build a long-connection data transmission channel for real-time transmission of financial information. During actual operation, the sensitive real-time data of financial institutions is first extracted, processed and assembled, and encrypted and protected by the server, and then the sensitive real-time data is transmitted to the local business application of the client through the API service via the long-connection data transmission channel. Based on the management terminal, a visual full-process control ability is provided. In the specific design, the embodiments of the present application are based on technologies such as message queues, stream computing, and WebSocket to solve the problem that financial information data service providers cannot provide real-time data security and efficient interface services, enabling users to quickly and accurately obtain the latest financial data information and realizing data connection and sharing.

[0017] Exemplarily, the functional architecture of the financial information reporting device is as Figure 2 shown. Among them, the server consists of a data source control module, a data control module, a real-time data module, a data assembly module, and a permission control module. The API service consists of an identity security module, a data receiving module, a running data collection module, and a coding language plugin.

[0018] The data source control module of the server is hierarchically designed with a metadata service and a data source service. Among them, the metadata service provides attribute information management of the data source, can perform resource identification, dynamic evaluation, and change tracking on networked data sources, and realizes the work requirements of managing a large amount of networked data source data. The data source service externally responds to data transmission requests to obtain corresponding raw data, and internally executes internal governance operations such as registration, update, and status monitoring of the data source. Through the decoupling of the metadata service and the data source service, and the double-layer architecture of separating internal and external services, the flexibility and efficiency of large-scale data source management are guaranteed.

[0019] The data control module of the server provides a call unit, a service control unit, a streaming transmission control unit, and an asset permission control unit. Among them, the call unit receives external data requests and internal scheduling instructions, and forwards metadata and raw data to the service control unit. The service control unit, on the one hand, relies on metadata and raw data for historical job synchronization, job status management, log management, and task allocation. On the other hand, it calls the data assembly module to execute data processing and relies on the streaming transmission control unit for real-time data flow. The streaming transmission control unit receives the streaming data transmitted by the data assembly module and dynamically controls the transmission status, transmission points, link scheduling, and traffic of the real-time data stream. The asset permission control unit provides unified permission management and security prevention and control capabilities for the service control unit and the API. The data control module improves the response ability to customized requirements and solves the problem of functional operation complexity based on task splitting, streaming processing, and refined log and permission management.

[0020] The real-time data module of the server is the underlying support for data storage and real-time data operation within financial institutions. It supports the integrated management of heterogeneous data sources such as Oracle, MySQL, Hive, DB2, and external APIs. It also performs data distribution and message passing based on message queues such as Kafka, where the Kafka message queue connects data sources and real-time data processing and storage devices such as Flink and Hudi. The real-time data processing and storage devices provide real-time data processing to ensure that the data quality and format meet expectations. Specifically, Flink obtains raw data from the Kafka message queue, performs data processing, format standardization, and dynamic stream conversion, and then transmits the processed data back to the Kafka message queue to convert static data source information into dynamic real-time data stream information.

[0021] The data assembly module of the server consists of a task service unit, a task executor, a task generation unit, a task scheduler, and a result output unit. Among them, the task service unit receives service requests from the invocation unit in the data control module, reports heartbeat information to the service control unit in a timely manner, and sends task invocations to the task executor. The task executor initiates specific task invocations to the task scheduler. The task scheduler controls task execution through a task scheduler and provides services such as task generation, task submission, task status update, and task metric update. Among them, task generation depends on the relevant services of the task generation unit. The task scheduler realizes the streaming conversion of data by invoking the execution engine service capabilities in the real-time data module. The task generation unit includes a task job generator and task job context management, providing more refined and accurate task control. The result output module is used to assemble result data into a streaming transmission data packet. After passing through the streaming transmission control in the data control module, it interacts with the outside through the invocation unit. Through the full-process management of task generation - scheduling - execution, it simplifies customer operations, enhances operation efficiency, and provides a technical foundation for data asset control.

[0022] The permission control module of the server serves as the security center of the entire solution, providing security control services for access requests and output content; providing password verification, verification code verification, and certificate verification functions for access requests; supporting dynamic encryption of access links based on dynamic tokens; providing message encryption for transmission links and data file encryption for transmission content; providing frame extraction verification services to sensitively identify whether there is eavesdropping or message interception and forgery behavior in the outgoing link. The permission control module effectively improves the system's security protection capabilities through multiple security mechanisms (such as dynamic encryption and frame extraction auditing).

[0023] Exemplarily, the identity security of the API service provides four major services: interface login, encrypted transmission, credential refresh, and certificate control. It cooperates with the permission control module on the server side to ensure the security of the API service, build an end-to-end security protection system, solve the problems of interface exposure risk and insufficient client security protection ability, and lay a trusted foundation for subsequent data transmission.

[0024] The data receiving module of the API service is a key module for realizing the effective transfer of real-time data to the client side, providing data stream reception, data stream verification, data queue sorting, and data landing services. After security analysis, the data receiving module obtains the data with the message structure converted from the Kafka message queue, and executes processes such as information acquisition, chunked transmission, automatic parsing, queue control, and data landing on the received data stream, and transmits it to the client in real time through the MCWebSocket protocol to ensure real-time and instantaneity.

[0025] The coding language plugin of the API service dynamically identifies development languages such as Java, Python, Go, C#, etc. according to the Accept-Language or User-Agent field in the client request header. Load the SDK of the corresponding language version from the plugin library, such as Java JAR package, Python Wheel package, etc., to generate a standardized API interface, eliminate the integration cost caused by programming language differences on the client side, and improve the solution customization ability.

[0026] The data collection module of the API service executes data collection operations by monitoring the running status of the terminal in real time, including indicators such as CPU / memory occupancy rate, network latency, and exception logs, to achieve a comprehensive perception of the client's running state data. By performing lightweight computing operations such as data cleaning, aggregation, and compression locally, the initial processing of the original data and the optimization of the transmission load are realized. It can be understood that the data collection module can execute the upload policy triggered during idle periods according to the device resource usage conditions such as battery power, network bandwidth, and CPU load through an intelligent scheduling algorithm to achieve the balanced control of data reporting and service performance.

[0027] Exemplarily, the management end provides management services such as process visualization, user management, permission management, asset group management, link monitoring, link reset, data retransmission, resume interrupted transfer, data statistics, traffic management, and plugin management based on the uploaded process data. At the same time, the problems of the data service process black box and insufficient visualization monitoring and disposal capabilities are solved through user management, permission management, and asset group management. Among them, the data acquisition module of the API service and the call unit in the data control module of the server end provide the function of uploading process data at both ends. The call unit not only collects the operation data generated by the data control module itself, but also collects real-time data and process data in the data assembly module, and completes data uploading at different times. It can be understood that the management end draws a complete data transmission service process through the disposal of the operation data at both ends, monitors the data service situation in real time, and provides various types of control services such as plugin control and emergency operations. It is responsible for the synchronous management of different data files, can set multiple synchronization strategies, select different synchronization strategies for each data file, and select whether to enable synchronization. For data that is not needed temporarily, users can choose to turn it off.

[0028] The device provided by the embodiment of the present application realizes the dynamic, low-latency, and high-reliability data transmission by constructing a long connection data transmission channel and a heartbeat service mechanism based on the MCWebSocket protocol. Moreover, through the intelligent division strategy of the exclusive channel and the shared channel, the metadata and business data are transmitted separately, which improves the utilization rate of channel resources while reducing the risk of network congestion caused by mixed data transmission. And through the collaborative processing of the Apache Flink streaming computing engine and the Kafka message queue, the real-time streaming conversion of structured messages is realized, improving the data transfer efficiency. In addition, the embodiment of the present application also improves the security protection ability of financial data transmission through multi-level encrypted transmission and progress synchronization mechanism, and eliminates the process black box through the visualization monitoring system of the management end, realizing the visualization tracking of the full-link transmission state.

[0029] The following introduces the financial information reporting method process. A financial information reporting method provided by the embodiment of the present application includes: 101. The API service forwards the data transmission request to the server end in response to the data transmission request.

[0030] In an embodiment of the present application, the API service can identify the development language of the client based on the Accept-Language or User-Agent field in the client request header, such as Java, Python, Go, C#, etc., and then match the corresponding communication plugin from the plugin library. The API service loads the target communication plugin to generate a standardized API interface, enabling the client to perform data interaction with the server through the communication plugin, achieving protocol compatibility transmission in a cross-language environment. Eliminate the integration cost caused by programming language differences in the client and improve the solution customization ability.

[0031] Further, the client submits a login request containing an identity identifier, a dynamic token signature, and credential information through the communication plugin. The API service calls the identity security module to parse the request message, extracts the identity identifier, dynamic token signature, and credential information, and passes them to the permission control and verification module on the server side. The permission control and verification module verifies the identity identifier, dynamic token signature, and credential information, including digital signature verification, credential validity check, and identity permission matching. After the verification result indicates passing, a time-limited access token is generated and passed to the API service.

[0032] It can be understood that the token contains client identifier, permission scope, and valid period information, and the token is digitally signed using an asymmetric encryption algorithm and returned to the client through a secure channel.

[0033] After the client obtains the token, the communication plugin automatically injects it into the protocol message header of subsequent requests, such as the HTTP Authorization header.

[0034] It can be understood that when the server processes the data request of the client subsequently, it can verify the validity and digital signature of the token in real time, and then complete the identity authentication and permission verification of the request.

[0035] 102. The server starts the creation process of the long connection data transmission channel to enable the API service to notify the client to create a data receiving node. When receiving the node number returned by the client, the server calls the MCWebSocket communication protocol to build a long connection data transmission channel for the node number and establishes a heartbeat service mechanism to detect the state of the long connection data transmission channel. Among them, the long connection data transmission channel includes an exclusive channel and a shared channel. The exclusive channel is used to transmit metadata, and the shared channel is used to transmit business data, task data, and process data.

[0036] In an embodiment of the present application, such as Figure 3 and Figure 4As shown, when the server receives a data transmission request, it prepares to initiate the creation workflow of the long connection data transmission channel. Based on the extended MCWebSocket protocol, the server classifies and manages the data and plans the long connection transmission channel. Specifically, the server first calls the data control module to initiate the creation process of the long connection data transmission channel. In this process, the API service is used to notify the client to create a metadata receiving node. It should be noted that the metadata contains key information describing the attributes of the transmitted data, such as data format, data volume, data structure, data field type, etc. It provides a basis for the client to receive, store, and use the data. And once the metadata changes, the server will promptly push the update to the client.

[0037] After the client completes the creation of the metadata receiving node NodeS, it returns the node number, such as NSid-XXX, to the server. After receiving this node number, the server calls the MCWebSocket protocol to construct the first long connection data transmission channel, such as NSid-XXX-Channel, for the NSid-XXX node. This channel is an exclusive channel dedicated to transmitting metadata. A heartbeat service mechanism is established between the two parties. The client initiates a ping command, and the server returns a pong command to confirm the establishment of the long connection. In particular, two primary and backup channels will be created for NSid-XXX-Channel to support active switching in case of anomalies.

[0038] Furthermore, the API service notifies the client to create a cluster of data receiving nodes, which includes task data receiving nodes, business data receiving nodes, and process data receiving nodes. After the client completes the creation of the node cluster, it returns the node number cluster to the server. The node number cluster includes the task data receiving node number NTid-XXX, the business data receiving node number NBid-XXX, and the process data receiving node number NMid-XXX. After receiving the node number cluster, the server calls the MCWebSocket protocol to construct the second long connection data transmission channel for transmitting business data streams, the third long connection data transmission channel for transmitting task data streams, and the fourth long connection data transmission channel for transmitting process data streams for the NBid-XXX, NTid-XXX, and NMid-XXX nodes respectively. These three channels are all shared channels. These shared channels achieve efficient sharing based on multiplexing technology and achieve balance in channel transmission capabilities through the channel load balancing capabilities integrated in the channel management component. A heartbeat service mechanism is also established between the two parties. The client initiates a ping command, and the server returns a pong command to establish a shared long connection. Moreover, two primary and backup channels will also be created for these three channels to support active switching in case of anomalies.

[0039] To ensure the reliability and stability of the long - connection data transmission channel, each long - connection data transmission channel is equipped with a primary transmission channel and a backup transmission channel. At the same time, a heartbeat service mechanism is established to detect the channel status in real - time. If the heartbeat of the primary transmission channel times out, the server will call the backup transmission channel to ensure that the API service can continue to transmit the corresponding data stream to be reported based on the backup transmission channel. This mechanism effectively avoids the problem of data transmission interruption caused by a single - channel failure, greatly improves the reliability of data transmission, and ensures the continuity and stability of financial information reporting.

[0040] It should be noted that the MCWebSocket communication protocol in the embodiments of this application is obtained by optimizing the transmission channel management, the transmission data compression algorithm, and the protocol message header extension on the basis of the standard WebSocket duplex communication protocol. It can improve the real - time performance, accuracy, security, and efficiency of the financial information reporting device, and specifically includes the following improvements: First, for the transmission channel management, it supports multi - category transmission channel customization, automatic transmission management, and multiplexing. By modifying the WebSocket constructor, it can parse out the multi - channel identification information mchlId (composed of the channel type chlType + single - channel identifier schlId) from a URL with a channel identifier ID (such as ws: / / mchlId / example.com / ws). The protocol reserved field RSV1 is enabled to indicate whether to use the multi - channel scheduling mechanism, and mchlId is passed as a parameter to the subsequent custom sendbychl(), onopenbychl(), onmessagebychl(), onerrorbychl(), and onclosebychl() events. The server adds a channel and data management component, assembles the mchlId information according to the principle of using exclusive channels for metadata and shared channels for other data, plans the channel load - balancing strategy based on the data volume and channel traffic, passes mchlId as a parameter to the constructor to complete the WebSocket construction, and calls five APIs such as sendbychl() based on the planned load - balancing strategy to complete the construction and hybrid management of multi - type long - connection channels. In the channel multiplexing part, a shared - channel mechanism is built by extending the MultiplexingExtension. By modifying the message - receiving receive() method, channel subscription subscribe(), channel cleanup cleanupChannels(), etc., a multi - channel sharing mechanism based on mchlId is realized. Based on the extended multiplexing, the channel pressure is balanced, and the channel transmission bandwidth is optimized to further improve the data transmission efficiency. The real - time performance and efficiency of data transmission are continuously improved through technical means such as reconstructing the constructor, modifying the API interface, adding channel and data management, and extending multiplexing.

[0041] Optimization of the transmission data compression algorithm. By rewriting the CompressionExtension constructor, parameters such as the compression threshold `threshold` and the compression algorithm level are set, and the compression ratio is continuously monitored through the `compression.ratio` parameter. In terms of the compression algorithm setting, considering the diversity of the transmission data formats in real-time data services, the configuration mechanism of the compression algorithm level parameter is extended, and the protocol reserved field RSV2 is enabled to indicate whether to adaptively select compression algorithms such as zlib and gzip. The data sending server supports autonomously switching compression algorithms according to different data types. For example, gzip is automatically selected for file stream compression, and zlib is automatically selected for data stream files. Subsequently, data compression is completed based on the selected algorithm, and the data is sent after the data compression is completed.

[0042] Extension of the protocol message header. The basic WebSocket message format consists of a message header and a message body. The message body part is the actual transmitted data, which supports text and binary data. This extension mainly targets the message header part, and the content of the extended message header is as follows: FIN: Indicates the integrity of the information, and 1 indicates the end of the message.

[0043] RSV1, RSV2, RSV3: Two reserved fields RSV1 and RSV2 are enabled. Among them, RSV1 stores the identification information corresponding to the multi-channel mchlId, RSV2 stores the compression algorithm selection identification information, and RSV3 remains unused as a reserved field.

[0044] Opcode: Identifies the type of the message, including text messages, binary messages, the client sending a ping message to the server, and the server responding with a pong message to the client. Among them, for...

[0045] Mask: Indicates whether the message is encrypted.

[0046] Payload length: Indicates the length of the message body.

[0047] Masking key: Only appears when the message needs to be encrypted and is used to encrypt and decrypt the message.

[0048] Payload data: Extended data is added. Among them, the 0 - 4 bytes represent the specific multi-channel information mchlId, which is used in conjunction with RSV1; the 5 - 6 bytes represent the specific compression algorithm, which is used in conjunction with RSV2. The subsequent bytes are used to store application data.

[0049] 103. The server obtains the data to be reported based on the data transmission request, standardizes the data to be reported into a structured message, divides the data to be reported into the Kafka message queue according to the data type, and based on the streaming computing engine Apache Flink, converts the static structured message in the Kafka message queue into a dynamic data stream to be reported, and transmits the data stream to be reported to the API service.

[0050] After receiving the data transmission request, the server calls the data control module to receive the request and transmits the data transmission request to the data assembly module. The data assembly module calls the data source control module according to the data identifier indicated by the data transmission request, obtains the metadata and the original data from the heterogeneous data source, and processes these data as the data to be reported. It should be noted that the original data comes from heterogeneous data sources, and the heterogeneous data sources include but are not limited to Oracle, MySQL, Hive, DB2, and external APIs. The data assembly module standardizes the data to be reported into a structured message, enabling data from different sources and in different formats to be processed in a unified format in subsequent processes, facilitating data storage, analysis, and transmission, and improving the system's compatibility and processing efficiency for various types of data. Furthermore, a data source identifier, a timestamp, and version information are added to the structured message, and the structured message with the added information is divided into the Kafka message queue. The server calls the real-time data module and uses the streaming computing engine Apache Flink to convert the static structured message in the Kafka message queue into a dynamic data stream to be reported, and transmits the data stream to be reported to the API service. Kafka has characteristics such as high throughput and strong scalability. It can quickly process a large amount of data, buffer and distribute the data to be processed, avoid blocking problems during data transmission, and improve the efficiency of data transmission. At the same time, Kafka supports the persistent storage of messages, ensuring that data will not be lost, providing a reliable guarantee for subsequent data processing. Flink has low latency and high concurrent processing capabilities and can process data streams in real time. By converting the static message into a dynamic data stream, the real-time transmission and processing of data are achieved, meeting business scenarios with high requirements for data real-time performance, such as financial transaction monitoring and real-time data analysis.

[0051] 104. The API service encrypts the data stream to be reported. When there is a metadata data stream in the data stream to be reported, it transmits the encrypted metadata data stream to the client through an exclusive channel and receives the data reception progress synchronized by the client. When the data reception progress indicates that the client has completed data reception, it transmits the other data streams to be reported except the encrypted metadata data stream to the client through a shared channel and receives the data reception progress synchronized by the client.

[0052] In the embodiments of the present application, the API service call data receiving module is responsible for receiving the data stream to be reported. At the same time, the API service call identity security module encrypts the data stream to be reported with a dynamic token and marks the encryption status in the header of the MCWebSocket communication protocol, so as to ensure the security of data during transmission. During actual operation, the data stream to be reported may include at least one of a metadata stream, a business data stream, a task data stream, and a process data stream. When there is a metadata stream in the data stream to be reported, the API service initiates real-time transmission of the metadata stream through the exclusive channel NSid-XXX-Channel. The client synchronously receives the data and timely sends the data reception progress to the API service. The client decrypts the encrypted metadata stream based on the Masking key field in the header of the MCWebSocket communication protocol to obtain the metadata. When the data reception progress indicates that the client has completed the reception of the metadata stream, the client completes the execution of the business logic related to the metadata, laying a foundation for subsequent data use. Then, the client notifies the server that the metadata has been successfully received and waits for the server's response regarding the start of the transmission of business data and task data streams.

[0053] After receiving the notification from the client that the metadata has been successfully received, the server transmits the other data streams to be reported except the encrypted metadata stream to the client in real time through the shared channel. The client synchronously receives the data and sends the data reception progress to the API service. At the same time, the client decrypts the encrypted data stream to be reported based on the Masking key field in the protocol header to obtain business data and task data. The client processes the business data according to the rules indicated by the task data. After the reception of the data stream to be reported is completed, the client notifies the server that the current round of data transmission is completed.

[0054] The API service receives the data reception progress sent by the client, updates the transmission status of the data stream to be reported according to this progress, and records the breakpoint sequence number. And the data reception progress is transmitted to the management end for visual display, facilitating the management personnel to grasp the data transmission situation in real time.

[0055] It can be understood that the client sends the data reception progress to the server at regular time intervals. If data loss is detected during the reception, the client will initiate a breakpoint resumption request. The API service re-transmits the missing data according to the breakpoint sequence number carried in the breakpoint resumption request to ensure the complete transmission of the data.

[0056] 105. The server and the API service collect the process data in the financial information reporting process and transmit it to the management end for visual display.

[0057] In the embodiments of the present application, the management terminal provides management services such as process visualization, user management, permission management, asset group management, link monitoring, link reset, data retransmission, breakpoint resumption, data statistics, traffic management, and plugin management based on the uploaded process data. At the same time, the problems of data service process black box and insufficient visualization monitoring and handling capabilities are solved through user management, permission management, and asset group management. Among them, the data acquisition module of the API service and the call unit in the data control module of the server provide the function of uploading process data at both ends. The call unit not only collects the operation data generated by the data control module itself, but also collects real-time data and process data in the data assembly module, and completes data uploading at different times. It can be understood that the management terminal draws a complete data transmission service process through the handling of the operation data at both ends, monitors the data service situation in real time, and provides various types of control services such as plugin control and emergency operations.

[0058] It can be understood that in the embodiments of the present application, the server can also obtain the data to be reported according to the scheduling plan in response to the autonomous scheduling instruction. Standardize the data to be reported into structured messages, and divide the data to be reported into Kafka message queues according to the data type. And based on the Apache Flink streaming computing engine, convert the static data to be reported in the Kafka message queue into dynamic streaming data, and encrypt the streaming data to obtain an encrypted data stream to be reported. When there is an encrypted metadata stream in the data stream to be reported, the server calls the API service to transmit the encrypted metadata stream to the client through an exclusive channel, and receives the data reception progress synchronized by the client. When the data reception progress indicates that the client has completed data reception, transmit the other data streams to be reported except the encrypted metadata stream to the client through a shared channel, and receive the data reception progress synchronized by the client. The server and the API service collect the process data in the financial information reporting process and transmit it to the management terminal for visual display.

[0059] The method provided by the embodiments of the present application realizes the dynamic, low-latency, and high-reliability data transmission by constructing a long connection data transmission channel and a heartbeat service mechanism based on the MCWebSocket protocol. Moreover, through the intelligent division strategy of the exclusive channel and the shared channel, the metadata and business data are classified and transmitted, which improves the utilization rate of channel resources while reducing the risk of network congestion caused by mixed data transmission. And through the collaborative processing of the Apache Flink streaming computing engine and the Kafka message queue, the real-time streaming conversion of structured messages is realized, which improves the data transfer efficiency. In addition, the embodiments of the present application also improve the security protection ability of financial data transmission through the multi-level encryption transmission and progress synchronization mechanism, and eliminate the process black box through the visualization monitoring system of the management terminal, realizing the visualization tracking of the full-link transmission status.

[0060] To solve the above technical problems, an embodiment of the present invention further provides a computer device. For details, please refer to Figure 5 , Figure 5 , which is a basic structural block diagram of the computer device in this embodiment.

[0061] As shown in Figure 5 , it is a schematic internal structure diagram of the computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected through a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. Control information sequences can be stored in the database. When the computer-readable instructions are executed by the processor, the processor can implement a data relationship reconstruction method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. Computer-readable instructions can be stored in the memory of the computer device. When the computer-readable instructions are executed by the processor, the processor can execute a data relationship reconstruction method. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand that Figure 5 The structure shown in

[0062] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0063] The present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, one or more processors are caused to execute the steps of the data relationship reconstruction method in any of the above embodiments.

[0064] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0065] Those skilled in the art can understand that the various operations, methods, steps, measures, and solutions in the processes discussed in this application can be alternated, changed, combined, or deleted. Further, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, changed, rearranged, decomposed, combined, or deleted. Further, those in the prior art that have steps, measures, and solutions in the various operations, methods, and processes disclosed in this application can also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0066] The above are only some embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A financial information reporting method, characterized in that, Including: The API service forwards the data transmission request to the server in response to the data transmission request. The server initiates a process for creating a long - connection data transmission channel to enable the API service to notify the client to create a data receiving node. When receiving the node number returned by the client, the server calls the MCWebSocket communication protocol to construct a long - connection data transmission channel for the node number, and establishes a heartbeat service mechanism to detect the status of the long - connection data transmission channel. Among them, the long - connection data transmission channel includes an exclusive channel and a shared channel. The exclusive channel is used to transmit metadata, and the shared channel is used to transmit business data, task data, and process data. Based on the data transmission request, the server obtains the data to be reported, standardizes the data to be reported into a structured message, divides the data to be reported into the Kafka message queue according to the data type, and based on the streaming computing engine Apache Flink, converts the static structured messages in the Kafka message queue into dynamic data streams to be reported, and transmits the data streams to be reported to the API service. The API service encrypts the data stream to be reported. When there is a metadata data stream in the data stream to be reported, it transmits the encrypted metadata data stream to the client through the exclusive channel and receives the data reception progress synchronized by the client. When the data reception progress indicates that the client has completed data reception, it transmits the other data streams to be reported except the encrypted metadata data stream to the client through the shared channel and receives the data reception progress synchronized by the client. The server and the API service collect process data in the financial information reporting process and transmit it to the management end for visual display.

2. The method according to claim 1, wherein Before the API service forwards the data transmission request to the server in response to the data transmission request, the method further includes: The API service calls a coding language plugin, determines a communication plugin matching the coding language type according to the coding language type of the client, and loads the communication plugin to enable the client to perform data interaction with the server through the communication plugin. The API service calls an identity security module, parses the login request message uploaded by the client through the communication plugin, obtains the identity identifier, dynamic token signature, and credential information carried in the login request message, and transmits the identity identifier, the dynamic token signature, and the credential information to the permission control and verification module of the server. The server calls the permission control and verification module to verify the identity identifier, the dynamic token signature, and the credential information. After the verification result indicates passing, it generates a time - limited access token and transmits the access token to the API service. The API service sends the access token to the client so that the client injects the access token into the protocol message header of the data transmission request through the communication plugin.

3. The method according to claim 1, wherein The server starts the creation process of the long - connection data transmission channel to enable the API service to notify the client to create a data receiving node, and when receiving the node number returned by the client, the server calls the MCWebSocket communication protocol to construct a long - connection data transmission channel for the node number, including: The server calls the data control module to start the creation process of the long - connection data transmission channel to enable the API service to notify the client to create a metadata receiving node, and passes the node number returned by the client to the server; When the server receives the node number, it creates a first long - connection data transmission channel, where the first long - connection data transmission channel is the exclusive channel; The API service notifies the client to create a data receiving node cluster, and passes the node number cluster returned by the client to the server; When the server receives the node number cluster, it creates a second long - connection data transmission channel for transmitting business data streams, a third long - connection data transmission channel for transmitting task data streams, and a fourth long - connection data transmission channel for transmitting process data streams. The second long - connection data transmission channel, the third long - connection data transmission channel, and the fourth long - connection data transmission channel are shared channels; Among them, each long - connection data transmission channel includes a main transmission channel and a standby transmission channel. A heartbeat service mechanism is established to detect the status of the long - connection data transmission channel. If it is detected that the heartbeat of the main transmission channel times out, the server is called to switch to the standby transmission channel, so that the API service transmits the corresponding data stream to be reported based on the standby transmission channel. The data receiving node cluster includes task data receiving nodes, business data receiving nodes, and process data receiving nodes; Among them, the MCWebSocket communication protocol is extended on the basis of the standard WebSocket duplex communication protocol. The MCWebSocket communication protocol distinguishes data transmission channel types through channel identifiers, selects a compression algorithm according to the data type, and identifies the compression algorithm type through the RSV2 field of the protocol header. An RSV1 field is added to the WebSocket header to store channel identification information, where the channel type includes the exclusive channel and the shared channel.

4. The method according to claim 1, characterized in that, Based on the data transmission request, the server obtains the data to be reported, standardizes the data to be reported into a structured message, divides the data to be reported into Kafka message queues according to the data type, and based on the streaming computing engine Apache Flink, converts the static structured messages in the Kafka message queues into dynamic data streams to be reported, and passes the data streams to be reported to the API service, including: The server calls the data control module to receive the data transmission request and passes the data transmission request to the data assembly module; The server, through the data assembly module, calls the data source control module to obtain metadata and raw data as the data to be reported according to the data identifier indicated by the data transmission request, standardizes the data to be reported into a structured message, adds a data source identifier, a timestamp, and version information to the structured message, divides the structured message with the added data source identifier, timestamp, and version information into Kafka message queues according to data types, and calls the real-time data module to convert the static structured message in the Kafka message queue into a dynamic data stream to be reported by using the streaming computing engine Apache Flink, and transmits the data stream to be reported to the API service; Among them, the raw data comes from heterogeneous data sources, and the heterogeneous data sources include but are not limited to Oracle, MySQL, Hive, DB2, and external APIs.

5. The method according to claim 1, wherein The API service encrypts the data stream to be reported. When there is a metadata stream in the data stream to be reported, it transmits the encrypted metadata stream to the client through the exclusive channel, and receives the data reception progress synchronized by the client. When the data reception progress indicates that the client has completed data reception, it transmits the other data streams to be reported except the encrypted metadata stream to the client through the shared channel, and receives the data reception progress synchronized by the client, including: The API service calls the data reception module to receive the data stream to be reported, calls the identity security module to perform dynamic token encryption on the data stream to be reported, and marks the encryption status in the MCWebSocket communication protocol header; When there is a metadata stream in the data stream to be reported, the API service transmits the encrypted metadata stream to the client through the exclusive channel; The client receives the encrypted metadata stream through the exclusive channel, sends the data reception progress to the API service, and decrypts the encrypted metadata stream based on the Masking key field in the protocol header to obtain the metadata; When the data reception progress indicates that the client has completed the reception of the metadata stream, the API service transmits the other data streams to be reported except the encrypted metadata stream to the client through the shared channel; The client receives the encrypted data stream to be reported through the shared channel, sends the data reception progress to the API service, and decrypts the encrypted data stream to be reported based on the Masking key field in the protocol header to obtain business data and task data, and processes the business data according to the rules indicated by the task data; The API service receives the data reception progress sent by the client, updates the transmission status of the data stream to be reported according to the data reception progress, records the breakpoint sequence number, and transmits the data reception progress to the management end for visual display.

6. The method according to claim 5, characterized in that The method further includes: The client sends the data reception progress to the server at fixed time intervals. If data loss is detected, a breakpoint resumption request is initiated so that the API service re-transmits the data based on the breakpoint sequence number carried in the breakpoint resumption request.

7. The method according to claim 1, characterized in that, The method further includes: The server, in response to an autonomous scheduling instruction, obtains the data to be reported, standardizes the data to be reported into a structured message, divides the data to be reported into Kafka message queues according to data types, and based on the Apache Flink streaming computing engine, converts the static data to be reported in the Kafka message queues into dynamic streaming data, and encrypts the streaming data to obtain an encrypted data stream to be reported; When there is an encrypted meta-data stream in the data stream to be reported, the server calls the API service to transmit the encrypted meta-data stream to the client through the exclusive channel, and receives the data reception progress synchronized by the client. When the data reception progress indicates that the client has completed data reception, the server transmits other data streams to be reported except the encrypted meta-data stream to the client through the shared channel, and receives the data reception progress synchronized by the client; The server and the API service collect the process data in the financial information reporting process and transmit it to the management end for visual display.

8. A financial information reporting device, characterized in that, It includes: API service, server and management end; The API service, in response to a data transmission request, forwards the data transmission request to the server, notifies the client to create a data reception node when detecting that the server starts the long connection data transmission channel creation process, encrypts the data stream to be reported. When there is a meta-data stream in the data stream to be reported, the API service transmits the encrypted meta-data stream to the client through the exclusive channel, and receives the data reception progress synchronized by the client. When the data reception progress indicates that the client has completed data reception, the API service transmits other data streams to be reported except the encrypted meta-data stream to the client through the shared channel, and receives the data reception progress synchronized by the client, and collects the process data in the financial information reporting process and transmits it to the management end for visual display; The server starts the creation process of the long connection data transmission channel. When receiving the node number returned by the client, for the node number, it calls the MCWebSocket communication protocol to construct the long connection data transmission channel, and establishes a heartbeat service mechanism to detect the status of the long connection data transmission channel. Based on the data transmission request, it obtains the data to be reported, standardizes the data to be reported into structured messages, divides the data to be reported into Kafka message queues according to the data type, and based on the streaming computing engine Apache Flink, converts the static structured messages in the Kafka message queue into dynamic data streams to be reported, and transmits the data streams to be reported to the API service. It also collects the process data in the financial information reporting process and transmits it to the management end for visual display. Among them, the long connection data transmission channel includes an exclusive channel and a shared channel. The exclusive channel is used to transmit metadata, and the shared channel is used to transmit business data, task data, and process data.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 7.

Images