Key service method, device, system and equipment for extracting data in real time and medium

Abstract

The invention relates to the technical field of data processing, can be applied to the fields of finance, medical treatment and the like, and discloses a key service method, device, system and equipment for extracting data in real time and a medium. The real-time extraction task end extracts data in real time, and when the data reaches a preset data volume, the data of the preset data volume is sent to the unified processing server end; the unified processing server calls the key management server in response to passively receiving the preset data volume of data; the key management server side performs key service on the data with the preset data volume and sends each piece of processed data to the unified processing server side in real time; the unified processing server responds to passively received each piece of processed data and sends each piece of processed data to a real-time extraction task end in real time; and the real-time extraction task end responds to passively received each piece of processed data and stores each piece of processed data into the data lake in real time. The resource consumption is reduced, and the data key service efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and can be applied to fields such as finance and healthcare. In particular, it relates to a key service method, device, system, equipment and medium for real-time extraction of data. Background Art

[0002] In the field of big data, after extracting multiple data sources into a data warehouse, the data is processed and then applied to various scenarios such as recommendation and reports to realize the value of the data. Based on the protection of data information, for sensitive information involved in the data, key services are usually required.

[0003] Generally, on the big data side, a third-party key management service is called based on a unified processing service to perform key services for the data. The resources of the third-party key management service are limited, and generally there is interface flow control to ensure the stability of the key service. Interface flow control will affect the efficiency of the data key service, and then cause the real-time extraction task to freeze and wait, affecting the efficiency of the real-time extraction task. If the resources of the third-party key management service are increased, it will lead to a large increase in costs.

[0004] Therefore, there is an urgent need to propose a key service method for real-time extraction of data to prevent the real-time extraction task from freezing and waiting and improve the efficiency of the data key service. Summary of the Invention

[0005] The present invention provides a key service method, device, system, equipment and medium for real-time extraction of data. When the data reaches a preset data volume, the unified processing server calls the key management server to perform key services, so that the real-time extracted data does not need to freeze and wait during the stage of accumulating the preset data volume; each piece of data after the service is completed is directly sent to the data lake in real time without waiting for all the data of the preset data volume to be serviced, improving the efficiency of the data key service.

[0006] In a first aspect, a key service method for real-time extraction of data is provided. The key service method is applied to a unified processing server, and the unified processing server communicates with a real-time extraction task server and a key management server respectively. The key service method includes:

[0007] In response to passively receiving data of a preset data volume sent by the real-time extraction task server, call the key management server to perform key services on the data of the preset data volume; wherein, the real-time extraction task server extracts data in real time;

[0008] In response to passively receiving each piece of data that has been serviced in real time sent by the key management server, send each piece of data that has been serviced in real time to the real-time extraction task server, so that the real-time extraction task server stores each piece of data that has been serviced in real time in the data lake.

[0009] Second aspect, a key service method for real-time data extraction is provided. The key service method is applied to the real-time extraction task side. The real-time extraction task side communicates with the unified processing server, and the unified processing server communicates with the key management server. The key service method includes:

[0010] Extract data in real time;

[0011] When the data reaches the preset data volume, send the data of the preset data volume to the unified processing server, so that the unified processing server calls the key management server to perform key services on the data of the preset data volume; wherein, the key management server sends each piece of data that has completed the service to the unified processing server in real time;

[0012] In response to passively receiving each piece of data that has completed the service sent by the unified processing server in real time, store each piece of data that has completed the service in the data lake in real time.

[0013] Third aspect, a key service device for real-time data extraction is provided. The key service device is deployed on the unified processing server. The unified processing server communicates with the real-time extraction task side and the key management server respectively. The key service device includes:

[0014] A calling unit, configured to call the key management server to perform key services on the data of the preset data volume in response to passively receiving the data of the preset data volume sent by the real-time extraction task side; wherein, the real-time extraction task side extracts data in real time;

[0015] An event stream unit, configured to send each piece of data that has completed the service to the real-time extraction task side in real time in response to passively receiving each piece of data that has completed the service sent by the key management server in real time, so that the real-time extraction task side stores each piece of data that has completed the service in the data lake in real time.

[0016] Fourth aspect, a key service device for real-time data extraction is provided. The key service device is deployed on the real-time extraction task side. The real-time extraction task side communicates with the unified processing server, and the unified processing server communicates with the key management server. The key service device includes:

[0017] A data extraction unit, configured to extract data in real time;

[0018] A request unit, configured to send the data of the preset data volume to the unified processing server when the data reaches the preset data volume, so that the unified processing server calls the key management server to perform key services on the data of the preset data volume; wherein, the key management server sends each piece of data that has completed the service to the unified processing server in real time;

[0019] A lake entry unit, configured to store each piece of data that has completed the service in the data lake in real time in response to passively receiving each piece of data that has completed the service sent by the unified processing server in real time.

[0020] In a fifth aspect, a key service system for real-time data extraction is provided, including: a real-time extraction task end, a unified processing server, and a key management server;

[0021] The real-time extraction task end is used to extract data in real time. When the data reaches a preset data volume, it sends data of the preset data volume to the unified processing server;

[0022] The unified processing server is used to call the key management server in response to passively receiving data of the preset data volume sent by the real-time extraction task end;

[0023] The key management server is used to perform key services on the data of the preset data volume and send each piece of processed data to the unified processing server in real time;

[0024] The unified processing server is used to send each piece of processed data to the real-time extraction task end in real time in response to passively receiving each piece of processed data sent by the key management server in real time;

[0025] The real-time extraction task end is used to store each piece of processed data in the data lake in real time in response to passively receiving each piece of processed data sent by the unified processing server in real time.

[0026] In a sixth aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned key service method for real-time data extraction are implemented.

[0027] In a seventh aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned key service method for real-time data extraction are implemented.

[0028] In the solutions implemented by the above-mentioned key service method, device, system, equipment, and medium for real-time data extraction, the real-time extraction task end extracts data in real time. When the data reaches a preset data volume, it sends data of the preset data volume to the unified processing server; the unified processing server calls the key management server in response to passively receiving data of the preset data volume sent by the real-time extraction task end; the key management server performs key services on the data of the preset data volume and sends each piece of processed data to the unified processing server in real time; the unified processing server sends each piece of processed data to the real-time extraction task end in real time in response to passively receiving each piece of processed data sent by the key management server in real time; the real-time extraction task end stores each piece of processed data in the data lake in real time in response to passively receiving each piece of processed data sent by the unified processing server in real time.

[0029] In the present invention, during the stage when the real-time extraction task end accumulates a batch of data, the real-time extraction task end can normally and real-time extract data without jamming and waiting; a connection request is created for one batch of data, reducing the resource consumption for creating connections; after a connection request is created for one batch of data, the unified processing server uniformly calls the key management server, reducing the concurrency of call requests, preventing the triggering of interface flow control, and ensuring the stability and real-time nature of the key service. It is not necessary to wait until all services for one batch of data are completed before it can be sent to the lake. Instead, each piece of data that has completed service is received in real-time and sent to the lake, reducing the waiting time for sending to the lake and improving the efficiency of the key service. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1 is a schematic working flowchart of a key service system for real-time extraction of data in the prior art;

[0032] Figure 2 is a schematic application environment diagram of a method for a key service for real-time extraction of data in an embodiment of the present invention;

[0033] Figure 3 is a schematic flowchart of a method for a key service for real-time extraction of data in an embodiment of the present invention;

[0034] Figure 4 is a schematic flowchart of a method for a key service for real-time extraction of data in another embodiment of the present invention;

[0035] Figure 5 is a schematic structural diagram of a key service device for real-time extraction of data in an embodiment of the present invention;

[0036] Figure 6 is a schematic structural diagram of a key service device for real-time extraction of data in another embodiment of the present invention;

[0037] Figure 7 is a schematic working flowchart of a key service system for real-time extraction of data in an embodiment of the present invention;

[0038] Figure 8 is a schematic structural diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] The working process of the key service method for real-time data extraction in the prior art will be introduced. Refer to Figure 1 as shown Figure 1 which shows the schematic diagram of the working process of the key service system for real-time data extraction in the prior art. The unified processing server communicates with the real-time extraction task server and the key management service server respectively.

[0041] The real-time extraction task server extracts data from the data source in real time.

[0042] For each piece of data extracted by the real-time extraction task server, the piece of data is sent to the unified processing server so that the unified processing server can call the key management service server.

[0043] After the key management service server performs key services on the piece of data, the piece of data after the service is sent to the unified processing server, and then the unified processing server sends the piece of data after the service to the real-time extraction task server.

[0044] The real-time extraction task server receives the piece of data after the service and stores it in the data lake.

[0045] The working process of the above prior art has the following drawbacks:

[0046] For each piece of data extracted by the real-time extraction task server, a connection request is created to the unified processing server, resulting in high resource consumption.

[0047] Real-time data extraction has high requirements for data real-time. When there are a large number of concurrent requests for key services, it will increase the interface pressure of the key management service server, trigger the interface rate limit of the key management service server, and lead to slow key services and reduced performance, etc.

[0048] Once the key management service server's interface is rate-limited and cannot provide key services efficiently, it will affect the efficiency of the data after the service entering the lake.

[0049] In order to improve the efficiency of the entire link. The embodiments of the present invention provide a key service method for real-time data extraction. This key service method can be applied in Figure 2In the application environment shown, the real-time extraction task end communicates with the unified processing server end, and the unified processing server end communicates with the key management server end. Since the key management server end is a third party accessed, the key service method for real-time extraction data provided by the embodiments of the present invention focuses on improving the unified processing server end and the real-time extraction task end.

[0050] The present invention will be described in detail below through specific embodiments.

[0051] Please refer to Figure 3 shown Figure 3 FIG. is a schematic flowchart of a key service method for real-time extraction data provided by an embodiment of the present invention. The key service method is applied to the unified processing server end, and the unified processing server end communicates with the real-time extraction task end and the key management server end respectively. The key service method includes the following steps:

[0052] Step S310: In response to passively receiving data of a preset data volume sent by the real-time extraction task end, call the key management server end to perform key service on the data of the preset data volume; wherein, the real-time extraction task end extracts data in real time.

[0053] The real-time extraction task end extracts data from the data source in real time for caching, and continuously records the quantity of the real-time extraction data during the caching process. If the quantity does not reach the preset data volume, the real-time extraction task end continues to cache. If the quantity reaches the preset data volume, the real-time extraction task end sends the data reaching the preset data volume as a batch of data to the unified processing server end.

[0054] The preset data volume can be a value flexibly set according to the key service performance of the key management server end. For example: the preset data volume can be 1000 pieces, 10000 pieces, etc.

[0055] After the real-time extraction task end sends a batch of data, the real-time extraction task end continues to start the extraction and caching of the next batch of data. In the prior art, when the real-time extraction task end extracts one piece of data, it creates a connection request to the unified processing server end. If the key management server end interface limits the flow rate and the key management server end cannot provide key service in time, it will affect the real-time extraction task end to extract the next piece of data, that is, cause the real-time extraction task end to freeze and wait. Through the improvement of the embodiments of the present application, during the stage when the real-time extraction task end accumulates a batch of data, the real-time extraction task end can extract data in real time without freezing and without waiting.

[0056] After passively receiving a batch of data sent by the real-time extraction task end through reactive programming, the unified processing server calls the key management server. In the prior art, every time the real-time extraction task end extracts a piece of data, it creates a connection request to the unified processing server. A connection request is created for each piece of data, resulting in high resource consumption. Through the improvement of the embodiments of the present application, a connection request is created for a batch of data, reducing the resource consumption of creating connections.

[0057] After being called, the key management server provides key services for a batch of data. In the prior art, after each piece of data creates a connection request, the unified processing server calls the key management server. Once the concurrent call requests of the unified processing server are excessive, it will trigger interface rate limiting of the key management server, resulting in a decline in the performance of key services. Through the improvement of the embodiments of the present application, after a connection request is created for a batch of data, the unified processing server uniformly calls the key management server, reducing the concurrency of call requests, preventing the trigger of interface rate limiting, and ensuring the stability and real-time nature of key services.

[0058] Step S320: In response to passively receiving each piece of data with the service completed sent by the key management server in real time, send each piece of data with the service completed to the real-time extraction task end in real time, so that the real-time extraction task end stores each piece of data with the service completed in the data lake in real time.

[0059] After the key management server completes the key service for each piece of data, it actively sends the piece of data with the service completed to the unified processing server in a timely manner. After the unified processing server passively receives the piece of data with the service completed, it actively sends it to the real-time extraction task end in a timely manner. The unified processing server can use the SSE (Server-Sent Events) technology to realize the active push of each piece of data with the service completed from the unified processing server to the real-time extraction task end.

[0060] After the real-time extraction task end passively receives the piece of data with the service completed, it stores it in the data lake in a timely manner. That is to say, the real-time extraction task end sends a batch of data to the unified processing server to call the key management server for key services, but receives each piece of data with the service completed in real time and stores it in the lake. Through the improvement of the embodiments of the present application, there is no need to wait until all the data in a batch is completely serviced before it can be stored in the lake. Instead, each piece of data with the service completed is received and stored in the lake in real time, reducing the waiting time for storage in the lake and improving the efficiency of key services.

[0061] In some alternative embodiments, the key management server includes multiple key management processors, and each key management processor communicates with the unified processing server. Calling the key management server to perform key services on a preset amount of data includes: circularly calling each key management processor to perform key services on each piece of data.

[0062] The key management server can provide key services through multiple key management processors. Since the resources of each key management processor are limited, key services for each piece of data can be performed by cyclically invoking each key management processor.

[0063] For example: if there are 5 key management processors. Then, the first 5 pieces of data in a batch of data can be successively invoked to 5 key management processors for key services, and the next 5 pieces of data in this batch of data can then be successively invoked to 5 key management processors for key services. This cycle continues until all the data in this batch of data has been served.

[0064] Another example: still with 5 key management processors. Then, the first 5 pieces of data in a batch of data can be successively invoked to 5 key management processors for key services, and the subsequent data in this batch of data is always successively invoked to the key management processor that becomes idle first. This cycle continues until all the data in this batch of data has been served.

[0065] In some alternative embodiments, the key management server includes multiple key management processors, and each key management processor communicates with a unified processing server. Invoking the key management server to perform key services on a preset quantity of data includes: parsing the key management service identification information of each piece of data, determining the corresponding key management processor for each piece of data according to the key management service identification information, and invoking the corresponding key management processor to perform key services on each piece of data.

[0066] The key management server can provide key services through multiple key management processors, and each key management processor can provide different types of key services. Therefore, key services for each piece of data can be performed by invoking the corresponding key management processor.

[0067] The unified processing server can parse the key management service identification information of each piece of data. For example, the attribute of each piece of data can contain the key management service identification information, and the unified processing server parses the attribute of each piece of data to determine the key management service identification information.

[0068] The key management service identification information can be identification information that clearly forms a corresponding relationship with the key management processor. For example, the key management service identification information clearly corresponds to the key management processor. The key management service identification information can also be identification information that indirectly forms a corresponding relationship with the key management processor. For example, the key management service identification information can be the key service type, and the key management processor that can provide key services can be corresponded through the key service type.

[0069] After the unified processing server determines the key management processor corresponding to each piece of data based on the key management service identification information, it can call the corresponding key management processor to perform key services on each piece of data.

[0070] For example: There are 5 key management processors, denoted as processor A, processor B, processor C, processor D, and processor E. If the first piece of data in a batch of data corresponds to processor A, then call processor A to perform key services on the first piece of data; if the second piece of data in this batch of data corresponds to processor D, then call processor D to perform key services on the second piece of data. This continues until all the data in this batch of data has been served.

[0071] In some alternative embodiments, the key service includes: encryption service or decryption service.

[0072] Taking the application in the financial scenario for encryption service as an example for specific illustration. For example, for a fund product, there is a large amount of transaction data stored in the data source. It is necessary to form a fund transaction dashboard based on this transaction data, but the personal information of the trading users needs to be encrypted.

[0073] The real-time extraction task end extracts transaction data from the data source in real time. When the number of transaction data reaches 1000, the real-time extraction task end sends these 1000 transaction data to the unified processing server.

[0074] In response to passively receiving these 1000 transaction data, the unified processing server cyclically calls multiple key management processors to encrypt these 1000 transaction data (especially the personal information of the trading users).

[0075] After each key management processor encrypts a piece of transaction data, it promptly sends the encrypted transaction data to the unified processing server; in response to passively receiving the encrypted transaction data, the unified processing server promptly sends the encrypted transaction data to the real-time extraction task end; in response to passively receiving the encrypted transaction data, the real-time extraction task end stores the encrypted transaction data in the data lake for use in forming the fund transaction dashboard.

[0076] Taking the application in the medical scenario for decryption service as an example for specific illustration. For example, for a certain disease, there is a large amount of treatment data stored in the data source (the relevant information of the patients has been encrypted). It is necessary to form a disease progression statistical model based on this treatment data, but the relevant information of the patients needs to be decrypted.

[0077] The real-time extraction task end extracts treatment data from the data source in real time. When the number of treatment data reaches 10000, the real-time extraction task end sends these 10000 treatment data to the unified processing server.

[0078] The unified processing server, in response to passively receiving the 10,000 pieces of treatment data, parses the encryption method of each piece of treatment data, determines the corresponding key management processor for each piece of treatment data according to the encryption method, and calls the corresponding key management processor to decrypt each piece of treatment data (especially the relevant information of the patient).

[0079] After each key management processor decrypts a piece of treatment data, it promptly sends the decrypted piece of treatment data to the unified processing server; the unified processing server, in response to passively receiving the decrypted piece of treatment data, promptly sends the decrypted piece of treatment data to the real-time extraction task end; the real-time extraction task end, in response to passively receiving the decrypted piece of treatment data, stores the decrypted piece of treatment data in the data lake for use in forming a disease course progress statistical model.

[0080] Please refer to Figure 4 as shown Figure 4 Fig. is another flow schematic diagram of the key service method for real-time extraction of data provided by an embodiment of the present invention. The key service method is applied to the real-time extraction task end. The real-time extraction task end communicates with the unified processing server, and the unified processing server communicates with the key management server. The key service method includes the following steps:

[0081] Step S410, real-time extraction of data;

[0082] Step S420, when the data reaches a preset data volume, send data of the preset data volume to the unified processing server so that the unified processing server calls the key management server to perform key services on the data of the preset data volume; wherein, the key management server sends each piece of data with completed services to the unified processing server in real time;

[0083] Step S430, in response to passively receiving each piece of data with completed services sent by the unified processing server in real time, store each piece of data with completed services in the data lake in real time.

[0084] For the specific limitations of the key service method applied to the real-time extraction task end in this embodiment, reference can be made to the limitations of the key service method applied to the unified processing server in the foregoing text, which will not be elaborated here.

[0085] In one embodiment, a key service device for real-time extraction of data is provided. The key service device is deployed on the unified processing server, and the unified processing server communicates with the real-time extraction task end and the key management server respectively. The key service device for real-time extraction of data corresponds one-to-one with the key service method applied to the unified processing server in the above embodiment. As Figure 5 shown, the key service device 500 includes:

[0086] A calling unit 501, configured to call a key management server to perform a key service on a preset amount of data in response to passively receiving the preset amount of data sent by a real-time extraction task end; wherein, the real-time extraction task end extracts data in real time.

[0087] An event stream unit 502, configured to, in response to passively receiving each piece of data that has completed the service sent by the key management server in real time, send each piece of data that has completed the service to the real-time extraction task end in real time, so that the real-time extraction task end stores each piece of data that has completed the service in the data lake in real time.

[0088] In some alternative embodiments, the key management server includes multiple key management processors, and each key management processor communicates with a unified processing server; the calling unit 501 is specifically configured to: circularly call each key management processor to perform a key service on each piece of data.

[0089] In some alternative embodiments, the key management server includes multiple key management processors, and each key management processor communicates with a unified processing server; the calling unit 501 is specifically configured to: parse the key management service identification information of each piece of data, determine the corresponding key management processor for each piece of data according to the key management service identification information, and call the corresponding key management processor to perform a key service on each piece of data.

[0090] In some alternative embodiments, the key service includes: an encryption service or a decryption service.

[0091] In one embodiment, a key service device for real-time data extraction is provided. The key service device is deployed at the real-time extraction task end. The real-time extraction task end communicates with a unified processing server, and the unified processing server communicates with a key management server. The key service device for real-time data extraction corresponds one-to-one with the key service method applied to the real-time extraction task end in the above embodiment. As Figure 6 shown, the key service device 600 includes:

[0092] A data extraction unit 601, configured to extract data in real time;

[0093] A request unit 602, configured to send a preset amount of data to the unified processing server when the data reaches the preset amount, so that the unified processing server calls the key management server to perform a key service on the preset amount of data; wherein, the key management server sends each piece of data that has completed the service to the unified processing server in real time.

[0094] A lake entry unit 603, configured to, in response to passively receiving each piece of data that has completed the service sent by the unified processing server in real time, store each piece of data that has completed the service in the data lake in real time.

[0095] For the specific limitations of the key service device for real-time data extraction, reference can be made to the limitations of the key service method for real-time data extraction in the foregoing text, which will not be elaborated here. Each module in the above key service device for real-time data extraction can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0096] In one embodiment, a key service system for real-time data extraction is provided. Please refer to Figure 7 as shown in Figure 7 FIG. [Figure number not provided in the original] is a schematic diagram of the working process of the key service system for real-time data extraction provided by an embodiment of the present invention. The key service system includes: a real-time extraction task end, a unified processing server, and a key management server;

[0097] The real-time extraction task end is used to extract data in real time. When the data reaches a preset data volume, it sends data of the preset data volume to the unified processing server;

[0098] The unified processing server is used to call the key management server in response to passively receiving data of the preset data volume sent by the real-time extraction task end;

[0099] The key management server is used to perform key services on the data of the preset data volume and send each piece of processed data to the unified processing server in real time;

[0100] The unified processing server is used to send each piece of processed data to the real-time extraction task end in real time in response to passively receiving each piece of processed data sent by the key management server in real time;

[0101] The real-time extraction task end is used to store each piece of processed data in the data lake in real time in response to passively receiving each piece of processed data sent by the unified processing server in real time.

[0102] In one embodiment, a computer device is provided. The internal structure diagram of the computer device can be as Figure 8As shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the key service method for real-time extraction of data applied to the unified processing server or the key service method for real-time extraction of data applied to the real-time extraction task end.

[0103] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0104] In response to passively receiving data of a preset data volume sent by the real-time extraction task end, call the key management server to perform key service on the data of the preset data volume; wherein, the real-time extraction task end extracts data in real time;

[0105] In response to passively receiving each piece of data whose service is completed and sent by the key management server in real time, send each piece of data whose service is completed to the real-time extraction task end in real time, so that the real-time extraction task end stores each piece of data whose service is completed in the data lake in real time.

[0106] Or,

[0107] Extract data in real time;

[0108] When the data reaches the preset data volume, send the data of the preset data volume to the unified processing server, so that the unified processing server calls the key management server to perform key service on the data of the preset data volume; wherein, the key management server sends each piece of data whose service is completed to the unified processing server in real time;

[0109] In response to passively receiving each piece of data whose service is completed and sent by the unified processing server in real time, store each piece of data whose service is completed in the data lake in real time.

[0110] It should be noted that the functions or steps that can be realized by the above computer-readable storage medium or computer device can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0111] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A key service method for real-time data extraction, characterized in that, The described key service method is applied to a unified processing server, which communicates with a real-time extraction task server and a key management server respectively. The key service method includes: In response to passively receiving data of a preset data volume sent by the real-time extraction task server, call the key management server to perform key services on the data of the preset data volume; wherein, the real-time extraction task server extracts data in real time; In response to passively receiving each piece of data with the service completed sent by the key management server in real time, send each piece of data with the service completed to the real-time extraction task server in real time, so that the real-time extraction task server stores each piece of data with the service completed in the data lake in real time.

2. The key service method for real-time data extraction according to claim 1, wherein The key management server includes multiple key management processors, and each key management processor communicates with the unified processing server; The step of calling the key management server to perform key services on the data of the preset data volume includes: Circularly call each key management processor to perform key services on each piece of data.

3. The key service method for real-time data extraction according to claim 1, characterized in that The key management server includes multiple key management processors, and each key management processor communicates with the unified processing server; The step of calling the key management server to perform key services on the data of the preset data volume includes: Parse the key management service identification information of each piece of data, determine the corresponding key management processor for each piece of data according to the key management service identification information, and call the corresponding key management processor to perform key services on each piece of data.

4. The key service method for real-time extraction of data according to any one of claims 1 to 3, characterized in that, The key service includes: encryption service or decryption service.

5. A key service method for real-time data extraction, characterized in that, The described key service method is applied to a real-time extraction task server, which communicates with the unified processing server, and the unified processing server communicates with the key management server. The key service method includes: Extract data in real time; When the data reaches the preset data volume, send the data of the preset data volume to the unified processing server, so that the unified processing server calls the key management server to perform key services on the data of the preset data volume; wherein, the key management server sends each piece of data with the service completed to the unified processing server in real time; In response to passively receiving each piece of data with the service completed sent by the unified processing server in real time, store each piece of data with the service completed in the data lake in real time.

6. A key service device for real-time data extraction, characterized in that, The key service device is deployed on the unified processing server, which communicates with the real-time extraction task server and the key management server respectively. The key service device includes: A calling unit, configured to call the key management server to perform key services on the data of the preset data volume in response to passively receiving the data of the preset data volume sent by the real-time extraction task server; wherein, the real-time extraction task server extracts data in real time; An event stream unit, configured to send each piece of data with the service completed to the real-time extraction task server in real time in response to passively receiving each piece of data with the service completed sent by the key management server in real time, so that the real-time extraction task server stores each piece of data with the service completed in the data lake in real time.

7. A key service device for real-time data extraction, characterized in that, The key service device is deployed at the real-time extraction task end. The real-time extraction task end communicates with the unified processing service end, and the unified processing service end communicates with the key management service end. The key service device includes: A data extraction unit for extracting data in real time; A request unit for sending data of a preset data volume to the unified processing service end when the data reaches the preset data volume, so that the unified processing service end invokes the key management service end to perform key services on the data of the preset data volume. Wherein, the key management service end sends each piece of data with completed service to the unified processing service end in real time; A data lake input unit for, in response to passively receiving each piece of data with completed service sent by the unified processing service end in real time, storing each piece of data with completed service into the data lake in real time.

8. A key service system for real-time data extraction, characterized in that, It includes: A real-time extraction task end, a unified processing service end and a key management service end; The real-time extraction task end is used for extracting data in real time and sending data of a preset data volume to the unified processing service end when the data reaches the preset data volume; The unified processing service end is used for, in response to passively receiving the data of the preset data volume sent by the real-time extraction task end, invoking the key management service end; The key management service end is used for performing key services on the data of the preset data volume and sending each piece of processed data to the unified processing service end in real time; The unified processing service end is used for, in response to passively receiving each piece of processed data sent by the key management service end in real time, sending each piece of processed data to the real-time extraction task end in real time; The real-time extraction task end is used for, in response to passively receiving each piece of processed data sent by the unified processing service end in real time, storing each piece of processed data into the data lake in real time.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the key service method for real-time data extraction according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the key service method for real-time data extraction according to any one of claims 1 to 5 are implemented.

Images