Public data processing method

By leveraging the collaborative work of the interface platform and the dual-check lock mechanism of the dynamic real-time loading module, the problem of government data being unable to be publicly used has been solved, achieving secure, available, and efficient data utilization.

CN119031037APending Publication Date: 2024-11-26HUIZHOU SMART BIG DATA CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410987829.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Because government-related data involves privacy and security issues, existing technologies cannot effectively disclose and utilize it, resulting in low data utilization rates.

Method used

Through the collaborative work of the business side, gateway side and data retrieval side of the interface platform, compliance verification, data anonymization and data integration of data usage requests are achieved. The dynamic real-time loading module and double-check lock mechanism ensure data security and stability, and ensure that data does not leave the domain during transmission and remains available.

Benefits of technology

It has achieved the security and availability of government data, ensuring that the data does not leave the domain and is available, thereby improving the utilization rate and value of the data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031037A_ABST
    Figure CN119031037A_ABST
Patent Text Reader

Abstract

The invention discloses a public data processing method. The method comprises the steps that S10, a service end of an interface platform receives a data use request; s20, the gateway end of the interface platform receives a data use request proposed by the service end; s30, the gateway end of the interface platform proposes a corresponding data use request to the data source end through the data access end; the data access end desensitizes the data provided by the data source end, and then sends the processed data to the gateway end; s40, the gateway end integrates the data sent by the access end and sends the integrated data to the service end; and S50, the service end outputs the received integrated data. The service end transmits a received request to the gateway end, the gateway end obtains corresponding data at the data source end through the data access end, the data access end desensitizes the data of the data source end and then outputs the data, it is ensured that the public data are not out of the domain and are available and invisible, the value of the public data is fully embodied and utilized, and the service life of the public data is prolonged. The problem of low utilization rate of traditional public data is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a public data processing method. Background Art

[0002] As big data matures, the means of data integration and acquisition increase. However, some government-related data are sensitive information due to privacy and security issues, and cannot be fully disclosed to the public. Once sensitive information is disclosed, it will lead to security issues such as data leakage. Therefore, the existing data cannot fully disclose sensitive and valuable data to external groups, resulting in low data utilization. Summary of the invention

[0003] In view of the deficiencies of the prior art, the present invention provides a public data processing method.

[0004] The present invention discloses a method for processing public data, comprising:

[0005] S10: The business end of the interface platform receives a data usage request;

[0006] S20, the gateway end of the interface platform receives the data use request proposed by the service end;

[0007] S30, the gateway end of the interface platform makes a corresponding data use request to the data source end through the data acquisition end; the data acquisition end performs desensitization processing on the data provided by the data source end, and then sends the processed data to the gateway end;

[0008] S40, the gateway integrates the data sent by the data acquisition end, and sends the integrated data to the business end;

[0009] S50: The business end outputs the received integrated data.

[0010] According to one embodiment of the present invention, S10 includes the following sub-steps:

[0011] S101. The verification module at the business end performs compliance verification on the information requested for data use.

[0012] According to one embodiment of the present invention, S10 includes the following sub-steps:

[0013] S102: The sampling inspection module at the business end randomly inspects the information of the data use request, and stops the data use request if it does not comply with the rules.

[0014] According to one embodiment of the present invention, S30 includes the following sub-steps:

[0015] S301, the data acquisition end background synchronously starts a daemon thread, and the daemon thread performs auxiliary work.

[0016] According to one embodiment of the present invention, it also includes:

[0017] S60. The data acquisition end is provided with a dynamic real-time loading module to realize hot update of the desensitizing service.

[0018] According to one embodiment of the present invention, S60 includes the following sub-steps:

[0019] S601, dynamic real-time loading module is realized through double-check lock mechanism.

[0020] According to one embodiment of the present invention, the double-check lock mechanism includes a desensitizing lock module and an update lock module. When an update task package is received, the desensitizing lock module is in a locked state and completes the currently unprocessed desensitizing task; after all desensitizing tasks are completed, the desensitizing lock is in an unlocked state, the update lock module is in a locked state, the update task enters and is executed, and new desensitizing tasks are prevented from entering.

[0021] According to an embodiment of the present invention, the gateway end and the data acquisition end are also connected via a monitoring interface, and the gateway end can capture the operation status information for monitoring.

[0022] According to an embodiment of the present invention, the auxiliary work is garbage collection or memory management.

[0023] According to an embodiment of the present invention, the gateway transmits a rule package to the dynamic real-time loading module through an interface corresponding to the dynamic real-time loading module to adjust and configure relevant service parameters.

[0024] The beneficial effects of the present invention are as follows: through the cooperation of the data acquisition end, the gateway end and the business end of the interface platform, the business end transmits the received request to the gateway end, the gateway end obtains the corresponding data at the data source end through the data acquisition end, and the data acquisition end outputs the data of the data source end after desensitizing processing, ensuring that the public data "does not leave the domain" and "is available but not visible", so that the value of public data can be fully reflected and utilized, solving the problem of low utilization rate of traditional public data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 A flowchart of a public data processing method;

[0027] Figure 2 A logical framework diagram of the public data processing method;

[0028] Figure 3Another logical framework diagram of a public data processing method. DETAILED DESCRIPTION

[0029] The following will disclose multiple embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, for the purpose of simplifying the drawings, some conventional structures and components will be depicted in a simple schematic manner in the drawings.

[0030] In addition, in the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes, and do not specifically refer to the order or sequence, nor are they used to limit the present invention. They are only used to distinguish components or operations described with the same technical terms, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0031] The application scenario of the public data processing method of this application is: there is currently a corresponding government sharing platform, which contains personal social security, real estate and other related information, and the government sharing platform cooperates with big data companies or publicly processes the information alone. The processing method of this application is built on an interface platform, and the interface platform includes a business end, a gateway end, and a data acquisition end. The interface platform is used to connect upstream and downstream. The upstream is the government sharing platform or big data company, and the downstream is the data user, such as major banks. That is, after the data user makes a request to the interface platform, the interface platform makes a request to the upstream, and processes the requested data and outputs it to the data user.

[0032] like Figure 1-Figure 3 As shown, Figure 1 A flowchart of a public data processing method; Figure 2 A logical framework diagram of the public data processing method; Figure 3 Another logical framework diagram of a public data processing method.

[0033] The public data processing method of the present application comprises the following steps:

[0034] S10: The data user submits a data use request to the interface platform, and the data use request is received by the service end. Specifically, the service end provides a service interface to the municipal bureau user so that the data user can submit a request.

[0035] Furthermore, S10 includes the following sub-steps: When the data user makes a request, it is necessary to upload the corresponding information, such as the signed authorization document, to the business port to ensure the compliance of the behavior. In addition, the business end also performs authority identification, whitelisting, communication link encryption, data encryption, etc. on the data user.

[0036] Furthermore, S10 also includes the following sub-steps: the business end randomly checks the data usage request information uploaded by the data user, that is, checks the integrity, authenticity and compliance of the corresponding information. If there is any non-compliance with the requirements, the business interface will be stopped and the data user will not be able to continue to make data usage requests; after the corresponding information is supplemented and improved, the corresponding business interface will be opened.

[0037] S20: The gateway receives the data usage request transmitted by the service end;

[0038] S30: The gateway transmits the received data usage request to the data source through the data acquisition end. Specifically, the gateway sends requests to different data sources according to the data usage requests raised by different products. The data acquisition end desensitizes the data provided by the data source and transmits the desensitized data to the gateway.

[0039] S40: After receiving the data sent by the data acquisition end, the gateway integrates the data and sends the integrated data to the business end. Specifically, since the data processing at the data acquisition end belongs to the mode of targeted single-field processing one by one, it mainly highlights the refined operation, and the gateway end is responsible for the data integration process, and finally encapsulates and outputs different processing results.

[0040] S50: The business end receives the integrated data and sends it to the data user.

[0041] In specific application, S30 also includes the following sub-steps: when the data acquisition end performs desensitization processing, the background synchronously starts the daemon thread, wherein the daemon thread only runs in the background, that is, it will not participate in the routine operation, to maintain the normal operation of the service and ensure the continuity of the system. Further explanation, the daemon thread is a thread that provides services in the background when the program is running. Compared with ordinary threads, the life cycle of the daemon thread is interrelated with the life cycle of the program, that is, when all non-daemon threads end, the daemon thread will also end. In this embodiment, the daemon thread is usually used to perform some auxiliary tasks, such as garbage collection, memory management, etc. In addition, the daemon thread runs in the background and will not block the exit of the program. At the same time, when all non-daemon threads end, the daemon thread will automatically exit, regardless of whether it has completed the task.

[0042] The public data processing method of this application also includes:

[0043] S60: A dynamic real-time loading module is set in the data acquisition end. The desensitization matching rules and desensitization services can be dynamically configured through the dynamic real-time loading module, thereby realizing hot-reload of the desensitization service and ensuring the stability of system operation.

[0044] S601: The dynamic real-time loading module realizes its function through a double-check lock mechanism, which is used to ensure that the update thread and the working thread will not conflict with each other, reduce the overhead of lock competition, and improve performance. Specifically, the double-check lock mechanism includes a desensitizing lock module and an update lock module. When receiving an update task package, the desensitizing task being processed is first locked. At this time, the desensitizing lock module is in a locked state. Since the desensitizing lock module is in a locked state, the execution of the update task will be delayed. Synchronously, the update lock module is in a locked state, which will prevent new desensitizing tasks from entering. After all the desensitizing tasks are processed, the desensitizing lock module is in an unlocked state, and the update task enters and starts to execute. In this embodiment, for the inspection order of the desensitizing lock module and the update lock module, the priority of the update lock module is higher than the priority of the desensitizing lock, that is, the state of the update lock module is checked first, and when the state of the update lock module is in a locked state, the reception of the new desensitizing task is suspended. Through the setting of the double-check lock mechanism, the orderly progress of the update process is guaranteed, and the data inconsistency or system instability caused by concurrent conflicts is avoided, and the effective management and utilization of resources are realized.

[0045] Furthermore, the gateway is provided with an interface corresponding to the dynamic real-time loading module, and through this interface, a rule package is sent to the data acquisition end, such as uploading a hot update rule package, modifying the binding rules of sensitive fields of each product, etc., so as to flexibly adjust and configure relevant service parameters, so that the gateway can dynamically adjust the configuration of the data acquisition end, and assist the desensitization rule configuration personnel to perform hot updates. In addition, the gateway also has the function of monitoring the operation of the data acquisition end, specifically: the gateway and the data acquisition end are also connected through a monitoring interface, through which the gateway can capture and record the key operating status information of the data acquisition end, and can also automatically detect and record abnormal alarms and system logs during operation.

[0046] To summarize, through the cooperation of the data acquisition end, gateway end and business end of the interface platform, the business end transmits the received request to the gateway end, the gateway end obtains the corresponding data at the data source end through the data acquisition end, and the data acquisition end outputs the data from the data source end after desensitizing, ensuring that public data "does not leave the domain" and "is available but not visible", so that the value of public data can be fully reflected and utilized, solving the problem of low utilization rate of traditional public data.

[0047] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A public data processing method, characterized in that: include: S10: The business end of the interface platform receives a data usage request; S20, the gateway end of the interface platform receives the data use request proposed by the service end; S30, the gateway end of the interface platform makes a corresponding data use request to the data source end through the data acquisition end; The data receiving end performs desensitization processing on the data provided by the data source end, and then sends the processed data to the gateway end; S40, the gateway integrates the data sent by the data acquisition end, and sends the integrated data to the business end; S50: The business end outputs the received integrated data.

2. The public data processing method according to claim 1, characterized in that: S10 includes the following sub-steps: S101. The verification module at the business end performs compliance verification on the information requested for data use.

3. The public data processing method according to claim 2, characterized in that: S10 includes the following sub-steps: S102: The sampling inspection module at the business end randomly inspects the information of the data use request, and stops the data use request if it does not comply with the rules.

4. The public data processing method according to claim 1, characterized in that: S30 includes the following sub-steps: S301, the data acquisition end background synchronously starts a daemon thread, and the daemon thread performs auxiliary work.

5. The public data processing method according to claim 1, characterized in that: Also includes: S60. The data acquisition end is provided with a dynamic real-time loading module to realize hot update of the desensitizing service.

6. The public data processing method according to claim 5, characterized in that: S60 includes the following sub-steps: S601, dynamic real-time loading module is realized through double-check lock mechanism.

7. The public data processing method according to claim 6, characterized in that: The double-check lock mechanism includes a desensitization lock module and an update lock module. When receiving an update task package, the desensitization lock module is in a locked state and completes the currently unprocessed desensitization task; After all desensitization tasks are completed, the desensitization lock is in an unlocked state, the update lock module is in a locked state, the update task enters and is executed, and new desensitization tasks are blocked from entering.

8. The public data processing method according to claim 5, characterized in that: The gateway end and the data acquisition end are also connected through a monitoring interface, and the gateway end can capture the operation status information for monitoring.

9. The public data processing method according to claim 4, characterized in that: Auxiliary work is garbage collection or memory management.

10. The public data processing method according to claim 5, characterized in that: The gateway transmits the rule package to the dynamic real-time loading module through the interface corresponding to the dynamic real-time loading module to adjust and configure relevant service parameters.