Data distribution method, device and equipment based on distributed server

Abstract

An embodiment of the present disclosure provides a data distribution method based on a distributed server, which can be applied to the fields of computer technology and big data technology. The method includes: obtaining original data, performing data standardization on the original data, and generating a data base table. Obtaining m user identifiers, where m is greater than or equal to 2 and m is an integer. Presetting the access permission information of the m user identifiers to the data in the data base table to generate an access permission table. The data base table and the access permission table are associated to generate a data management view table. According to the data management view table, target data is obtained from the data base table, and data distribution is performed on the target data. The present disclosure also provides a data distribution device, computing device, medium, and program product based on a distributed server.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of big data technology, and in particular to a data distribution method, apparatus, device, medium, and program product based on a distributed server. Background Art

[0002] With the development of internet computing technology, the demand for data distribution is increasing among businesses, including banks. The current technical solution involves using SQL statements in Flink to define queries and operations on Kafka tables, then incorporating access control logic into these SQL statements to achieve data distribution. However, this approach requires writing the same or similar access control logic in every SQL statement, resulting in redundant, complex, and difficult-to-maintain code. Furthermore, due to the lack of a unified and standardized access control mechanism, each SQL statement requires its own access control logic, increasing development and maintenance costs and difficulties. This approach, which lacks flexibility and scalability in access control data distribution, can only determine whether certain data can be distributed based on fixed conditions, and cannot dynamically adjust access control rules and policies to suit different scenarios and needs. The lack of efficient and easy-to-use access control tools requires writing complex and tedious SQL statements to implement access control, hindering efficient and accurate data distribution. Summary of the Invention

[0003] In view of the above problems, the present disclosure provides a data distribution method, apparatus, device, medium and program product based on a distributed server.

[0004] According to the first aspect of the present disclosure, a data distribution method based on a distributed server is provided, characterized in that the method includes: obtaining original data, performing data standardization on the original data, and generating a data base table; obtaining m user identifiers, where m is greater than or equal to 2 and m is an integer; presetting access permission information of the m user identifiers to data in the data base table to generate an access permission table; associating the data base table and the access permission table to generate a data management view table; obtaining target data from the data base table according to the data management view table, and distributing the target data.

[0005] According to an embodiment of the present disclosure, it is characterized in that the original data is standardized to generate a data basic table, including: obtaining the server address and port number of the distributed streaming processing platform; connecting to the distributed streaming processing platform based on the server address and port number of the distributed streaming processing platform; and standardizing the original data through the distributed streaming processing platform to generate a data basic table.

[0006] According to an embodiment of the present disclosure, it is characterized in that m user identifiers are user identifiers of all clients in the database, and n user identifiers are user identifiers of n clients that need to receive distributed data. The data basic table and the access permission table are associated to generate a data management view table, including: obtaining n user identifiers from the m user identifiers, where n is greater than or equal to 2, n is an integer and n is less than or equal to m; obtaining n user access permission information corresponding to the n user identifiers from the access permission table; establishing control logic for the access permission of the n users in the data basic table based on the n user access permission information; and generating a data management view table according to the control logic for the access permission of the n users.

[0007] According to an embodiment of the present disclosure, it is characterized in that obtaining n user identifiers from the m user identifiers includes: presetting a custom function; confirming, through the custom function, n user identifiers whose data needs to be obtained from the data stream; and obtaining the n user identifiers from the m user identifiers.

[0008] According to an embodiment of the present disclosure, it is characterized in that according to the data management view table, target data is obtained from the data basic table, and data distribution is performed on the target data, including: presetting a special data blacklist in the data management view table; obtaining target data from the data basic table according to the data management view table; performing special data detection on the target data; if special data in the special data blacklist is detected in the target data, stopping the special data distribution and returning the result; and if special data in the special data blacklist is detected in the target data, performing data distribution on the target data.

[0009] According to an embodiment of the present disclosure, it is characterized in that distributing the target data includes: performing stream processing on the target data to generate a target data stream; and distributing the target data stream.

[0010] According to the second aspect of the present disclosure, a data distribution device based on a distributed server is provided, characterized in that the device includes: a first acquisition module, used to acquire original data, perform data standardization on the original data, and generate a data base table; a second acquisition module, used to acquire m user identifiers, where m is greater than or equal to 2 and m is an integer; a first generation module, used to preset access permission information of the m user identifiers to the data in the data base table to generate an access permission table; a second generation module, used to associate the data base table with the access permission table to generate a data management view table; a third acquisition module, used to acquire target data from the data base table according to the data management view table, and distribute the target data.

[0011] According to an embodiment of the present disclosure, it is characterized in that the first acquisition module includes: a fourth acquisition module, used to obtain the server address and port number of the distributed streaming processing platform; a connection module, used to connect to the distributed streaming processing platform based on the server address and port number of the distributed streaming processing platform; and a third generation module, used to perform data standardization on the original data through the distributed streaming processing platform to generate a data basic table.

[0012] According to an embodiment of the present disclosure, it is characterized in that the m user identifiers are user identifiers of all clients in the database, and the n user identifiers are user identifiers of n clients that need to receive distributed data, and the second generation module includes: a fifth acquisition module, used to obtain n user identifiers from the m user identifiers, wherein n is greater than or equal to 2, n is an integer and n is less than or equal to m; a sixth acquisition module, used to obtain n user access permission information corresponding to the n user identifiers from the access permission table; a fourth generation module, used to establish control logic for the access permissions of the n users in the data basic table based on the n user access permission information; and a fifth generation module, used to generate a data management view table according to the control logic for the access permissions of the n users.

[0013] According to an embodiment of the present disclosure, it is characterized in that the fifth acquisition module includes: a first preset module, used to preset a custom function; a seventh acquisition module, used to confirm n user identifiers whose data needs to be obtained from the data stream through the custom function; and an eighth acquisition module, used to obtain the n user identifiers from the m user identifiers.

[0014] According to an embodiment of the present disclosure, it is characterized in that the third acquisition module includes: a second preset module, used to preset a special data blacklist in the data management view table; a ninth acquisition module, used to obtain target data from the data basic table according to the data management view table; a detection module, used to perform special data detection on the target data; a stop module, used to stop the special data distribution and return the result if it is detected that the special data in the special data blacklist exists in the target data; and a first distribution module, used to distribute the target data if it is detected that the special data in the special data blacklist does not exist in the target data.

[0015] According to an embodiment of the present disclosure, it is characterized in that the first distribution module includes: a sixth generation module, used to perform stream processing on the target data to generate a target data stream; and a second distribution module, used to distribute the target data stream.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned distributed server-based data distribution method.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the above-mentioned data distribution method based on a distributed server.

[0018] According to a fifth aspect of the present disclosure, a computer program product is also provided, including a computer program, which implements the above-mentioned distributed server-based data distribution method when executed by a processor.

[0019] The disclosed embodiments, through the technical means of associating the data base table with the access permission table to generate a data management view table and distributing data through the data management view table, achieve the technical effects of simplifying complex code, improving the speed and accuracy of data distribution, conserving computer memory resources, and improving computing efficiency. This solves the technical problem of writing control logic for identical or similar access permissions, which leads to redundant, complex, and difficult-to-maintain code and hinders users from quickly and conveniently analyzing and processing data. Furthermore, the efficiency and accuracy of data distribution are improved, enhancing work efficiency and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0021] Figure 1 A diagram schematically illustrates an application scenario of a data distribution method and apparatus based on a distributed server according to an embodiment of the present disclosure;

[0022] Figure 2 Schematically shows a flow chart of a data distribution method based on a distributed server according to an embodiment of the present disclosure;

[0023] Figure 3 Schematically shows a flow chart of generating a data base table in a data distribution method based on a distributed server according to an embodiment of the present disclosure;

[0024] Figure 4 A flowchart of generating a data management view table in a data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 5 Schematically shows a flow chart of obtaining n user identifiers in a data distribution method based on a distributed server according to an embodiment of the present disclosure;

[0026] Figure 6 Schematically shows a flow chart of performing special data detection in a data distribution method based on a distributed server according to an embodiment of the present disclosure;

[0027] Figure 7 A flowchart of distributing target data in a data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 8 The following schematically illustrates a flow chart for implementing data distribution based on Flink in a distributed server-based data distribution method according to an embodiment of the present disclosure;

[0029] Figure 9 The following schematically shows a structural block diagram of a data distribution device based on a distributed server according to an embodiment of the present disclosure;

[0030] Figure 10 Schematically shows a structural block diagram of a first acquisition module in a data distribution device based on a distributed server according to an embodiment of the present disclosure;

[0031] Figure 11 Schematically shows a structural block diagram of a second generation module in a data distribution device based on a distributed server according to an embodiment of the present disclosure;

[0032] Figure 12 Schematically shows a structural block diagram of a fifth acquisition module in a data distribution device based on a distributed server according to an embodiment of the present disclosure;

[0033] Figure 13Schematically shows a structural block diagram of a third acquisition module in a data distribution device based on a distributed server according to an embodiment of the present disclosure;

[0034] Figure 14 Schematically shows a structural block diagram of a first distribution module in a data distribution device based on a distributed server according to an embodiment of the present disclosure; and

[0035] Figure 15 A block diagram of an electronic device suitable for implementing a data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0037] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0038] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0039] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0040] The accompanying drawings illustrate some block diagrams and / or flow charts. It should be understood that some blocks in the block diagrams and / or flow charts, or combinations thereof, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable control device, so that when executed by the processor, these instructions may create a device for implementing the functions / operations described in the block diagrams and / or flow charts.

[0041] First, let’s explain the technical terms that appear in this article as follows:

[0042] Kafka: Positioned as a distributed stream processing platform, it is widely used for its high throughput, persistence, horizontal scalability, and support for stream data processing. Kafka can be used for message queuing, stream processing, log aggregation, and other functions. Kafka employs a highly flexible publish-subscribe model, enabling real-time and flexible scaling for both the server and client sides. Its offset mechanism precisely controls the client's read position in the message queue.

[0043] Flink: At its core, it is a distributed streaming data flow engine written in Java. Flink executes arbitrary streaming data programs in a data-parallel, pipelined manner. Flink's pipeline runtime system can execute both batch and stream processing programs. In addition, Flink's runtime itself supports the execution of iterative algorithms. Flink provides a high-throughput, low-latency streaming data engine as well as support for event-time processing and state management. Flink applications are fault-tolerant in the event of machine failures and support exactly-once semantics. Programs can be written in languages ​​such as Java, Scala, Python, and SQL, and are automatically compiled and optimized into data flow programs that run in clusters or cloud environments.

[0044] View table: A virtual table that does not store actual data but generates data dynamically based on a query statement. This means that when a user queries a view table, the SQL statement that defines the view is actually executed in real time. The data in the view table comes from one or more real data tables called base tables. The process of creating a view table involves writing SQL queries and has no direct impact on the base tables. The advantage of a view table is that it can simplify complex SQL operations. You don't need to worry about query details, you only need to focus on the required data. View tables provide a flexible and secure way to access and manage data in the database, and are particularly suitable for large projects and complex query scenarios.

[0045] UDF (User-Defined Function): A user-defined function (UDF) allows you to write specific logic and algorithms based on your needs to process and transform data. It is a programming mechanism that allows you to write specific functionality to extend software applications. This provides users with greater flexibility and customization options.

[0046] SQL (Structured Query Language) is a standard computer language used to access and process databases.

[0047] DDL (Data Definition Language) statements are data definition statements, mainly used for creating, deleting and modifying data objects.

[0048] Stream processing is a data distribution technology that allows data producers to write data into an ordered data stream in real time, and data consumers can read the data in real time in the same order.

[0049] An embodiment of the present disclosure provides a data distribution method based on a distributed server, the method comprising: obtaining raw data, performing data standardization on the raw data, and generating a data base table. m user identifiers are obtained, where m is an integer and greater than or equal to 2. Access rights information for the m user identifiers to data in the data base table is preset to generate an access rights table. The data base table and the access rights table are associated to generate a data management view table. According to the data management view table, target data is obtained from the data base table, and data distribution is performed on the target data.

[0050] The disclosed embodiments, based on the technical means of associating the data base table with the access permission table to generate a data management view table, and distributing data through the data management view table, achieve the technical effects of simplifying complex code, improving the speed and accuracy of data distribution, conserving computer memory resources, and improving computing efficiency. This solves the technical problem of writing control logic for identical or similar access permissions, which leads to redundant, complex, and difficult-to-maintain code and hinders users from quickly and conveniently analyzing and processing data. Furthermore, the efficiency and accuracy of data distribution are improved, enhancing work efficiency and user experience.

[0051] Figure 1 The following schematically illustrates an application scenario diagram of a data distribution method and apparatus based on a distributed server according to an embodiment of the present disclosure. Figure 1 The examples shown are merely examples of scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0052] like Figure 1 As shown, the application scenario 100 according to this embodiment may include multiple application terminals and application servers. For example, the multiple application terminals include application terminal 101, application terminal 102, application terminal 103, and so on. Network 104 is used to provide a medium for communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0053] Users can use application terminal devices 101, 102, and 103 to interact with application server 105 via network 104 to receive or send messages, etc. Various applications can be installed on application terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0054] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0055] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0056] It should be noted that the distributed server-based data distribution method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the distributed server-based data distribution device provided in the embodiment of the present disclosure can generally be set in the server 105. The distributed server-based data distribution method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the distributed server-based data distribution device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0057] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0058] The following will be based on Figure 1 The scene described by Figures 2 to 8 The data distribution method based on a distributed server of the disclosed embodiment is described in detail. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0059] Figure 2 The flowchart of the data distribution method based on the distributed server according to the embodiment of the present disclosure is schematically shown.

[0060] like Figure 2 As shown, the method 200 includes steps S201 to S205.

[0061] Step S201: acquiring original data, performing data standardization on the original data, and generating a data basic table.

[0062] Figure 3 The flowchart of generating a data basic table in the data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0063] like Figure 3 As shown, the method 300 includes steps S301 to S303.

[0064] Step S301: Obtain the server address and port number of the distributed streaming processing platform.

[0065] For example, the distributed stream processing platform can adopt the Kafka platform, and the server address and port number of the Kafka platform can be obtained.

[0066] Step S302: Connect to the distributed streaming processing platform based on the server address and port number of the distributed streaming processing platform.

[0067] Step S303: Standardize the original data through the distributed streaming processing platform to generate a data basic table.

[0068] For example, use the DDL statements in Flink to define the structure and properties of the Kafka table, such as field name, field type, partition key, offset, time attribute, etc., to create the Kafka table through the Kafka platform and use the Kafka table as the base table.

[0069] By utilizing the high throughput characteristics of the distributed streaming processing platform, the speed and accuracy of generating data base tables can be improved.

[0070] Return to reference Figure 2In step S202, m user identifiers are obtained, where m user identifiers are user identifiers of all clients in the database, where m is greater than or equal to 2 and is an integer.

[0071] Step S203: presetting the access permission information of the m user identifiers to the data in the data basic table to generate an access permission table.

[0072] For example, the access permission information of the m user identifiers to the data in the data basic table may be preset to generate an access permission table, or other intelligent analysis technologies may be used to obtain relevant permission control information from the outside to generate an access permission table.

[0073] Step S204: Associating the data basic table with the access permission table to generate a data management view table.

[0074] Figure 4 The flowchart of generating a data management view table in the data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 4 As shown, the method 400 includes steps S401 to S404.

[0076] Step S401 : Obtain n user identifiers from the m user identifiers, where the n user identifiers are user identifiers of n clients that need to receive distributed data, wherein n is greater than or equal to 2, is an integer, and is less than or equal to m.

[0077] Figure 5 The flowchart of obtaining n user identifiers in the data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0078] like Figure 5 As shown, the method 500 includes steps S501 to S503.

[0079] Step S501: preset a custom function.

[0080] For example, a custom UDF function can be preset to implement access permission control logic.

[0081] Step S502: confirming, through the user-defined function, n user identifiers whose data needs to be obtained from the data stream.

[0082] For example, you can use a custom user-defined function (UDF) to process and transform data in a Kafka table. Within the custom UDF, you can implement access control logic, such as determining whether to allow access to certain fields or records based on user identity, role, department, and other information. The custom UDF can also be used to identify the n user IDs whose data needs to be retrieved from the data stream.

[0083] Step S503: Obtain the n user identifiers from the m user identifiers.

[0084] By confirming the client's n data identifiers through a custom function, the accuracy of data distribution can be improved.

[0085] Return to reference Figure 4 In step S402, n user access permission information corresponding to the n user identifiers is obtained from the access permission table.

[0086] Step S403: Based on the access permission information of the n users, control logic for the access permission of the n users is established in the data basic table.

[0087] Step S404: Generate a data management view table according to the control logic of the access rights of the n users.

[0088] For example, a data management view table can be a view table. You can use the DDL statement in Flink to define the structure and properties of the view table, such as field name, field type, query statement, etc., to create the view table.

[0089] By obtaining the access rights information of the user IDs of n clients that need to receive distributed data, establishing access control logic in the basic table and generating a data management view table, we can dynamically establish a management view table, improving the flexibility and practicality of access control allocation. Furthermore, establishing access control logic can improve reusability and save computing resources.

[0090] Return to reference Figure 2 In step S205, target data is obtained from the data basic table according to the data management view table, and data distribution is performed on the target data.

[0091] Figure 6 The flowchart of performing special data detection in the data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0092] like Figure 6 As shown, the method 600 includes steps S601 to S605.

[0093] Step S601: Preset a special data blacklist in the data management view table.

[0094] For example, a special data blacklist can be preset in the view table. The special data may include sensitive data, confidential data, etc., which can facilitate fuzzy processing of special data.

[0095] Step S602: Obtain target data from the data foundation table according to the data management view table.

[0096] For example, you can use a view table and use Flink's SQL statements to query and operate on the view table, thereby controlling access permissions to the Kafka table and retrieving target data from it. In this process, you only need to call the view table's name or identifier without worrying about the view table's internal logic and algorithms, nor do you need to write the same or similar access control logic in every SQL statement or API.

[0097] Step S603: Perform special data detection on the target data.

[0098] Step S604: If it is detected that the target data contains special data in the special data blacklist, the distribution of the special data is stopped and a result is returned.

[0099] Step S605: If it is detected that the target data does not contain the special data in the special data blacklist, the target data is distributed.

[0100] Figure 7 The flowchart of distributing target data in the data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 7 As shown, the method 700 includes steps S701 to S702.

[0102] Step S701: perform stream processing on the target data to generate a target data stream.

[0103] For example, you can use Flink to perform stream processing on a Kafka table so that the processed data can be output to different target systems or storage media.

[0104] Step S702: Distribute the target data stream.

[0105] By stream processing the target data, we can ensure stable data distribution under high concurrency and high load conditions, thereby improving the reliability of data distribution. In addition, through real-time special data detection, we can improve the security and standardization of data and enhance the reliability of data distribution.

[0106] Figure 8 The flowchart of implementing data distribution based on Flink in the data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0107] like Figure 8 As shown, the method 800 includes steps S801 to S804.

[0108] Step S801: Create a Kafka table using Link.

[0109] For example, you can use Kafka as a data source and use DDL statements in Flink to define the structure and properties of the Kafka table, such as field name, field type, partition key, offset, time attribute, etc.

[0110] Step S802: Use the UFD function package to package the Kafka table and generate a view table.

[0111] For example, use DDL statements in Flink to define the structure and properties of a view table, such as field names, field types, and query statements. In query statements, you can reference Kafka tables and use custom User-Defined Functions (UDFs) to process and transform the data in these tables. Specifically, this can include pre-setting a UDF package that reads environment variables from the user's environment information to confirm the user's information. The logic in the UDF package matches the user, limiting the data the user can access. After wrapping Kafka with the UDF package, a view table is generated. Within the custom UDF, you can implement access control logic, such as determining whether access to certain fields or records is permitted based on the user's identity, role, department, and other information, and whether special data needs to be obfuscated, and returning the corresponding results.

[0112] Step S803: Use the view table to control the Kafka table.

[0113] For example, you can use a view table to control a Kafka table and use Flink SQL statements to query and operate the view table, thereby controlling access permissions to the Kafka table. In this process, you only need to call the view table name without worrying about the logic and algorithms within the view table, nor do you need to write the same or similar access control logic in every SQL statement or API.

[0114] Step S804: Implement data distribution.

[0115] For example, data distribution can be achieved by using view tables.

[0116] According to the embodiments of the present disclosure, view tables and customized user-defined functionalities (UDFs) can be used in Flink to encapsulate queries and operations on Kafka tables. Access control logic can be implemented in the view tables and UDFs, thus avoiding the need to write the same or similar access control logic in every SQL statement, reducing code redundancy, complexity, and difficulty in maintenance. The logic and algorithms of view tables and UDFs can be customized according to different scenarios and requirements, thereby dynamically adjusting access control rules and policies to meet different data security and compliance requirements. View tables and UDFs can be called using simple and intuitive SQL statements to control access rights on Kafka tables, making it easy for users to quickly and conveniently analyze and process data.

[0117] Figure 9 The structural block diagram of a data distribution device based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 9 As shown, the apparatus 900 includes: a first acquisition module 901 , a second acquisition module 902 , a first generation module 903 , a second generation module 904 and a third acquisition module 905 .

[0119] The first acquisition module 901 is used to acquire original data, perform data standardization on the original data, and generate a data basic table. In one embodiment, the first acquisition module 901 can be used to execute step S201 described above.

[0120] Figure 10 The structural block diagram of the first acquisition module in the data distribution device based on the distributed server according to an embodiment of the present disclosure is schematically shown.

[0121] like Figure 10 As shown, the first acquisition module 901 includes: a fourth acquisition module 1001 , a connection module 1002 and a third generation module 1003 .

[0122] The fourth acquisition module 1001 is used to acquire the server address and port number of the distributed streaming processing platform. In one embodiment, the fourth acquisition module 1001 can be used to execute the above-described step S301, which will not be described in detail here.

[0123] The connection module 1002 is configured to connect to the distributed streaming processing platform based on the server address and port number of the distributed streaming processing platform. In one embodiment, the connection module 1002 may be configured to execute step S302 described above, which will not be described in detail here.

[0124] The third generating module 1003 is configured to perform data standardization on the raw data through the distributed stream processing platform to generate a data base table. In one embodiment, the third generating module 1003 may be configured to execute step S303 described above, which will not be described in detail here.

[0125] Return to reference Figure 9 The second acquisition module 902 is configured to acquire m user identifiers, where m is an integer and greater than or equal to 2. In one embodiment, the second acquisition module 902 may be configured to execute step S202 described above, which will not be described in detail herein.

[0126] The first generating module 903 is configured to preset the access permission information of the m user identifiers to the data in the data basic table to generate an access permission table. In one embodiment, the first generating module 903 may be configured to execute the above-described step S203, which will not be described in detail here.

[0127] The second generating module 904 is configured to associate the data basic table with the access permission table to generate a data management view table. In one embodiment, the second generating module 904 may be configured to execute step S204 described above.

[0128] Figure 11 The structural block diagram of the second generation module in the data distribution device based on the distributed server according to an embodiment of the present disclosure is schematically shown.

[0129] like Figure 11 As shown, the second generating module 904 includes: a fifth acquiring module 1101 , a sixth acquiring module 1102 , a fourth generating module 1103 and a fifth generating module 1104 .

[0130] The fifth acquisition module 1101 is configured to acquire n user identifiers from the m user identifiers, where n is greater than or equal to 2, is an integer, and is less than or equal to m. In one embodiment, the fifth acquisition module 1101 may be configured to execute step S401 described above.

[0131] Figure 12 The structural block diagram of the fifth acquisition module in the data distribution device based on the distributed server according to the embodiment of the present disclosure is schematically shown.

[0132] like Figure 12As shown, the fifth acquisition module 1101 includes: a first preset module 1201 , a seventh acquisition module 1202 and an eighth acquisition module 1203 .

[0133] The first preset module 1201 is used to preset a custom function. In one embodiment, the first preset module 1201 can be used to execute the step S501 described above, which will not be described in detail here.

[0134] The seventh acquisition module 1202 is configured to use the user-defined function to identify n user identifiers whose data needs to be acquired from the data stream. In one embodiment, the seventh acquisition module 1202 may be configured to execute the aforementioned step S502, which will not be described in detail herein.

[0135] The eighth acquisition module 1203 is configured to acquire the n user identifiers from the m user identifiers. In one embodiment, the eighth acquisition module 1203 may be configured to execute step S503 described above, which will not be described in detail here.

[0136] Return to reference Figure 11 The sixth acquisition module 1102 is configured to acquire n user access rights information corresponding to the n user identifiers from the access rights table. In one embodiment, the sixth acquisition module 1102 may be configured to execute step S402 described above, which will not be described in detail herein.

[0137] The fourth generating module 1103 is configured to establish control logic for the access rights of the n users in the data basic table based on the access rights information of the n users. In one embodiment, the fourth generating module 1103 may be configured to execute step S403 described above, which will not be described in detail here.

[0138] The fifth generating module 1104 is configured to generate a data management view table according to the control logic of the access rights of the n users. In one embodiment, the fifth generating module 1104 may be configured to execute the aforementioned step S404, which will not be described in detail here.

[0139] Return to reference Figure 9 The third acquisition module 905 is configured to acquire target data from the data basic table according to the data management view table and distribute the target data. In one embodiment, the third acquisition module 905 may be configured to execute step S205 described above.

[0140] Figure 13 The structural block diagram of the third acquisition module in the data distribution device based on the distributed server according to an embodiment of the present disclosure is schematically shown.

[0141] like Figure 13As shown, the third acquisition module 905 includes: a second preset module 1301 , a ninth acquisition module 1302 , a detection module 1303 , a stop module 1304 and a first distribution module 1305 .

[0142] The second preset module 1301 is configured to preset a special data blacklist in the data management view table. In one embodiment, the second preset module 1301 may be configured to execute step S601 described above, which will not be described in detail here.

[0143] The ninth acquisition module 1302 is configured to acquire target data from the data foundation table according to the data management view table. In one embodiment, the ninth acquisition module 1302 may be configured to execute the aforementioned step S602, which will not be described in detail herein.

[0144] The detection module 1303 is configured to perform special data detection on the target data. In one embodiment, the detection module 1303 may be configured to execute the aforementioned step S603, which will not be described in detail here.

[0145] The stop module 1304 is configured to stop the distribution of the special data and return a result if the target data contains special data in the special data blacklist. In one embodiment, the stop module 1304 may be configured to execute step S604 described above, which will not be described in detail here.

[0146] The first distribution module 1305 is configured to distribute the target data if it is detected that the target data does not contain the special data in the special data blacklist. In one embodiment, the first distribution module 1305 may be configured to execute step S605 described above.

[0147] Figure 14 The structural block diagram of the first distribution module in the data distribution device based on the distributed server according to an embodiment of the present disclosure is schematically shown.

[0148] like Figure 14 As shown, the first distribution module 1305 includes: a sixth generation module 1401 and a second distribution module 1402 .

[0149] The sixth generating module 1401 is configured to perform stream processing on the target data to generate a target data stream. In one embodiment, the sixth generating module 1401 may be configured to execute step S701 described above, which will not be described in detail herein.

[0150] The second distribution module 1402 is configured to distribute the target data stream. In one embodiment, the second distribution module 1402 may be configured to execute the aforementioned step S702, which will not be described in detail herein.

[0151] According to an embodiment of the present disclosure, any multiple modules among the first acquisition module 901, the second acquisition module 902, the first generation module 903, the second generation module 904, and the third acquisition module 905 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 901, the second acquisition module 902, the first generation module 903, the second generation module 904, and the third acquisition module 905 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 901 , the second acquisition module 902 , the first generation module 903 , the second generation module 904 and the third acquisition module 905 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0152] Figure 15 A block diagram of an electronic device suitable for implementing a data distribution method based on a distributed server according to an embodiment of the present disclosure is schematically shown.

[0153] like Figure 15 As shown, the electronic device 1500 according to an embodiment of the present disclosure includes a processor 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage portion 1508 into a random access memory (RAM) 1503. The processor 1501 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1501 may also include an onboard memory for caching purposes. The processor 1501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0154] Various programs and data required for the operation of the electronic device 1500 are stored in the RAM 1503. The processor 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. The processor 1501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1502 and / or the RAM 1503. It should be noted that the programs may also be stored in one or more memories other than the ROM 1502 and the RAM 1503. The processor 1501 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0155] According to an embodiment of the present disclosure, electronic device 1500 may further include an input / output (I / O) interface 1505, which is also connected to bus 1504. Electronic device 1500 may further include one or more of the following components connected to I / O interface 1505: an input portion 1506 including a keyboard, mouse, etc.; an output portion 1507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage portion 1508 including a hard disk; and a communication portion 1509 including a network interface card such as a LAN card or a modem. Communication portion 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to I / O interface 1505 as needed. Removable media 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in drive 1510 as needed, so that computer programs read from the removable media can be installed into storage portion 1508 as needed.

[0156] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0157] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1502 and / or RAM 1503 described above and / or one or more memories other than ROM 1502 and RAM 1503.

[0158] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the distributed server-based data distribution method provided by the embodiments of the present disclosure.

[0159] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 1501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0160] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal over a network medium, downloaded and installed via communication portion 1509, and / or installed from removable media 1511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0161] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1509 and / or installed from the removable medium 1511. When the computer program is executed by the processor 1501, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0162] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0165] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data distribution method based on a distributed server, characterized in that: The method includes: Obtaining original data, performing data standardization on the original data, and generating a data basic table; Get m user IDs, where m is greater than or equal to 2 and is an integer; Presetting access permission information of the m user identifiers to the data in the data basic table to generate an access permission table; Associating the data basic table with the access permission table to generate a data management view table; and According to the data management view table, target data is obtained from the data basic table, and the target data is distributed. The m user identifiers are user identifiers of all clients in the database, and the n user identifiers are user identifiers of n clients that need to receive distributed data. The data basic table and the access permission table are associated to generate a data management view table, including: Obtaining n user identifiers from the m user identifiers, where n is greater than or equal to 2, n is an integer, and n is less than or equal to m; Acquire n user access permission information corresponding to the n user identifiers from the access permission table; Based on the n user access authority information, establishing control logic for the n users' access authority in the data foundation table; and Generate a data management view table based on the control logic of the access rights of the n users, The step of obtaining n user identifiers from the m user identifiers includes: Preset custom functions; Confirming n user identifiers that need to obtain data from the data stream through the custom function; and The n user identifiers are obtained from the m user identifiers.

2. The method according to claim 1, characterized in that The raw data is standardized to generate a data base table, including: Get the server address and port number of the distributed streaming processing platform; Connecting to the distributed streaming processing platform based on the server address and port number of the distributed streaming processing platform; and The raw data is standardized through the distributed streaming processing platform to generate a data base table.

3. The method according to claim 1 or 2, characterized in that According to the data management view table, target data is obtained from the data foundation table, and data distribution is performed on the target data, including: Presetting a special data blacklist in the data management view table; According to the data management view table, obtaining target data from the data basic table; performing special data detection on the target data; If it is detected that the target data contains special data in the special data blacklist, stopping the distribution of the special data and returning a result; and If it is detected that the target data does not contain the special data in the special data blacklist, the target data is distributed.

4. The method according to claim 3, characterized in that Distributing the target data includes: performing stream processing on the target data to generate a target data stream; and The target data stream is distributed.

5. A data distribution device based on a distributed server, characterized in that: The device includes: A first acquisition module is used to acquire original data, perform data standardization on the original data, and generate a data basic table; A second acquisition module is configured to acquire m user identifiers, where m is greater than or equal to 2 and is an integer; A first generating module, configured to preset access permission information of the m user identifiers to the data in the data basic table to generate an access permission table; A second generating module is configured to associate the data basic table with the access permission table to generate a data management view table; and The third acquisition module is used to acquire target data from the data basic table according to the data management view table and distribute the target data. The m user identifiers are user identifiers of all clients in the database, and the n user identifiers are user identifiers of n clients that need to receive distributed data. The data basic table and the access permission table are associated to generate a data management view table, including: Obtaining n user identifiers from the m user identifiers, where n is greater than or equal to 2, n is an integer, and n is less than or equal to m; Acquire n user access permission information corresponding to the n user identifiers from the access permission table; Based on the n user access authority information, establishing control logic for the n users' access authority in the data foundation table; and Generate a data management view table based on the control logic of the access rights of the n users, The step of obtaining n user identifiers from the m user identifiers includes: Preset custom functions; Confirming n user identifiers that need to obtain data from the data stream through the custom function; and The n user identifiers are obtained from the m user identifiers.

6. An electronic device comprising: an or processor; A storage device for storing a program or When the one or program is executed by the one or processor, the one or processor executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images