Data processing method and related device

Abstract

The invention discloses a data processing method and a related device, and relates to the technical field of message queues, a target client requests to carry out target operation on business data of a target theme, a server verifies whether a permission mapping relation matched with a permission authentication request exists in first-class permission data, and if yes, the first-class permission data is sent to the server; the server searches target message queue cluster information matched with the target theme in second-class authority data and sends the target message queue cluster information to the target client, the second-class authority data comprises a plurality of routing relationships, and the routing relationships comprise corresponding relationships between the theme and each message queue cluster in the message queue system; and the target client side executes the target operation in the target message queue cluster according to the target message queue cluster information, so that the flexibility of accessing multiple clusters by each client side is improved through centralized maintenance of authority data by the server side.

Description

Technical Field

[0001] This application relates to the field of message queue technology, and in particular to a data processing method and related apparatus. Background Technology

[0002] Message queues, as a high-performance data transmission system, are widely used in scenarios such as real-time data transmission, log collection, and big data processing. To achieve simultaneous access to multiple clusters, traditional message queue clients need to manage the relationship between topics and message queue client instances themselves. However, due to the increasing number of topics and the growing scale of queue clusters, existing message queue clients lack flexibility in accessing multiple cluster queues, and the complexity of migrating and upgrading existing applications is high. Summary of the Invention

[0003] In view of the above problems, this application provides a data processing method and related apparatus to achieve flexible access to multiple cluster queues by message queue clients. The specific solution is as follows:

[0004] The first aspect of this application provides a data processing method applied to a data processing system, the data processing system including a server and at least one client, comprising:

[0005] The target client sends an authentication request to the server. The authentication request includes the client identifier, target topic, and target operation of the target client, and is used to request the target operation to be performed on the business data of the target topic. The target client can be any client in the data processing system.

[0006] The server verifies whether there is a permission mapping relationship in the first type of permission data that matches the permission authentication request. The first type of permission data includes multiple permission mapping relationships, and the permission mapping relationship includes the correspondence between client identifier, topic and operation.

[0007] If it exists, the server searches for the target message queue cluster information that matches the target topic in the second type of permission data, and sends the target message queue cluster information to the target client. The second type of permission data includes multiple routing relationships, and the routing relationships include the correspondence between topics and various message queue clusters in the message queue system.

[0008] After receiving the target message queue cluster information, the target client executes the target operation on the target message queue cluster according to the target message queue cluster information.

[0009] In one possible implementation, the target operations include read and write operations, and the permission mapping relationship includes subscription and publish relationships; the data processing method further includes:

[0010] The server receives a publishing request for a first topic from a first client. Based on the second type of permission data and the dynamic load of each message queue cluster in the message queue system, it determines the message queue cluster information corresponding to the first topic, constructs a first publishing relationship and a first routing relationship, and updates the first type of permission data and the second type of permission data.

[0011] Wherein, the first publishing relationship includes the correspondence between the client identifier of the first client, the first topic, and the write operation; the first routing relationship includes the correspondence between the first topic and its corresponding message queue cluster information;

[0012] The server receives a subscription request for a second topic from a second client, determines whether a second routing relationship exists, and if so, constructs a first subscription relationship and updates the first type of permission data.

[0013] Wherein, the second routing relationship includes the routing relationship of the second topic; the first subscription relationship includes the client identifier of the second client, the second topic, and the correspondence of the read operation;

[0014] Both the first client and the second client are any client in the data processing system.

[0015] In one possible implementation, the server determines the message queue cluster information corresponding to the first topic based on the second type of permission data and the dynamic load of each message queue cluster within the message queue system, including:

[0016] Based on the business information corresponding to the first topic, predict the data volume level of the first topic;

[0017] Based on the second type of permission data, the number of topics corresponding to each message queue cluster in the message queue system is determined.

[0018] Based on the data volume level of the first topic, the number of topics corresponding to each message queue cluster in the message queue system, and the dynamic load, the message queue cluster information corresponding to the first topic is determined according to the preset load balancing rules.

[0019] In one possible implementation, after receiving the target message queue cluster information, the target client performs the target operation on the target message queue cluster according to the target message queue cluster information, including:

[0020] The target client receives the target message queue cluster information and, based on the target message queue cluster information, determines whether a target client instance exists in the instance cache;

[0021] If it exists, the target client invokes the target client instance in the instance cache to perform the target operation in the target message queue cluster;

[0022] If it does not exist, the target client creates the target client instance based on the target message queue cluster information, calls the target client instance to perform the target operation in the target message queue cluster, and caches the target client instance in the instance cache.

[0023] In one possible implementation, after the target client receives the target message queue cluster information, the data processing method further includes:

[0024] The target client caches the target cluster mapping relationship in the permission data cache. The target cluster mapping relationship includes the correspondence between the target topic, the target operation, and the target message queue cluster information.

[0025] The target client sends an authentication request to the server, including:

[0026] Determine whether there is a cluster mapping relationship in the permission data cache that matches the permission authentication request. If not, send the permission authentication request to the server.

[0027] In one possible implementation, the data processing method also includes:

[0028] If a cluster mapping relationship exists that matches the permission authentication request, the target message queue cluster information is obtained from the cluster mapping relationship that matches the permission authentication request, and the target operation is performed on the target message queue cluster according to the target message queue cluster information.

[0029] In one possible implementation, the data processing method also includes:

[0030] If the cache duration of the target client instance exceeds a preset first cache duration, the target client will clear the target client instance.

[0031] If the cache duration of the cluster mapping relationship exceeds a preset second cache duration or exceeds a preset validity period, the target client cleans up the target client instance.

[0032] According to a preset query cycle, the target client requests the server to query whether the cluster mapping relationship is valid. If the cluster mapping relationship is invalid, the cluster mapping relationship is cleared.

[0033] A second aspect of this application provides a data processing apparatus, the data processing apparatus comprising a first data processing unit configured on a server and second data processing units respectively configured on at least one client, including:

[0034] The second data processing unit is used to: send an authorization authentication request to the first data processing unit, wherein the authorization authentication request includes the client identifier of the target client, the target topic, and the target operation, and is used to request the target operation to be performed on the business data of the target topic, wherein the target client is any second data processing unit in the data processing system;

[0035] The first data processing unit is configured to: verify whether there is a permission mapping relationship matching the permission authentication request in the first type of permission data; if so, search for target message queue cluster information matching the target topic in the second type of permission data, and send the target message queue cluster information to the first data processing unit; the first type of permission data includes multiple permission mapping relationships, which include the correspondence between client identifier, topic, and operation; the second type of permission data includes multiple routing relationships, which include the correspondence between topics and various message queue clusters in the message queue system.

[0036] The second data processing unit is configured to: receive the target message queue cluster information and, according to the target message queue cluster information, execute the target operation in the target message queue cluster.

[0037] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the data processing method described in the first aspect or any implementation thereof.

[0038] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the data processing method of the first aspect or any implementation thereof.

[0041] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the data processing method described in the first aspect or any implementation thereof.

[0042] Using the above technical solution, this application provides a data processing method and related apparatus, applied to a data processing system. The data processing system includes a server and at least one client. A target client sends an authentication request to the server. The authentication request includes the target client's client identifier, target topic, and target operation, and is used to request a target operation on business data of the target topic. The target client can be any client in the data processing system. The server verifies whether a permission mapping relationship matching the authentication request exists in the first type of permission data. If it exists, the server searches for target message queue cluster information matching the target topic in the second type of permission data and sends the target message queue cluster information to the target client. After receiving the target message queue cluster information, the target client executes the target operation in the target message queue cluster according to the target message queue cluster information. Since the first type of permission data includes multiple permission mapping relationships, including the correspondence between client identifier, topic, and operation, and the second type of permission data includes multiple routing relationships, including the correspondence between topics and various message queue clusters in the message queue system. Therefore, by centrally maintaining permission data on the server side, when a target client initiates a target operation on the business data of the target topic, the access permission is verified based on the first type of permission data, and the access address is determined based on the second type of permission data. This enables permission control and flexibility for each client to access multiple clusters. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0045] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0046] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating the specific implementation flow of a data processing method provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the specific structure of a data processing system provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of permission data provided in an embodiment of this application;

[0050] Figure 7 A schematic diagram illustrating access control provided in an embodiment of this application;

[0051] Figure 8 A schematic diagram illustrating a cluster expansion effect provided in an embodiment of this application;

[0052] Figure 9 A schematic diagram illustrating the specific structure of another data processing system provided in this application embodiment;

[0053] Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0054] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] This application can be applied to the field of message queue technology, specifically to scenarios where message queue clients can flexibly access multiple message queue clusters.

[0059] This application may be applied, but is not limited to, to applications with data processing capabilities or cloud services provided by cloud-side servers, which will be described in detail below:

[0060] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the methods provided in the embodiments of this application to one or more terminals.

[0061] The terminal 100 may be equipped with a data processing application. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the data processing interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0062] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0063] The following description Figure 1 The product form of the mid-terminal 100;

[0064] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0065] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.

[0066] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0067] Among them, the input device can receive input data, etc.

[0068] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display the interface for data processing, processing results, etc.

[0069] The memory can be used to store software code related to the data processing method, the processor can execute the steps of the data processing method, and can also schedule other units (such as the input unit and display unit mentioned above) to achieve the corresponding functions.

[0070] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0071] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.

[0072] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0073] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.

[0074] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0075] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.

[0076] The memory can be used to store software code related to data processing methods, the processor can execute the steps of the chip's data processing methods, and it can also schedule other units to achieve corresponding functions.

[0077] The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0078] Reference Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 2 As shown in the embodiment of this application, a data processing method is provided and applied to a data processing system. The data processing system includes a server and at least one client, and may include steps S201 to S204. These steps are described in detail below.

[0079] S201. The target client sends an authentication request to the server.

[0080] In this embodiment, the authorization request includes the client identifier of the target client, the target topic, and the target operation, and is used to request a target operation on the business data of the target topic. The target client can be any client in the data processing system.

[0081] S202. The server verifies whether there is a permission mapping relationship in the first type of permission data that matches the permission authentication request.

[0082] In this embodiment, the first type of permission data includes multiple permission mapping relationships, which include the correspondence between client identifier, topic, and operation.

[0083] S203. If it exists, the server searches for the target message queue cluster information that matches the target topic in the second type of permission data, and sends the target message queue cluster information to the target client.

[0084] In this embodiment, the second type of permission data includes multiple routing relationships, which include the correspondence between topics and various message queue clusters in the message queue system.

[0085] S204. After receiving the target message queue cluster information, the target client executes the target operation in the target message queue cluster according to the target message queue cluster information.

[0086] In this embodiment, the target message queue cluster information includes the address of the target message queue cluster, such as IP and port information.

[0087] As can be seen from the above technical solutions, the data processing method provided in this application embodiment includes a first type of permission data comprising multiple permission mapping relationships, which include the correspondence between client identifiers, topics, and operations; and a second type of permission data comprising multiple routing relationships, which include the correspondence between topics and various message queue clusters in the message queue system. Therefore, through centralized maintenance of the permission data on the server side, when a target client initiates a target operation on business data of a target topic, access permissions are verified based on the first type of permission data, and the access address is determined based on the second type of permission data. This achieves both permission control and flexibility for each client accessing multiple clusters.

[0088] Based on the above embodiments, this application also includes steps for generating and updating permission data, see [link to documentation]. Figure 3 , Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 3 This illustrates one possible implementation of the steps for generating and updating permission data on the server side. The target operations include read and write operations, and the corresponding permission mapping relationships include subscription and publish relationships. For example... Figure 3 As shown, this method specifically includes steps S301 to S305, which are described in detail below.

[0089] S301, Receive the first topic publication request sent by the first client.

[0090] S302. Based on the second type of permission data and the dynamic load of each message queue cluster in the message queue system, after determining the message queue cluster information corresponding to the first topic, construct the first publishing relationship and the first routing relationship.

[0091] In this embodiment, the first publishing relationship includes the correspondence between the client identifier of the first client, the first topic, and the write operation. The first routing relationship includes the correspondence between the first topic and its corresponding message queue cluster information.

[0092] In one optional embodiment, the server determines the message queue cluster information corresponding to the first topic based on the second type of permission data and the dynamic load of each message queue cluster within the message queue system, including:

[0093] A1. Based on the business information corresponding to the first topic, predict the data volume level of the first topic;

[0094] A2. Based on the second type of permission data, determine the number of topics corresponding to each message queue cluster in the message queue system;

[0095] A3. Based on the data volume level of the first topic, the number of topics corresponding to each message queue cluster in the message queue system, and the dynamic load, determine the message queue cluster information corresponding to the first topic according to the preset load balancing rules.

[0096] The load balancing rules include ensuring that the number of topics corresponding to any message queue cluster does not exceed a topic number threshold, that all load parameters meet their respective load parameter limits, and that the load balancing conditions are met. Load parameters include message backlog, throughput, CPU utilization, memory utilization, and network bandwidth usage.

[0097] Specifically, based on the data volume level of the first topic (data volume and data writing frequency, etc.), for message queue clusters with fewer topics than the topic number threshold, the load parameters and load balancing degree are predicted after binding the first topic with the message queue cluster. If there is no existing message queue cluster that meets the above load balancing rules, a cluster expansion instruction is triggered, causing the message queue system to add a new message queue cluster. The newly added message queue cluster is then used as the message queue cluster corresponding to the first topic, and the information of the message queue cluster corresponding to the first topic, i.e., the target message queue cluster information, is obtained.

[0098] S303, Update the first type of permission data and the second type of permission data.

[0099] S304. Receive the subscription request for the second topic from the second client and determine whether a second routing relationship exists.

[0100] In this embodiment, the second routing relationship is a routing relationship that includes the second topic.

[0101] S305. If so, establish the first subscription relationship and update the first type of permission data.

[0102] In this embodiment, the first subscription relationship includes the client identifier of the second client, the second topic, and the correspondence of read operations.

[0103] It should be noted that both the first client and the second client are any client in the data processing system.

[0104] As can be seen from the above technical solutions, the data processing method provided in this application embodiment can uniformly manage the first type of permission data and the second type of permission data related to the message queue cluster through the server, maintain the mapping relationship between producers, consumers and topics, establish the routing relationship between topics and clusters, and perform more flexible permission management through the above relationship. At the same time, through the routing relationship between topics and clusters, different topics can be routed to different message queue clusters, thereby achieving the goal of business isolation and enabling clients to access multiple message queue clusters simultaneously.

[0105] Based on the above embodiments, see Figure 4 , Figure 4 This is a schematic diagram illustrating the specific implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 4 As shown, this method specifically includes steps S401 to S409, which are described in detail below.

[0106] S401. The target client determines whether a cluster mapping relationship matching the authorization authentication request exists in the authorization data cache.

[0107] S402. If it does not exist, send an authentication request to the server.

[0108] S403. The server verifies whether there is a permission mapping relationship in the first type of permission data that matches the permission authentication request.

[0109] In this embodiment, the first type of permission data includes multiple permission mapping relationships, which include the correspondence between client identifier, topic, and operation.

[0110] S404. If it exists, the server searches for the target message queue cluster information that matches the target topic in the second type of permission data, and sends the target message queue cluster information to the target client.

[0111] In this embodiment, the second type of permission data includes multiple routing relationships, which include the correspondence between topics and various message queue clusters in the message queue system.

[0112] S405. After receiving the target message queue cluster information, the target client caches the target cluster mapping relationship in the permission data cache.

[0113] S406. If a cluster mapping relationship matching the authorization request exists, the target client obtains the target message queue cluster information from the cluster mapping relationship matching the authorization request.

[0114] S407. The target client determines whether a target client instance exists in the instance cache based on the target message queue cluster information.

[0115] S408. If it exists, the target client calls the target client instance in the instance cache to perform the target operation in the target message queue cluster.

[0116] S409. If it does not exist, the target client creates a target client instance based on the target message queue cluster information, calls the target client instance to perform the target operation in the target message queue cluster, and caches the target client instance in the instance cache.

[0117] It should be noted that by using instance caching and permission data caching within the client, the efficiency of permission authentication and routing authentication is improved, thus enhancing cluster access efficiency. Furthermore, a cache cleanup mechanism for these instance and permission data caches is used to clean up the cache, reducing storage resource consumption and improving cache effectiveness. Optionally, the cache cleanup mechanism includes:

[0118] If the cache duration of a cluster mapping relationship exceeds a preset second cache duration or exceeds a preset expiration date, the target client cleans up the target client instance. According to a preset query cycle, the target client requests the server to check the validity of the cluster mapping relationship; if the cluster mapping relationship is invalid, it is cleaned up. If the cache duration of a target client instance exceeds a preset first cache duration, the target client cleans up the target client instance.

[0119] In summary, the server is responsible for maintaining message queue cluster information, including the access permissions between producers, consumers, and topics, as well as the routing relationships between topics and clusters. Simultaneously, the server receives authentication requests from clients, performs authentication, and returns the allowed cluster information to the client. The client sends an authentication request to the server, establishing a connection with the cluster information returned by the server. Before accessing the message queue cluster, the client sends request information such as producer application, consumer application, topic, and consumer group to the server for authentication. The server returns all successfully authenticated cluster information to the client, and the client establishes connections with multiple message queue clusters through the message queue instance manager.

[0120] As can be seen from the above technical solution, the server in this application provides unified management of message queue cluster information and maintains application-level permission data for producers and consumers. Based on the existing message queue system's permission control mechanism, it further enhances the flexibility of permission authentication. It can adapt to enterprise-wide permission control scenarios for application access to topics, making access to the message queue cluster more secure and controllable. Based on the routing relationship between topics and clusters, expanding the cluster can achieve non-interference with existing clusters. Simultaneously, it can establish routing relationships between different business topics and different clusters, achieving resource isolation between different businesses. This ensures that a single cluster failure does not affect the entire business, enhancing the system's fault tolerance.

[0121] In this application, the client implements the construction and management of multiple message queue client instances through a message queue client instance processor and a message queue client instance cache. Each message queue client instance establishes a connection with its corresponding message queue cluster, thereby enabling the client to transparently access multiple message queue clusters, significantly reducing development costs and the complexity of modification and migration. Furthermore, before constructing each message queue client instance, it first checks whether the corresponding client instance for that cluster exists in the cache; if it exists, it is reused; otherwise, a new instance is created, reducing the performance pressure on the application and the message queue cluster.

[0122] Figure 5 This is a schematic diagram of the specific structure of a data processing system provided in an embodiment of this application, such as... Figure 5 As shown, the data processing system includes a server 51, at least one client 52, and a message queue cluster 53. The server 51 consists of two parts: a permission data processor 211 and a permission authentication processor 512. The client 52 consists of two parts: a permission authentication sender / receiver processor 521 and a message queue client instance processor 522. The message queue cluster 53 includes n message queue clusters, for example, n=4, which can be independent of the server or configured within the server.

[0123] The following combination Figure 5 The functions of the data processing system will be described in detail below:

[0124] In this embodiment, the permission data processor maintains the permission data. The permission data includes a first type of permission data and a second type of permission data. Specifically, the first type of permission data includes the publishing relationship between producers and topics (i.e., producer information) and the subscription relationship between consumers and topics (i.e., consumer information). The second type of permission data includes the routing relationship between topics and message queue cluster information. That is, the second type of permission data specifically includes the routing relationship between topics and the message queue cluster (i.e., topic information) and message queue cluster information (i.e., cluster information).

[0125] Figure 6 This is a schematic diagram of permission data provided in an embodiment of this application, such as... Figure 6As shown, in the interface service management of the management mode, in the producer information, producer 1 has established a publishing relationship with topic 1, producer 2 has established a publishing relationship with topic 2, and the publishing relationship between producer 1 and topic 3 is under publishing review (the review mechanism can be found in existing technology). In the consumer information, consumer 1 and consumer 2 both have subscription relationships with topic 1, consumer 2 and consumer 3 both have subscription relationships with topic 2, and the subscription relationship between consumers and topic 1 is under subscription application (the application mechanism can be found in existing technology). In the topic information, topic 1 has a routing relationship with cluster 1, and topic 2 has a routing relationship with cluster 4. The cluster information includes the addresses of clusters 1 to 4.

[0126] like Figure 6 As shown, through the publish and subscribe relationships in the above example, in runtime access control, producer 1 is allowed to write data to topic 1, but not to topic 2. Producer 2 can write data to topic 2, but not to topic 1. As consumers, consumer 1 is allowed to consume data from topic 1, but cannot consume from other topics. Consumer 2 can consume data from both topic 1 and topic 2 simultaneously. Consumer 3 is only allowed to consume data from topic 2, and not from topic 1. This achieves access control at the producer and consumer application levels.

[0127] like Figure 6 As shown, the runtime permission data is integrated with the enterprise's internal interface service management system, and is transformed from the management-level permission data, enabling more comprehensive access control and interface data standardization management capabilities.

[0128] In this embodiment, message queue cluster resource isolation is achieved through topic and cluster routing management. Figure 7 This is a schematic diagram of access control provided in an embodiment of this application, such as... Figure 7 As shown, topic 1 for service A is routed to cluster 1, and topic 2 for service B is routed to cluster 4. When cluster 1 fails, only service A is affected, and cluster 4 containing topic 2 is not affected; service B is unaffected.

[0129] In this embodiment, when it is necessary to expand the cluster, the server only needs to maintain the newly added cluster and the routing relationship between the incremental topic and the newly added cluster to achieve seamless expansion of the cluster without affecting the existing cluster. Figure 8 This is a schematic diagram of a cluster expansion effect provided in an embodiment of this application. The expanded message queue resource is numbered as cluster 5, and an incremental topic (topic 3) is mapped to this cluster. The producers (producer 2) and consumers (consumer 3) of this topic will be allowed to establish connections with the newly added cluster (cluster 5) without being aware of other clusters.

[0130] In this embodiment, when the permission authentication processor receives the permission authentication request sent by the permission authentication sending and receiving processor from the client, it performs permission authentication on the permission authentication request based on the permission data, and after the permission authentication is successful, it returns the corresponding message queue cluster information (i.e., address) to the permission authentication sending and receiving processor, so that the message queue client instance processor can access the cluster based on the message queue cluster information.

[0131] In this embodiment, the permission authentication processor executes the permission authentication logic as follows:

[0132] Receive authentication requests sent by clients. If the authentication request is for a producer and a topic, it means that the user is requesting to write to the topic as a producer. If the access request is for a consumer and a topic, it means that the user is requesting to read from the topic as a consumer.

[0133] If the access request information is for a producer and a topic, and the producer's permissions are authenticated, then query the permission data to see if the publishing relationship between the producer and the topic exists.

[0134] If the access request information is for a consumer and a topic, authenticate the consumer's permissions, and then query the permission data to see if the subscription relationship between the consumer and the topic exists.

[0135] If the publish or subscribe relationship does not exist, the current permission authentication request is determined to be unauthorized access, the process ends, and the client cannot establish a connection with the message queue cluster.

[0136] If a publish relationship exists between the producer and the topic, or a subscription relationship exists between the consumer and the topic, then continue to query the permission data to see if a routing relationship exists between the topic and the cluster.

[0137] If it exists, the corresponding message queue cluster information (including address, such as IP and port) is returned based on the routing information so that the client can load this information to access the message queue cluster.

[0138] If it does not exist, it is determined that the current topic is not authorized for access, the authorization authentication fails, the process ends, and the client cannot establish a connection.

[0139] Figure 9 This is a schematic diagram illustrating the specific structure of another data processing system provided in this application embodiment, specifically showing the structure of the client. See also... Figure 9 A detailed introduction to the client's specific structure and functions:

[0140] In this embodiment, the authorization authentication sender / receiver processor can construct an authorization authentication request based on the topic accessed by the producer / consumer application and send it to the server for authorization authentication. It also receives message queue cluster information returned by the server and places it into the message queue cluster cache. Based on the message queue cluster information and authorization authentication success information, an authorized cluster mapping relationship is constructed, and this cluster mapping relationship is placed into the authorization data cache.

[0141] In this embodiment, the message queue client instance processor can read message queue cluster information from the message queue cluster cache, iterate through each message queue cluster in the cache, create a message queue client instance sequentially, and place the successfully created message queue client instance into the message queue client instance cache (i.e., the instance cache). Each message queue instance establishes a connection with its corresponding message queue cluster, thereby enabling simultaneous access to multiple message queue clusters. Simultaneously, the message queue client instance cache enables message queue instance reuse, reducing the performance pressure on the application and the message queue cluster. Figure 9 As shown, the message queue client instance processor constructs a message queue client instance based on the message queue cluster cache, puts it into the message queue client instance cache, reads message queue client instance 1 and message queue client instance 2 from the cache, accesses cluster 1 through message queue client instance 1, and accesses cluster 2 through message queue client instance 2.

[0142] It should be noted that the functions of the data processing system can be described in reference to the methods of the above embodiments.

[0143] The above describes a data processing method provided by an embodiment of this application. The following describes an apparatus for performing the above data processing method.

[0144] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 10 As shown, the data processing device 1000 includes: a first data processing unit 1001 configured on the server and a second data processing unit 1002 configured on at least one client. The first data processing unit includes an authentication unit 1003 and a cluster information sending unit 1004. The second data processing unit includes an authentication request unit 1005 and an operation execution unit 1006.

[0145] The authorization authentication request unit 1005 is used to: send an authorization authentication request to the first data processing unit, wherein the authorization authentication request includes the client identifier of the target client, the target topic and the target operation, and is used to request the target operation to be performed on the business data of the target topic, wherein the target client is any second data processing unit in the data processing system;

[0146] The permission authentication unit 1003 is used to: verify whether there is a permission mapping relationship in the first type of permission data that matches the permission authentication request;

[0147] The cluster information sending unit 1004 is used to: if it exists, search for target message queue cluster information matching the target topic in the second type of permission data, and send the target message queue cluster information to the first data processing unit. The first type of permission data includes multiple permission mapping relationships, and the permission mapping relationships include the correspondence between client identifier, topic and operation. The second type of permission data includes multiple routing relationships, and the routing relationships include the correspondence between topics and various message queue clusters in the message queue system.

[0148] The operation execution unit 1006 is used to: receive the target message queue cluster information and then execute the target operation in the target message queue cluster according to the target message queue cluster information.

[0149] In one possible implementation, the target operations include read and write operations, and the permission mapping relationship includes subscription and publish relationships; the first data processing unit further includes: a permission data processing unit, used for:

[0150] Upon receiving a publishing request for a first topic from the second data processing unit, and based on the second type of permission data and the dynamic load of each message queue cluster within the message queue system, determining the message queue cluster information corresponding to the first topic, constructing a first publishing relationship and a first routing relationship, and updating the first type of permission data and the second type of permission data;

[0151] Wherein, the first publishing relationship includes the correspondence between the client identifier of the first client, the first topic, and the write operation; the first routing relationship includes the correspondence between the first topic and its corresponding message queue cluster information;

[0152] Receive a subscription request for a second topic from the second data processing unit, determine whether a second routing relationship exists, and if so, construct a first subscription relationship and update the first type of permission data;

[0153] Wherein, the second routing relationship includes the routing relationship of the second topic; the first subscription relationship includes the client identifier of the second client, the second topic, and the correspondence of the read operation;

[0154] In one possible implementation, the permission data processing unit is configured to: determine the message queue cluster information corresponding to the first topic based on the second type of permission data and the dynamic load of each message queue cluster within the message queue system, specifically configured to:

[0155] Based on the business information corresponding to the first topic, predict the data volume level of the first topic;

[0156] Based on the second type of permission data, the number of topics corresponding to each message queue cluster in the message queue system is determined.

[0157] Based on the data volume level of the first topic, the number of topics corresponding to each message queue cluster in the message queue system, and the dynamic load, the message queue cluster information corresponding to the first topic is determined according to the preset load balancing rules.

[0158] In one possible implementation, the operation execution unit, after receiving the target message queue cluster information, performs the target operation on the target message queue cluster according to the target message queue cluster information, specifically by:

[0159] Receive the target message queue cluster information, and based on the target message queue cluster information, determine whether the target client instance exists in the instance cache;

[0160] If it exists, invoke the target client instance in the instance cache to perform the target operation in the target message queue cluster;

[0161] If it does not exist, create the target client instance based on the target message queue cluster information, call the target client instance to perform the target operation in the target message queue cluster, and cache the target client instance in the instance cache.

[0162] In one possible implementation, the client further includes a caching unit, which caches the target cluster mapping relationship in the permission data cache after the second data processing unit receives the target message queue cluster information. The target cluster mapping relationship includes the correspondence between the target topic, the target operation, and the target message queue cluster information.

[0163] In one possible implementation, when the permission authentication request unit sends a permission authentication request to the first data processing unit, it is specifically used to determine whether there is a cluster mapping relationship in the permission data cache that matches the permission authentication request; if not, it sends the permission authentication request to the first data processing unit.

[0164] In one possible implementation, the authentication request unit is also used for:

[0165] If a cluster mapping relationship exists that matches the permission authentication request, the target message queue cluster information is obtained from the cluster mapping relationship that matches the permission authentication request, and the target operation is performed on the target message queue cluster according to the target message queue cluster information.

[0166] In one possible implementation, the cache unit is also used for:

[0167] If the cache duration of the target client instance exceeds a preset first cache duration, the target client instance is cleaned up.

[0168] If the cache duration of the cluster mapping relationship exceeds the preset second cache duration or exceeds the preset validity period, the target client instance is cleaned up.

[0169] According to the preset query cycle, a query is sent to the first data processing unit to check whether the cluster mapping relationship is valid. If the cluster mapping relationship is invalid, the cluster mapping relationship is cleared.

[0170] This application also provides an electronic device in its embodiments. (See reference...) Figure 11 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0171] like Figure 11As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 into a random access memory (RAM) 1103. When the electronic device is powered on, the RAM 1103 also stores various programs and data required for the operation of the electronic device. The processing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0172] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1108 including, for example, memory cards, hard drives, etc.; and communication devices 1109. Communication device 1109 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 11 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0173] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the data processing methods provided in this application.

[0174] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data processing methods provided in this application.

[0175] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0177] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0178] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, Applied to a data processing system, the data processing system including a server and at least one client, including: The target client sends an authentication request to the server. The authentication request includes the client identifier, target topic, and target operation of the target client, and is used to request the target operation to be performed on the business data of the target topic. The target client can be any client in the data processing system. The server verifies whether there is a permission mapping relationship in the first type of permission data that matches the permission authentication request. The first type of permission data includes multiple permission mapping relationships, and the permission mapping relationship includes the correspondence between client identifier, topic and operation. If it exists, the server searches for the target message queue cluster information that matches the target topic in the second type of permission data, and sends the target message queue cluster information to the target client. The second type of permission data includes multiple routing relationships, and the routing relationships include the correspondence between topics and various message queue clusters in the message queue system. After receiving the target message queue cluster information, the target client executes the target operation on the target message queue cluster according to the target message queue cluster information.

2. The data processing method according to claim 1, characterized in that, The target operations include read operations and write operations; the permission mapping relationship includes subscription relationships and publish relationships; the data processing method further includes: The server receives a publishing request for a first topic from a first client. Based on the second type of permission data and the dynamic load of each message queue cluster in the message queue system, it determines the message queue cluster information corresponding to the first topic, constructs a first publishing relationship and a first routing relationship, and updates the first type of permission data and the second type of permission data. Wherein, the first publishing relationship includes the correspondence between the client identifier of the first client, the first topic, and the write operation; the first routing relationship includes the correspondence between the first topic and its corresponding message queue cluster information; The server receives a subscription request for a second topic from a second client, determines whether a second routing relationship exists, and if so, constructs a first subscription relationship and updates the first type of permission data. Wherein, the second routing relationship includes the routing relationship of the second topic; the first subscription relationship includes the client identifier of the second client, the second topic, and the correspondence of the read operation; Both the first client and the second client are any client in the data processing system.

3. The data processing method according to claim 2, characterized in that, The server determines the message queue cluster information corresponding to the first topic based on the second type of permission data and the dynamic load of each message queue cluster within the message queue system, including: Based on the business information corresponding to the first topic, predict the data volume level of the first topic; Based on the second type of permission data, the number of topics corresponding to each message queue cluster in the message queue system is determined. Based on the data volume level of the first topic, the number of topics corresponding to each message queue cluster in the message queue system, and the dynamic load, the message queue cluster information corresponding to the first topic is determined according to the preset load balancing rules.

4. The data processing method according to claim 1, characterized in that, After receiving the target message queue cluster information, the target client executes the target operation on the target message queue cluster according to the target message queue cluster information, including: The target client receives the target message queue cluster information and, based on the target message queue cluster information, determines whether a target client instance exists in the instance cache; If it exists, the target client invokes the target client instance in the instance cache to perform the target operation in the target message queue cluster; If it does not exist, the target client creates the target client instance based on the target message queue cluster information, calls the target client instance to perform the target operation in the target message queue cluster, and caches the target client instance in the instance cache.

5. The data processing method according to claim 4, characterized in that, After the target client receives the target message queue cluster information, the data processing method further includes: The target client caches the target cluster mapping relationship in the permission data cache. The target cluster mapping relationship includes the correspondence between the target topic, the target operation, and the target message queue cluster information. The target client sends an authentication request to the server, including: Determine whether a cluster mapping relationship matching the permission authentication request exists in the permission data cache. If not, send the permission authentication request to the server.

6. The data processing method according to claim 5, characterized in that, The data processing method further includes: If a cluster mapping relationship exists that matches the permission authentication request, the target message queue cluster information is obtained from the cluster mapping relationship that matches the permission authentication request, and the target operation is performed on the target message queue cluster according to the target message queue cluster information.

7. The data processing method according to claim 5, characterized in that, The data processing method further includes: If the cache duration of the target client instance exceeds a preset first cache duration, the target client will clear the target client instance. If the cache duration of the cluster mapping relationship exceeds a preset second cache duration or exceeds a preset validity period, the target client cleans up the target client instance. According to a preset query cycle, the target client requests the server to query whether the cluster mapping relationship is valid. If the cluster mapping relationship is invalid, the cluster mapping relationship is cleared.

8. A data processing apparatus, characterized in that, The data processing device includes a first data processing unit configured on the server and a second data processing unit configured on at least one client. The first data processing unit includes an authentication unit and a cluster information sending unit, and the second data processing unit includes an authentication request unit and an operation execution unit. The authorization authentication request unit is used to: send an authorization authentication request to the first data processing unit, wherein the authorization authentication request includes the client identifier of the target client, the target topic, and the target operation, and is used to request the target operation to be performed on the business data of the target topic, wherein the target client is any second data processing unit in the data processing system; The permission authentication unit is used to: verify whether there is a permission mapping relationship in the first type of permission data that matches the permission authentication request; The cluster information sending unit is used to: if it exists, search for target message queue cluster information matching the target topic in the second type of permission data, and send the target message queue cluster information to the first data processing unit. The first type of permission data includes multiple permission mapping relationships, and the permission mapping relationships include the correspondence between client identifier, topic and operation. The second type of permission data includes multiple routing relationships, and the routing relationships include the correspondence between topics and various message queue clusters in the message queue system. The operation execution unit is used to: receive the target message queue cluster information, and then execute the target operation on the target message queue cluster according to the target message queue cluster information.

9. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the data processing method as described in any one of claims 1 to 7.

Images