Data processing method, device, equipment and computer readable storage medium

Abstract

The embodiments of the present invention provide a data processing method, apparatus, device and computer-readable storage medium, wherein different processing node groups are set to correspond to different data visibility permissions, and the processing nodes in each processing node group only cache the part of the data for which they have processing permissions according to the configured data visibility permissions, thereby avoiding repeated caching of data as much as possible, improving the utilization rate of cache resources and reducing costs; on this basis, when a data query request is received, the corresponding target processing node group can be directly determined according to the permission judgment information carried by the data query request and the correspondence between the processing node group and the data visibility permissions, thereby directly initiating a data query to the target processing node group, avoiding the use of a blind data query method for data query, and the query has a smaller data range, and is faster and more accurate, which can further improve resource utilization.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the continuous development of 5G (5th-Generation, the fifth generation of mobile communication technology) networks, the scale of networks is getting larger and larger, and network management software is facing massive data processing. With the extensive application of microservice architecture, the client of the software is becoming simpler and the computing tasks undertaken by the server are increasing. The general behavior mode of the software is that a thin client similar to a WEB browser initiates a REST (Representational State Transfer, a design and development method for network applications) request. After receiving the request, the server performs complex calculations and returns the results. When it comes to massive data and high concurrent access by clients, the server faces great pressure to respond to requests and return results in a timely manner.

[0003] Internet companies first face scenarios with thin clients (such as programs with B / S architecture), massive data, and high concurrent access. The solutions they adopt are usually to distribute requests to multiple service nodes and to use a cache architecture based on ultra-large memory. This solution divides the cache into multiple layers, and each layer of cache consists of a distributed cache cluster deployed on multiple server nodes. For example, if there are 100 million pieces of raw data in the database, and multi-user high-concurrency queries on the database cannot meet the efficiency requirements, then three layers of distributed cache are added, each consisting of multiple server nodes. When a user initiates a query, it is first diverted to the server node of the first layer of cache for blind data query. If the data hits, it is returned; otherwise, the data query request is sent to (i.e., penetrated to) the server node of the second layer of cache for blind query, and so on. In this data processing architecture, there are problems with query accuracy and efficiency; at the same time, the data cached in each layer of cache has a high degree of duplication, resulting in problems with cache resource utilization and high data processing costs. Summary of the invention

[0004] The embodiments of the present invention mainly provide a data processing method, device, equipment and computer-readable storage medium, which are intended to solve the problems of high cache data duplication, low resource utilization, high data processing cost, and low data query accuracy and efficiency in the data processing architecture of related technologies.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a data processing device, including:

[0006] A processing node group, wherein each processing node group includes at least one processing node, and different processing node groups correspond to different data visibility permissions; each processing node group receives a data stream from a data bus, and when it determines that it has the permission to process the received data stream according to its corresponding data visibility permission and the first permission judgment information carried by the data stream, it caches the data in the data stream; the data visibility permission includes configuration information indicating which data is authorized to be processed;

[0007] The node manager is used to determine the corresponding target data visibility permission according to the second permission judgment information carried by the data query request when receiving a data query request, determine the target processing node group corresponding to the target data visibility permission according to the corresponding relationship between the processing node group and the data visibility permission and the target data visibility permission, and initiate a data query to the target processing node group based on the data query request.

[0008] To solve the above technical problems, an embodiment of the present invention further provides a data processing method, including:

[0009] A processing node group is configured, wherein each processing node group includes at least one processing node, and different processing node groups correspond to different data visibility permissions; each processing node group receives a data stream from a data bus, and when it determines that it has the permission to process the received data stream according to its corresponding data visibility permission and the first permission judgment information carried by the data stream, the data in the data stream is cached; the data visibility permission includes configuration information indicating which data is authorized to be processed;

[0010] When a data query request is received, the corresponding target data visibility permission is determined according to the second permission judgment information carried by the data query request, and the target processing node group corresponding to the target data visibility permission is determined according to the correspondence between the processing node group and the data visibility permission and the target data visibility permission, and a data query is initiated to the target processing node group based on the data query request.

[0011] To solve the above technical problems, an embodiment of the present invention further provides a data processing device, including a processor, a memory, and a communication bus connecting the processor and the memory;

[0012] The memory stores a computer program, and the computer program can be executed by the processor to implement the steps of the data processing method as described above.

[0013] To solve the above technical problem, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processing node to implement the steps of the data processing method as described above.

[0014] The data processing method, apparatus, device and computer-readable storage medium provided by the embodiments of the present invention are configured with processing node groups, and different processing node groups correspond to different data visibility permissions; each processing node group can receive a data stream from a data bus, and when it determines that it has the permission to process the received data stream based on its corresponding data visibility permissions and the first permission judgment information carried by the data stream, it caches the data in the data stream; that is, the processing nodes in each processing node group only cache the part of the data that it has the processing permission to process, so that repeated caching of data can be avoided as much as possible, the utilization rate of cache resources can be improved and the cost can be reduced; on this basis, when a data query request is received, the corresponding target processing node group can be directly determined based on the second permission judgment information carried by the data query request and the correspondence between the processing node group and the data visibility permissions, so as to directly initiate a data query to the target processing node group, avoiding the use of data blind query for data query, and the query data range is smaller, the query is also faster and more accurate, and the resource utilization rate can be further improved.

[0015] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least part of the beneficial effects become obvious from the description in the specification of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the structure of a data processing device provided in Embodiment 1 of the present invention;

[0017] Figure 2 A schematic diagram of the cascade connection of processing nodes provided in the first embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a data flow processing flow provided in the first embodiment of the present invention;

[0019] Figure 4 A schematic diagram of the addressing information processing flow provided in the first embodiment of the present invention;

[0020] Figure 5 A schematic diagram of a data deletion task processing flow provided in the first embodiment of the present invention;

[0021] Figure 6 A schematic diagram of a process for sending an addressing information deletion notification provided in the first embodiment of the present invention;

[0022] Figure 7 A schematic diagram of the addressing information deletion notification processing flow provided in the first embodiment of the present invention;

[0023] Figure 8 A schematic diagram of a data migration process provided in the first embodiment of the present invention;

[0024] Fig. 9 A schematic diagram of a process flow for adding data visibility permissions provided in the first embodiment of the present invention;

[0025] Fig.10 A schematic diagram of a data query processing flow provided in the first embodiment of the present invention;

[0026] Fig.11 A schematic diagram of the structure of a data processing device provided in Embodiment 2 of the present invention;

[0027] Fig.12 A schematic diagram of processing data stream D1 provided in Embodiment 2 of the present invention;

[0028] Fig.13 A schematic diagram of processing data stream D2 provided in Embodiment 2 of the present invention;

[0029] Fig.14 A schematic diagram of processing data stream D3 provided in Embodiment 2 of the present invention;

[0030] Fig.15 A schematic diagram of data query processing provided by Embodiment 2 of the present invention;

[0031] Fig.16 This is a schematic diagram of the structure of a data processing device provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following is a further detailed description of the embodiments of the present invention through specific implementation methods combined with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] Embodiment 1:

[0034] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and implementation examples.

[0035] The data processing architecture in the related art has the problems of high cached data duplication, low resource utilization, high data processing cost, and low data query accuracy and efficiency. The present embodiment provides a data processing device with a new data processing architecture, the data processing device is provided with a processing node group, and different processing node groups correspond to different data visibility permissions, so that the processing nodes in each processing node group only cache the part of the data that they have processing permissions according to the configured data visibility permissions, thereby avoiding repeated caching of data as much as possible. When a data query request is received, it can be directly determined in which processing node group the data to be queried by the data query request is, and directly initiate a data query to the corresponding processing node group. Therefore, it can improve the utilization rate of cache resources and reduce costs, and can also improve query efficiency and accuracy.

[0036] For ease of understanding, this embodiment is described below in conjunction with Figure 1 The data processing device shown is used for illustration purposes only.

[0037] See also Figure 1 As shown, the data processing device illustrated in this embodiment includes a processing node group and a node manager, wherein:

[0038] A processing node group includes at least one processing node, and different processing node groups correspond to different data visibility permissions. In this embodiment, the data visibility permission includes configuration information that characterizes which data is authorized to be processed. For example, in Example 1, there are currently 10 network elements, three processing node groups are configured, and the first processing node group is configured to have the authority to process the data of 3 of the network elements. The configuration information corresponding to the first processing node group includes relevant configuration information for identifying the data of the 3 network elements, such as but not limited to the location information, identity information, service information, network element binding or corresponding user information of the network element. Similarly, assuming that the second processing node group is configured to have the authority to process the data of 5 of the remaining network elements, the configuration information corresponding to the second processing node group includes relevant configuration information for identifying the data of the 5 network elements. Assuming that the third processing node group is configured to have the authority to process the data of the last remaining 2 network elements, the configuration information corresponding to the third processing node group includes relevant configuration information for identifying the two 2 data. Correspondingly, the number of processing nodes configured in the first processing node group, the second processing node group and the third processing node group can also be flexibly set according to demand. For example, it can be set to 1 according to demand, or it can be set to multiple according to demand.

[0039] The above examples are based on the setting of the processing node group according to the network element. It should be understood that the setting basis of the processing node group is not limited to the network element, and can be flexibly set according to specific needs. For example, it can be set according to but not limited to at least one of the network element, service type, geographical location where the data is generated, time, and user service authority. For example, in Example 2, there are currently 20 types of business types of data, and two processing node groups are configured. The first processing node group is configured to have the authority to process the data corresponding to 12 of the business types. The configuration information corresponding to the first processing node group includes the relevant configuration information for identifying the data of these 12 business types. Similarly, assuming that the second processing node group is configured to have the authority to process the data of the remaining 8 business types, the configuration information corresponding to the second processing node group includes the relevant configuration information for identifying the data of these 8 business types.

[0040] For another example, in Example 3, there are currently 12 types of user business permissions, and three processing node groups are configured. The first processing node group is configured to have the permission to process data corresponding to two of the user business permissions. The configuration information corresponding to the first processing node group includes relevant configuration information for identifying the data of these two types of user business permissions. Similarly, assuming that the second processing node group is configured to have the permission to process data corresponding to five of the remaining ten types of user business permissions, the configuration information corresponding to the second processing node group includes relevant configuration information for identifying the data corresponding to these five types of user business permissions. Assuming that the third processing node group is configured to have the permission to process data corresponding to the last five types of user business permissions, the configuration information corresponding to the third processing node group includes relevant configuration information for identifying the data corresponding to the last five types of user business permissions.

[0041] It can be seen that in the above examples, there is no intersection between the visible permissions of sub-data corresponding to different processing node groups. In this way, it can be ensured as much as possible that the data cached and processed by different processing node groups are different, thereby avoiding repeated caching of data as much as possible, improving resource utilization and reducing costs. Of course, it should be understood that in some application scenarios, it is also possible to set that there are partial intersections between the visible permissions of sub-data corresponding to some or all processing node groups. For example, it can be set that the first processing node group in Example 2 has the authority to process data corresponding to 12 types of business types, and the second processing node group has the authority to process data of 10 types of business types (8 of which are the remaining 8 types of business types, and the other 2 are two types of business types overlapping with the first processing node group). This setting method may result in repeated caching of a small amount of data, but relative to related technologies, it can also reduce repeated caching of data to a certain extent. It can be understood that the specific setting can be flexibly set according to specific needs when applied, and it is not limited here.

[0042] In this embodiment, the configuration of the above-mentioned processing node group can be automatically configured by the node manager according to but not limited to the rules shown in the above-mentioned examples. It can also be manually configured by the user. When the configuration is completed by the node manager, the node manager can generate the corresponding processing node group according to the acquired data visibility permission configuration information. For example, in one example, the node manager is connected to the external permission management component to obtain the various information for data visibility permissions shown in the above-mentioned examples but not limited to the above-mentioned examples, and generates the corresponding processing node group according to the acquired data visibility permission configuration information.

[0043] See also Figure 1As shown, in this embodiment, each processing node group receives a data stream from the data bus, and when it determines that it has the authority to process the received data stream based on its own corresponding data visibility permissions and the first permission judgment information carried by the data stream (it should be understood that the first permission judgment information includes various information for identifying which data visibility permissions the data stream belongs to, which corresponds to the configuration information included in the data visibility permissions), it caches the data in the data stream.

[0044] In this embodiment, each processing node group can receive data streams from the data bus in parallel and process the received data streams in parallel, thereby doubling the efficiency of data cache processing. Of course, in some application scenarios, one of the processing node groups can also receive data streams from the data bus and then send them to other processing node groups in a serial or parallel manner.

[0045] For example, in the above example three, the first processing node group, the second processing node group and the third processing node group can receive data streams from the data bus in parallel. For the first processing node group, it only caches the data corresponding to the 12 business types that it has the authority to process, and does not cache the received data that does not correspond to these 12 business types. Similarly, the second processing node group and the third processing node group also each only cache the data corresponding to the business types that they have the authority to process, so that the data can be cached according to different visual data authority levels (or classifications), which can not only avoid data duplication caching as much as possible, but also directly reduce the query scope of subsequent data queries, thereby improving the accuracy and efficiency of data queries.

[0046] For example, when the node manager receives a data query request, it can determine the corresponding target data visibility permission based on the second permission judgment information carried by the data query request (it should be understood that the second permission judgment information also includes various information for identifying which data visibility permission the data flow belongs to, which corresponds to the configuration information included in the data visibility permission), determine the target processing node group corresponding to the target data visibility permission based on the correspondence between the processing node group and the data visibility permission and the target data visibility permission, and initiate a data query to the target processing node group based on the data query request.

[0047] In this embodiment, the correspondence between the processing node group and the data visibility permission can be stored locally in the node manager, or in any location that the node manager can obtain, as long as the node manager can call it normally when needed. The correspondence between the processing node group and the data visibility permission can include but is not limited to the data visibility permission corresponding to each processing node group in the above three examples.

[0048] In this embodiment, see Figure 1As shown, the data processing device may also include a cache service module, which can be connected to the outside (referring to the outside of the data processing device) to receive data query requests and send them to the node manager, that is, the data query request received by the node manager may come from the cache service module. After the node manager determines the target processing node group in the above manner, the method of initiating a data query to the target processing node group based on the data query request may include but is not limited to the following two methods:

[0049] Method 1: The node manager directly sends the data query task to the corresponding target processing node group based on the data query request. After the processing node in the target processing node obtains the corresponding data, it directly sends it to the cache service module, or sends it to the node manager, and the node manager sends it to the cache service module;

[0050] Method 2: The node manager sends the determined target processing node group information to the cache service module, and the cache service module sends the data query task to the corresponding target processing node group based on the data query request; after the processing node in the target processing node obtains the corresponding data, it directly sends it to the cache service module, or sends it to the node manager, and the node manager sends it to the cache service module.

[0051] It should be understood that the processing node in this embodiment can be implemented by a software module having the above-mentioned data processing function, or by a chip (for example, including but not limited to a central processing unit) or a device (for example, including but not limited to a server node) having the above-mentioned data processing function. Correspondingly, the node manager can also be implemented by a software module having the above-mentioned function, or by a chip or a device having the above-mentioned processing function. That is, in this embodiment, no limitation is imposed on the specific implementation forms of the processing node and the node manager.

[0052] In this embodiment, when a processing node group includes at least two processing nodes, the at least two processing nodes can be cascaded in sequence to form a stream topology chain. Correspondingly, the data visibility permission corresponding to the processing node group includes at least two sub-data visibility permissions, one processing node corresponds to one sub-data visibility permission, and different processing nodes correspond to different sub-data visibility permissions.

[0053] For example, see Figure 2As shown, it is assumed that a certain processing node group includes 3 processing nodes, which include processing node 1, processing node 2, and processing node 3, which are cascaded in sequence to form a streaming topology chain. Processing node 1 is the first-level processing node in the streaming topology chain, that is, the starting point; processing node 3 is the last-level processing node in the streaming topology chain, that is, the end point. Correspondingly, the data visibility authority corresponding to the processing node group includes 3 sub-data visibility authorities corresponding to processing node 1, processing node 2, and processing node 3, respectively, and the sub-data visibility authorities corresponding to different processing nodes are different. Of course, it should be understood that when a processing node group includes multiple processing nodes, it is not limited to the cascade method of each processing node, and a part of the processing nodes can also be connected in parallel and then cascaded with other processing nodes (similar to the series-parallel combination method in the circuit).

[0054] In some application scenarios, the sub-data visibility permissions between different processing nodes can be set to have no intersection. For example, following the above example 3, assuming that the first processing node group is Figure 2 In the processing node group shown, the sub-data visible permission corresponding to processing node 1 is to cache the data corresponding to 4 of the above 12 business types, the sub-data visible permission corresponding to processing node 2 is to cache the data corresponding to 4 of the remaining 8 business types, and the sub-data visible permission corresponding to processing node 3 is to cache the data corresponding to the last remaining 4 business types.

[0055] In other application scenarios, the sub-data visibility permissions between different processing nodes can be set to have intersections. For example, following the above example 3, assuming that the first processing node group is Figure 2 In the processing node group shown, the sub-data visible permission corresponding to processing node 1 is to cache data corresponding to 4 of the above-mentioned 12 business types, the sub-data visible permission corresponding to processing node 2 is to cache data corresponding to 4 of the remaining 8 business types, and the sub-data visible permission corresponding to processing node 3 is to cache data corresponding to 8 of the business types. Among these 8 business types, 4 are the last 4 remaining business types, and the other 4 business types are business types repeated with the first processing node group and / or the second processing node group.

[0056] Optionally, in this embodiment, for at least two processing nodes in the streaming topology chain, when there is a part of the processing nodes with greater sub-data visibility permissions than another part of the processing nodes (for example, the sub-data visibility permissions of processing node 3 in the above-mentioned other application scenario are greater than the sub-data visibility permissions of processing nodes 2 and 1), the processing nodes with greater sub-data visibility permissions are located further back in the cascade in the streaming topology chain. Of course, the cascade rules of each processing node in the streaming topology chain are not limited to the rules of this example, for example, random cascades or a method in which the processing nodes with greater sub-data visibility permissions are located further forward in the cascade in the streaming topology chain can be adopted.

[0057] In this embodiment, cache space may be allocated to each processing node respectively, and the allocated cache space may be composed of, but not limited to, the memory space and distributed cache space allocated to each processing node.

[0058] In some examples of this embodiment, when each processing node group receives a data stream from a data centerline, the first-level processing node in a stream topology chain within the processing node group may receive the data stream from the data centerline.

[0059] In some examples of the present embodiment, after a processing node in a streaming topology chain within a processing node group receives a data stream, the processing node may determine whether it has the authority to process the received data stream based on its own corresponding sub-data visual permissions and the first permission judgment information carried by the data stream. If so, the data in the data stream is cached in its own corresponding cache space, and addressing information corresponding to the data is generated and cached; otherwise, when it is determined that it is not the last-level processing node in the streaming topology chain, the data stream is sent to the next-level processing node.

[0060] In some examples of this embodiment, when a processing node caches data in a data stream in its corresponding cache space and generates addressing information corresponding to the data, the generated addressing information includes the sub-data visibility permission of the processing node itself; after generating the addressing information, the processing node can also send the addressing information to the next-level processing node when it determines that it is not the last-level processing node in the streaming topology chain; in this way, when the processing node in the streaming topology chain receives the addressing information sent by its previous-level processing node, it sends the addressing information to the next-level processing node when it determines that it is not the last-level processing node in the streaming topology chain, and when it determines that it has the permission to process the data corresponding to the addressing information based on its own sub-data visibility permission and the sub-data visibility permission included in the addressing information, it caches the addressing information in its corresponding cache space. In some application scenarios, before caching the addressing information in its corresponding cache space, the processing node can also first determine whether the same addressing information has been cached in its own cache space, and if so, it can no longer cache it. In other application scenarios, the processing node may directly use an overwriting storage method for caching even if the same addressing information is already stored.

[0061] For an example of how a processing node processes a data stream, see Figure 3 As shown, including but not limited to:

[0062] S301: A processing node receives a data stream (when the processing node is not a first-level processing node, the data stream comes from a previous-level processing node; when the processing node is a first-level processing node, the data stream comes from a previous-level processing node).

[0063] S302: The processing node determines whether it has the authority to process the data flow. If so, go to S303; otherwise, go to S304.

[0064] S303: Cache the data of the data stream, including caching the data, statistical calculation, and constructing addressing information, wherein the constructed addressing information also includes the sub-data visibility rights corresponding to the data. Then, the process goes to S307.

[0065] S304: The processing node determines whether it is the last-level processing node. If so, it goes to S305; otherwise, it goes to S306.

[0066] S305: The processing node discards the data flow.

[0067] S306: The processing node sends the data stream to the next-level processing node.

[0068] S307: The processing node determines whether it is the last-level processing node. If so, go to S309; ​​otherwise, go to S308.

[0069] S308: The processing node sends the addressing information to the next level processing node.

[0070] S309: The processing node does not send addressing information.

[0071] Optionally, in some examples, in the above S303, after constructing the address information, the processing node may first determine whether its own cache space has already cached the same address information. If so, it will no longer cache it (or use overlay caching) and send it down.

[0072] For an example processing flow after a processing node in a stream topology chain within a processing node group receives addressing information, see Figure 4 As shown, including:

[0073] S401: The processing node receives the addressing information and proceeds to S402 and S405 respectively. S402 and S405 may be executed in parallel or not.

[0074] S402: The processing node determines whether it is the last-level processing node. If so, it goes to S403; otherwise, it goes to S404.

[0075] S403: The processing node does not send the received addressing information.

[0076] S404: The processing node sends the addressing information to the next level processing node.

[0077] S405: When it is determined that the system has the authority to process the data corresponding to the addressing information according to its own sub-data visibility authority and the sub-data visibility authority included in the addressing information, if yes, go to S406; otherwise, go to S408.

[0078] S406: The processing node determines whether the same addressing information has been cached, if so, go to S408; otherwise, go to S407.

[0079] S407: The processing node caches the addressing information, which may be cached in the addressing cache space.

[0080] S408: The processing node does not cache the addressing information.

[0081] In some examples of the present embodiment, when a processing node in the streaming topology chain within the processing node group receives a data flow deletion task, when it is determined that there is data to be deleted corresponding to the data flow deletion task cached in its own cache space, the corresponding data to be deleted will be deleted from its own cache space; otherwise, when it is determined that it is not the last level processing node in the streaming topology chain, the data flow deletion task will be sent to the next level processing node. That is, in this embodiment, the data flow cached by the processing node can also be dynamically cleared to make more reasonable use of storage space. In some examples of the present embodiment, when the processing node deletes all the data corresponding to the visible permission of a sub-data in the cache according to the data flow deletion task, the processing node can also delete the addressing information corresponding to the part of the data, and construct an addressing information deletion notification to notify the processing nodes of other levels to also delete the addressing information. Avoid other processing nodes from making invalid queries based on the addressing information.

[0082] For an example processing flow after a processing node in a stream topology chain in a processing node group receives a data stream deletion task, see Figure 5 As shown, including:

[0083] S501: The processing node receives a data flow deletion task (the data flow deletion task may be received from a data bus or from a node controller).

[0084] S502: The processing node determines whether it has corresponding data cached, and if so, goes to S503; otherwise, goes to S504.

[0085] S503: The processing node clears the corresponding data from the cache space.

[0086] S504: The processing node determines whether it is the last-level processing node. If so, the process goes to S505; otherwise, the process goes to S506.

[0087] S505: The processing node does not send the data flow deletion task to the processing node at the next level.

[0088] S506: The processing node sends a data flow deletion task to the next level processing node.

[0089] In some examples of this embodiment, when a processing node in the streaming topology chain within the processing node group receives a data visible permission deletion notification, the processing node deletes the target data and target addressing information corresponding to the visible permission of the sub-data to be deleted indicated in the data visible permission deletion notification in its own cache space, and sends an addressing information deletion notification including the target addressing information to other processing nodes in the streaming topology chain;

[0090] When a processing node in a streaming topology chain receives an addressing information deletion notification, when it determines that it is not the last level processing node in the streaming topology chain, it sends an addressing information deletion notification to the next level processing node, and when it determines that the target addressing information is cached in its own cache space, it deletes the cached target addressing information. That is, in this embodiment, the rights of the processing nodes can also be dynamically adjusted and managed. The above data visible rights deletion notification can be, but is not limited to, sent directly to the processing node that needs to be adjusted by the node manager, or it can be sent to the processing node through the data bus. For an example processing flow, see Figure 6 As shown, including:

[0091] S601: The processing node receives an addressing information deletion notification (the addressing information deletion notification may be received from a data bus or from a node controller).

[0092] S602: The processing node clears the corresponding data from the cache space and generates an addressing information deletion notification.

[0093] S603: Send the addressing information deletion notification to other processing nodes in the streaming topology chain.

[0094] When a processing node in the stream topology chain within the processing node group receives an addressing information deletion notification, an example processing flow is shown in Figure 7 As shown, including:

[0095] S701: The processing node receives the addressing information deletion notification and proceeds to S702 and S705 respectively. S702 and S705 may be executed in parallel or not.

[0096] S702: The processing node determines whether it is the last-level processing node. If so, it goes to S703; otherwise, it goes to S704.

[0097] S703: The processing node does not send the received addressing information deletion notification.

[0098] S704: The processing node sends an addressing information deletion notification to a next-level processing node.

[0099] S705: The processing node determines whether it has cached the target addressing information in the addressing information deletion notification. If so, go to S706; otherwise, go to S707.

[0100] S706: The processing node deletes the target addressing information.

[0101] S707: Do not perform the addressing information deletion action, and end.

[0102] In an example of this embodiment, when a processing node in the streaming topology chain receives a notification of deletion of visible rights of sub-data, it can also migrate the data to be deleted to other processing nodes. When a processing node receives a notification of deletion of visible rights of sub-data, it is also used to generate a data migration notification containing target data and send it to other processing nodes in the streaming topology chain (including but not limited to the next level processing nodes);

[0103] When a processing node in a streaming topology chain receives a data migration notification, it determines that it has the authority to process the target data in the data migration notification based on its corresponding sub-data visibility permissions. It caches the target data in its corresponding cache space and generates and caches the addressing information corresponding to the data. Otherwise, when it determines that it is not the last level processing node in the streaming topology chain, it sends the data migration notification to the next level processing node. For a processing process, please refer to Figure 8 As shown, including:

[0104] S801: The processing node receives a data migration notification.

[0105] S802: The processing node determines whether it has the authority to process the data in the data migration notification. If so, go to S803; otherwise, go to S804.

[0106] S803: Cache the data in the data migration notification, including caching data, statistical calculation, and constructing addressing information, where the constructed addressing information also includes the sub-data visibility rights corresponding to the data. Then go to S807.

[0107] S804: The processing node determines whether it is the last-level processing node. If so, go to S805; otherwise, go to S806.

[0108] S805: The processing node discards the data migration notification.

[0109] S806: The processing node sends a data migration notification to a next-level processing node.

[0110] S807: The processing node determines whether it is the last-level processing node. If so, go to S809; otherwise, go to S808.

[0111] S808: The processing node sends the addressing information to the next level processing node.

[0112] S809: The processing node does not send addressing information.

[0113] Optionally, in some examples, in the above S803, after constructing the address information, the processing node may first determine whether its own cache space has already cached the same address information. If so, it will no longer cache it (or use overlay caching) and send it down.

[0114] In some examples of this embodiment, when a processing node in the stream topology chain within the processing node group receives a notification of data visible permission increase, the processing node may extract and cache the data corresponding to the sub-data visible permission to be increased indicated in the notification of data visible permission increase from the data pool, cache it in its own cache space, and generate and cache the addressing information corresponding to the data. Optionally, for the generated addressing information, when the processing node determines that it is not the last level processing node in the stream topology chain, it may also send the generated addressing information to the next level processing node;

[0115] or,

[0116] The addressing information of the data corresponding to the sub-data visible permission to be added indicated in the data visible permission addition notification is obtained from other processing nodes in the streaming topology chain and cached in its own cache space.

[0117] That is, in this embodiment, the rights of the processing nodes can also be dynamically expanded and managed. The above notification of increase in data visibility permissions can be, but is not limited to, directly sent by the node manager to the processing node that needs to be adjusted, or can be sent to the processing node through the data bus. When the node manager sends the notification of increase in data visibility permissions, it can instruct the processing node to be adjusted to directly extract the data corresponding to the sub-data visibility permissions to be added from the data pool (when it is determined that other processing nodes in the streaming topology chain do not have the data). When it is determined that other processing nodes in the streaming topology chain cache the data corresponding to the sub-data visibility permissions to be added, it can instruct the processing node to be adjusted to directly obtain the corresponding addressing information from the responding processing node and cache it.

[0118] For an example of a processing node processing a notification of increased data visibility permissions, see Fig. 9 As shown, including:

[0119] S901: The processing node receives a notification of adding data visibility permissions.

[0120] S902: The processing node determines whether to extract corresponding data from the data pool according to the notification of the increase of the data visibility permission. If yes, go to S903; otherwise, go to S904;

[0121] S903: The processing node extracts the corresponding data from the data pool for cache processing, including caching data, statistical calculation, and constructing addressing information. The constructed addressing information also includes the sub-data visibility rights corresponding to the data.

[0122] S904: The processing node obtains and caches the corresponding addressing information from other level processing nodes indicated in the data visibility permission increase notification.

[0123] As shown in the above example, in this embodiment, the node manager can directly issue data query tasks to the corresponding target processing node group based on the data query request, and can also issue data query tasks to the corresponding target processing node group based on the data query request through the cache service module. And in an example of the present embodiment, the node manager can determine the target data visibility permissions based on the second permission judgment information carried in the data query request, and according to the corresponding relationship between the processing node and the sub-data visibility permissions contained in the corresponding relationship between the processing node group and the data visibility permissions, determine the target processing node corresponding to the target data visibility permissions, and initiate a data query task to the target processing node based on the data query request through any of the two methods described in the above examples, but not limited to. For an example of the data query request processing process, please refer to Fig.10 As shown, including:

[0124] S1001: The cache service module BFF API receives a data query request and sends the data query request to the node manager.

[0125] S1002: The node manager determines the corresponding target processing node and sends it to the cache service module.

[0126] S1003: The cache service module directly initiates a data query task to the target processing node.

[0127] S1004: The target processing node confirms whether its own cache space has cached all the data requested by the data query task. If so, go to S1005; otherwise, go to S1006.

[0128] S1005: extract corresponding data from the cache space of the target processing node and send it to the cache service module.

[0129] S1006: The target processing node obtains corresponding data from other level processing nodes according to the corresponding addressing information, and sends the data to the cache service module after obtaining all the data.

[0130] It can be seen that the data processing device provided in this embodiment has at least the following advantages:

[0131] The data processing device caches the data received from the data bus in a hierarchical cache mode according to data visible permissions, and the total amount of data remains unchanged; and compared with the architecture of related technologies, it can reduce or even avoid the generation of repeated data cache, greatly reducing resource consumption;

[0132] When the data processing device receives a data query request, it can directly determine the corresponding target processing node group according to the second authority judgment information carried by the data query request and the corresponding relationship between the processing node group and the data visible authority, thereby directly initiating a data query to the target processing node group, avoiding the use of blind data query for data query, and the query data range is smaller, and the query is faster and more accurate, which can further improve resource utilization. In addition, when querying data, the corresponding processing node can be addressed once through addressing information between each processing node, reducing overhead; and when querying data, an addressing operation is generated, because the processing node with smaller data visible authority on the streaming topology chain is always addressed, the data set cached by the addressed processing node is smaller, which can further increase the hit efficiency.

[0133] In addition, the processing nodes on the streaming topology chain intercept data according to data visibility rights, which can also greatly reduce the amount of data interaction between systems.

[0134] Embodiment 2:

[0135] The present embodiment provides a data processing method, including: configuring a processing node group, wherein a processing node group includes at least one processing node, and different processing node groups correspond to different data visibility permissions; each processing node group receives a data stream from a data bus, and when it determines that it has the permission to process the received data stream based on its own corresponding data visibility permissions and first permission judgment information carried by the data stream, caches the data in the data stream; the data visibility permissions include configuration information that characterizes which data is authorized to be processed; wherein the configuration of the processing node group can be configured by but is not limited to the node processor configuration, and some examples can be shown in the above embodiments.

[0136] When a data query request is received, the corresponding target data visual permission is determined according to the second permission judgment information carried by the data query request, and the target processing node group corresponding to the target data visual permission is determined according to the corresponding relationship between the processing node group and the data visual permission and the target data visual permission, and a data query is initiated to the target processing node group based on the data query request. The data query process is shown in the above embodiment and will not be repeated here.

[0137] For ease of understanding, this embodiment below illustrates the data processing method with reference to a specific application example.

[0138] In this application example, it is assumed that the external condition management component is a user authority management component. It includes two regions, Central China and North China. There are two users in the Central China region: user U1 has authority over network element ME1, user U2 has authority over network element ME3, and user U2 has authority over network elements ME1, ME2, and ME3. There are two users in the North China region: user U4 has authority over network element ME4, and user U5 has authority over network elements ME5 and ME6. For this application scenario, please refer to a data processing architecture. Fig.11 As shown, it includes: the node manager queries the user authority information from the user authority management component. The node manager generates data visibility authority 1 (ie, Huazhong visibility authority group) and data visibility authority 2 (ie, Huabei visibility authority group) according to the queried user authority information.

[0139] The node manager configures the processing node group according to data visibility permission 1 and data visibility permission 2, where data visibility permission 1 corresponds to processing node group PG1, and data visibility permission 2 corresponds to processing node group PG2. Processing nodes P1, P2, and P3 are added to processing node group PG1 and cascaded in sequence; the constructed streaming topology chain has P1 as the starting point and P3 as the end point. The sub-data visibility permission of processing node P1 is the data of network element ME1, and the sub-data visibility permission of processing node P2 is the data of network element ME3; the sub-data visibility permission of processing node P3 is the data of network elements ME1, ME2, and ME3. Processing nodes P4 and P5 are added to processing node group PG2, and the streaming topology constructed in sequence is P3 as the starting point and P5 as the end point. The sub-data visibility permission of processing node P3 is the data of network element ME4, and the sub-data visibility permission of processing node P4 is the data of network elements ME5 and ME6.

[0140] For ease of understanding, the following Fig.11 The processing node group PG1 shown in FIG is used as an example for explanation. Based on the above example, it can be seen that the processing node group PG1 processes nodes P1, P2, and P3. Suppose the sub-data visibility permissions corresponding to the processing nodes P1, P2, and P3 are R1, R2, and R3 respectively; according to Fig.11 As shown, R1 and R2 have no intersection, and R1 and R2 are subsets of R3. The process of processing node group PG1 processing data flow (the following is an example of alarm data) is described as follows. Fig.12 As shown:

[0141] Processing node P1 receives alarm data D1 from the data bus. Assume that the data visibility permission of D1 corresponds to R1.

[0142] D1 flows through the processing node P1. After the processing node P1 confirms that D1 corresponds to R1, it caches D1, increases the statistical count by one, generates addressing information ADDR1 containing R1, and sends ADDR1 to the next-level processing node P2.

[0143] ADDR1 flows through processing node P2, which checks and confirms that R2 does not contain R1 and transparently transmits ADDR1 to processing node P3.

[0144] ADDR1 is transmitted to the processing node P3. After the processing node P3 confirms that its corresponding R3 contains R1, ADDR1 is cached and the process ends.

[0145] exist Fig.12 Based on the example shown, please refer to Fig.13 As shown, including:

[0146] The processing node P1 receives the alarm data D2 from the data bus. Assume that the data visibility permission of D2 corresponds to R2.

[0147] D2 flows through processing node P1. Processing node P1 confirms that it has no authority to process D2 and transparently transmits D2 to processing node P2.

[0148] D2 flows through processing node P2. After processing node P2 confirms that D2 corresponds to R2, it caches D2, increases the statistical count by one, generates ADDR2 containing R2 addressing information, and sends ADDR2 to processing node P3.

[0149] ADDR2 flows through processing node P3. After processing node P3 checks and confirms that its corresponding R3 contains R1, it caches ADDR2 and the process ends.

[0150] exist Fig.12 and Fig.13 Based on the example shown, for the processing of reporting new alarm data D3, please refer to Fig.14 As shown, including:

[0151] The processing node P1 receives the alarm data D3 from the data bus. Assume that the data visibility permission of D3 corresponds to R1.

[0152] D3 flows through processing node P1. After processing node P1 confirms that D3 corresponds to R1, it caches D3, increases the statistical count by one, generates addressing information ADDR1 containing R1, and sends ADDR1 to P2.

[0153] ADDR1 flows through processing node P2. After processing node P2 checks the visibility rights, it transparently transmits ADDR1 to processing node P3.

[0154] ADDR1 flows through processing node P3. After processing node P3 confirms that its corresponding R3 contains R1, it finds that ADDR1 has been cached and discards the new ADDR1. Of course, the new ADDR1 can also be used to overwrite the old ADDR1, and the process ends.

[0155] exist Fig.12 , Fig.13and Fig.14 Based on the example shown, assume that processing node P3 has cached alarm data D4 that is not included in the data visibility rights of R1 and R2, and assume that D4 corresponds to R3. Fig.15 As shown, including:

[0156] User U3, who has the right to view R3 data, initiates a query request.

[0157] The cache service module API receives the query request from U3, obtains the processing node P3 with R3 data visibility permission as the target processing node through the node manager, and initiates a query request to the processing node P3 (that is, a way of issuing data query tasks).

[0158] Processing node P3 receives the query request, checks its own cache, and finds through the addressing information that some data is cached by processing nodes P1 and P2. It then initiates a query request to P1 and P2, including the addressing information corresponding to this part of the data.

[0159] Processing nodes P1 and P2 return data to processing node P3, which aggregates its own cache data, aggregates the statistical information returned by P1 and P2 with its own statistical information, and returns it to the cache service module API, which returns it to U3.

[0160] This embodiment also provides a data processing device, which can be, but is not limited to, a data processing platform or a server. Fig.16 As shown, it includes a processor 1601, a memory 1602 and a communication bus 1603 connecting the processor 1601 and the memory 1602;

[0161] The memory 1602 stores a computer program, which can be executed by the processor 1601 to implement the steps of the data processing method as described above.

[0162] This embodiment further provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processing node to implement the steps in the data processing method shown in the above embodiments.

[0163] The computer-readable storage medium in this embodiment may be arranged on, but not limited to, a base station, for example, on a BBU of the base station, and includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0164] This embodiment also provides a computer program (or computer software), which can be distributed on a computer-readable medium and executed by a computing device to implement the steps in the data processing method as described above; and in some cases, at least one of the steps shown or described can be executed in an order different from that described in the above embodiment.

[0165] This embodiment further provides a computer program product, including a computer readable device, on which any computer program as shown above is stored. In this embodiment, the computer readable device may include the computer readable storage medium as shown above.

[0166] It can be seen that those skilled in the art should understand that all or some of the steps, systems, and functional modules / units in the above disclosed methods can be implemented as software (which can be implemented with computer program code executable by a computing device), firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processing node, such as a central processing node, a digital signal processing node, or a microprocessing node, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit.

[0167] In addition, it is well known to those skilled in the art that communication media generally contain computer readable instructions, data structures, computer program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery media. Therefore, the present invention is not limited to any specific hardware and software combination.

[0168] The above contents are further detailed descriptions of the embodiments of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A data processing device, characterized in that: include: A processing node group, wherein each processing node group includes at least one processing node, and different processing node groups correspond to different data visibility permissions; Each of the processing node groups receives a data stream from the data bus, and when it determines that it has the authority to process the received data stream according to its corresponding data visibility authority and the first authority judgment information carried by the data stream, caches the data in the data stream; the data visibility authority includes configuration information indicating which data is authorized to be processed; The node manager is used to determine the corresponding target data visibility permission according to the second permission judgment information carried in the data query request when receiving the data query request, determine the target processing node group corresponding to the target data visibility permission according to the corresponding relationship between the processing node group and the data visibility permission and the target data visibility permission, and initiate a data query to the target processing node group based on the data query request; Wherein, when a certain processing node group includes at least two processing nodes, the at least two processing nodes are sequentially cascaded to form a streaming topology chain, and the corresponding data visibility permissions in the processing node group include at least two sub-data visibility permissions, one processing node corresponds to one sub-data visibility permission, and different processing nodes correspond to different sub-data visibility permissions; After receiving the data stream, the processing node in the streaming topology chain determines that it has the authority to process the received data stream based on its own corresponding sub-data visual permissions and the first permission judgment information carried by the data stream. It caches the data in the data stream in its own corresponding cache space and generates and caches the addressing information corresponding to the data; otherwise, when it determines that it is not the last-level processing node in the streaming topology chain, it sends the data stream to the next-level processing node.

2. The data processing device according to claim 1, characterized in that When, among at least two processing nodes of the streaming topology chain, a part of the processing nodes has greater sub-data visibility authority than another part of the processing nodes, the processing nodes with greater sub-data visibility authority are positioned later in the cascade in the streaming topology chain.

3. The data processing device according to claim 2, characterized in that When the processing node caches the data in the data stream in its corresponding cache space and generates addressing information corresponding to the data, the generated addressing information includes the sub-data visibility authority of the processing node itself; After generating the addressing information, the processing node sends the addressing information to the next level processing node when determining that it is not the last level processing node in the streaming topology chain; After receiving the addressing information sent by the previous-level processing node, the processing node in the streaming topology chain sends the addressing information to the next-level processing node when it determines that it is not the last-level processing node in the streaming topology chain, and caches the addressing information in its corresponding cache space when it determines that it has the authority to process the data corresponding to the addressing information based on its own sub-data visibility permissions and the sub-data visibility permissions included in the addressing information.

4. The data processing device according to any one of claims 1 to 3, characterized in that: When a processing node in the streaming topology chain receives a data stream deletion task, if it determines that there is data to be deleted corresponding to the data stream deletion task cached in its own cache space, it will delete the corresponding data to be deleted from its own cache space; otherwise, if it determines that it is not the last-level processing node in the streaming topology chain, it will send the data stream deletion task to the next-level processing node.

5. The data processing device according to any one of claims 1 to 3, characterized in that: When a processing node in the streaming topology chain receives a data visible permission deletion notification, the processing node deletes the target data and target addressing information corresponding to the sub-data visible permission to be deleted indicated in the data visible permission deletion notification in its own cache space, and sends an addressing information deletion notification including the target addressing information to other processing nodes in the streaming topology chain; When a processing node in the streaming topology chain receives an addressing information deletion notification, when it determines that it is not the last-level processing node in the streaming topology chain, it sends the addressing information deletion notification to the next-level processing node, and when it determines that the target addressing information is cached in its own cache space, it deletes the cached target addressing information.

6. The data processing device according to claim 5, characterized in that When a processing node in the streaming topology chain receives a notification of deletion of the visible permission of the sub-data, the processing node is further used to generate a data migration notification containing the target data and send it to other processing nodes in the streaming topology chain; When a processing node in the streaming topology chain receives a data migration notification, it determines that it has the authority to process the target data in the data migration notification according to its corresponding sub-data visibility authority, caches the target data in its corresponding cache space, and generates and caches addressing information corresponding to the data; Otherwise, when it is determined that the node itself is not the last level processing node in the streaming topology chain, the data migration notification is sent to the next level processing node.

7. The data processing device according to any one of claims 1 to 3, characterized in that: When a processing node in the streaming topology chain receives a notification of adding data visibility permissions, it extracts and caches data corresponding to the sub-data visibility permissions to be added indicated in the notification of adding data visibility permissions from a data pool, caches the data in its own cache space, and generates and caches addressing information corresponding to the data; or, The addressing information of the data corresponding to the sub-data visible permission to be added indicated in the data visible permission addition notification is obtained from other processing nodes in the streaming topology chain and cached in its own cache space.

8. The data processing device according to any one of claims 1 to 3, characterized in that: There is no intersection between the sub-data visible permissions corresponding to different processing node groups.

9. The data processing device according to any one of claims 1 to 3, characterized in that: The node manager determines the target processing node corresponding to the target data visibility permission based on the target data visibility permission and the correspondence between the processing node and the sub-data visibility permission contained in the correspondence between the processing node group and the data visibility permission, and initiates a data query task to the target processing node based on the data query request.

10. The data processing device according to any one of claims 1 to 3, characterized in that: The processing node group is generated by the node manager according to the acquired data visibility permission configuration information.

11. A data processing method, comprising: Configuring a processing node group, wherein one processing node group includes at least one processing node, and different processing node groups correspond to different data visibility permissions; Each of the processing node groups receives a data stream from the data bus, and when it determines that it has the authority to process the received data stream according to its corresponding data visibility authority and the first authority judgment information carried by the data stream, caches the data in the data stream; the data visibility authority includes configuration information indicating which data is authorized to be processed; When receiving a data query request, determining the corresponding target data visibility permission according to the second permission judgment information carried in the data query request, determining the target processing node group corresponding to the target data visibility permission according to the corresponding relationship between the processing node group and the data visibility permission and the target data visibility permission, and initiating a data query to the target processing node group based on the data query request; Wherein, when a certain processing node group includes at least two processing nodes, the at least two processing nodes are sequentially cascaded to form a streaming topology chain, and the corresponding data visibility permissions in the processing node group include at least two sub-data visibility permissions, one processing node corresponds to one sub-data visibility permission, and different processing nodes correspond to different sub-data visibility permissions; After receiving the data stream, the processing node in the streaming topology chain determines that it has the authority to process the received data stream based on its own corresponding sub-data visual permissions and the first permission judgment information carried by the data stream. It caches the data in the data stream in its own corresponding cache space and generates and caches the addressing information corresponding to the data; otherwise, when it determines that it is not the last-level processing node in the streaming topology chain, it sends the data stream to the next-level processing node.

12. A data processing device, characterized in that: comprising a processor, a memory and a communication bus connecting the processor and the memory; The memory stores a computer program, and the computer program can be executed by the processor to implement the steps of the data processing method according to claim 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a first computer program, which can be executed by the first processing node to implement the steps of the data processing method according to claim 11.

Images