Big data resource service allocation management system

Abstract

The invention discloses a big data resource service allocation management system, which comprises the following steps: after receiving a user resource request, extracting user, task and resource attributes, constructing a resource candidate set, establishing attribute association among users, tasks and resources through a formal concept analysis technology, and generating a concept lattice structure, a legal allocation domain meeting constraints is exported from the concept lattice, an improved multi-arm tiger machine algorithm is introduced, each form concept in the concept lattice serves as a strategy unit to construct a concept arm, priorities of different concept arms are evaluated according to historical feedback information, and an optimal resource service combination is dynamically selected in the legal domain to execute allocation; according to the system, the concept lattice can be incrementally maintained when the attributes change, and dynamic optimization and structural evolution of the resource allocation strategy are realized. According to the method, the decision basis can be incrementally updated when the attributes change, and resource management requirements in a complex change environment are met.

Description

Technical Field

[0001] This invention relates to the field of big data resource management technology, and in particular to a big data resource service allocation and management system. Background Technology

[0002] With the widespread application of cloud computing and big data technologies, a large number of multi-source, heterogeneous, and dynamically changing data processing tasks have emerged in scenarios such as government affairs, finance, energy, and industrial internet. This has created high-concurrency, high-adaptability, and high-security collaborative management requirements for resource scheduling and service allocation. Currently, big data resource management systems generally adopt allocation mechanisms based on static policy configuration, manual rule definition, or centralized optimization models, relying on preset permission rules and service level matching methods to make resource allocation decisions.

[0003] Existing technologies have significant limitations in complex multi-user, multi-task environments. On the one hand, traditional permission rule configuration methods struggle to express the multidimensional and polymorphic access relationships between users, tasks, and resources, easily leading to policy conflicts, configuration redundancy, or compliance issues. On the other hand, scheduling algorithms based on centralized learning or global optimization struggle to balance policy legitimacy, security boundaries, and online adaptive capabilities when facing dynamic changes in resource states and real-time feedback requirements, resulting in low resource utilization efficiency and delayed policy response. Furthermore, existing methods lack structural support for feedback-driven policy evolution and dynamic updates to allocation boundaries, failing to provide the ability to finely model and continuously evolve access policies.

[0004] Therefore, how to provide a big data resource service allocation and management system is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a big data resource service allocation and management system. This invention constructs the attribute association relationship between users, tasks and resources, uses formal concept analysis to generate a legal allocation domain for resource services, and dynamically selects the optimal allocation strategy through a structured online learning mechanism. It has the advantages of clear strategy expression, strong allocation compliance and high resource utilization.

[0006] The big data resource service allocation and management system according to an embodiment of the present invention includes the following modules: The resource pool management module constructs a resource pool consisting of computing, storage, and network resources, generates resource identifiers, and stores the corresponding resource capability attributes. The request parsing module receives resource service requests and generates a request identifier, and parses the request to obtain user attributes, task attributes, service level parameters and resource requirement parameters. The resource information acquisition module reads the resource capability attributes based on the resource identifier and collects the resource operating status to form a set of basic resource information. The candidate set generation module matches and filters the set of basic resource information based on resource requirement parameters, and generates a set of candidate resources and their status parameters bound to the request identifier. The formal background construction module consists of a set of objects composed of user attributes, task attributes, and candidate resources, and an attribute set composed of service authorization attributes mapped by service level parameters and resource capability attributes mapped by status parameters. A binary relationship is established to generate the formal background. The concept lattice generation and legal domain derivation module performs formal concept analysis on the formal background to generate a concept lattice structure, and derives the legal allocation domain and constraint set of resource services bound to the request identifier from the concept lattice structure; The online decision-making and execution feedback module constructs an online decision action set based on a valid allocation domain. It selects target actions and generates allocation instructions to execute resource and service configurations based on an improved multi-armed slot machine algorithm. It collects feedback data to update the decision status and incrementally updates the concept lattice structure when the object set or attribute set changes, and outputs the resource and service allocation results.

[0007] Optionally, the resource pool management module includes: The building unit is used to access computing resources, storage resources and network resources, and establish corresponding resource sub-pools. The computing resources include node entities with processing and memory capabilities, the storage resources include data storage devices with read and write capabilities, and the network resources include connection components that provide transmission capabilities. The identifier generation unit is used to generate resource identifiers for accessed computing resources, storage resources and network resources. The resource identifier includes resource type information, logical location information and a unique number, which is used to uniquely identify each resource instance in the resource pool. The capability attribute management unit is used to associate and store the resource capability attributes of each resource identifier. The resource capability attributes include computing resources, storage resources, and network resources. The computing resources include the number of processing cores, memory capacity, virtualization support, and current load limit. The storage resources include capacity, media type, data access speed, and redundancy configuration. The network resources include maximum bandwidth, connection topology, availability level, and isolation attributes.

[0008] Optionally, the request parsing module includes: The request receiving unit is used to receive resource service requests initiated by users. The resource service requests include user identifier, task identifier, service level declaration and resource usage instructions. The identifier generation unit is used to generate a request identifier based on the resource service request. The request identifier is generated by combining the request timestamp, user identifier code and sequence number, and is used to uniquely identify the request in the system. The user attribute extraction unit is used to extract user attributes based on the user identifier. The user attributes include the user's organization, permission level, and authentication method, and are bound to the request identifier. The task attribute extraction unit is used to extract task type, task priority, expected execution time and concurrency limit information based on the task identifier, generate task attributes, and bind them to the request identifier; The service level resolution unit is used to extract service level parameters from the service level declaration, including response time targets, service availability requirements, and isolation policy configuration. The resource requirement parsing unit is used to extract resource requirement parameters from the resource usage description. These resource requirement parameters include computing specifications, storage capacity, network bandwidth, and deployment location preferences.

[0009] Optionally, the resource information acquisition module includes: The capability attribute reading unit is used to read the resource capability attributes corresponding to the resource identifier from the resource pool management module based on the resource identifier; The status acquisition unit is used to collect the operating status of the resource corresponding to the resource identifier. The operating status of the resource includes the current resource availability, load level, connection status and fault marking information. The resource information integration unit is used to structurally integrate the resource capability attributes and resource operating status of each resource to form a resource attribute record; The basic information generation unit is used to organize multiple resource attribute records into a resource basic information set, the set is grouped according to resource type, and an index mapping of resource identifiers is established.

[0010] Optionally, the candidate set generation module includes: The parameter receiving unit is used to receive resource requirement parameters bound to the request identifier, including computing specifications, storage capacity, network bandwidth, and deployment location preference. The matching execution unit is used to classify and traverse the set of basic resource information based on resource type, and perform matching filtering according to the following conditions: For computing resources, determine whether the number of processing cores and memory capacity are not lower than the computing specifications in the resource requirement parameters; For storage resources, determine whether the remaining capacity meets the storage capacity requirements and whether the media type is consistent with the types allowed by the task attributes; For network resources, determine whether the maximum bandwidth meets the network bandwidth requirements and whether the logical location information matches the deployment location preference; The filtering processing unit is used to filter resource attribute records that meet all matching conditions to form an intermediate candidate list; The status extraction unit is used to extract the resource operating status from the intermediate candidate list. The resource operating status includes the current resource availability, load level, connection status and fault marking information, which are used to form candidate status parameters. The candidate set construction unit is used to bind the resource identifiers in the intermediate candidate list to their resource running status, generate structured candidate resource objects, and organize all candidate resource objects into candidate resource sets according to resource type.

[0011] Optionally, the formal background construction module includes: The object collection construction unit is used to receive user attributes, task attributes and candidate resource sets bound to the request identifier, extract user identifier, task identifier and candidate resource identifier, and construct an object collection using the three types of identifiers as elements. The service authorization attribute generation unit is used to generate service authorization attributes based on service level parameters. The service authorization attributes include access permission level, isolation policy requirements and service availability target, which are used to describe the service boundaries that users and tasks can access. The allocation constraint attribute generation unit is used to generate allocation constraint attributes based on the resource capability attributes and resource operating status of each resource in the candidate resource set. The allocation constraint attributes include resource load limits, quota constraints, location affinity constraints, and task isolation requirements. The attribute set building unit is used to merge service authorization attributes and allocation constraint attributes to generate a unified attribute set; The binary relation establishment unit is used to establish a binary relation matrix based on the field matching relationship between the object set and the attribute set. The matching relationship includes recording a positive association for service authorization when the corresponding user attribute or task attribute in the object set satisfies the service authorization attribute, recording a positive association for resource constraint when the candidate resource set corresponding to the object set satisfies the allocation constraint attribute, and recording non-association in other cases. Formal background output unit, used to construct a formal background from a set of objects, a set of attributes and their binary relation matrix.

[0012] Optionally, the concept lattice generation and legal domain derivation module includes: A formal background receiving unit is used to receive a formal background bound to a request identifier, wherein the formal background includes a triplet consisting of an object set, an attribute set, and their binary relation matrix; The derivation operator building unit is used to calculate object derivation operators and attribute derivation operators based on object sets and attribute sets. The object derivation operator is used to map the object set to a common subset of attributes, and the attribute derivation operator is used to map the attribute set to a subset of objects that commonly satisfy the attribute subset. The closure computation unit is used to perform bidirectional closure operations on the derived operators of the object subset and the attribute subset, filter the closure pairs that satisfy the closure conditions, and form a formal concept set. Each formal concept includes an object subset and its corresponding attribute subset. The closure pairs satisfy that the object subset and the attribute subset are derived from each other. Partial order building unit is used to establish partial order relations based on the inclusion relations between subsets of objects in a formal concept set, and to build a concept lattice structure, with hierarchical links between concept lattice nodes; The concept node selection unit is used to select the target concept node that best matches the subset of objects bound to the request identifier in the concept lattice structure, and use it as the logical starting point for the legal allocation space. The legitimate domain derivation unit is used to limit the subset of objects contained in the target concept node to resource identifiers in the candidate resource set, combine the corresponding resource identifiers with task identifiers to form a resource-service combination, and output the legitimate allocation domain of the resource service. The constraint set extraction unit is used to extract service authorization attributes and allocation constraint attributes from the attribute subset of the target concept node, and form a constraint set corresponding to the legal allocation domain of the resource service.

[0013] Optionally, the improved multi-armed slot machine algorithm includes: A concept arm construction unit is used to construct each formal concept into a concept arm based on a concept lattice structure. Each concept arm corresponds to a set of object subsets and attribute subsets. The concept arm is used to represent resource service composition generation rules and serves as a structural unit for online learning. The concept arm status maintenance unit is used to collect historical execution status information associated with each concept arm. The historical execution status information consists of feedback data from resource-service instances that have been generated and executed under the concept arm. The concept arm evaluation unit is used to calculate evaluation values ​​based on the historical execution status information of each concept arm, and to measure the relative priority of each concept arm as a strategy selection unit. The evaluation value is obtained by summarizing and calculating the historical execution status information associated with the concept arm. The summarization includes weighting the response time, resource consumption and task completion status in multiple execution feedbacks to form a single value. A concept arm selection unit is used to select a target concept arm from all concept arms based on the evaluation value; The instance generation unit is used to generate a specific resource-service instance by combining the resource running status and attribute subset bound to the current request identifier within the object subset of the target concept arm. The specific resource-service instance is not used as a learning object, does not participate in the evaluation and update process, and only exists as an execution action. An action execution unit is used to execute the specific resource-service instance and complete resource and service configuration operations; The feedback collection unit collects service response information, resource usage information, and task completion status information corresponding to the executed actions, forming feedback data. The concept arm update unit is used to assign the feedback data to the target concept arm that generates the execution action, and update its historical execution state information and evaluation value. The feedback data is not bound to a specific resource-service instance and is only used for concept arm-level learning state adjustment.

[0014] Optionally, the collected feedback data updates the decision state, and incrementally updates the concept lattice structure when the object set or attribute set changes, outputting resource service allocation results including: The feedback data collection unit is used to receive service response information, resource usage information and task completion status information collected during the execution of specific resource-service instances, form structured feedback data, and establish a mapping relationship with the conceptual arm that generates the specific resource-service instance. The state update unit is used to append the structured feedback data to the historical execution state information set of the corresponding concept arm, and recalculate the evaluation value of the target concept arm based on the updated historical execution state information. The evaluation value is used as the input parameter of the subsequent concept arm selection strategy. The set change monitoring unit is used to monitor whether the user attributes, task attributes or candidate resource sets bound to the request identifier have changed. When the content of the object set or attribute set is updated, the incremental update process is triggered. The concept increment calculation unit is used to perform bidirectional closure operation of derived operators on newly added or updated elements when incremental changes in the object set or attribute set are detected, to generate new formal concepts and identify their insertion positions; The structural insertion unit is used to insert new formal concepts into the existing concept lattice structure and adjust the link structure with the adjacent concept nodes based on the partial order relationship, so as to maintain the orderliness and hierarchical integrity of the concept lattice structure and generate the updated concept lattice structure. The legal domain update unit is used to re-derive the legal allocation domain and constraint set of the resource service corresponding to the current request identifier based on the updated conceptual lattice structure, and replace the original allocation domain for subsequent policy action generation; The result generation unit is used to combine the evaluation value of the target concept arm with the updated resource service legal allocation domain, select resource-service combinations from the action set, and form the resource service allocation result for the current round.

[0015] The beneficial effects of this invention are: This invention introduces formal concept analysis technology to construct attribute relationships between users, tasks, and resources, forming a complete and consistent formal background and conceptual lattice structure. It can automatically derive the legal allocation domain and constraint set of resource services, fundamentally solving the problem of expressing complex permission relationships and policy conflicts under traditional rule configuration methods.

[0016] This invention employs a structured, improved multi-armed slot machine algorithm, which uses formal concepts in the concept lattice as strategy units to construct concept arms. It combines historical feedback to dynamically evaluate the performance of different strategies, thereby achieving adaptive allocation of resource services within compliance constraints. This effectively avoids the problems of unauthorized access and unbalanced allocation that are easily triggered in traditional reinforcement learning algorithms.

[0017] This invention designs an incremental update mechanism for concept lattices under changes in object sets and attribute sets. Combined with a feedback data-driven strategy evolution method, it supports dynamic optimization of resource allocation strategies under continuous changes in resource status and task requirements, thereby improving the system's real-time response capability and resource utilization efficiency in large-scale, multi-task environments. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 The flowchart of the big data resource service allocation and management system proposed in this invention is shown below; Figure 2 This is an execution structure diagram of the improved multi-armed slot machine algorithm in this invention for online strategy selection and feedback updates within the legal domain; Figure 3 This is a schematic diagram illustrating the process of generating a concept lattice structure and a legally allocated domain through formal concept analysis in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 The Big Data Resource Service Allocation and Management System includes the following modules: The resource pool management module constructs a resource pool consisting of computing, storage, and network resources, generates resource identifiers, and stores the corresponding resource capability attributes. The request parsing module receives resource service requests and generates a request identifier, and parses the request to obtain user attributes, task attributes, service level parameters and resource requirement parameters. The resource information acquisition module reads the resource capability attributes based on the resource identifier and collects the resource operating status to form a set of basic resource information. The candidate set generation module matches and filters the set of basic resource information based on resource requirement parameters, and generates a set of candidate resources and their status parameters bound to the request identifier. The formal background construction module consists of a set of objects composed of user attributes, task attributes, and candidate resources, and an attribute set composed of service authorization attributes mapped by service level parameters and resource capability attributes mapped by status parameters. A binary relationship is established to generate the formal background. The concept lattice generation and legal domain derivation module performs formal concept analysis on the formal background to generate a concept lattice structure, and derives the legal allocation domain and constraint set of resource services bound to the request identifier from the concept lattice structure; The online decision-making and execution feedback module constructs an online decision action set based on a valid allocation domain. It selects target actions and generates allocation instructions to execute resource and service configurations based on an improved multi-armed slot machine algorithm. It collects feedback data to update the decision status and incrementally updates the concept lattice structure when the object set or attribute set changes, and outputs the resource and service allocation results.

[0021] In this embodiment, the resource pool management module includes: The building unit is used to access computing resources, storage resources and network resources, and establish corresponding resource sub-pools. The computing resources include node entities with processing and memory capabilities, the storage resources include data storage devices with read and write capabilities, and the network resources include connection components that provide transmission capabilities. The identifier generation unit is used to generate resource identifiers for accessed computing resources, storage resources and network resources. The resource identifier includes resource type information, logical location information and a unique number, which is used to uniquely identify each resource instance in the resource pool. The capability attribute management unit is used to associate and store the resource capability attributes of each resource identifier. The resource capability attributes include computing resources, storage resources, and network resources. The computing resources include the number of processing cores, memory capacity, virtualization support, and current load limit. The storage resources include capacity, media type, data access speed, and redundancy configuration. The network resources include maximum bandwidth, connection topology, availability level, and isolation attributes.

[0022] In this embodiment, the request parsing module includes: The request receiving unit is used to receive resource service requests initiated by users. The resource service requests include user identifier, task identifier, service level declaration and resource usage instructions. The identifier generation unit is used to generate a request identifier based on the resource service request. The request identifier is generated by combining the request timestamp, user identifier code and sequence number, and is used to uniquely identify the request in the system. The user attribute extraction unit is used to extract user attributes based on the user identifier. The user attributes include the user's organization, permission level, and authentication method, and are bound to the request identifier. The task attribute extraction unit is used to extract task type, task priority, expected execution time and concurrency limit information based on the task identifier, generate task attributes, and bind them to the request identifier; The service level resolution unit is used to extract service level parameters from the service level declaration, including response time targets, service availability requirements, and isolation policy configuration. The resource requirement parsing unit is used to extract resource requirement parameters from the resource usage description. These resource requirement parameters include computing specifications, storage capacity, network bandwidth, and deployment location preferences.

[0023] In this embodiment, the resource information acquisition module includes: The capability attribute reading unit is used to read the resource capability attributes corresponding to the resource identifier from the resource pool management module based on the resource identifier; The status acquisition unit is used to acquire the operating status of resources corresponding to resource identifiers. The resource operating status includes current resource availability, load level, connection status, and fault marking information, wherein: The operational status of computing resources includes CPU utilization, memory usage, and interrupt history; The operational status of storage resources includes remaining capacity, IO rate, and read / write error rate; The operational status of network resources includes bandwidth utilization, connection packet loss rate, and link stability indicators. The resource information integration unit is used to structurally integrate the resource capability attributes and resource operating status of each resource to form a resource attribute record; Structured integration refers to merging the resource capability attributes of each resource with the collected operating status into a unified data record according to the field correspondence, organizing it in the form of key-value pairs or table fields, and creating an index according to the resource identifier; The basic information generation unit is used to organize multiple resource attribute records into a resource basic information set, the set is grouped according to resource type, and an index mapping of resource identifiers is established; Grouping by resource type refers to classifying resource attribute records in the resource basic information set into computing resource groups, storage resource groups, or network resource groups based on the resource type information contained in the resource identifier, and then organizing the storage within each group according to the order of the resource identifier.

[0024] In this embodiment, the candidate set generation module includes: The parameter receiving unit is used to receive resource requirement parameters bound to the request identifier, including computing specifications, storage capacity, network bandwidth, and deployment location preference. The matching execution unit is used to classify and traverse the set of basic resource information based on resource type, and perform matching filtering according to the following conditions: For computing resources, determine whether the number of processing cores and memory capacity are not lower than the computing specifications in the resource requirement parameters; For storage resources, determine whether the remaining capacity meets the storage capacity requirements and whether the media type is consistent with the types allowed by the task attributes; For network resources, determine whether the maximum bandwidth meets the network bandwidth requirements and whether the logical location information matches the deployment location preference; The filtering processing unit is used to filter resource attribute records that meet all matching conditions to form an intermediate candidate list; The status extraction unit is used to extract the resource operating status from the intermediate candidate list. The resource operating status includes the current resource availability, load level, connection status and fault marking information, which are used to form candidate status parameters. The candidate set construction unit is used to bind the resource identifiers in the intermediate candidate list to their resource running status, generate structured candidate resource objects, and organize all candidate resource objects into candidate resource sets according to resource type.

[0025] In this embodiment, the formal background construction module includes: The object collection construction unit is used to receive user attributes, task attributes and candidate resource sets bound to the request identifier, extract user identifier, task identifier and candidate resource identifier, and construct an object collection using the three types of identifiers as elements. The service authorization attribute generation unit is used to generate service authorization attributes based on service level parameters. The service authorization attributes include access permission level, isolation policy requirements and service availability target, which are used to describe the service boundaries that users and tasks can access. The allocation constraint attribute generation unit is used to generate allocation constraint attributes based on the resource capability attributes and resource operating status of each resource in the candidate resource set. The allocation constraint attributes include resource load limits, quota constraints, location affinity constraints, and task isolation requirements. The attribute set building unit is used to merge service authorization attributes and allocation constraint attributes to generate a unified attribute set; The binary relation establishment unit is used to establish a binary relation matrix based on the field matching relationship between the object set and the attribute set. The matching relationship includes recording a positive association for service authorization when the corresponding user attribute or task attribute in the object set satisfies the service authorization attribute, recording a positive association for resource constraint when the candidate resource set corresponding to the object set satisfies the allocation constraint attribute, and recording non-association in other cases. Formal background output unit, used to construct a formal background from a set of objects, a set of attributes and their binary relation matrix.

[0026] In this embodiment, the concept lattice generation and legal domain derivation module includes: A formal background receiving unit is used to receive a formal background bound to a request identifier, wherein the formal background includes a triplet consisting of an object set, an attribute set, and their binary relation matrix; The derivation operator building unit is used to calculate object derivation operators and attribute derivation operators based on object sets and attribute sets. The object derivation operator is used to map the object set to a common subset of attributes, and the attribute derivation operator is used to map the attribute set to a subset of objects that commonly satisfy the attribute subset. The object export operator iterates through the attribute columns associated with the object and takes the intersection of them as the export result; the attribute export operator iterates through all objects that have the attribute and extracts their common attribute set. The closure computation unit is used to perform bidirectional closure operations on the derived operators of the object subset and the attribute subset, filter the closure pairs that satisfy the closure conditions, and form a formal concept set. Each formal concept includes an object subset and its corresponding attribute subset. The closure pairs satisfy that the object subset and the attribute subset are derived from each other. First, perform the object derivation operator on the object subset to obtain the attribute subset, then perform the attribute derivation operator on the attribute subset to obtain a new object subset. Repeat this process until the object subset and the attribute subset no longer change, forming a closure result. The closure condition is that the subset of objects obtained after continuous operation of the object export and property export operators is the same as the initial subset of objects, and the corresponding property subset does not change when it is exported again. Partial order building unit is used to establish partial order relations based on the inclusion relations between subsets of objects in a formal concept set, and to build a concept lattice structure, with hierarchical links between concept lattice nodes; A partial order relation refers to the inclusion relationship between two formal concepts whose subsets of objects exist. If the subset of objects of concept A is contained in the subset of objects of concept B, then A is defined as less than or equal to B. The concept node selection unit is used to select the target concept node that best matches the subset of objects bound to the request identifier in the concept lattice structure, and use it as the logical starting point for the legal allocation space. The most matching target concept node is the concept node in the concept lattice structure whose object subset contains the user identifier, task identifier, and candidate resource identifier bound to the request identifier, and whose object subset size is the smallest. The legitimate domain derivation unit is used to limit the subset of objects contained in the target concept node to resource identifiers in the candidate resource set, combine the corresponding resource identifiers with task identifiers to form a resource-service combination, and output the legitimate allocation domain of the resource service. The constraint set extraction unit is used to extract service authorization attributes and allocation constraint attributes from the attribute subset of the target concept node, and form a constraint set corresponding to the legal allocation domain of the resource service.

[0027] In this embodiment, the improved multi-armed slot machine algorithm includes: A concept arm construction unit is used to construct each formal concept into a concept arm based on a concept lattice structure. Each concept arm corresponds to a set of object subsets and attribute subsets. The concept arm is used to represent resource service composition generation rules and serves as a structural unit for online learning. The concept arm status maintenance unit is used to collect historical execution status information associated with each concept arm. The historical execution status information consists of feedback data from resource-service instances that have been generated and executed under the concept arm. The concept arm evaluation unit is used to calculate evaluation values ​​based on the historical execution status information of each concept arm, and to measure the relative priority of each concept arm as a strategy selection unit. The evaluation value is obtained by summarizing and calculating the historical execution status information associated with the concept arm. The summarization includes weighting the response time, resource consumption and task completion status in multiple execution feedbacks to form a single value. A concept arm selection unit is used to select a target concept arm from all concept arms based on the evaluation value; The instance generation unit is used to generate a specific resource-service instance by combining the resource running status and attribute subset bound to the current request identifier within the object subset of the target concept arm. The specific resource-service instance is not used as a learning object, does not participate in the evaluation and update process, and only exists as an execution action. An action execution unit is used to execute the specific resource-service instance and complete resource and service configuration operations; Completing resource and service configuration operations refers to performing quota allocation, access permission binding, and service parameter loading operations on selected resources based on the generated specific resource-service instance, and starting or adjusting the service instance on the corresponding resource node to ensure that the resource and service configuration status is consistent with the parameters defined in the specific resource-service instance; The feedback collection unit collects service response information, resource usage information, and task completion status information corresponding to the executed actions, forming feedback data. Service response information includes actual response time and service availability; resource usage information includes CPU, memory and bandwidth utilization; and task completion status information includes execution result, error code and status flag. The concept arm update unit is used to assign the feedback data to the target concept arm that generates the execution action, and update its historical execution state information and evaluation value. The feedback data is not bound to a specific resource-service instance and is only used for concept arm-level learning state adjustment. Updating its historical execution status information and evaluation value means: adding the current round of execution feedback to the feedback dataset of the target concept arm, and recalculating the evaluation value of the concept arm based on the updated feedback data to reflect the current expectation of its strategy performance.

[0028] In this embodiment, the process of collecting feedback data to update the decision state and incrementally updating the concept lattice structure when the object set or attribute set changes, and outputting resource service allocation results, includes: The feedback data collection unit is used to receive service response information, resource usage information and task completion status information collected during the execution of specific resource-service instances, form structured feedback data, and establish a mapping relationship with the conceptual arm that generates the specific resource-service instance. Establishing a mapping relationship means recording the identifier of the selected target concept arm when generating a specific resource-service instance, and associating and storing the identifier with the corresponding generated resource-service instance for subsequent feedback on attribution and status updates; The state update unit is used to append the structured feedback data to the historical execution state information set of the corresponding concept arm, and recalculate the evaluation value of the target concept arm based on the updated historical execution state information. The evaluation value is used as the input parameter of the subsequent concept arm selection strategy. Recalculating the evaluation value of the target concept arm refers to summarizing and processing multiple execution feedbacks associated with the concept arm based on the updated historical execution status information, and generating an evaluation value that reflects its current strategy performance. The set change monitoring unit is used to monitor whether the user attributes, task attributes or candidate resource sets bound to the request identifier have changed. When the content of the object set or attribute set is updated, the incremental update process is triggered. Triggering the incremental update process means that when a new, deleted, or changed element is detected in the object collection or attribute collection, the formal concept corresponding to the affected element is recalculated and the concept grid structure update processing step is initiated. The concept increment calculation unit is used to perform bidirectional closure operation of derived operators on newly added or updated elements when incremental changes in the object set or attribute set are detected, to generate new formal concepts and identify their insertion positions; The bidirectional closure operation of the derived operator is to perform the object derived operator on the newly added or changed object subset to obtain the corresponding attribute subset, and then perform the attribute derived operator on the attribute subset to obtain the final object subset. If the two results are consistent, the concept of closure is formed. The structural insertion unit is used to insert new formal concepts into the existing concept lattice structure and adjust the link structure with the adjacent concept nodes based on the partial order relationship, so as to maintain the orderliness and hierarchical integrity of the concept lattice structure and generate the updated concept lattice structure. Adjusting the link structure with adjacent concept nodes above and below refers to updating the link pointers with the directly superior and inferior concept nodes after inserting a new formal concept, based on the inclusion of its object subset in the partial order relation, to ensure the orderliness and integrity of the concept lattice hierarchy. The legal domain update unit is used to re-derive the legal allocation domain and constraint set of the resource service corresponding to the current request identifier based on the updated conceptual lattice structure, and replace the original allocation domain for subsequent policy action generation; The result generation unit is used to combine the evaluation value of the target concept arm with the updated resource service legal allocation domain, select resource-service combinations from the action set, and form the resource service allocation result for the current round.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a smart manufacturing platform where multiple enterprise users share the same big data infrastructure, including edge computing nodes, high-performance storage servers, a containerized microservice framework, and high-speed network channels. Each user may concurrently submit a large number of data processing, model training, simulation calculation, and visualization rendering tasks at different times. These tasks have highly heterogeneous requirements for computing specifications, memory resources, storage bandwidth, and security policies.

[0030] For example, Company A submits a task involving defect detection of multiple batches of industrial camera images. This task has high priority and high requirements for GPU computing resources and I / O bandwidth. Company B initiates a task for batch processing trajectory modeling, which requires a high number of CPU cores but has a high tolerance for latency. Company C runs a predictive maintenance task involving multi-level data fusion, which has significant requirements for resource location affinity and must be deployed on computing resources close to the edge nodes of the workshop. In such scenarios, traditional static rule allocation mechanisms are difficult to adapt quickly to the diverse needs of tasks, and are prone to policy conflicts or scheduling failures when tasks explode or resource status fluctuates, resulting in resource idleness or latency backlog.

[0031] In this scenario, the big data resource service allocation and management system of this invention is deployed in the scheduling and control layer of the intelligent manufacturing platform, serving as a unified resource allocation hub to receive resource service requests from different enterprise users. The system first uses a request parsing module to extract the user's organization identifier, task identifier, declared service level (e.g., response time less than 10 seconds, high service isolation level), and resource requirement parameters (e.g., computing specifications of 8 cores and 16GB of memory, required storage of 200GB, bandwidth requirement of 1Gbps, deployment preference of Edge-Node-03). For example, when processing an image processing task from Company A, the module identifies its GPU type requirement as NVIDIA T4 and its acceptable node deployment location as the edge group, and binds these to generate a request identifier RID-0001.

[0032] The system maintains information on currently available resources through a resource pool management module and collects the running status of each resource instance in real time. The resource information acquisition module integrates data from multiple dimensions, including computing power, storage capacity, and network connection status, forming a structured set of basic resource information. Currently, the platform has 128 active resource nodes, including 64 CPU nodes, 32 GPU nodes, and 32 edge nodes. The system detected that among the GPU nodes, 10 devices currently have an idle rate higher than 70%, normal temperature, and no fault markers, meeting the resource requirements of Company A's task.

[0033] The candidate set generation module filters the aforementioned basic information, eliminating nodes that are unstable, have insufficient bandwidth, or are overloaded, ultimately determining 12 candidate resource instances. The formal context construction module combines the user attributes parsed from the request (such as tenant identifier as Tenant-A and permission level as Level-3) and task attributes (such as task category as image segmentation and task time limit as T≤15s) with the candidate resource instances to generate an object set; and generates service authorization attributes and allocation constraint attributes based on service level and resource capability parameters, forming an attribute set. The system uses this data to generate a formal context matrix.

[0034] The concept lattice generation and legal domain derivation module performs formal concept analysis on the formal background, calculates multiple formal concept nodes through the derivation operator, and establishes a concept lattice structure containing 40 nodes. The system identifies the concept node CN-005 that best matches request RID-0001. The resource objects covered by this node are all GPU edge nodes that meet the requirements of high response level and strong isolation, and contain a subset of attributes such as "GPU model matching", "latency <10ms", and "no historical faults". This node serves as the starting point for generating the legal allocation domain.

[0035] In the online decision-making module, the system initializes a set of concept arms with concept nodes as structural units, forming a total of 40 concept arms. The improved multi-armed slot machine algorithm completes policy learning through a two-layer structure: the upper layer evaluates the historical feedback performance of each concept arm, with CN-005 currently having a historical success rate of 92% and an average resource utilization rate of 83%; the lower layer combines the current requested resource status to generate multiple executable resource-service configuration instances. The system ultimately selects the resource service combination instance numbered RSI-0053 and issues an allocation instruction.

[0036] After task execution, the feedback acquisition module collects service response information (response latency 8.3ms), resource usage information (GPU utilization 85%, peak memory usage 6.8GB), and task completion status (normal completion, no interruption) as execution feedback data for this round of concept arm CN-005. The system writes this data into the concept arm's history record and re-evaluates the arm's priority assessment value (increasing it by 1.2%) for use in the next round of decision optimization.

[0037] During operation, the system also detected that the new task type attribute "trajectory compression" added by Company B did not appear in the original task attribute set. The system triggered the incremental update process of the object set, performed bidirectional closure operation and partial order structure adjustment on the affected formal concepts, added a concept node CN-048, and simultaneously expanded the concept arm set.

[0038] After 72 hours of continuous testing, the system processed approximately 4,260 service requests, successfully matched and executed 4,126 resource service instances, achieving a scheduling success rate of 96.85%. Compared to the traditional static policy system (scheduling success rate of 84.71%), the success rate increased by over 12%; average resource utilization improved by 11.3%, with average GPU utilization increasing from 63% to 74%; average service response latency decreased by approximately 17.6%, effectively ensuring task timeliness requirements; there were zero task-level unauthorized or non-compliant scheduling events, achieving a 100% policy legality compliance rate.

[0039] This embodiment demonstrates that the present invention can effectively solve problems such as difficulty in expressing strategy conflicts, poor dynamic adaptation capabilities, and lack of feedback mechanisms in resource service allocation under complex multi-user environments. It has stability, accuracy, and scalability, and is applicable to various scenarios such as government cloud, industrial cloud, and scientific research computing platform.

[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A big data resource service allocation and management system, characterized in that, Includes the following modules: The resource pool management module constructs a resource pool consisting of computing, storage, and network resources, generates resource identifiers, and stores the corresponding resource capability attributes. The request parsing module receives resource service requests and generates a request identifier, and parses the request to obtain user attributes, task attributes, service level parameters and resource requirement parameters. The resource information acquisition module reads the resource capability attributes based on the resource identifier and collects the resource operating status to form a set of basic resource information. The candidate set generation module matches and filters the set of basic resource information based on resource requirement parameters, and generates a set of candidate resources and their status parameters bound to the request identifier. The formal background construction module consists of a set of objects composed of user attributes, task attributes, and candidate resources, and an attribute set composed of service authorization attributes mapped by service level parameters and resource capability attributes mapped by status parameters. A binary relationship is established to generate the formal background. The concept lattice generation and legal domain derivation module performs formal concept analysis on the formal background to generate a concept lattice structure, and derives the legal allocation domain and constraint set of resource services bound to the request identifier from the concept lattice structure; The online decision-making and execution feedback module constructs an online decision action set based on a valid allocation domain. It selects target actions and generates allocation instructions to execute resource and service configurations based on an improved multi-armed slot machine algorithm. It collects feedback data to update the decision status and incrementally updates the concept lattice structure when the object set or attribute set changes, and outputs the resource and service allocation results.

2. The big data resource service allocation and management system according to claim 1, characterized in that, The resource pool management module includes: The building unit is used to access computing resources, storage resources and network resources, and establish corresponding resource sub-pools. The computing resources include node entities with processing and memory capabilities, the storage resources include data storage devices with read and write capabilities, and the network resources include connection components that provide transmission capabilities. The identifier generation unit is used to generate resource identifiers for accessed computing resources, storage resources and network resources. The resource identifiers include resource type information, logical location information and a unique number. The capability attribute management unit is used to associate and store the resource capability attributes of each resource identifier. The resource capability attributes include computing resources, storage resources, and network resources. The computing resources include the number of processing cores, memory capacity, virtualization support, and current load limit. The storage resources include capacity, media type, data access speed, and redundancy configuration. The network resources include maximum bandwidth, connection topology, availability level, and isolation attributes.

3. The big data resource service allocation and management system according to claim 1, characterized in that, The request parsing module includes: The request receiving unit is used to receive resource service requests initiated by users. The resource service requests include user identifier, task identifier, service level declaration and resource usage instructions. The identifier generation unit is used to generate a request identifier based on the resource service request. The request identifier is generated by combining the request timestamp, user identifier code and sequence number. The user attribute extraction unit is used to extract user attributes based on the user identifier. The user attributes include the user's organization, permission level, and authentication method, and are bound to the request identifier. The task attribute extraction unit is used to extract task type, task priority, expected execution time and concurrency limit information based on the task identifier, generate task attributes, and bind them to the request identifier; The service level resolution unit is used to extract service level parameters from the service level declaration, including response time targets, service availability requirements, and isolation policy configuration. The resource requirement parsing unit is used to extract resource requirement parameters from the resource usage description. These resource requirement parameters include computing specifications, storage capacity, network bandwidth, and deployment location preferences.

4. The big data resource service allocation and management system according to claim 1, characterized in that, The resource information acquisition module includes: The capability attribute reading unit is used to read the resource capability attributes corresponding to the resource identifier from the resource pool management module based on the resource identifier; The status acquisition unit is used to collect the operating status of the resource corresponding to the resource identifier. The operating status of the resource includes the current resource availability, load level, connection status and fault marking information. The resource information integration unit is used to structurally integrate the resource capability attributes and resource operating status of each resource to form a resource attribute record; The basic information generation unit is used to organize multiple resource attribute records into a resource basic information set, the set is grouped according to resource type, and an index mapping of resource identifiers is established.

5. The big data resource service allocation and management system according to claim 1, characterized in that, The candidate set generation module includes: The parameter receiving unit is used to receive resource requirement parameters bound to the request identifier, including computing specifications, storage capacity, network bandwidth, and deployment location preference. The matching execution unit is used to classify and traverse the set of basic resource information based on resource type, and perform matching filtering according to the following conditions: For computing resources, determine whether the number of processing cores and memory capacity are not lower than the computing specifications in the resource requirement parameters; For storage resources, determine whether the remaining capacity meets the storage capacity requirements and whether the media type is consistent with the types allowed by the task attributes; For network resources, determine whether the maximum bandwidth meets the network bandwidth requirements and whether the logical location information matches the deployment location preference; The filtering processing unit is used to filter resource attribute records that meet all matching conditions to form an intermediate candidate list; The status extraction unit is used to extract the resource operating status from the intermediate candidate list. The resource operating status includes the current resource availability, load level, connection status and fault marking information, which are used to form candidate status parameters. The candidate set construction unit is used to bind the resource identifiers in the intermediate candidate list to their resource running status, generate structured candidate resource objects, and organize all candidate resource objects into candidate resource sets according to resource type.

6. The big data resource service allocation and management system according to claim 1, characterized in that, The formal background construction module includes: The object collection construction unit is used to receive user attributes, task attributes and candidate resource sets bound to the request identifier, extract user identifier, task identifier and candidate resource identifier, and construct an object collection using the three types of identifiers as elements. The service authorization attribute generation unit is used to generate service authorization attributes based on service level parameters. The service authorization attributes include access permission level, isolation policy requirements, and service availability targets. The allocation constraint attribute generation unit is used to generate allocation constraint attributes based on the resource capability attributes and resource operating status of each resource in the candidate resource set. The allocation constraint attributes include resource load limits, quota constraints, location affinity constraints, and task isolation requirements. The attribute set building unit is used to merge service authorization attributes and allocation constraint attributes to generate a unified attribute set; The binary relation establishment unit is used to establish a binary relation matrix based on the field matching relationship between the object set and the attribute set. The matching relationship includes recording a positive association for service authorization when the corresponding user attribute or task attribute in the object set satisfies the service authorization attribute, recording a positive association for resource constraint when the candidate resource set corresponding to the object set satisfies the allocation constraint attribute, and recording non-association in other cases. Formal background output unit, used to construct a formal background from a set of objects, a set of attributes and their binary relation matrix.

7. The big data resource service allocation and management system according to claim 1, characterized in that, The concept lattice generation and legal domain derivation module includes: A formal background receiving unit is used to receive a formal background bound to a request identifier, wherein the formal background includes a triplet consisting of an object set, an attribute set, and their binary relation matrix; The derivation operator building unit is used to calculate object derivation operators and attribute derivation operators based on object sets and attribute sets. The object derivation operator is used to map the object set to a common subset of attributes, and the attribute derivation operator is used to map the attribute set to a subset of objects that commonly satisfy the attribute subset. The closure computation unit is used to perform bidirectional closure operations on subsets of objects and subsets of attributes, filter closure pairs that satisfy the closure conditions, and form a set of formal concepts. Partial order building units are used to establish partial order relations based on the inclusion relations between subsets of objects in a formal concept set, and to construct concept lattice structures; The concept node selection unit is used to select the target concept node in the concept lattice structure that best matches the subset of objects bound to the request identifier; The legitimate domain derivation unit is used to limit the subset of objects contained in the target concept node to resource identifiers in the candidate resource set, combine the corresponding resource identifiers with task identifiers to form a resource-service combination, and output the legitimate allocation domain of the resource service. The constraint set extraction unit is used to extract service authorization attributes and allocation constraint attributes from the attribute subset of the target concept node, and form a constraint set corresponding to the legal allocation domain of the resource service.

8. The big data resource service allocation and management system according to claim 1, characterized in that, The improved multi-armed slot machine algorithm includes: Concept arm building unit, used to construct each formal concept as a concept arm based on the concept lattice structure, each concept arm corresponds to a set of object subsets and attribute subsets; The concept arm status maintenance unit is used to collect historical execution status information associated with each concept arm. The historical execution status information consists of feedback data from resource-service instances that have been generated and executed under the concept arm. The concept arm evaluation unit is used to calculate the evaluation value based on the historical execution status information of each concept arm. The evaluation value is obtained by summarizing and calculating the historical execution status information associated with the concept arm. The summarization includes weighting the response time, resource consumption and task completion status in multiple execution feedbacks to form a single value. A concept arm selection unit is used to select a target concept arm from all concept arms based on the evaluation value; The instance generation unit is used to generate a specific resource-service instance by combining the resource running status and attribute subset bound to the current request identifier within the object subset of the target concept arm; An action execution unit is used to execute the specific resource-service instance and complete resource and service configuration operations; The feedback collection unit collects service response information, resource usage information, and task completion status information corresponding to the executed actions, forming feedback data. The concept arm update unit is used to assign the feedback data to the target concept arm that generates the execution action and update its historical execution status information and evaluation value.

9. The big data resource service allocation and management system according to claim 8, characterized in that, The collected feedback data updates the decision state, and the concept lattice structure is incrementally updated when the object set or attribute set changes. The output resource service allocation results include: The feedback data collection unit is used to receive service response information, resource usage information and task completion status information collected during the execution of specific resource-service instances, form structured feedback data, and establish a mapping relationship with the conceptual arm that generates the specific resource-service instance; The state update unit is used to append the structured feedback data to the historical execution state information set of the corresponding concept arm, and recalculate the evaluation value of the target concept arm based on the updated historical execution state information. The set change monitoring unit is used to monitor whether the user attributes, task attributes or candidate resource sets bound to the request identifier have changed. When the content of the object set or attribute set is updated, the incremental update process is triggered. The concept increment calculation unit is used to perform bidirectional closure operation of derived operators on newly added or updated elements when incremental changes in the object set or attribute set are detected, to generate new formal concepts and identify their insertion positions; The structural insertion unit is used to insert new formal concepts into the existing concept lattice structure and adjust the link structure with the adjacent concept nodes based on the partial order relationship to generate the updated concept lattice structure. The legal domain update unit is used to re-derive the legal allocation domain and constraint set of the resource service corresponding to the current request identifier based on the updated conceptual lattice structure. The result generation unit is used to combine the evaluation value of the target concept arm with the updated resource service legal allocation domain, select resource-service combinations from the action set, and form the resource service allocation result for the current round.

Images