Data security acquisition scheduling system based on big data

Abstract

The embodiment of the invention provides a data security acquisition scheduling system based on big data, and the method comprises the steps: an encryption strategy determination step: customizing an encryption rule through the data type and sensitivity analysis of a big data source; a data acquisition step: encrypting information extracted from a data source by using the determined encryption strategy; based on application steps of a task scheduling algorithm with dual requirements of security and efficiency, the algorithm optimizes resource utilization and distributes encrypted data sets among multiple nodes to ensure a rapid and safe data processing process. Through the scheme of the embodiment of the invention, the problem of how to safely and efficiently perform data acquisition and scheduling in a big data environment can be solved.

Description

Technical Field

[0001] The present application relates to data security and privacy protection, and specifically to a data security collection and scheduling system based on big data. Background Art

[0002] In response to the application needs of big data in the current information society, the data security acquisition and scheduling system based on big data is a specially designed technical solution to ensure that the required data information can be efficiently and securely acquired, processed and transmitted in a big data environment. This method monitors and analyzes big data streams, uses encryption technology, authentication mechanisms, and efficient storage management and data scheduling algorithms to achieve comprehensive protection and optimal use of data.

[0003] However, this field still faces some key issues that need to be solved: First, to achieve high-throughput data transmission while ensuring big data privacy and compliance, it is necessary to further explore secure encryption technologies and access control strategies applicable to various application scenarios; second, to coordinate concurrent processing between multiple data sources in a distributed architecture environment, it is necessary to strengthen the research of intelligent algorithms to achieve automated fault diagnosis and self-healing capabilities; third, massive and complex data sets bring processing performance bottlenecks, so it is necessary to design a more powerful and flexible software and hardware infrastructure support system to support efficient and reliable scheduling operations. Therefore, carrying out safe and effective data acquisition and dispatching work in the context of big data is still a very challenging and promising research direction. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a data security collection and scheduling system based on big data, which at least partially solves the problems existing in the prior art.

[0005] The data security collection and dispatching system based on big data includes:

[0006] A step of determining an encryption strategy, in which encryption rules are customized by analyzing the data type and sensitivity of the big data source;

[0007] A data collection step, using the encryption strategy determined above to encrypt the information extracted from the data source;

[0008] The application steps of the task scheduling algorithm based on the dual requirements of security and efficiency, the algorithm optimizes resource utilization and distributes encrypted data sets among multiple nodes to ensure fast and secure data processing.

[0009] Preferably, the step of determining an encryption strategy specifically includes:

[0010] Analyzing data types and structural characteristics of the big data sources to identify key information domains;

[0011] Defining a corresponding encryption level according to the importance level of the key information domain;

[0012] Determine the encryption algorithms and key lengths used to implement each encryption level;

[0013] Develop abnormal data detection mechanisms to prevent illegal access of malicious data and adjust encryption rules accordingly;

[0014] Regularly review the effectiveness of encryption rules and dynamically update them based on the current security situation.

[0015] Preferably, the data collection step specifically includes:

[0016] Use predetermined data extraction criteria to capture target data packets from large data sets from different sources;

[0017] Verify that the metadata in the data package is consistent with the predetermined standards to ensure its validity;

[0018] Encrypt the collected target data packets using corresponding algorithms according to encryption strategies;

[0019] Generate a digital signature for the encrypted data packet to ensure integrity, and attach a timestamp to ensure the accuracy of time records;

[0020] The encrypted and digitally signed data packets are safely stored in the intermediate buffer waiting for task scheduling.

[0021] Preferably, the application steps of the task scheduling algorithm specifically include:

[0022] Execute a priority evaluation algorithm on the received encrypted data set to determine the task scheduling order;

[0023] Determine the currently available computing power of data processing nodes and predict the estimated time required to complete specific processing;

[0024] Use a distributed consistent hashing algorithm to determine the storage location of data blocks and balance the load distribution to each data processing node;

[0025] Apply fault-tolerance algorithms to detect failed data node tasks and redirect unfinished subtasks through a replication mechanism;

[0026] When the processed data set is uploaded to the result aggregation node, encryption is used to ensure the security of the communication link in a multi-node environment.

[0027] Preferably, the application steps of the task scheduling algorithm based on the dual requirements of security and efficiency include:

[0028] Determine the mission's importance based on pre-set security and encryption levels;

[0029] Prioritize the scheduling of processor resources for encryption processing for tasks with high priority levels;

[0030] If the importance level of the task does not reach the predetermined safety threshold, the task is placed in the waiting queue and marked as a task with low safety requirements;

[0031] For tasks in the waiting queue, resources are called for processing according to priority when safety requirements are met.

[0032] Preferably, the task scheduling based on the dual requirements of safety and efficiency further includes:

[0033] Set the encryption strength function (F_{encrypt}(Data,SecurityLevel)) to represent the encryption strength determined according to the security level;

[0034] When the task (T i SecurityLevel(T i )>MinimumSecurity), a higher encryption algorithm level is used to increase the output value of (F_{encrypt}) to enhance the security factor of the task;

[0035] Otherwise, use basic or standard encryption to reduce the output of (F_{encrypt}) in order to improve processing efficiency;

[0036] In (F_{encrypt}(T i ,SecurityLevel(T i ))) must be greater than or equal to (MinimumEncryptedStrength) before (T i ) is placed in the scheduling task pool to wait for further processing.

[0037] Preferably, the specific steps of dynamically adjusting the priority sequence of tasks according to security requirements include:

[0038] Set security score (SScore) = (BaseScore + (F_{encrypt}(Task,SecurityLevel(Task))Thr esholdLevel)) to evaluate task security performance;

[0039] Assume that a task to be scheduled (Task j SScore j geMinSScore), then improve the sorting position of (Task_j) in the waiting queue to ensure that higher security requirements are executed first;

[0040] Conversely, if the (SScore of a certain task (Task_k) k < MinSScore), then it is lowered in the waiting queue to reduce the processor time and increase the throughput of the scheduler;

[0041] The tasks updated based on the security score are re - sorted by (SScore) to ensure that the encryption requirements of the tasks can be considered preferentially.

[0042] Preferably, the specific method for realizing dynamic allocation of multi - node resources and task scheduling includes the steps:

[0043] Calculate the available resource ratio (Availability_i) of all running nodes and set a minimum resource threshold (Threshold_{Resource}) to avoid overloading a single node;

[0044] If (Availability_i >= Threshold_{Resource}), then select node (i) as the encryption or task execution node and send the waiting tasks to this node;

[0045] Otherwise, select a node with more sufficient resources according to the resource situation and consider load balancing to make the resources be evenly used;

[0046] If the cumulative time when (SScore_n < MinimumScoreForHighPriority) on node (n) exceeds (TimeThreshold), then transfer all tasks in the waiting queue of low - priority and non - security - related currently to other nodes that meet higher security conditions for processing to ensure the balance and stability of processing capabilities.

[0047] The embodiments of the present disclosure provide a data security collection and scheduling system based on big data, including: a step of determining an encryption policy, customizing encryption rules by analyzing the data type and sensitivity of the big data source; a data collection step, encrypting the information extracted from the data source using the determined encryption policy above; an application step of a task scheduling algorithm based on dual requirements of security and efficiency, which optimizes resource utilization and distributes the encrypted data sets among multiple nodes to ensure a fast and secure data processing process. Through the solution of the embodiments of the present disclosure, the problem of how to perform data collection and scheduling safely and efficiently in a big data environment can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0049] Figure 1 This is a flow chart of the data security collection and scheduling system based on big data;

[0050] Figure 2 It is a detailed flowchart of the steps for determining an encryption strategy;

[0051] Figure 3 It is a flow chart with detailed steps of data collection;

[0052] Figure 4 It is a detailed flowchart of the application steps of the task scheduling algorithm;

[0053] Figure 5 It is a flow chart of the application steps of the task scheduling algorithm based on the dual requirements of safety and efficiency;

[0054] Figure 6 It is a further flowchart of task scheduling based on the dual requirements of safety and efficiency;

[0055] Figure 7 It is a flowchart of the specific steps for dynamically adjusting the priority sequence of tasks according to safety requirements;

[0056] Figure 8 It is a flowchart of the specific method for realizing dynamic allocation of multi-node resources and task scheduling, which includes the following steps: DETAILED DESCRIPTION

[0057] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0058] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0059] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0060] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0061] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0062] The data security collection and scheduling system based on big data is a solution for the secure transmission and efficient processing of massive and diverse data in the current Internet environment. Its key steps mainly include: formulating encryption strategies to ensure data security, encrypting data through specific methods, and realizing efficient transmission and utilization of data in a distributed processing environment through reasonable scheduling algorithms while ensuring security.

[0063] Steps to determine an encryption strategy:

[0064] Encryption strategy refers to the process and guidelines of customizing encryption by differentiating the types of information in big data sources and according to the importance or confidentiality (sensitivity) of the information. This step is the forefront of data security.

[0065] For example, a company needs to protect its users' personal information from being illegally accessed and used. First, analyze the company's main data streams. For example, user names and contact information are medium-level private data, while users' specific financial information such as bank account numbers require more stringent security measures and are highly sensitive information. Based on these analyses, different levels or mechanisms of encryption are used for different types of data to ensure that even if sensitive information is intercepted during transmission, the specific content cannot be directly parsed.

[0066] A data collection step:

[0067] Once the encryption strategy is determined, the next step is to start collecting data and implement pre-designed encryption technology on the collected data to prevent data leakage during transmission and subsequent analysis.

[0068] Taking the above enterprise as an example, it may have data from multiple sources to be integrated and analyzed, such as social media comments and transaction records in internal databases. These data are immediately processed according to pre-set encryption standards when they are obtained (such as using DES to encrypt medium-risk data and using the RSA algorithm to process high-level sensitive data). The encryption process ensures that even if someone tries to eavesdrop on the transmission or processing of data, the information obtained will be worthless. This helps companies ensure privacy compliance while analyzing user behavior data and market trends.

[0069] Apply scheduling algorithms based on safety and efficiency requirements:

[0070] Data security and efficiency are the most critical requirements in a big data environment. An effective scheduling algorithm can ensure data scheduling that is both secure and efficient. For example, using the Dijkstra algorithm or priority queue as the basis for scheduling, the encrypted data packets are sent to the designated computing unit for processing under the premise that node resources allow. This helps to maximize the use of resources.

[0071] Assume that an enterprise uses a distributed storage network consisting of multiple servers, and each node can undertake computing tasks. The factors that the scheduling algorithm needs to consider include not only the data transmission time, but also the actual load of the target node and the communication cost between nodes. In order to optimize this, a scoring system can be designed for the algorithm to assign scores to indicators with different weights. For example, nodes with closer distances can be given higher scores to reduce the impact of communication delay time; and nodes with strong processing capabilities can also get more points to fully utilize resources and shorten the total processing time. In addition, machine learning can be introduced to predict the future load of nodes to make scheduling more accurate and real-time. Doing so can not only improve the response efficiency of the entire cluster, but also reduce the extra workload caused by repeated operations.

[0072] In summary, the data security collection and efficient scheduling method based on big data can ensure that enterprises or other organizations can effectively manage their massive information, while also taking into account the requirements of data security and transmission performance. This method not only provides a reliable solution framework for the enterprise, but also proves to the outside world that the company is strict and responsible in data governance, further improving user trust and enhancing the competitiveness of the enterprise.

[0073] Next, the specific steps for determining the encryption strategy are described.

[0074] In a typical implementation case, determining an effective encryption strategy is a critical step in the design of a secure collection and dispatch system based on big data. This process aims to protect sensitive data from unauthorized access and ensure the confidentiality and integrity of key information. It is mainly divided into the following steps:

[0075] 1. First, it is necessary to conduct a detailed analysis of large data sets from different sources. The focus of this stage is to understand the specific types of data sources (such as transaction records, social media interactions, sensor outputs, etc.) and the characteristics of the data structure. Through in-depth analysis, it is possible to accurately identify key areas or fields that need to be encrypted and protected. For example, a medical information system may contain a large amount of patient health records and medical records. At this time, fields such as patient diagnosis and test results and medication details are usually regarded as highly critical information with high security requirements, so they will be marked as the scope of subsequent key encryption.

[0076] 2. Secondly, once the critical information domains are identified, the corresponding encryption requirements need to be formulated according to the different levels of security in each field. The importance is usually defined by the enterprise information security management system (ISMS) and relevant industry standards: if certain specific data is classified as confidential, then this type of domain will receive the highest level of encryption strength (such as 256-bit AES). In our example, the details of the patient's medical records are classified as confidential information; at the same time, considering the transmission efficiency, basic information such as the date can be reduced in priority and only 128-bit encryption is applied.

[0077] 3. Next, after clarifying which parts should be given strong or weak protection, it is necessary to select a suitable algorithm for implementation. Currently, common algorithms such as Advanced Encryption Standard (AES) symmetric algorithm and Rivest Shamir Adleman (RSA) public key system are available for use. At the same time, a suitable key length, such as 128 / 192 or 256 bits, is required to ensure security while minimizing the impact on daily business operations. Here, AES-256 is selected to encrypt the above medical data.

[0078] 4. At the same time, a sound mechanism needs to be established to prevent data loss caused by abnormal phenomena such as malicious attacks or erroneous input; for example, a logging system should be established to track all encryption operation processes for later investigation, and a special module can be set up to detect potential risk data types (such as SQL injection instruction texts, etc.) and block them at the source. Once suspicious behavior occurs, relevant rules need to be adjusted in a timely manner.

[0079] 5. The last step is to regularly review the effectiveness of existing protection mechanisms to cope with ever-changing security challenges. In practice, you can regularly perform simulated attack assessments (penetration testing) to check whether they comply with the latest IT industry best practices or regulatory requirements (GDPR, HIPPAA, etc.). When necessary, change the current encryption mechanism to ensure that the protection effect keeps pace with the times and avoid hidden dangers caused by changes in external environmental factors.

[0080] By operating these five key points, a complete and effective data security protection framework can be built to ensure that all types of sensitive resources can be properly protected, thereby better promoting the long-term and healthy development of the enterprise.

[0081] Next, the data collection steps are described in detail:

[0082] In order to effectively collect and process large amounts of data from different sources, a data collection procedure specifically includes the following key steps. First, according to the preset capture rules, data packets are accurately obtained from various data sources to ensure that the captured content meets the required standards; then the content of the data packet is verified, which usually involves checking the metadata in the data packet to ensure that the necessary data elements exist and follow the unified data format and specification requirements. Subsequently, the data is protected by adopting appropriate encryption algorithms to avoid unauthorized access and information leakage; after encryption, digital signatures and accurate timestamps are added to maintain its integrity and accuracy traceability; finally, the encrypted data with signatures and time information is stored in a specific security buffer area for subsequent data scheduling operations.

[0083] For example, an e-commerce company may want to analyze data from different channels, such as website logs, social media comments, and behavioral records on mobile apps. First, it is necessary to set standards and conditions so that only useful information can be extracted from relevant sources. These standards can be specified to extract only data items about user browsing history and transaction details, such as date timestamps, product IDs, or category identifiers. Next, ensure that the collected data is clean, such as using regular expressions to verify that the date format is correct or whether there are errors such as null values ​​in the ID field to ensure data quality. Once the verified data is screened and determined, the company will use standard encryption algorithms such as AES to ensure that only authorized personnel can access it. In order to keep the pre-encryption state unchanged and prevent tampering, a unique digital signature is also required on each piece of data; at the same time, the current accurate system time is added to the data as a mark, so that the timeliness of the data can be clearly identified when processing delays. After the above preparations, the data will be temporarily stored in the company's encrypted file library or other types of data lakes until they are allocated to the data analysis system for further processing according to the preset plan. This process effectively ensures information security throughout the entire collection process and subsequent effective use.

[0084] Next, the application steps of the task scheduling algorithm are described as follows:

[0085] The priority evaluation algorithm is executed on the received encrypted data sets to determine the order of task scheduling. This step mainly assigns priority tags to each data set that needs to be processed according to its urgency, importance or other custom standards, and data sets with higher priorities are processed first. For example, if a system receives a batch of log files that need to be analyzed, and several log records involving sensitive information leakage are urgently marked with higher priorities, then these records will be scheduled to the corresponding processing unit for processing first.

[0086] Determine the currently available computing power of the data processing nodes and predict the estimated time required to complete specific processing. In this step, by real-time monitoring of each server's CPU load, memory utilization, storage space and other information, determine whether they currently have sufficient capacity to receive more data and process it. For example, if the monitoring system shows that the current processor occupancy rate of a server is less than 60%, it is considered that it is capable of taking on the computing tasks to be assigned. In addition, the time required to process a single data packet can be estimated based on past data and statistical methods, and this estimated time can be used as a consideration for subsequent load distribution.

[0087] Use a distributed consistent hashing algorithm to determine the storage location of data blocks and balance the load distribution to each data processing node. This involves distributing data to multiple different physical or logical machines for concurrent processing, and maintaining a balanced load to avoid overheating and crashing due to excessive concentration of resources in a certain part of the equipment. For example, when processing a large amount of data, hash values ​​may be used to distribute data blocks. Assume that a group of server node IDs form an ordered sequence after a consistent hashing algorithm; for new data blocks, the node sequence number obtained by calculating its hash falls on the corresponding machine in the sequence. If some points fail to provide service, since the design concept of consistent hashing is to try to keep the existing mapping unchanged, the new hash value can still point to a normally working server to ensure that the characteristics of high throughput and fault tolerance are reflected.

[0088] Apply fault-tolerant algorithms to detect failed data node tasks and redirect unfinished subtasks through a replica mechanism. To prevent any single server from stopping the execution of the entire job due to hardware failure, the system needs to be designed with recovery capabilities. In this step, when a task that is executing a data packet is found to be interrupted, it should be immediately checked whether the backup is complete and can continue to run. Once verified, the remaining healthy machines should be arranged to take over the unfinished transactions. For example, for a video file analysis process, if the analysis of a certain segment in the middle fails to end correctly for some reason, and a backup copy has been saved to another location before, the segment will automatically continue to be executed from the location of the backup copy until the entire analysis process is successfully completed.

[0089] When the processed data set is uploaded to the result aggregation node, encryption is used to ensure the security of the communication link in a multi-node environment. After the data is processed and reaches the preset output format, it must be merged for further data cleaning or display. This step not only emphasizes the aggregation of results, but also requires maintaining a high degree of privacy in the data flow between different components, that is, using transmission protection technologies such as TLS to implement full-process encryption measures for information flowing through the network to avoid the negative impact of data leakage; for example, the intermediate information extracted from multiple analyzers will first be converted by their respective encryption engines before being sent to the next site, until the last stage when all results are brought together as a whole.

[0090] Next, the application steps of the task scheduling algorithm based on the dual requirements of security and efficiency are described. This process ensures that efficient operating processes can be maintained while processing sensitive or critical data.

[0091] The first step is to determine the importance level of the task. The system predefines a set of criteria to measure the security level and encryption level of a task, and classifies it into tasks of different priorities accordingly. For example, a task involving processing sensitive personal information will be assessed as having high security and encryption requirements and marked as a high-level task.

[0092] Then, for those tasks that have been identified as high priority and require a higher level of protection, the system will first schedule computing resources for them to implement encryption processing measures, such as using advanced algorithms such as AES256 to encrypt sensitive data. This ensures that the most important and sensitive tasks can always be encrypted and processed under the most stringent protection conditions.

[0093] Other tasks that are considered less in need of strict protection - that is, those relatively less important work units that do not touch the established security threshold - will not receive resources immediately and will be placed in a waiting queue. They will also be specially marked to indicate that these are low-security projects that can be postponed or do not need to be processed immediately.

[0094] The last step is to schedule the tasks in the queue. Once there are enough resources and certain security conditions are met - for example, when the internal deployment server is idle during a certain fixed period of time or during a time window with low external threats - the waiting queue will be activated and resources will be allocated and executed according to the natural order of the waiting tasks themselves or the priority order set in advance.

[0095] A specific example is a financial company's transaction recording system. Whenever a new bank transfer request is submitted, it will go through an initial security classification, where large transfers containing a lot of financial information may be considered highly important operations and given the highest priority due to the need for higher levels of data encryption and support. In contrast, routine small queries such as checking account balances are judged not to require immediate attention and are placed in a low priority queue where appropriate. They are only processed when system resources allow, ensuring that each transaction is carried out safely and with full consideration of the risks without reducing overall efficiency. This approach ensures that critical information is protected from threats while maintaining a fast turnaround capability.

[0096] Next, we describe the process of implementing this strategy based on task scheduling based on the dual requirements of security and efficiency. When processing tasks in a big data environment, it is crucial to ensure data security. At the same time, in order to take into account the efficiency of the system, it is necessary to balance the contradiction between security and performance. By introducing security levels and encryption mechanisms, we can make appropriate choices between different levels of security requirements and processing performance.

[0097] First, define an encryption strength function (F_{encrypt}(Data,SecurityLevel)), which determines the required encryption strength level according to different security levels. For example, if the given data (Data) has a security requirement level of 3 (a higher level), this function will give a higher encryption strength value that is appropriate to the security level.

[0098] Considering the specific implementation process, for each safety task (T i ), if (SecurityLevel(T i )>MinimumSecurity)—When the security level of this task exceeds the minimum threshold defined by the system, the system will use encryption methods with higher security standards and add the result of the function (F_{encrypt}(Data,SecurityLevel)) based on this. This setting ensures that highly sensitive critical tasks receive additional protection.

[0099] On the contrary, when encountering a low-sensitivity task, that is, when the security level of a task is lower than the system preset threshold (SecurityLevel(T i )<=MinimumSecurity)), there is no need to take extreme encryption measures, so a relatively simple encryption algorithm or standard settings will be selected to protect the data, thereby reducing the pressure on computing resources brought by the encryption process. In this way, on the one hand, the system's computing load can be saved to ensure high task throughput, while at the same time not giving up the judgment of importance.

[0100] Finally, after encryption, the security factor (i.e., (F_{encrypt}(T i ,SecurityLevel(T i )))) must be no less than the minimum required strength requirement ((MinimumEncryptedStrength)) before it can be submitted to the scheduling queue. This condition is intended to ensure that each job to be completed meets at least the basic level of encryption strength. In other words, any processing that does not meet the standard is not allowed to be added to the next work link for processing to avoid the risk of leaking confidential information or suffering unauthorized access. Assuming a specific numerical example, when executing a transmission task (T_j) in an environment where the data flow security factor is required to be no less than 70 points, if the encryption calculation result only gets a score of 68, this job will not enter the subsequent queue sequence unless it reaches 70 or higher after additional reinforcement and other operations.

[0101] Such an arrangement not only ensures the effective protection of sensitive information but also minimizes the impact on non-highly confidential matters and the efficient execution capability of the overall system. It is one of the practical solutions proposed to address practical problems.

[0102] Next, the specific steps for dynamically adjusting the priority sequence of tasks based on security requirements are described. This process involves the calculation of task security scores and the dynamic adjustment of tasks in the queue order, thereby ensuring that the collection and processing of highly sensitive data are given priority, while also improving the overall efficiency of the system. First, a baseline score (BaseScore) needs to be defined, which is the basic score for each task; in addition, a function (F_{encrypt}) based on the task itself and its security level (SecurityLevel(Task)) requirements is introduced. This function is used to evaluate the value of the encryption operation required for the task to contribute to security, and at the same time sets a threshold level (ThresholdLevel) to quantify the security score of the task. Based on these basic elements, the security score (SScore) can be calculated using the above formula:

[0103] [SScore=BaseScore+F_{encrypt}(Task,SecurityLevel(Task))cdotThresholdLevel]

[0104] Specifically, suppose there are a series of tasks waiting to be scheduled that need to perform secure data collection, each with a different level of sensitivity or privacy requirements. To simplify the explanation and give an example:

[0105] 1. First set a default (BaseScore) for all tasks under consideration, for example to 5.

[0106] 2. Assume that (Task_a) is a highly demanding task (for example, involving bank account information processing), then its (SecurityLevel(Task_a)) score will be higher than that of a general data collection task.

[0107] 3. The encryption requirement corresponding to the function (F_{encrypt}) will also be given a higher coefficient.

[0108] 4. Assume that (Task_b) is a relatively insensitive task (such as recording weather data changes), so its (SecurityLevel(Task_b)) score will be lower, and the weight of (F_{encrypt}) will be correspondingly smaller.

[0109] Next, a minimum security score threshold (MinSScore) is set, such as 15.

[0110] -If the task to be scheduled (Task_j) is calculated to have a very high (SScore_j) (such as greater than or equal to 15 points), indicating that its safety score meets the requirements, then it will be given priority, that is, (Task_j) will be moved forward in the queue.

[0111] For example:

[0112] -Assume (BaseScore = 5, SecurityLevel(Task_a) = 7), and assume that (F_{encrypt}) gives a 2x multiplier bonus based on high security, and set (ThresholdLevel = 100).

[0113] -At this time, the score of (Task_a) is (SScore_a=5+2cdot7cdot100)

[0114] -Since this score definitely exceeds the minimum guaranteed score (MinSScore=15)

[0115] Therefore (Task_a) is scheduled to be executed earlier.

[0116] on the contrary:

[0117] -Assuming (BaseScore=5, SecurityLevel(Task_b)=2), if (F_{encrypt}) is given 0.5 times the normal level, the calculated value is (SScore_b=5+0.5cdot2cdot100=5+100=105).

[0118] -This score obviously exceeds (MinSScore)

[0119] -However, due to the low importance and priority of the task itself, no further increase is required.

[0120] - On the contrary, if the score of a task (SScores) is less than (MinSScore), for example, another data cleaning task (Task_c) that does not involve sensitive information but is computationally intensive, even if (SecurityLevel(Task_c)=1), its score is far from reaching the (MinSScore) threshold.

[0121] Through this mechanism, the system can effectively distinguish between sensitive jobs that truly require priority protection and more common operations, and perform intelligent scheduling to optimize processor time allocation and ultimately improve the overall efficiency of data security collection scheduling.

[0122] Next, the specific method for implementing dynamic multi-node resource allocation and task scheduling is described, including the following steps:

[0123] Calculate the proportion of available resources of all running nodes (Availability_i) and set a minimum resource threshold (Threshold_{Resource}) to avoid overloading a single node;

[0124] If (Availability_i >= Threshold_{Resource}), select node (i) as the encryption or task execution node and send the waiting tasks to this node;

[0125] Otherwise, select a node with more abundant resources according to the resource situation and consider load balancing to make the resources evenly used;

[0126] If the cumulative time when (SScore_n < MinimumScoreForHighPriority) on node (n) exceeds (TimeThreshold), transfer all tasks in the current low-priority non-security-related waiting queue to other nodes that meet higher security conditions for processing to ensure the balance and stability of processing capabilities.

[0127] First, it is necessary to calculate the current available resource proportion ((Availability_i)) of each running node by monitoring the actual usage status and resource situation of all nodes (such as CPU usage rate, memory occupancy rate, remaining storage, etc.). This proportion is obtained by the ratio of the difference between the actual usage value of the resource and the total resource amount to the total amount. At the same time, it is also necessary to set a minimum resource threshold ((Threshold_{Resource})) to determine whether a node is suitable for accepting more tasks.

[0128] Next, check (Availability_i) for each node one by one. If (Availability_i >= Threshold_{Resource}) for a node, then it is considered that this node has enough remaining capacity to handle more encryption operations or ordinary tasks, so the queued tasks are assigned to this node; if the detection result is (Availability_i < Threshold_{Resource}), it means that the node has reached the warning line level of resource allocation or is close to the overload risk. At this time, new compute-intensive tasks should not be distributed to this node. Instead, continue to check the (Availability) of other more suitable and highly idle available nodes until the condition is met; at the same time, the system will automatically consider the load differences among the nodes in the overall cluster to achieve optimized load configuration and keep the workloads of the nodes in the cluster roughly matched.

[0129] Further, if there are some tasks that need special attention to their security attributes, a security scoring standard can be introduced. For example, a parameter (SScore_n) is set to represent the score of all security-related operations on a node, as well as a predefined security level threshold ((MinimumScoreForHighPriority)) and a time threshold ((TimeThreshold)). When it is detected that the average (SScore) of all security-related operations on the node (n) cannot reach the predetermined high-level minimum security requirement level ((MinimumScoreForHighPriority)) for a long period of time, that is, (SScore_nltMinimumScoreForHighPriority), and the duration exceeds a certain threshold, all non-highly sensitive tasks with lower priority in the waiting queue that do not currently meet the security standard restrictions should be actively migrated from this node (n) to other working nodes in the cluster that are more suitable for executing the corresponding tasks and have better security performance. This mechanism is used to ensure the balanced utilization of tasks and resources between nodes and improve the overall security level and stability. For example, in a distributed system test environment, there are three working nodes (Node_1) (Node_2) (Node_3) with different purposes, each of which is responsible for encryption processing tasks, general query services, and real-time streaming data recording functions. For encryption processing tasks, after evaluation, (Threshold_{Resource} = 20%) is set. If it is found that (Node_1) still has more than 60% idle CPU and half of the free memory space at this time, it is considered that new data security encryption requests can be safely added. For real-time large-scale streaming data capture, when it is found that (Node_3) has its overall availability degraded to 10% or even worse due to a temporary sudden large flow, and it can no longer respond to all new tasks in time, the relatively low-level tasks will be transferred from (Node_3) to (Node_1) (Node_2) with better idleness.

[0130] The above processes and steps can ensure that task allocation in a big data environment can not only meet the requirements of load balancing and stability, but also take into account the efficient support for special application scenarios with higher critical tasks and security requirements.

[0131] The data security collection and scheduling system based on big data of the present invention includes: first, through the step of determining an encryption strategy, specifically, this method performs in-depth analysis based on the data type and sensitivity of the big data source, and customizes encryption rules based on this, aiming to protect the security of sensitive information and prevent various risks that may be encountered during the data flow process. Secondly, in the data collection stage under the guidance of the above encryption rules, by executing encryption processing measures for specific types of data, it can ensure that the acquired data is fully protected at the time of initial collection. Subsequently, this method distributes the encrypted data effectively and evenly to each processing node for distributed computing or analysis through a task scheduling algorithm designed specifically to ensure security and optimize efficiency. The algorithm not only takes into account the speed of task completion but also ensures the security and compliance of data distribution.

[0132] This method effectively solves a major technical problem in the current big data processing environment, that is, how to achieve the best balance between data processing security and efficiency. In practical application, sophisticated encryption rules can be used to enhance the protection of sensitive information in a targeted manner without increasing redundancy costs; in terms of task scheduling, the use of efficient algorithms can not only avoid excessive pressure on a single resource under traditional methods, but also make full use of idle computing power in the network. Such a solution design ensures that information can be transmitted securely and used efficiently in a large-scale data operation environment, even in the face of increasingly complex and changing data sources and their attributes.

[0133] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the embodiments of the present disclosure. It should be understood that the above description is only the specific implementation method of the embodiments of the present disclosure and is not intended to limit the protection scope of the embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the protection scope of the embodiments of the present disclosure.

Claims

1. A data security collection and scheduling system based on big data, characterized in that: include: A step for determining an encryption strategy, in which encryption rules are customized by analyzing the data type and sensitivity of the big data source; A data collection step, using the encryption strategy determined above to encrypt the information extracted from the data source; The application steps of the task scheduling algorithm based on the dual requirements of security and efficiency, the algorithm optimizes resource utilization and distributes encrypted data sets among multiple nodes to ensure fast and secure data processing.

2. The data security acquisition and scheduling system based on big data according to claim 1 is characterized in that: The steps of determining an encryption strategy specifically include: Analyzing data types and structural characteristics of the big data sources to identify key information domains; Defining a corresponding encryption level according to the importance level of the key information domain; Determine the encryption algorithms and key lengths used to implement each encryption level; Develop abnormal data detection mechanisms to prevent illegal access of malicious data and adjust encryption rules accordingly; Regularly review the effectiveness of encryption rules and dynamically update them based on the current security situation.

3. The data security collection and scheduling system based on big data according to claim 1 is characterized in that: The data collection step specifically includes: Use predetermined data extraction criteria to capture target data packets from large data sets from different sources; Verify that the metadata in the data package is consistent with the predetermined standards to ensure its validity; Encrypt the collected target data packets using corresponding algorithms according to encryption strategies; Generate a digital signature for the encrypted data packet to ensure integrity, and attach a timestamp to ensure the accuracy of time records; The encrypted and digitally signed data packets are safely stored in the intermediate buffer waiting for task scheduling.

4. The data security collection and scheduling system based on big data according to claim 1 is characterized in that: The application steps of the task scheduling algorithm specifically include: Execute a priority evaluation algorithm on the received encrypted data set to determine the task scheduling order; Determine the currently available computing power of data processing nodes and predict the estimated time required to complete specific processing; Use a distributed consistent hashing algorithm to determine the storage location of data blocks and balance the load distribution to each data processing node; Apply fault-tolerance algorithms to detect failed data node tasks and redirect unfinished subtasks through a replication mechanism; When the processed data set is uploaded to the result aggregation node, encryption is used to ensure the security of the communication link in a multi-node environment.

5. The data security collection and scheduling system based on big data according to claim 1 is characterized in that: The application steps of the task scheduling algorithm based on the dual requirements of safety and efficiency include: Determine the mission's importance based on pre-set security and encryption levels; Prioritize the scheduling of processor resources for encryption processing for tasks with high priority levels; If the importance level of the task does not reach the predetermined safety threshold, the task is placed in the waiting queue and marked as a task with low safety requirements; For tasks in the waiting queue, resources are called for processing according to priority when safety requirements are met.

6. The data security collection and scheduling system based on big data according to claim 5 is characterized in that: Task scheduling based on the dual requirements of safety and efficiency further includes: Set the encryption strength function (F_{encrypt}(Data,SecurityLevel)) to represent the encryption strength determined according to the security level; When the task (T i SecurityLevel(T i )>MinimumSecurity), a higher encryption algorithm level is used to increase the output value of (F_{encrypt}) to enhance the security factor of the task; Otherwise, use basic or standard encryption to reduce the output of (F_{encrypt}) in order to improve processing efficiency; In (F_{encrypt}(T i ,SecurityLevel(T i ))) must be greater than or equal to (MinimumEncryptedStrength) before (T i ) is placed in the scheduling task pool to wait for further processing.

7. The data security collection and scheduling system based on big data according to claim 6 is characterized in that: The specific steps to dynamically adjust the priority of tasks according to safety requirements include: Setting the safety score (SScore) = (BaseScore + (F_{encrypt}(Task, SecurityLevel(Task)) ThresholdLevel)) to evaluate the task security performance; Assume that for a to-be-scheduled task (Task_j), (SScore_j >= MinSScore), then increase the sorting position of (Task_j) in the waiting queue to ensure that higher security requirements are executed first; On the contrary, if for a task (Task_k), (SScore_k < MinSScore), then lower its position in the waiting queue to reduce the processor time and increase the throughput of the scheduler; The tasks updated based on the security score are re-sorted by (SScore) to ensure that the encryption requirements of the tasks can be considered first.

8. The data security collection and scheduling system based on big data according to claim 7 is characterized in that: The specific method for implementing multi-node resource dynamic allocation and task scheduling includes the steps: Calculate the available resource ratio (Availability_i) of all running nodes and set the minimum resource threshold (Threshold_{Resource}) to avoid overloading a single node; If (Availability_i >= Threshold_{Resource}), then select node (i) as the encryption or task execution node and send the waiting tasks to this node; Otherwise, select a node with more sufficient resources according to the resource situation and consider load balancing to make the resources be evenly used; If the cumulative time of (SScore_n < MinimumScoreForHighPriority) on node (n) exceeds (TimeThreshold), then transfer all tasks in the current low-priority non-security-related waiting queue to other nodes that meet higher security conditions for processing to ensure the balance and stability of the processing capacity.

Images