Development data query management method and system based on data security

Abstract

The invention discloses a development data query management method and system based on data security, and relates to the technical field of computers, the method comprises the following steps: obtaining a query task configured by a user at a client based on a target demand; performing access permission pre-verification from the dimensions of a user, a role and a data resource through a three-level permission model, and performing security processing on a query task and associated data after verification succeeds to generate a standardized query task with a security identifier; extracting multi-dimensional features of request execution time, task emergency degree, data scale and data type; generating a storage node priority sequence; a plurality of mainstream databases are adapted through a unified adaptation interface, and task analysis, conversion and execution are completed; monitoring a task execution state in real time; according to the method, access permission preprocessing can be carried out from multiple dimensions of users, roles and data resources through a three-level permission model and security processing, and the risks of unauthorized access and data leakage are avoided.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a data query management method and system based on data security. Background Technology

[0002] In the field of computer and auxiliary equipment repair, the development of data query management involves systematically building and maintaining a data query system to achieve efficient collection, storage, processing, and retrieval of various types of data generated during the repair process.

[0003] Currently, the development data query management mainly adopts structured query task control and cluster node scheduling. Efficient data query and management are achieved through task allocation and priority scheduling. During task allocation, users configure target structured query tasks on the client based on their needs. The client uploads the task and execution request information to the server. The server allocates the task to the storage node in the real-time development data cluster that matches the time period according to the execution request information. Priority scheduling is based on the priority order of storage node classification, storage type, etc., to execute query tasks to obtain development data for real-time management of development data.

[0004] However, since the allocation of storage nodes depends on the execution request time and time period, and does not take into account the real-time load of the nodes, it is easy to cause some nodes to be overloaded and some nodes to be idle, reducing the overall utilization of the cluster and thus affecting the quality of data query management. Summary of the Invention

[0005] The purpose of this invention is to provide a development data query management method and system based on data security, which can improve the overall utilization of the cluster and ensure the quality of development data query management, thereby solving the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A data security-based development data query management method includes the following steps:

[0008] S1. Obtain the query task configured by the user on the client based on the target requirements. The query task supports structured data query and unstructured data query.

[0009] S2. Access permissions are pre-verified from the dimensions of users, roles and data resources through a three-level permission model. After successful verification, the query task and related data are processed for security, and a standardized query task with a security identifier is generated.

[0010] S3. Real-time collection of status data of all storage nodes in the real-time development data cluster, parsing of execution request information of standardized query tasks, and extraction of multi-dimensional features such as execution request time, task urgency, data scale and data type;

[0011] S4. Perform weighted calculations based on multi-dimensional features to generate a priority ranking of storage nodes, and allocate standardized query tasks to the storage nodes with the highest priority and load within a preset threshold.

[0012] S5. Adapt to multiple mainstream databases through a unified adaptation interface, complete task parsing, transformation and execution, and back up the core data after parsing and transformation to multiple nodes;

[0013] S6. Monitor the task execution status in real time. If the task execution fails, the retry mechanism is automatically triggered. If the retry fails, the task is switched to the backup storage node for re-execution. If the retry is successful or the task execution is successful, the result is processed and the process proceeds to S7.

[0014] S7. Query the execution results, perform secondary permission verification and secondary security processing, and then send the processing results back to the client for display.

[0015] S8: Real-time monitoring of the entire task status and quantitative indicators, recording and archiving the entire operation log, and generating operation and maintenance reports on a regular basis.

[0016] Preferably, when a query task is retrieved, the user configures the query task based on business objectives through a visual interface on the client, and sets the task urgency, execution mode and result output format, supporting multiple types of data query configurations.

[0017] Preferably, the security process involves using TLS / SSL encryption and automatic desensitization of sensitive data during data transmission. During encryption, when the client transmits the query task and related data to the server, the TLS / SSL encryption protocol is activated to encrypt the transmitted data, preventing it from being intercepted and leaked during transmission. Then, the related data of the query task is desensitized.

[0018] When automatically desensitizing sensitive data, a character replacement desensitization algorithm is used to automatically desensitize the sensitive data associated with the query task. Sensitive fields are replaced according to preset rules to generate secure data, and then a standardized query task with a unique security identifier is generated.

[0019] Preferably, the three-level permission model is constructed by binding users to roles and roles to data resource permissions, as follows:

[0020] A1. Collect all data resources within the system, classify and classify them according to sensitivity resistance and business affiliation, and generate a data resource list and permission levels;

[0021] A2. Define roles based on business job requirements, assign corresponding data resource permissions to each role, and form a role-permission mapping table.

[0022] A3. Associate system users with corresponding roles. A user can be bound to multiple roles, and the permissions are the set of permissions of all bound roles. Generate a user-role mapping table to complete the construction of a three-level permission model.

[0023] Preferably, the three-level permission model performs the following access permission pre-verification process:

[0024] B1. Extract user ID and target data resource information from the user-configured query task;

[0025] B2. Query all roles bound to the user through the user-role mapping table;

[0026] B3. Based on the bound role, obtain the range of data resources that the user can access and the operation permissions through the role-permission mapping table;

[0027] B4. Compare the target data resource of the query task with the user's access permission range. If the target resource is within the permission range and the operation complies with the permissions, the verification passes; otherwise, the verification fails and an insufficient permission prompt is returned.

[0028] Preferably, when collecting status data of all storage nodes in the real-time development data cluster, a distributed data collection framework combined with node agent deployment is adopted to realize the real-time capture and aggregation of storage node status data. When the distributed data collection framework is combined with node agent deployment, a lightweight Agent program is deployed on each storage node to collect node hardware status and task running data at a frequency of 1 second / time. The status database of the dynamic scheduling module is transmitted in real time through a message queue to ensure that the data latency is less than 500ms. The collected status data includes CPU load, memory usage, remaining storage capacity, current task queue length, data storage type, and node health status.

[0029] The preferred standard query task execution request information parsing process is as follows:

[0030] C1. Receive standardized query tasks with security identifiers and extract the execution request information field from the task header;

[0031] C2. Parse the request header metadata and extract the execution request time, task ID, and security identifier;

[0032] C3. Parse the core information of the task body and extract the task urgency, data size, data type and target database type;

[0033] C4. Perform format validation on the parsed results to ensure the fields are complete and generate a structured request information parsing report.

[0034] Preferably, during multi-dimensional feature extraction, the feature dimensions to be extracted are first determined based on the parsed execution request information. Based on the feature dimensions, features such as execution request time, task urgency, data size, and data type are extracted. Then, non-numerical features are converted into standardized numerical values, and structured feature vectors are generated based on the converted standardized numerical values ​​and numerical features.

[0035] Preferably, when generating the storage node priority ranking, the feature values ​​are weighted and summed based on the structured feature vector and the real-time load data of the nodes to obtain the priority score of each node, and then the nodes are sorted in descending order of priority score to generate the storage node priority ranking table.

[0036] When standardizing query task allocation, the node with the highest score is selected as a candidate node from the priority sorting table. The real-time load rate of the candidate node is checked to see if it is ≤ a preset threshold. If it is ≤ a preset threshold, the task is directly allocated to the node and the node task queue is updated. Otherwise, the node is skipped and the next priority node in the sorting is selected for re-verification until a node that meets the load requirements is found.

[0037] Based on the above-described data security-based development data query management method, this invention also provides a data security-based development data query management system, including: a security control module, which integrates a three-level permission model, performs access permission pre-verification and security processing from the dimensions of users, roles and data resources, records full-link operation logs, and configures the log retention period;

[0038] The dynamic scheduling module collects the status data of storage nodes in real time and performs weighted calculations to generate a priority ranking of storage nodes, and allocates standardized query tasks to the optimal storage node.

[0039] The fault-tolerant execution module monitors storage node failures and task execution status in real time, triggers an automatic task retry mechanism, and performs multi-node backup storage of core data.

[0040] A multi-source adaptation module that adapts to multiple mainstream databases and completes task parsing, conversion, and execution;

[0041] The task lifecycle management module is used to display the full status of tasks, collect quantitative indicators of task execution, and provide configuration of multi-mode scheduling strategies.

[0042] The storage cluster module is used to store the core data of each node;

[0043] The interactive module provides a query window that displays task status, quantitative indicators, and operation logs in real time, as well as report generation and export.

[0044] The present invention proposes a data security-based development data query management method and system, which has the following advantages compared with the prior art:

[0045] 1. This invention, through a three-level permission model and security processing, can preprocess access permissions from multiple dimensions such as users, roles, and data resources, avoiding the risks of unauthorized access and data leakage;

[0046] 2. This invention avoids blind allocation caused by unknown load by collecting the status data of all storage nodes in the real-time development data cluster in real time. The node priority ranking is based on multi-dimensional characteristics such as execution request time, task urgency, data size and data type. Moreover, the allocation clearly limits the storage node with the highest priority and the load within the preset threshold, ensuring that the priority ranking result will favor the nodes with low load. This avoids high-load nodes being continuously assigned tasks, which not only prevents high-load nodes from being overloaded by the continued assignment of tasks, but also directs tasks to idle nodes, thereby improving the overall utilization of the cluster.

[0047] 3. Through the coordinated operation of the security management module, dynamic scheduling module, fault-tolerant execution module, multi-source adaptation module, task lifecycle management module, storage cluster module and interaction module, this invention can achieve efficient utilization and value transformation of development data, while ensuring the security of data during the value transformation process through security management. Attached Figure Description

[0048] Figure 1 A flowchart of a data query management method developed according to an embodiment of the present invention is shown;

[0049] Figure 2 A flowchart illustrating the construction process of a three-level permission model according to an embodiment of the present invention is shown.

[0050] Figure 3 A flowchart illustrating the pre-verification process of the three-level permission model according to an embodiment of the present invention is shown.

[0051] Figure 4 A flowchart illustrating the execution request information parsing process for a standardized query task according to an embodiment of the present invention is shown.

[0052] Figure 5 A block diagram of a development data query management system according to an embodiment of the present invention is shown. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This invention provides, for example Figure 1-4 The data security-based development data query management method shown includes the following steps:

[0055] S1. Obtain the query task configured by the user on the client based on the target requirements. The query task supports structured data query and unstructured data query.

[0056] The target requirements are used to clarify the core objectives that you want to achieve through data querying based on actual business scenarios or data processing needs, including data extraction requirements, data statistical analysis requirements, data association and integration requirements, unstructured data processing requirements, and data output and application requirements.

[0057] A query task is when a user translates their target requirements into specific configuration items that the client can recognize and execute. These specific configuration items include query object configuration, query conditions and logic configuration, execution parameter configuration, and output and security configuration. Among them, the query object configuration is used to specify the data resources to be accessed, including the database type, data storage location, and data range. For structured data, table relationships need to be specified, and for unstructured data, file types and metadata fields need to be specified.

[0058] When configuring query conditions and logic, set data filtering and processing rules. Structured data is queried using SQL statements, while unstructured data is filtered using keywords and attribute conditions. At the same time, configure data aggregation rules, such as summation, counting, and grouping statistics.

[0059] The execution parameter configuration includes task urgency, execution mode, and retry mechanism parameters. The retry mechanism parameters include the number of retries on failure and the retry interval. The execution mode includes real-time execution, scheduled execution, and dependency-triggered execution. Scheduled execution requires a specific time to be specified, and dependency-triggered execution requires association with a preceding task.

[0060] Output and security configuration is used to specify the output format, storage path, and sensitive data de-identification rules and result caching duration.

[0061] When a query task is retrieved, the user configures the query task based on business objectives through a visual interface on the client side. At the same time, the user sets the task urgency, execution mode, and result output format. The system supports multiple types of data query configurations to meet diverse business needs.

[0062] S2. Access permissions are pre-verified from the dimensions of users, roles and data resources through a three-level permission model. If the verification is successful, the query task and related data are processed for security and a standardized query task with a security identifier is generated. If the verification fails, the access permission pre-verification process is returned and no security processing is performed.

[0063] The three-level permission model binds users to roles and roles to data resource permissions, enabling fine-grained and flexibly configurable data access permission control, avoiding the management chaos caused by directly assigning permissions to individual users;

[0064] The three-level permission model is constructed as follows:

[0065] A1. Collect all data resources within the system, classify and categorize them according to sensitivity tolerance and business affiliation, and generate a data resource list and permission levels. Among them, data resources include databases, data tables, data fields, and files. Permission levels include low-sensitivity data, medium-sensitivity data, and high-sensitivity data. Low-sensitivity data can be edited, queried, and exported; medium-sensitivity data can be queried and exported; and high-sensitivity data can only be viewed.

[0066] A2. Define roles based on business job requirements, assign corresponding data resource permissions to each role, and form a role-permission mapping table. Roles include ordinary maintenance personnel, senior maintenance personnel, and auditors. When allocating data resource permissions, ordinary maintenance personnel can only access low-sensitivity data in their own business line, while senior administrators can access all business data.

[0067] A3. Associate system users with corresponding roles. A user can be bound to multiple roles, and the permissions are the set of permissions of all bound roles. Generate a user-role mapping table to complete the construction of a three-level permission model.

[0068] The three-level permission model performs the following access permission pre-verification process:

[0069] B1. Extract user ID and target data resource information from the user-configured query task;

[0070] B2. Query all roles bound to the user through the user-role mapping table;

[0071] B3. Based on the bound role, obtain the range of data resources that the user can access and the operation permissions through the role-permission mapping table;

[0072] B4. Compare the target data resource of the query task with the user's access permission range. If the target resource is within the permission range and the operation complies with the permissions, the verification passes; otherwise, the verification fails and an insufficient permission prompt is returned.

[0073] The security process involves using TLS / SSL encryption and automatic desensitization of sensitive data during data transmission. During encryption, when the client transmits the query task and related data to the server, the TLS / SSL encryption protocol is activated to encrypt the transmitted data, preventing interception and leakage during transmission. Then, the related data of the query task is desensitized.

[0074] When automatically de-identifying sensitive data, a character replacement de-identification algorithm is used to automatically de-identify the sensitive data associated with the query task. Sensitive fields are replaced according to preset rules to generate secure data. Then, a standardized query task with a unique security identifier is generated. The preset rules are standardized processing rules that are configured in advance. Fixed and flexible de-identification logic is formulated for different types of sensitive data to ensure that the de-identification process is uniform, compliant and efficient, without the need for manual configuration by the user. For example, the format of matching 18 characters of ID card number with 6-digit address code, 8-digit birth date and 4-digit check code.

[0075] Character replacement desensitization algorithms include formulas for desensitizing ID card numbers, mobile phone numbers, and general strings. The formula for desensitizing ID card numbers is as follows:

[0076] D=S1S2…S6+'******'+S13…S18,

[0077] Where D is the anonymized ID number, S1-S18 are the 18 characters of the original ID number, retaining the first 6 address digits and the last 4 check digits, and replacing the middle 6 digits with *.

[0078] The formula for de-identifying mobile phone numbers is:

[0079] P = P1P2P3 + '****' + P8…P11,

[0080] Where P is the de-identified mobile phone number, P1-P11 are the original 11-digit mobile phone numbers, retaining the first 3 digits of the carrier code and the last 4 digits, and replacing the middle 4 digits with *.

[0081] The general formula for desensitizing strings is:

[0082] ,

[0083] in, This is the anonymized string. L represents the original string, and L represents the length of the original string. To round down, the first 1 / 3 and the last 1 / 3 of the characters are retained, and the middle is replaced with *.

[0084] S3. Real-time collection of status data of all storage nodes in the real-time development data cluster, parsing of execution request information of standardized query tasks, and extraction of multi-dimensional features such as execution request time, task urgency, data scale and data type;

[0085] When collecting status data of all storage nodes in the real-time development data cluster, a distributed data collection framework combined with node agent deployment is adopted to realize the real-time capture and aggregation of storage node status data. When the distributed data collection framework is combined with node agent deployment, a lightweight Agent program is deployed on each storage node to collect node hardware status and task running data at a frequency of 1 second / time. The status database of the dynamic scheduling module is transmitted in real time through a message queue to ensure that the data latency is less than 500ms. The collected status data includes CPU load, memory usage, remaining storage capacity, current task queue length, data storage type, and node health status.

[0086] The process for parsing the execution request information of a standardized query task is as follows:

[0087] C1. Receive standardized query tasks with security identifiers and extract the execution request information field from the task header;

[0088] C2. Parse the request header metadata and extract the execution request time, task ID, and security identifier;

[0089] C3. Parse the core information of the task body and extract the task urgency, data size, data type and target database type;

[0090] C4. Perform format validation on the parsed results to ensure the fields are complete and generate a structured request information parsing report;

[0091] When extracting multi-dimensional features, the feature dimensions to be extracted are first determined based on the parsed execution request information. Based on the feature dimensions, features such as execution request time, task urgency, data size and data type are extracted. Then, non-numerical features are converted into standardized values, and structured feature vectors are generated based on the converted standardized values ​​and numerical features.

[0092] The feature extraction formula for request time is:

[0093] ,

[0094] in, The time matching degree is given, where t is the execution time of the request. This represents the center value of the optimal processing time window for the node. The width of the time window;

[0095] The feature extraction formula for task urgency is:

[0096] E = k * E0, where E is the quantified task urgency, k is the normalization coefficient, and E0 is the user's original configuration urgency.

[0097] The feature extraction formula for data scale is:

[0098] ,

[0099] Where S is the size of the quantized data. Estimate the amount of data for the task. This represents the maximum amount of data that the system can support in a single task.

[0100] The feature extraction formula for data types is:

[0101] ,

[0102] in, For data type compatibility, A is the data type and B is the node storage type. When the data type and the node storage type are consistent, the data type compatibility is 1; otherwise, the data type compatibility is 0.3.

[0103] S4. Perform weighted calculations based on multi-dimensional features to generate a priority ranking of storage nodes, and allocate standardized query tasks to the storage nodes with the highest priority and load within a preset threshold. When the target storage nodes are all loaded, the task is automatically diverted to the backup storage nodes.

[0104] When generating storage node priority ranking, the feature values ​​are weighted and summed based on structured feature vectors and real-time node load data to obtain a priority score for each node. These nodes are then sorted in descending order of priority score to generate a storage node priority ranking table. The priority score calculation formula is as follows:

[0105] ,

[0106] in, Score based on priority. … These are the feature weights, and + + + + =1, To achieve time matching accuracy, To assess the urgency of the task, To adapt to data scale, For data type compatibility, H represents the real-time load of the node;

[0107] When standardizing query task allocation, the node with the highest score is selected from the priority sorting table as a candidate node. The real-time load rate of the candidate node is checked to see if it is ≤ a preset threshold. If it is ≤ a preset threshold, the task is directly allocated to the node and the node task queue is updated. Otherwise, the node is skipped and the next priority node in the sorting is selected for re-verification until a node that meets the load requirements is found.

[0108] The preset threshold is a dynamic threshold setting based on node hardware performance and historical load data, and is not a fixed value. Hardware performance includes the number of CPU cores, memory capacity, and disk I / O rate. Historical load data is the average peak load and load fluctuation variance of the node over the past 30 days. When the dynamic threshold is adjusted, node load data is automatically calculated every day at midnight. If the average load is below 50% of the threshold for three consecutive days, the threshold will be increased by 5%. If the load exceeds the limit three times in a row, the threshold will be decreased by 5%.

[0109] S5. Adapt to multiple mainstream databases through a unified adaptation interface, complete task parsing, transformation and execution, and back up and store the core data after parsing and transformation on multiple nodes.

[0110] When adapting to multiple mainstream databases, first load the preset mainstream database drivers, establish a driver pool and a connection pool, and each database has an independent connection configuration, including IP, port, account and password. Receive the assigned standardized query tasks, extract the target database type identifier from the standardized query tasks, and match the corresponding database connection and driver.

[0111] During task parsing and transformation, the core logic of standardized query tasks is first parsed, then format conversion and adaptation are performed, and finally the legality of the converted task syntax is checked. If the verification passes, an executable task is generated; if it fails, an error log is returned, triggering the task retry mechanism. When parsing the core logic of standardized query tasks, SQL statements and parameters are extracted for structured tasks, and file paths, filtering conditions, and metadata fields are extracted for unstructured tasks. Format conversion and adaptation are based on the target database syntax rules, converting standardized query logic into compatible syntax, such as converting general SQL into HiveSQL's specific function format, and converting unstructured query conditions into file retrieval syntax supported by the target database.

[0112] During execution, the executable task is sent to the target database through the connection pool, and the execution status and intermediate results returned by the database are received. The execution status and intermediate results returned by the database are synchronized in real time. After the execution is completed, the final query result is obtained and temporarily stored in the cache area.

[0113] When backing up and parsing core data across multiple nodes, the core data is extracted from the cache, split according to the sharding rules, and synchronized to the backup nodes through an encrypted transmission channel. The backup nodes receive and return an acknowledgment flag. Once all backup nodes have confirmed receipt, the data backup is marked as successful and the data backup status is updated. If any backup node fails, the backup is re-initiated to the standby node to ensure at least two valid backups.

[0114] S6. Monitor the task execution status in real time. If the task execution fails, the retry mechanism is automatically triggered. If the retry fails, the task is switched to the backup storage node for re-execution. If the retry is successful or the task execution is successful, the result is processed and the process proceeds to S7.

[0115] The task execution status monitoring includes node heartbeat detection, real-time monitoring of execution logs, and proactive reporting of task status. During node heartbeat detection, a heartbeat request is sent to the storage node where the task resides every second. If the node is normal, it returns a response signal containing the load status. If no response is received for three consecutive times, the node is marked as faulty and a retry mechanism is triggered. During execution log monitoring, the database execution log and task output log are captured in real time, and key status identifiers are extracted, such as in progress, completed, and execution error. When an execution error identifier is extracted, a retry mechanism is triggered.

[0116] When a task status is actively reported, the status reporting logic built into the executable task is used to actively report the status. The status reporting logic reports the execution progress every 5 seconds and pushes the final status and result summary immediately after the execution is completed.

[0117] The task is considered successful if all of the following conditions are met; otherwise, the task is considered to have failed:

[0118] 1) The task execution terminal returns a clear success status indicator, and there are no error records in the execution log;

[0119] 2) The generated query results are complete and valid, and the core data has been backed up on multiple nodes;

[0120] 3) The task execution time did not exceed the preset timeout threshold, and the retry mechanism was not triggered or the task was successfully executed after retry. The preset timeout threshold is configured according to the task type. For example, the default timeout is 300 seconds for structured query tasks and 600 seconds for unstructured query tasks.

[0121] The retry mechanism defaults to 3 retries, with the retry interval increasing by 10 seconds multiplied by the number of retries. If all 3 retries fail, the task is automatically added to the pending queue. Operations personnel can trace the cause of the failure through logs and manually trigger re-execution after repair.

[0122] S7. Query the execution results, perform secondary permission verification and secondary security processing, and then send the processing results back to the client for display.

[0123] When querying execution results, the final execution result is retrieved from the target storage node, carrying the unique task ID and security identifier to ensure accurate matching of the query object and avoid result confusion.

[0124] The secondary permission verification uses a three-level permission model to verify access permissions. If the verification passes, it proceeds to the security processing stage. If the verification fails, the result is directly intercepted, an insufficient permission prompt is returned, and an exception log is recorded. The secondary security processing uses TLS / SSL encryption and automatic desensitization of sensitive data to process the transmitted data. The specific process of the secondary permission verification is the same as the permission pre-verification process in step S2, and the secondary security processing process is the same as the security processing process in step S2. Therefore, it will not be described again here.

[0125] S8: Real-time monitoring of the entire task status and quantitative indicators, recording and archiving the entire chain of operation logs, and generating operation and maintenance reports on a regular basis;

[0126] The full status of a task includes task allocation or distribution status, task execution status, task retry or node switching status, and permission verification and security processing status; quantitative indicators include execution time, resource consumption, data transmission volume, number of failure retries, and number of backup nodes.

[0127] Scheduled maintenance report generation automatically triggers report generation tasks according to a preset cycle, which is defined by the administrator. It extracts core data from the quantitative indicator library and log library, performs data statistics, and generates standardized maintenance reports based on Excel or HTML format.

[0128] By collecting real-time status data of all storage nodes in the real-time development data cluster, blind allocation caused by unknown load is avoided from the source. Node priority ranking is based on multi-dimensional characteristics such as execution request time, task urgency, data size and data type. In addition, the storage nodes with the highest priority and load within the preset threshold are clearly defined during allocation. This ensures that the priority ranking results will favor nodes with low load and prevent high-load nodes from being continuously assigned tasks. This not only prevents high-load nodes from being overloaded by the continued assignment of tasks, but also directs tasks to idle nodes, thereby improving the overall utilization of the cluster.

[0129] Based on the above-described data security-based development data query management method, this invention also provides a data security-based development data query management system, such as... Figure 5As shown, the system includes a security management module, a dynamic scheduling module, a fault-tolerant execution module, a multi-source adaptation module, a task lifecycle management module, a storage cluster module, and an interaction module. The security management module integrates a three-level permission model, performing access permission pre-verification and security processing from the dimensions of users, roles, and data resources, recording full-link operation logs, and configuring log retention periods. The dynamic scheduling module collects the status data of storage nodes in real time and performs weighted calculations to generate storage node priority rankings, allocating standardized query tasks to the optimal storage node. The fault-tolerant execution module monitors storage node failures and task execution status in real time, triggering an automatic task retry mechanism, while simultaneously performing multi-node backup storage of core data. The multi-source adaptation module adapts to multiple mainstream databases, completing task parsing, transformation, and execution. The task lifecycle management module displays the full status of tasks, statistically analyzes task execution metrics, and provides multi-mode scheduling strategy configurations. The storage cluster module stores the core data of each node. The interaction module provides a query window to display task status, quantitative metrics, and operation logs in real time, as well as report generation and export.

[0130] The security control module and the task lifecycle management module are electrically connected to the dynamic scheduling module, the fault-tolerant execution module, the multi-source adaptation module, the storage cluster module, and the interaction module, respectively. The security control module is also electrically connected to the task lifecycle management module. Through the coordinated operation of the security control module, the dynamic scheduling module, the fault-tolerant execution module, the multi-source adaptation module, the task lifecycle management module, the storage cluster module, and the interaction module, efficient utilization and value transformation of development data can be achieved, while security control ensures the security of data during the value transformation process.

[0131] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data security-based development data query management method, characterized in that: Includes the following steps: S1. Obtain the query task configured by the user on the client based on the target requirements. The query task supports structured data query and unstructured data query. S2. Access permissions are pre-verified from the dimensions of users, roles and data resources through a three-level permission model. After successful verification, the query task and related data are processed for security, and a standardized query task with a security identifier is generated. S3. Real-time collection of status data of all storage nodes in the real-time development data cluster, parsing of execution request information of standardized query tasks, and extraction of multi-dimensional features such as execution request time, task urgency, data scale and data type; S4. Perform weighted calculations based on multi-dimensional features to generate a priority ranking of storage nodes, and allocate standardized query tasks to the storage nodes with the highest priority and load within a preset threshold. S5. Adapt to multiple mainstream databases through a unified adaptation interface, complete task parsing, transformation and execution, and back up the core data after parsing and transformation to multiple nodes; S6. Monitor the task execution status in real time. If the task execution fails, the retry mechanism is automatically triggered. If the retry fails, the task is switched to the backup storage node for re-execution. If the retry is successful or the task execution is successful, the result is processed and the process proceeds to S7. S7. Query the execution results, perform secondary permission verification and secondary security processing, and then send the processing results back to the client for display. S8: Real-time monitoring of the entire task status and quantitative indicators, recording and archiving the entire operation log, and generating operation and maintenance reports on a regular basis.

2. The data security-based development data query management method according to claim 1, characterized in that: When a query task is retrieved, the user configures the query task based on business objectives through a visual interface on the client, and sets the task urgency, execution mode and result output format, supporting configuration of multiple types of data queries.

3. The data security-based development data query management method according to claim 2, characterized in that: The security process involves using TLS / SSL encryption and automatic desensitization of sensitive data during data transmission. During encryption, when the client transmits the query task and related data to the server, the TLS / SSL encryption protocol is activated to encrypt the transmitted data, preventing interception and leakage during transmission. Then, the related data of the query task is desensitized. When automatically desensitizing sensitive data, a character replacement desensitization algorithm is used to automatically desensitize the sensitive data associated with the query task. Sensitive fields are replaced according to preset rules to generate secure data, and then a standardized query task with a unique security identifier is generated.

4. The data security-based development data query management method according to claim 3, characterized in that: The three-tier permission model binds users to roles and roles to data resource permissions. The three-tier permission model is constructed as follows: A1. Collect all data resources within the system, classify and classify them according to sensitivity resistance and business affiliation, and generate a data resource list and permission levels; A2. Define roles based on business job requirements, assign corresponding data resource permissions to each role, and form a role-permission mapping table; A3. Associate system users with corresponding roles. A user can be bound to multiple roles, and the permissions are the set of permissions of all bound roles. Generate a user-role mapping table to complete the construction of a three-level permission model.

5. The data security-based development data query management method according to claim 4, characterized in that: The three-level permission model performs the following access permission pre-verification process: B1. Extract user ID and target data resource information from the user-configured query task; B2. Query all roles bound to the user through the user-role mapping table; B3. Based on the bound role, obtain the range of data resources that the user can access and the operation permissions through the role-permission mapping table; B4. Compare the target data resource of the query task with the user's access permission range. If the target resource is within the permission range and the operation complies with the permissions, the verification passes; otherwise, the verification fails and an insufficient permission prompt is returned.

6. The data security-based development data query management method according to claim 5, characterized in that: When collecting status data of all storage nodes in the real-time development data cluster, a distributed data collection framework combined with node agent deployment is adopted to realize the real-time capture and aggregation of storage node status data. When the distributed data collection framework is combined with node agent deployment, a lightweight Agent program is deployed on each storage node to collect node hardware status and task running data at a frequency of 1 second / time. The status database of the dynamic scheduling module is transmitted in real time through a message queue to ensure that the data latency is less than 500ms. The collected status data includes CPU load, memory usage, remaining storage capacity, current task queue length, data storage type, and node health status.

7. A data security-based development data query management method according to claim 6, characterized in that: The process for parsing the execution request information of a standardized query task is as follows: C1. Receive standardized query tasks with security identifiers and extract the execution request information field from the task header; C2. Parse the request header metadata and extract the execution request time, task ID, and security identifier; C3. Parse the core information of the task body and extract the task urgency, data size, data type and target database type; C4. Perform format validation on the parsed results to ensure the fields are complete and generate a structured request information parsing report.

8. A data security-based development data query management method according to claim 7, characterized in that: When extracting multi-dimensional features, the feature dimensions to be extracted are first determined based on the parsed execution request information. Based on the feature dimensions, features such as execution request time, task urgency, data size, and data type are extracted. Then, non-numerical features are converted into standardized numerical values, and structured feature vectors are generated based on the converted standardized numerical values ​​and numerical features.

9. A data security-based development data query management method according to claim 8, characterized in that: When generating the storage node priority ranking, the feature values ​​are weighted and summed based on the structured feature vector and the real-time load data of the nodes to obtain the priority score of each node. Then, the nodes are sorted in descending order of priority scores to generate the storage node priority ranking table. When standardizing query task allocation, the node with the highest score is selected as a candidate node from the priority sorting table. The real-time load rate of the candidate node is checked to see if it is ≤ a preset threshold. If it is ≤ a preset threshold, the task is directly allocated to the node and the node task queue is updated. Otherwise, the node is skipped and the next priority node in the sorting is selected for re-verification until a node that meets the load requirements is found.

10. A data security-based development data query management system, used to implement the data security-based development data query management method according to any one of claims 1-9, characterized in that: include: The security management module integrates a three-level permission model, performs access permission pre-verification and security processing from the dimensions of users, roles and data resources, records full-link operation logs, and configures the log retention period; The dynamic scheduling module collects the status data of storage nodes in real time and performs weighted calculations to generate a priority ranking of storage nodes, and allocates standardized query tasks to the optimal storage node. The fault-tolerant execution module monitors storage node failures and task execution status in real time, triggers an automatic task retry mechanism, and performs multi-node backup storage of core data. A multi-source adaptation module that adapts to multiple mainstream databases and completes task parsing, conversion, and execution; The task lifecycle management module is used to display the full status of tasks, collect quantitative indicators of task execution, and provide configuration of multi-mode scheduling strategies. The storage cluster module is used to store the core data of each node; The interactive module provides a query window that displays task status, quantitative indicators, and operation logs in real time, as well as report generation and export.

Images