Multi-strategy caching and automatic data generation method and system based on Node environment

Abstract

The invention provides a multi-strategy caching and automatic data generation method and system based on a Node environment, and relates to the technical field of data processing, and the method comprises the following steps: receiving a client request, obtaining a data generation template, constructing a data dependency graph to analyze a field dependency relationship, and determining a field generation sequence by adopting a recursive generation strategy and depth-first search; data validity verification is executed, a data generation thread pool is created by adopting an adaptive load balancing mechanism based on the data dependency graph, and asynchronous processing is realized through double buffer queues, so that the data generation efficiency is improved, the data logic integrity is ensured, and the system resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-strategy caching and automatic data generation method and system based on a Node environment. Background Art

[0002] With the rapid development of information technology, data simulation and test data generation have become crucial components of software development and testing. In the Node.js environment, developers require large amounts of simulated data for application testing, demonstration, and development. Currently, a variety of data generation tools are available on the market. These tools typically generate test data based on predefined templates or random algorithms. These tools are widely used in scenarios such as web application development, interface testing, and performance evaluation.

[0003] In real-world development environments, especially when building large-scale applications or microservices-based systems, data generation tools must handle complex data structures, inter-field dependencies, and high-performance data generation. Developers often need to simulate the data interactions and data flows of real business scenarios. This requires data generation tools to not only provide basic random data but also support complex data models and efficient caching mechanisms.

[0004] However, the existing data generation technology based on the Node environment has the following defects and shortcomings: the existing technology lacks a systematic method for processing complex data structures and dependencies between fields, and usually adopts hard coding or simple rules to process the relationship between fields. It cannot effectively handle deep data dependencies, resulting in problems with the logical consistency of the generated data, which is difficult to meet the testing requirements of complex business scenarios; the existing data generation tools are insufficient in performance optimization, especially in large data volume generation scenarios, lack effective concurrent processing mechanisms and load balancing strategies, resulting in uneven resource utilization and low generation efficiency during the data generation process, which cannot meet the needs of high-performance application development; the existing technology is relatively simple in data caching strategies, usually adopts a fixed caching mechanism, lacks flexible multi-level caching strategies and cache management mechanisms, and cannot dynamically adjust the caching strategy according to different data request characteristics and business scenarios, resulting in system resource waste or low cache hit rate. Summary of the Invention

[0005] The embodiments of the present invention provide a multi-strategy caching and automatic data generation method and system based on a Node environment, which can solve the problems in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a multi-strategy caching and automatic data generation method based on a Node environment, comprising:

[0007] Receive a data request sent by a client, wherein the data request includes a data type identifier, a data volume parameter, and a cache strategy parameter;

[0008] Obtaining a corresponding data generation template from a preset data template library according to the data type identifier, and performing type verification and value range restriction on each data field according to the data structure definition and field constraint rules in the data generation template;

[0009] A data dependency graph is constructed based on the data generation template and the verified field constraint rules. The dependency relationship between data fields is analyzed through a graph traversal algorithm. A recursive generation strategy is adopted for data fields with dependency relationships. During the recursive generation process, a depth-first search is used to determine the field generation order. Data validity verification is performed in each recursive layer according to the constraint rules defined in the data generation template to ensure the logical integrity of the generated target data. The generated target data is structurally matched with the data generation template to ensure that the target data meets the preset data structure requirements.

[0010] Based on the field generation order of the data dependency graph and the data volume parameter, an adaptive load balancing mechanism is used to dynamically create a data generation thread pool in the Node environment; each thread in the thread pool executes the target data generation task according to the field generation order, and asynchronous processing of target data generation and cache writing is achieved through a double-buffer queue mechanism;

[0011] The target data is stored according to the cache policy parameters and returned to the client.

[0012] Obtaining a corresponding data generation template from a preset data template library according to the data type identifier includes:

[0013] Extracting multi-dimensional feature information based on the data type identifier, performing dimensionality reduction processing on the multi-dimensional feature information to obtain main feature components, calculating information importance based on the uniqueness and discrimination of the main feature components, and combining the main feature components whose information importance is higher than a preset importance threshold to generate initial semantic features;

[0014] A hierarchical index structure of a data template library is constructed based on the initial semantic features, the conditional probability between each index node and its upper-level node is calculated, and a node importance score is obtained by combining the index depth weight value; a dual-channel retrieval network is established in the hierarchical index structure according to the node importance score;

[0015] Inputting a search request into the dual-channel search network, respectively calculating the transmission probability of the search request in the direct matching channel and the associated matching channel of the dual-channel search network; and preliminarily screening the data generation template according to the transmission probability;

[0016] The feature similarity of the candidate templates obtained by screening is calculated, and the feature similarity is weightedly fused with the historical matching accuracy of the template to obtain a comprehensive score. Finally, the data generation template with the highest comprehensive score is selected from the data template library as the retrieval result output.

[0017] A data dependency graph is constructed based on the data generation template and the verified field constraint rules, and the dependency relationships between data fields are analyzed using a graph traversal algorithm, including:

[0018] The data fields in the verified field constraint rules are set as graph nodes, the dependencies between the fields are set as graph edges, the dependency eigenvalues ​​between each pair of nodes are calculated, and the dependency eigenvalues ​​are attenuated and assigned to the corresponding graph edges as weights, thereby constructing a data dependency graph;

[0019] The data dependency graph is divided into a source node layer, an intermediate node layer, and a target node layer according to the ratio of the node in-degree to the node out-degree. The nodes in each layer are sorted according to the total connection weight. Starting from the source node in the highest layer, the adjacent nodes in each layer are traversed using a breadth-first strategy and all nodes whose distance to the current node is less than a preset distance threshold are recorded to obtain the local dependency node set of the layer.

[0020] Taking a node in the local dependency node set as the starting point, and under the constraints that the path length is less than a preset length threshold and the edge weights on the path are all greater than the preset minimum weight requirements, traverse to the lower-level nodes through a depth-first strategy to obtain a complete cross-level dependency path set; accumulate and calculate the dependency strength along the forward and reverse dependency paths respectively, multiply the dependency strengths in the two directions to obtain a dependency reliability measure value, and output the dependency relationship whose dependency reliability measure value is higher than the preset dependency threshold as the final verification result.

[0021] The order of field generation is determined by depth-first search during the recursive generation process, including:

[0022] During the recursive generation process, the number of dependent fields of the field node to be processed is obtained; the hierarchical depth of the field node in the current recursive path is calculated; the number of dependent fields and the hierarchical depth are normalized and weighted to obtain the hierarchical value of the field node;

[0023] Entering the depth-first field generation phase, for each field on the current search path, obtain the penalty factor corresponding to the level where the field is located; combine the level value of the field node with the cumulative result of the penalty factors of each level before the current level to obtain the path score of the field;

[0024] During the search process, the number of visited fields and the number of fields to be visited are counted in real time, and the depth limit coefficient is dynamically adjusted according to the ratio of the number of visited fields to the total number of fields; the pre-set baseline search depth is combined with the depth limit coefficient to obtain the maximum depth limit of the current search stage;

[0025] When the search reaches the maximum depth limit, the backtracking mechanism is triggered to return to the nearest branch point, and a new search direction is selected to continue exploring until a field generation order that satisfies the dependency constraints is found.

[0026] Perform structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements, including:

[0027] The target data is constructed into a target data tree according to the node relationship, and the data generation template is constructed into a template data tree according to the node relationship; for each node in the target data tree, the cosine distance of the corresponding node in the template data tree is used as the node-level structure matching score, and the structure matching scores of all nodes are accumulated to obtain a structure matching basic score;

[0028] Structural layer features are extracted based on the hierarchical distribution relationship of the target data tree, semantic layer features are extracted based on the association strength between nodes in the target data tree, and constraint layer features are obtained by performing constraint analysis on the target data tree; a comprehensive feature score reflecting the multidimensional characteristics of the data structure is obtained by weighted combination of the three features and their corresponding weight coefficients;

[0029] When the comprehensive feature score is greater than a preset score threshold, it is confirmed that the target data meets the preset data structure requirements.

[0030] Adopting adaptive load balancing mechanism, dynamically creating data generation thread pool in Node environment, including:

[0031] Obtaining a system baseline thread number, a system current load, and a system maximum load, multiplying a ratio of the system current load to the system maximum load by the system baseline thread number to obtain an initial thread number; and creating a data generation thread pool based on the initial thread number;

[0032] Obtaining the load variance and response time of the data generation thread pool; calculating a load weight coefficient and a response weight coefficient based on historical operation data; obtaining a load score by multiplying the load variance and the load weight coefficient, multiplying the response time by the response weight coefficient to obtain a response score, and adding the load score and the response score to obtain a load balancing score;

[0033] When the load balancing score exceeds the preset load threshold, the load balancing operation of the data generation task is triggered; when the load increases, data generation threads are dynamically added to the data generation thread pool; otherwise, idle threads are removed; the adjusted running data is used as the new historical running data for the next round of data generation and resource adjustment.

[0034] A second aspect of an embodiment of the present invention provides a multi-strategy caching and automatic data generation system based on a Node environment, comprising:

[0035] The first unit is configured to receive a data request sent by a client, wherein the data request includes a data type identifier, a data volume parameter, and a cache strategy parameter;

[0036] The second unit is configured to obtain a corresponding data generation template from a preset data template library according to the data type identifier, and perform type verification and value range restriction on each data field according to the data structure definition and field constraint rules in the data generation template;

[0037] The third unit is used to construct a data dependency graph based on the data generation template and the verified field constraint rules, analyze the dependency relationship between data fields through a graph traversal algorithm, adopt a recursive generation strategy for data fields with dependencies, determine the field generation order through depth-first search during the recursive generation process, and perform data validity verification according to the constraint rules defined in the data generation template in each recursive layer to ensure the logical integrity of the generated target data; perform structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements;

[0038] The fourth unit is configured to dynamically create a data generation thread pool in a Node environment using an adaptive load balancing mechanism based on the field generation order of the data dependency graph and the data volume parameter; each thread in the thread pool executes the target data generation task according to the field generation order, and implements asynchronous processing of target data generation and cache writing through a double-buffered queue mechanism;

[0039] The fifth unit is used to store the target data according to the cache policy parameters and return it to the client.

[0040] According to a third aspect of the embodiments of the present invention,

[0041] An electronic device is provided, comprising:

[0042] processor;

[0043] a memory for storing processor-executable instructions;

[0044] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0045] According to a fourth aspect of the embodiments of the present invention,

[0046] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0047] The beneficial effects of this application are as follows:

[0048] The present invention implements a multi-strategy caching and automatic data generation method based on the Node environment, which can flexibly generate data that conforms to specific structures and constraints according to the parameters in the client request, greatly improving the automation and adaptability of data generation, reducing the manual operations of developers, and significantly improving development and testing efficiency.

[0049] The data dependency graph and depth-first search algorithm constructed by the present invention ensure the logical integrity and consistency of the generated data. By adopting a recursive generation strategy for data fields with dependencies, the precise construction of complex data structures is achieved, which effectively solves the technical problem of traditional data generation methods that are difficult to handle dependencies between complex data, and improves data quality.

[0050] The present invention adopts an adaptive load balancing mechanism and double-buffer queue technology to achieve efficient asynchronous processing of data generation and cache writing, fully utilizing the asynchronous non-blocking characteristics of the Node environment, greatly improving the system's concurrent processing capability and response speed. It is particularly suitable for large-scale data generation scenarios, and at the same time optimizes data storage and access efficiency through a multi-strategy caching mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a multi-strategy caching and automatic data generation method based on a Node environment according to an embodiment of the present invention;

[0052] Figure 2 Flowchart for data template selection based on hierarchical dual-channel retrieval. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0054] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes in some embodiments will not be repeated.

[0055] Figure 1 Schematic diagram of the process of the multi-strategy caching and automatic data generation method based on the Node environment according to the embodiment of the present invention. Figure 1 As shown, the method includes:

[0056] Receive a data request sent by a client, wherein the data request includes a data type identifier, a data volume parameter, and a cache strategy parameter;

[0057] Obtaining a corresponding data generation template from a preset data template library according to the data type identifier, and performing type verification and value range restriction on each data field according to the data structure definition and field constraint rules in the data generation template;

[0058] A data dependency graph is constructed based on the data generation template and the verified field constraint rules. The dependency relationship between data fields is analyzed through a graph traversal algorithm. A recursive generation strategy is adopted for data fields with dependency relationships. During the recursive generation process, a depth-first search is used to determine the field generation order. Data validity verification is performed in each recursive layer according to the constraint rules defined in the data generation template to ensure the logical integrity of the generated target data. The generated target data is structurally matched with the data generation template to ensure that the target data meets the preset data structure requirements.

[0059] Based on the field generation order of the data dependency graph and the data volume parameter, an adaptive load balancing mechanism is used to dynamically create a data generation thread pool in the Node environment; each thread in the thread pool executes the target data generation task according to the field generation order, and asynchronous processing of target data generation and cache writing is achieved through a double-buffer queue mechanism;

[0060] The target data is stored according to the cache policy parameters and returned to the client.

[0061] In an optional implementation, obtaining a corresponding data generation template from a preset data template library according to the data type identifier includes:

[0062] Extracting multi-dimensional feature information based on the data type identifier, performing dimensionality reduction processing on the multi-dimensional feature information to obtain main feature components, calculating information importance based on the uniqueness and discrimination of the main feature components, and combining the main feature components whose information importance is higher than a preset importance threshold to generate initial semantic features;

[0063] A hierarchical index structure of a data template library is constructed based on the initial semantic features, and the conditional probability between each index node and its upper-level node is calculated. The node importance score is obtained by combining the index depth weight value, and the node importance score is used to measure the influence of the index node on the retrieval results; a dual-channel retrieval network is established in the hierarchical index structure based on the node importance score, wherein the direct matching channel is constructed based on the direct association relationship between index nodes, and the association matching channel is constructed based on the indirect association relationship between nodes;

[0064] Inputting a search request into the dual-channel search network, calculating the transmission probability of the search request in the direct matching channel and the associated matching channel respectively, wherein the transmission probability decays exponentially with the extension of the path; and performing preliminary screening of the data generation template according to the transmission probability;

[0065] The feature similarity of the candidate templates obtained by screening is calculated, and the feature similarity is weightedly fused with the historical matching accuracy of the template to obtain a comprehensive score. Finally, the data generation template with the highest comprehensive score is selected from the data template library as the retrieval result output.

[0066] To extract multidimensional feature information based on data type identifiers, the system first analyzes the semantic structure of the data type identifier, extracting feature information encompassing dimensions such as data format, data granularity, business domain, and usage scenario. For example, for the "Transactional Financial Data" type identifier, the system extracts feature dimensions such as "Financial Domain," "Transaction Type," "Numeric Data," and "Time Series Characteristics." To reduce computational complexity, the system uses principal component analysis to reduce the dimensionality of these multidimensional features, selecting the feature components that most effectively distinguish data types. In implementation, the system calculates the contribution of each feature component to data type distinction and quantifies this contribution as information importance. Assuming the system sets an importance threshold of 0.75, the "Financial Domain" feature with an importance of 0.83 and the "Transaction Type" feature with an importance of 0.79 will be retained and combined into the initial semantic features for "Financial Transaction," while the "Time Series Characteristics" feature with an importance of 0.62 will be temporarily filtered out.

[0067] When constructing a hierarchical index structure for the data template library based on initial semantic features, the system organizes the data templates in a tree structure, with the top-level nodes representing the most basic feature categories, such as "industry," the second level representing "data type," and the third level representing "specific application scenario." Within this index structure, the system calculates the conditional probability between each child node and its parent node, representing the probability of that child node appearing given the parent node. An index depth weighting factor is also introduced to appropriately increase the importance of deeper nodes, preventing excessive retrieval bias toward top-level features. For example, under the "Finance" parent node, the conditional probability of the "Transaction Data" child node is 0.45. Combined with a depth weight of 1.2, the node's importance score is calculated to be 0.54. Based on these node importance scores, the system constructs a dual-channel retrieval network: a direct matching channel based on direct parent-child or sibling relationships between nodes, and an associated matching channel based on indirect semantic relationships between nodes, enabling the capture of cross-level feature correlations.

[0068] During the retrieval process, the system converts the data type identifier into a search request vector and inputs it into the dual-channel retrieval network. In the direct matching channel, the system calculates the transfer probability along the hierarchical structure of the index tree. For example, when searching for "bank transaction records," the transfer probability of the retrieved information from the "Finance" node to the "Transaction Data" node is 0.85, and the transfer probability of the retrieved information to the "Banking Business" node is 0.72. As the path extends, the transfer probability decays exponentially. This is achieved by multiplying the transfer probability by a decay factor less than 1, such as 0.9, with each node transfer. In the association matching channel, the system considers indirect connections between nodes. For example, the semantic connection between "bank transaction records" and "payment system logs" results in an indirect transfer probability of 0.63. The system combines the transfer probabilities from both channels and selects the template with the highest overall transfer probability as the candidate set.

[0069] For the candidate template set, the system further calculates the feature vector similarity between the search request and the templates, typically using the cosine similarity method to quantify this similarity. For the search request "bank transaction records," the feature similarity for Template A is 0.88, and for Template B it is 0.76. The system also considers the templates' historical matching accuracy: for example, Template A has a historical accuracy of 92%, while Template B has a historical accuracy of 87%. Finally, the system performs a weighted fusion using weighting factors (e.g., a feature similarity weight of 0.7 and a historical accuracy weight of 0.3) to calculate a comprehensive score. For Template A, the comprehensive score is 0.88 × 0.7 + 0.92 × 0.3 = 0.892; for Template B, the comprehensive score is 0.76 × 0.7 + 0.87 × 0.3 = 0.793. Therefore, the system selects Template A, which has the higher score, as the final search result and returns it to the user.

[0070] In practical application scenarios, this method can adapt to template retrieval requirements for diverse data types. For example, when a user submits a "Mobile Call Record" data type identifier, the system first extracts feature dimensions such as "Communications Domain," "User Behavior," and "Time Series." After filtering by importance, it retains the core feature combinations. In a dual-channel retrieval network, the system can simultaneously match two candidate templates: a "Call Detail Template" and a "User Behavior Analysis Template." By calculating feature similarities (0.91 and 0.83, respectively) and combining these with historical accuracy rates (89% and 94%, respectively), the system selects the "Call Detail Template" with a combined score of 0.91 × 0.7 + 0.89 × 0.3 = 0.904 as the best match.

[0071] The above method achieves accurate retrieval of data generation templates through technical means such as multi-dimensional feature extraction, hierarchical index construction, dual-channel retrieval and comprehensive scoring. It can quickly locate the most suitable data template based on the data type identifier provided by the user, thereby improving data generation efficiency and accuracy.

[0072] In an optional embodiment, a data dependency graph is constructed based on the data generation template and the verified field constraint rules, and the dependency relationship between data fields is analyzed by a graph traversal algorithm, including:

[0073] The data fields in the verified field constraint rules are set as graph nodes, the dependencies between the fields are set as graph edges, the dependency eigenvalues ​​between each pair of nodes are calculated, and the dependency eigenvalues ​​are attenuated and assigned to the corresponding graph edges as weights, thereby constructing a data dependency graph;

[0074] The data dependency graph is divided into a source node layer, an intermediate node layer, and a target node layer according to the ratio of the node in-degree to the node out-degree. The nodes in each layer are sorted according to the total connection weight. Starting from the source node in the highest layer, the adjacent nodes in each layer are traversed using a breadth-first strategy and all nodes whose distance to the current node is less than a preset distance threshold are recorded to obtain the local dependency node set of the layer.

[0075] Taking a node in the local dependency node set as the starting point, and under the constraints that the path length is less than a preset length threshold and the edge weights on the path are all greater than the preset minimum weight requirements, traverse to the lower-level nodes through a depth-first strategy to obtain a complete cross-level dependency path set; accumulate and calculate the dependency strength along the forward and reverse dependency paths respectively, multiply the dependency strengths in the two directions to obtain a dependency reliability measure value, and output the dependency relationship whose dependency reliability measure value is higher than the preset dependency threshold as the final verification result.

[0076] When constructing a data dependency graph, the system first sets the data fields in the validated field constraint rules as graph nodes. For example, in an e-commerce order system, fields such as "User ID," "Order Amount," and "Payment Status" are all set as independent graph nodes. The system then sets the dependencies between fields as graph edges. For example, because "Order Amount" depends on "Product Unit Price" and "Product Quantity," directed edges are established between the "Product Unit Price" node and the "Order Amount" node, and between the "Product Quantity" node and the "Order Amount" node, respectively.

[0077] For each pair of nodes with a dependency relationship, the system calculates a dependency eigenvalue. This calculation is based on multiple metrics, including data correlation coefficient, co-occurrence frequency, and conditional probability. For example, for "unit price" and "order amount," the system calculated a correlation coefficient of 0.85 and a co-occurrence frequency of 0.92 based on historical data, resulting in a dependency eigenvalue of 0.88. To account for the gradual weakening of dependencies as they pass through hierarchical levels, the system introduces a decay mechanism that exponentially attenuates the dependency eigenvalue. For example, if the decay coefficient is set to 0.9, the dependency eigenvalue between nodes with a distance of 2 will become 0.81 times its original value. These attenuated dependency eigenvalues ​​are assigned to the corresponding graph edges as weights, completing the construction of the data dependency graph.

[0078] Once constructed, the system divides the data dependency graph into different levels based on the in-degree to out-degree ratio of nodes. In-degree refers to the number of edges pointing to a node, while out-degree refers to the number of edges pointing to other nodes. Nodes with an in-degree to out-degree ratio less than 0.3 are classified as source nodes, such as "User ID" and "Product ID." Nodes with an in-degree to out-degree ratio greater than 3 are classified as target nodes, such as "Order Status" and "Logistics Status." The remaining nodes are classified as intermediate nodes, such as "Payment Amount" and "Shipping Address." Within each level, the system ranks nodes by their total connection weight (the sum of the weights of all edges connected to the node), with nodes with higher weights ranking higher.

[0079] Graph traversal begins at the source node at the highest level. Using a breadth-first strategy, the system begins with the highest-weighted node at the source node level (e.g., "User ID") and traverses its adjacent nodes. If the distance between a node and the current node is less than a preset distance threshold (e.g., 2), it is recorded in the local dependency node set. For example, starting from the "User ID" node, one can find adjacent nodes with a distance of 1, such as "User Level" and "Number of Historical Orders," as well as nodes reachable further through these nodes, such as "Discount Rate" (via "User Level"). Together, these nodes constitute the local dependency node set at the source node level.

[0080] Starting from a node within the local dependency node set, the system further performs a depth-first traversal to construct a complete set of cross-level dependency paths. During this traversal, the system imposes three constraints: the path length must be less than a preset length threshold (e.g., 4), the weight of each edge along the path must be greater than a preset minimum weight requirement (e.g., 0.5), and the traversal direction must be from higher layers to lower layers. Starting from "User ID," the system finds the following dependency path: "User ID" → "User Level" → "Discount Rate" → "Amount Paid." Each path represents a specific data flow and dependency chain.

[0081] For each identified dependency path, the system assesses its reliability by cumulatively calculating dependency strength. This is done by calculating the forward and reverse dependency strengths separately, then multiplying the strengths in both directions to obtain a dependency reliability metric. The forward dependency strength is the cumulative product of all edge weights along the path, reflecting the influence of the source node on the target node. The reverse dependency strength is the cumulative product of all edge weights along the reverse path, reflecting the target node's dependence on the source node. For example, if the path "User ID → User Level → Discount Rate → Actual Payment Amount" has a forward dependency strength of 0.75 and a reverse dependency strength of 0.68, the dependency reliability metric is 0.51. The system ultimately outputs the dependency reliability metric for any dependency relationship above the preset dependency threshold (e.g., 0.5) as the final verification result.

[0082] Through the above implementation methods, the present invention can comprehensively and accurately analyze the dependencies between data fields, effectively support application scenarios such as data quality monitoring, anomaly detection and data lineage analysis, and improve the efficiency and accuracy of data governance.

[0083] In an optional embodiment, determining the field generation order by depth-first search during the recursive generation process includes:

[0084] During the recursive generation process, the number of dependent fields of the field node to be processed is obtained, where the number of dependent fields represents the strength of association between the field and other fields; the hierarchical depth of the field node in the current recursive path is calculated, where the hierarchical depth reflects the recursive dependency relationship between fields; the number of dependent fields and the hierarchical depth are normalized and weighted to obtain the hierarchical value of the field node;

[0085] Entering the depth-first field generation phase, for each field on the current search path, obtain the penalty factor corresponding to the level where the field is located; combine the level value of the field node with the cumulative result of the penalty factors of each level before the current level to obtain the path score of the field, which is used to guide the selection of the search direction;

[0086] During the search process, the number of visited fields and the number of fields to be visited are counted in real time, and the depth limit coefficient is dynamically adjusted according to the ratio of the number of visited fields to the total number of fields; the pre-set baseline search depth is combined with the depth limit coefficient to obtain the maximum depth limit of the current search stage;

[0087] When the search reaches the maximum depth limit, the backtracking mechanism is triggered to return to the nearest branch point, and a new search direction is selected to continue exploring until a field generation order that satisfies the dependency constraints is found.

[0088] During implementation, the system first constructs a field dependency graph, where each node represents a field to be generated, and the edges between nodes represent the dependencies between fields. Suppose there is a set of fields F = {f1, f2, ..., fn}, where each field depends on other fields. For example, in a product information system, the "Product Discount Price" field depends on the "Product Original Price" field and the "Discount Rate" field.

[0089] When the system initiates the recursive generation process, it begins with the currently processed field node and retrieves the number of dependent fields for that node. The number of dependent fields is extracted directly from the dependency graph and represents the number of other fields directly related to the field. For example, if field f3 depends on f1 and f2, then the number of dependent fields for f3 is 2. This value represents the strength of the field's association with other fields; the more dependent fields there are, the higher the degree of association.

[0090] At the same time, the system calculates the hierarchical depth of the field node in the current recursive path. Hierarchical depth refers to the length of the path from the search starting point to the current node, reflecting the recursive dependency relationship between fields. For example, if the current search path is f1 → f2 → f5, the hierarchical depth of f5 is 3. A greater hierarchical depth indicates that the field is further back in the dependency chain.

[0091] To make these two metrics comparable, the system normalizes the number of dependent fields and the hierarchical depth. The normalized value of the number of dependent fields is equal to the number of dependencies of the field divided by the maximum number of dependencies in the field set. The normalized value of the hierarchical depth is equal to the current depth divided by the preset maximum search depth limit. Assuming that the number of dependencies of a field is 3 and the maximum number of dependencies is 5, the normalized number of dependent fields is 0.6; if the hierarchical depth of the field is 2 and the maximum search depth is 10, the normalized hierarchical depth is 0.2.

[0092] The system performs a weighted calculation on the normalized number of dependent fields and the hierarchical depth to determine the hierarchical value of a field node. Specifically, the hierarchical value is equal to the normalized number of dependent fields multiplied by the weight α plus the normalized hierarchical depth multiplied by the weight β, where α and β are preset weight parameters, and α + β = 1. In practice, these two parameters can be adjusted based on business needs. For example, α = 0.7 and β = 0.3 indicates a greater emphasis on inter-field dependencies.

[0093] After entering the depth-first field generation phase, the system obtains the penalty factor corresponding to the level of each field in the current search path. The penalty factor is used to control the balance between breadth and depth of the search and generally increases with the depth of the level. For example, the penalty factor for the first level can be set to 0.1, the second level to 0.2, and so on.

[0094] The system combines the field node's level value with the cumulative penalty factors of each level before the current level to calculate the field's path score. Specifically, the path score is equal to the field's level value multiplied by (1 minus the cumulative value of the penalty factors of each level). For example, if a field has a level value of 0.8, is currently at the third level, and the cumulative penalty factors of the first two levels are 0.3, the field's path score is 0.8 × (1 - 0.3) = 0.56. A higher path score indicates a higher priority for the field in the current search path.

[0095] During the search process, the system counts the number of fields visited and the number of fields to be visited in real time. Assuming that the total number of fields is 100 and 30 have been visited, the percentage of fields visited is 0.3. The system dynamically adjusts the depth limit coefficient based on this ratio. The depth limit coefficient can be calculated using a decreasing function. For example, the depth limit coefficient is equal to 1 minus the square of the percentage of fields visited, which in the above example is 1-(0.3). 2 =0.91.

[0096] The system combines a pre-set baseline search depth with a depth limit coefficient to determine the maximum depth limit for the current search phase. For example, if the baseline search depth is 15 and the depth limit coefficient is 0.91, the current maximum depth limit is 15 × 0.91 ≈ 13. This dynamic adjustment mechanism ensures deeper exploration in the early stages of the search. As the number of visited fields increases, the search depth gradually decreases, prompting the algorithm to backtrack and explore other paths.

[0097] When the search reaches the maximum depth limit, the system triggers a backtracking mechanism, returning to the nearest branch point and selecting a new search direction to continue exploration. Specifically, the system rolls back to the previous level and starts a new search from the next unvisited neighbor of the current node. If all neighbors have been visited, the system continues backtracking until it finds a node with unvisited neighbors.

[0098] The system selects the next field to visit based on the path score. At each branch point, the system calculates the path scores of all optional fields and selects the field with the highest score as the next target. If multiple fields have the same score, they can be sorted based on additional information such as their identifier or creation time.

[0099] Through the depth-first search process described above, the system ultimately determines a field generation order that satisfies all dependency constraints. For example, for a system containing 10 fields, the resulting generation order might be f1 → f3 → f2 → f5 → f4 → f7 → f6 → f9 → f8 → f10. This ensures that each field is generated after its dependent fields, thus guaranteeing data consistency and integrity.

[0100] In an optional embodiment, performing structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements includes:

[0101] The target data is constructed into a target data tree according to the node relationship, and the data generation template is constructed into a template data tree according to the node relationship; for each node in the target data tree, the cosine distance of the corresponding node in the template data tree is used as the node-level structure matching score, and the structure matching scores of all nodes are accumulated to obtain a structure matching basic score;

[0102] Structural layer features are extracted based on the hierarchical distribution relationship of the target data tree, semantic layer features are extracted based on the association strength between nodes in the target data tree, and constraint layer features are obtained by performing constraint analysis on the target data tree; a comprehensive feature score reflecting the multidimensional characteristics of the data structure is obtained by weighted combination of the three features and their corresponding weight coefficients;

[0103] When the comprehensive feature score is greater than a preset score threshold, it is confirmed that the target data meets the preset data structure requirements.

[0104] In this embodiment, the target data and data generation template are first obtained. Both are data sets with specific structural relationships. The target data can be the JSON format product description information output by the intelligent generation system, and the data generation template is a pre-defined standard data structure template.

[0105] When constructing the target data into a target data tree based on node relationships, the system analyzes the hierarchical, reference, and logical relationships between the data items in the target data to construct a tree structure. For example, for an electronic device description, the data contains main nodes such as "Product Name," "Technical Parameters," and "Functional Features," while "Technical Parameters" contains sub-nodes such as "Processor," "Memory," and "Storage." At this point, the system organizes each node into a tree structure based on its subordinate relationships, with "Product Name," "Technical Parameters," and "Functional Features" as the first-level nodes and "Processor," "Memory," and "Storage" as the second-level nodes. Similarly, the data generation template is constructed into a template data tree in the same manner.

[0106] For each node in the target data tree, the system will search for the corresponding node in the template data tree and calculate the cosine distance between the two nodes as the node-level structural matching score. In specific implementation, the attributes, values, types and other information of the node are converted into vector representations, and then the cosine similarity between the vectors is calculated. For example, the "processor" node in the target data tree contains three attributes: "model", "main frequency", and "number of cores", with values ​​of "A12", "2.4GHz", and "8" respectively. The corresponding node in the template data tree also contains the same three attributes but with values ​​of "B11", "2.2GHz", and "6". The system will convert this information into a vector and calculate the cosine distance, obtaining a matching score of 0.85 for the node. The same operation is performed on all nodes in the target data tree, and the basic structural matching score is accumulated, for example, the accumulated score is 45.6.

[0107] Next, the system extracts features from three dimensions and performs a comprehensive score. When extracting structural features, the hierarchical distribution of the target data tree is analyzed, including the tree depth, the number of nodes per layer, and the branching factor. For example, for the aforementioned electronic device description data, the tree depth is 3, with 3 nodes on the first layer, 8 nodes on the second layer, and 15 nodes on the third layer. The calculated structural feature score is 0.92.

[0108] When extracting semantic features, the system analyzes the strength of associations between nodes in the target data tree. Node association strength is calculated based on factors such as inter-node reference relationships and content relevance. For example, the "Memory" node and the "Performance" node have a reference relationship and high content relevance, resulting in an association strength of 0.88. However, the "Product Color" and "Processor Model" nodes have a low association strength of 0.12. Combining all inter-node association strengths yields a semantic feature score of 0.85.

[0109] When extracting constraint-layer features, the system performs constraint analysis on the target data tree, checking whether the data meets constraints such as business rules, data integrity, and value ranges. For example, it checks whether the value of the "Price" node is a positive number or whether the "Production Date" conforms to a time format. By verifying these various constraints, the constraint-layer feature score is 0.95.

[0110] The system calculates a comprehensive feature score by weighting the three features with their corresponding weight coefficients. In this example, the weight of the structural layer feature is 0.4, the weight of the semantic layer feature is 0.3, and the weight of the constraint layer feature is 0.3. The calculated comprehensive feature score is 0.91, calculated as 0.92 × 0.4 + 0.85 × 0.3 + 0.95 × 0.3 = 0.91.

[0111] Assuming that the preset scoring threshold is 0.8, since the comprehensive feature score of 0.91 is greater than the preset scoring threshold of 0.8, the system confirms that the target data meets the preset data structure requirements and can enter the subsequent processing flow.

[0112] This method allows for flexible adjustments to weight coefficients and scoring thresholds for different application scenarios. For example, in the financial sector, which strictly requires data integrity, the constraint layer feature weight can be increased to 0.5 and the scoring threshold to 0.9. In the content creation sector, the constraint layer requirements can be appropriately lowered while the semantic layer weight can be increased.

[0113] This method can also provide optimization suggestions for data that doesn't meet requirements. For example, if a target data score is 0.75, which falls below the 0.8 threshold, the system will identify the dimensions and nodes with low scores, such as the "Technical Parameters" node that lacks the necessary "Compatibility" information, and provide specific correction suggestions.

[0114] Through the above technical solution, the present invention achieves structured verification of generated data, ensuring data quality and improving the reliability of subsequent data processing and application. This method is particularly suitable for large-scale data generation scenarios, such as intelligent content creation, automatic product information generation, and knowledge graph construction.

[0115] In an optional implementation, an adaptive load balancing mechanism is used to dynamically create a data generation thread pool in a Node environment, including:

[0116] Obtaining a system baseline thread number, a system current load, and a system maximum load, multiplying a ratio of the system current load to the system maximum load by the system baseline thread number to obtain an initial thread number; and creating a data generation thread pool based on the initial thread number;

[0117] Obtaining the load variance and response time of the data generation thread pool; calculating a load weight coefficient and a response weight coefficient based on historical operation data; obtaining a load score by multiplying the load variance and the load weight coefficient, multiplying the response time by the response weight coefficient to obtain a response score, and adding the load score and the response score to obtain a load balancing score;

[0118] When the load balancing score exceeds the preset load threshold, the load balancing operation of the data generation task is triggered; when the load increases, data generation threads are dynamically added to the data generation thread pool; otherwise, idle threads are removed; the adjusted running data is used as the new historical running data for the next round of data generation and resource adjustment.

[0119] The system first needs to obtain the system baseline thread count, the system's current load, and the system's maximum load. The system baseline thread count can be determined by obtaining the number of CPU cores in the Node.js environment, for example, using the os.cpus().length method to obtain the number of physical CPU cores. For example, on a server with an 8-core processor, the system baseline thread count is 8. The system's current load can be determined by using the os.loadavg() method to obtain the average load values ​​for the past 1, 5, and 15 minutes, with the 1-minute average load used as the current load indicator. For example, the current 1-minute average load is 3.5. The system's maximum load is typically related to the number of CPU cores and can be set to twice the number of CPU cores, or 16 in this example.

[0120] The system calculates the initial number of threads by multiplying the ratio of the current load to the maximum load by the system's baseline number of threads. In the above example, the current load is 3.5, the maximum load is 16, the ratio is 0.21875, and the baseline number of threads is 8. Therefore, the initial number of threads is 8 × 0.21875 = 1.75, rounded up to 2. Based on this calculated initial number of threads, the system creates a data generation thread pool. In the Node.js environment, the worker_threads module can be used to implement multithreaded processing, creating a thread pool consisting of two worker threads. Each worker thread is responsible for independent data generation tasks and communicates with the main thread through message passing.

[0121] After the thread pool is created, the system continuously monitors its operating status to obtain load variance and response time. Load variance reflects the balance of load distribution among threads and can be obtained by regularly sampling the CPU utilization of each thread and calculating its variance. For example, if the CPU utilization of two threads is 65% and 55%, respectively, with an average of 60%, the load variance is 25. Response time represents the average processing time of data generation tasks and can be obtained by recording the time interval between task submission and completion. For example, the average response time of the current task is 120 milliseconds.

[0122] The system calculates the load weight coefficient and response weight coefficient based on historical operation data. Historical operation data includes load variance and response time records over a period of time in the past. The load weight coefficient indicates the degree of influence of load balancing on system performance, which can be determined based on the correlation between load variance and overall system performance in historical data. In this embodiment, analysis of historical data found that for every 10-point decrease in load variance, the system throughput increases by approximately 8%, and based on this, the load weight coefficient can be set to 0.8. The response weight coefficient indicates the degree of influence of response time on system performance, which can be determined based on the correlation between response time and user experience satisfaction in historical data. Analysis found that for every 20 milliseconds shortening of response time, user satisfaction increases by approximately 5%, and based on this, the response weight coefficient is set to 0.25.

[0123] The system calculates the load score by multiplying the load variance by the load weight factor. In the example above, the load variance is 25, the load weight factor is 0.8, and the load score is 25 × 0.8 = 20. The response score is calculated by multiplying the response time by the response weight factor. For a response time of 120 milliseconds and a response weight factor of 0.25, the response score is 120 × 0.25 = 30. Adding the load score and response score gives the load balancing score: 20 + 30 = 50.

[0124] The system has preset a load balancing threshold, for example, 45. When the calculated load balancing score is 50, which exceeds the preset load threshold of 45, the system triggers the load balancing operation of the data generation task. The system determines the current load trend by comparing the load values ​​through multiple consecutive samplings. If the load is found to be on an upward trend, such as the load values ​​of the last three samples are 3.2, 3.5, and 3.8, respectively, showing a clear upward trend, the system dynamically adds data generation threads to the data generation thread pool. The number of threads added can be determined by the degree to which the load balancing score exceeds the threshold. If the score exceeds the threshold by 5-15 points, 1 thread is added, if it exceeds by 15-30 points, 2 threads are added, and if it exceeds by more than 30 points, 3 threads are added. In this example, the score exceeds the threshold by 5 points, and the system adds 1 thread to the thread pool, bringing the total number of threads to 3.

[0125] Conversely, if the load shows a downward trend, such as when the load values ​​for the last three samples were 3.8, 3.5, and 3.2, respectively, indicating a significant downward trend, the system will remove idle threads from the thread pool. The system monitors the activity of each thread to identify threads that have been idle for a period of time. Idleness is determined when the thread's CPU usage remains below 20% for multiple consecutive samples. When removing threads, ensure that at least one active thread remains in the thread pool to ensure proper system operation.

[0126] After each load balancing operation, the system records the adjusted operational data, including the new load variance, response time, number of threads, and other information, as new historical operational data for use in the next round of data generation and resource adjustments. Historical operational data is stored using a sliding window, retaining data from the most recent period (e.g., the past 24 hours), ensuring that the system can make accurate resource adjustment decisions based on the latest operational conditions.

[0127] Through the above-mentioned adaptive load balancing mechanism, the system can dynamically adjust the resource configuration of the data generation thread pool according to the real-time load conditions, while ensuring the efficiency of data generation, optimizing the utilization of system resources and improving the overall performance.

[0128] A second aspect of an embodiment of the present invention provides a multi-strategy caching and automatic data generation system based on a Node environment, comprising:

[0129] The first unit is configured to receive a data request sent by a client, wherein the data request includes a data type identifier, a data volume parameter, and a cache strategy parameter;

[0130] The second unit is configured to obtain a corresponding data generation template from a preset data template library according to the data type identifier, and perform type verification and value range restriction on each data field according to the data structure definition and field constraint rules in the data generation template;

[0131] The third unit is used to construct a data dependency graph based on the data generation template and the verified field constraint rules, analyze the dependency relationship between data fields through a graph traversal algorithm, adopt a recursive generation strategy for data fields with dependencies, determine the field generation order through depth-first search during the recursive generation process, and perform data validity verification according to the constraint rules defined in the data generation template in each recursive layer to ensure the logical integrity of the generated target data; perform structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements;

[0132] The fourth unit is configured to dynamically create a data generation thread pool in a Node environment using an adaptive load balancing mechanism based on the field generation order of the data dependency graph and the data volume parameter; each thread in the thread pool executes the target data generation task according to the field generation order, and implements asynchronous processing of target data generation and cache writing through a double-buffered queue mechanism;

[0133] The fifth unit is used to store the target data according to the cache policy parameters and return it to the client.

[0134] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including:

[0135] processor;

[0136] a memory for storing processor-executable instructions;

[0137] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0138] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0139] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-strategy caching and automatic data generation method based on the Node environment, characterized in that: include: Receive a data request sent by a client, wherein the data request includes a data type identifier, a data volume parameter, and a cache strategy parameter; Obtaining a corresponding data generation template from a preset data template library according to the data type identifier, and performing type verification and value range restriction on each data field according to the data structure definition and field constraint rules in the data generation template; A data dependency graph is constructed based on the data generation template and the verified field constraint rules. The dependency relationship between data fields is analyzed through a graph traversal algorithm. A recursive generation strategy is adopted for data fields with dependencies. During the recursive generation process, a depth-first search is used to determine the field generation order. Data validity verification is performed at each recursive level according to the constraint rules defined in the data generation template to ensure the logical integrity of the generated target data. Performing structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements; Based on the field generation order of the data dependency graph and the data volume parameter, an adaptive load balancing mechanism is adopted to dynamically create a data generation thread pool in the Node environment; Each thread in the thread pool executes the target data generation task according to the field generation order, and realizes asynchronous processing of target data generation and cache writing through a double buffer queue mechanism; The target data is stored according to the cache policy parameters and returned to the client.

2. The method according to claim 1, characterized in that Obtaining a corresponding data generation template from a preset data template library according to the data type identifier includes: Extracting multi-dimensional feature information based on the data type identifier, performing dimensionality reduction processing on the multi-dimensional feature information to obtain main feature components, calculating information importance based on the uniqueness and discrimination of the main feature components, and combining the main feature components whose information importance is higher than a preset importance threshold to generate initial semantic features; A hierarchical index structure of a data template library is constructed based on the initial semantic features, the conditional probability between each index node and its upper-level node is calculated, and a node importance score is obtained by combining the index depth weight value; a dual-channel retrieval network is established in the hierarchical index structure according to the node importance score; Inputting a search request into the dual-channel search network, respectively calculating the transmission probability of the search request in the direct matching channel and the associated matching channel of the dual-channel search network; and preliminarily screening the data generation template according to the transmission probability; The feature similarity of the candidate templates obtained by screening is calculated, and the feature similarity is weightedly fused with the historical matching accuracy of the template to obtain a comprehensive score. Finally, the data generation template with the highest comprehensive score is selected from the data template library as the retrieval result output.

3. The method according to claim 1, characterized in that A data dependency graph is constructed based on the data generation template and the verified field constraint rules, and the dependency relationships between data fields are analyzed using a graph traversal algorithm, including: The data fields in the verified field constraint rules are set as graph nodes, the dependencies between the fields are set as graph edges, the dependency eigenvalues ​​between each pair of nodes are calculated, and the dependency eigenvalues ​​are attenuated and assigned to the corresponding graph edges as weights, thereby constructing a data dependency graph; The data dependency graph is divided into a source node layer, an intermediate node layer, and a target node layer according to the ratio of the node in-degree to the node out-degree. The nodes in each layer are sorted according to the total connection weight. Starting from the source node in the highest layer, the adjacent nodes in each layer are traversed using a breadth-first strategy and all nodes whose distance to the current node is less than a preset distance threshold are recorded to obtain the local dependency node set of the layer. Taking a node in the local dependency node set as the starting point, and under the constraints that the path length is less than a preset length threshold and the edge weights on the path are all greater than the preset minimum weight requirements, traverse to the lower-level nodes through a depth-first strategy to obtain a complete cross-level dependency path set; accumulate and calculate the dependency strength along the forward and reverse dependency paths respectively, multiply the dependency strengths in the two directions to obtain a dependency reliability measure value, and output the dependency relationship whose dependency reliability measure value is higher than the preset dependency threshold as the final verification result.

4. The method according to claim 1, wherein The order of field generation is determined by depth-first search during the recursive generation process, including: During the recursive generation process, the number of dependent fields of the field node to be processed is obtained; the hierarchical depth of the field node in the current recursive path is calculated; the number of dependent fields and the hierarchical depth are normalized and weighted to obtain the hierarchical value of the field node; Entering the depth-first field generation phase, for each field on the current search path, obtain the penalty factor corresponding to the level where the field is located; combine the level value of the field node with the cumulative result of the penalty factors of each level before the current level to obtain the path score of the field; During the search process, the number of visited fields and the number of fields to be visited are counted in real time, and the depth limit coefficient is dynamically adjusted according to the ratio of the number of visited fields to the total number of fields; the pre-set baseline search depth is combined with the depth limit coefficient to obtain the maximum depth limit of the current search stage; When the search reaches the maximum depth limit, the backtracking mechanism is triggered to return to the nearest branch point, and a new search direction is selected to continue exploring until a field generation order that satisfies the dependency constraints is found.

5. The method according to claim 1, wherein Perform structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements, including: The target data is constructed into a target data tree according to the node relationship, and the data generation template is constructed into a template data tree according to the node relationship; for each node in the target data tree, the cosine distance of the corresponding node in the template data tree is used as the node-level structure matching score, and the structure matching scores of all nodes are accumulated to obtain a structure matching basic score; Structural layer features are extracted based on the hierarchical distribution relationship of the target data tree, semantic layer features are extracted based on the association strength between nodes in the target data tree, and constraint layer features are obtained by performing constraint analysis on the target data tree; a comprehensive feature score reflecting the multidimensional characteristics of the data structure is obtained by weighted combination of the three features and their corresponding weight coefficients; When the comprehensive feature score is greater than a preset score threshold, it is confirmed that the target data meets the preset data structure requirements.

6. The method according to claim 1, characterized in that Adopting adaptive load balancing mechanism, dynamically creating data generation thread pool in Node environment, including: Obtaining a system baseline thread number, a system current load, and a system maximum load, multiplying a ratio of the system current load to the system maximum load by the system baseline thread number to obtain an initial thread number; and creating a data generation thread pool based on the initial thread number; Obtaining the load variance and response time of the data generation thread pool; calculating a load weight coefficient and a response weight coefficient based on historical operation data; obtaining a load score by multiplying the load variance and the load weight coefficient, multiplying the response time by the response weight coefficient to obtain a response score, and adding the load score and the response score to obtain a load balancing score; When the load balancing score exceeds the preset load threshold, the load balancing operation of the data generation task is triggered; when the load increases, data generation threads are dynamically added to the data generation thread pool; otherwise, idle threads are removed; the adjusted running data is used as the new historical running data for the next round of data generation and resource adjustment.

7. A multi-strategy caching and automatic data generation system based on a Node environment, for implementing the method according to any one of claims 1 to 6, characterized in that: include: The first unit is configured to receive a data request sent by a client, wherein the data request includes a data type identifier, a data volume parameter, and a cache strategy parameter; The second unit is configured to obtain a corresponding data generation template from a preset data template library according to the data type identifier, and perform type verification and value range restriction on each data field according to the data structure definition and field constraint rules in the data generation template; The third unit is used to construct a data dependency graph based on the data generation template and the verified field constraint rules, analyze the dependency relationship between data fields through a graph traversal algorithm, adopt a recursive generation strategy for data fields with dependencies, determine the field generation order through depth-first search during the recursive generation process, and perform data validity verification according to the constraint rules defined in the data generation template at each recursive level to ensure the logical integrity of the generated target data; Performing structural matching verification on the generated target data and the data generation template to ensure that the target data meets the preset data structure requirements; The fourth unit is used to dynamically create a data generation thread pool in the Node environment using an adaptive load balancing mechanism based on the field generation order of the data dependency graph and the data volume parameter; Each thread in the thread pool executes the target data generation task according to the field generation order, and realizes asynchronous processing of target data generation and cache writing through a double buffer queue mechanism; The fifth unit is used to store the target data according to the cache policy parameters and return it to the client.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images