A dynamic collaboration and deployment method for enterprise digital tool chain based on cloud computing

Abstract

The present application relates to a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method. According to the data mapping relationship change list, a rule-based reasoning method is used to analyze the changes in interface protocols and data fields, and according to the pre-configured interface protocol adaptation rules, a data mapping script between tools of different versions is generated; after deleting the obsolete interface, a graph-based shortest path search algorithm is used to bypass the deleted obsolete interface, re-plan the data transfer path between tools, and re-form a complete tool chain data transfer path; adjust the data segment size and transmission batch parameters to optimize the data flow efficiency in the tool chain, and dynamically adjust the execution order and concurrency of each tool in the tool chain according to the optimized data flow plan.

Description

Technical Field

[0001] The present application relates to the field of electrical digital data technology, and in particular to a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method. Background Art

[0002] In the dynamic collaboration scenario of an enterprise tool chain, when the upgrade of a core tool in the tool chain causes changes in the upstream and downstream interface protocols, how to efficiently adjust the data mapping relationship between tools is a technical problem that needs to be solved urgently;

[0003] The newly added data fields and abandoned interfaces have brought great challenges to the automatic adjustment mechanism of the tool chain collaboration system. The system needs to accurately identify the impact scope of interface changes and dynamically generate new mapping rules based on business logic and data dependencies. This process involves complex data structure comparison, semantic understanding and logical reasoning, which places high demands on the intelligence level of the algorithm. At the same time, frequent interface changes will also lead to continuous updating and adjustment of mapping rules, which in turn affects the execution efficiency of the entire tool chain.

[0004] How to minimize the impact of mapping adjustments on toolchain performance while ensuring data consistency and correctness is a technical problem that requires in-depth research and trade-offs;

[0005] This requires that the collaborative system fully considers the priority and calling frequency of data dependencies during the mapping adjustment process, reasonably sets adjustment strategies and time windows, and performs targeted optimization of mapping rules on critical paths, so as to find the best balance between dynamic adaptation and efficient execution and ensure the smooth operation of the tool chain. Summary of the invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the purpose of this application is to provide a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method.

[0007] The present application discloses a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method, comprising the following steps:

[0008] S101. Monitor the version changes of the core tools in the tool chain, obtain the newly added data fields and abandoned interface information after the upgrade, determine the differences between the interface protocols before and after the upgrade through a semantic analysis method, adjust the data mapping relationship according to the differences, obtain the adjustment content, and generate a data mapping relationship change list;

[0009] S102, according to the data mapping relationship change list in step S101, using a rule-based reasoning method, analyzing the changes in the interface protocol and data fields, and generating data mapping scripts between different versions of tools according to pre-configured interface protocol adaptation rules;

[0010] S103. When the tool version upgrade causes the interface protocol in step S102 to be changed, the data mapping relationship is adjusted. During the adjustment process, a similarity analysis is performed between the newly added fields and the existing fields to infer the relevance between the newly added fields and the existing fields, and the newly added fields are included in the data field matcher to construct a data mapping relationship, and the abandoned interface is deleted;

[0011] S104, after deleting the abandoned interface in step S103, using a graph-based shortest path search algorithm to bypass the deleted abandoned interface, re-plan the data transfer path between tools, and re-form the tool chain data transfer path;

[0012] S105, collecting the data processing delay and resource consumption of the tool nodes in the re-formed tool chain data transmission path in step S104, analyzing the average processing delay and throughput before and after the interface protocol change, determining whether there is network delay and bandwidth limitation, and if so, obtaining the data shard size and transmission batch parameters in the data processing process;

[0013] S106. According to the adjustment of the data shard size and the transmission batch parameters in step S105, the data flow efficiency in the tool chain is optimized, and according to the optimized data flow plan, the execution order and concurrency of each tool in the tool chain are dynamically adjusted.

[0014] Preferably, in the step S101, a field change set is extracted from the version management warehouse according to the patch package serial number, and a field change index table is established. The name and type of the newly added field in the index table are identified by regular matching, and a feature vector calculator is used to obtain semantic feature values, field pairs with similarity greater than a threshold are selected, and the type compatibility of the field is determined by a data type verifier. Based on the type compatibility, a data conversion compensator is used to generate a data migration rule, and a rule verifier is used to perform data consistency verification to mark the reliability level of the mapping rule, and generate a mapping relationship list with a reliability level mark;

[0015] For the mapping rules whose reliability level is lower than the threshold in the mapping relationship list with reliability level mark, a review is conducted, and the reliability level of the mapping rules is updated according to the review result to generate a data mapping change list.

[0016] Preferably, in step S102, a rule parser is used to read the interface change rule set from the data mapping relationship change list, and the field mapping relationship is parsed by the rule interpreter to construct a rule syntax tree;

[0017] Generate an initial rule relationship graph based on the rule syntax tree, and verify the integrity of node connections through an integrity checker;

[0018] Based on the verification results of the integrity checker, matching rules are selected from the rule template library to complete the mapping relationship between rule nodes to obtain a completed rule relationship graph. The completed rule relationship graph is converted into a data processing instruction sequence using a mapping template generator. The data processing instruction sequence is verified for the correctness of data conversion by a data sample tester to obtain a data mapping script.

[0019] Preferably, in step S103, the new and old versions of the interface definition files are read by a protocol change parser, the field names and types are extracted, and a text vectorization tool is used to generate field feature vectors, and the cosine similarity between the newly added fields and the existing fields is calculated;

[0020] If the similarity exceeds the threshold, a set of field mapping rules is generated;

[0021] If the similarity does not exceed the threshold, no field mapping rule set is generated;

[0022] Use the data sampler to obtain data samples of the newly added fields, and calculate the data distribution feature vector to determine the accuracy of the field mapping rules. Identify the dependencies of the abandoned interfaces by calling the link analyzer, and use the topological sorting algorithm to generate a cleanup order table. Confirm the deactivated status of the cleanup interface by calling the validator.

[0023] Preferably, in step S104, a detection request is sent to a node in the tool chain by a node detector, and a topological structure of the tool chain is obtained according to a comparison between a response time of the node and a preset response time threshold;

[0024] Calculating data flow paths between the nodes using a connectivity detector, and determining weighted shortest paths based on transmission rates and processing rates of the paths;

[0025] The path scorer calculates the scores of transmission delay, packet loss rate and load balancing according to the weighted shortest path, and determines the optimal transmission path according to the weights of the scores;

[0026] The configuration sender updates the node's routing configuration table according to the optimal transmission path. Meanwhile, the performance collector is responsible for collecting transmission performance indicators. If an abnormal performance indicator is detected, it switches to an alternative path.

[0027] Preferably, in step S105, a performance collector is used to obtain processing delay data and resource consumption data from a tool chain node, and a data normalization processor is used to obtain unified-scale performance indicator data according to the processing delay data and the resource consumption data, wherein the performance indicator data includes a processing delay indicator and a resource consumption indicator;

[0028] Send detection packets through the network detector to obtain network transmission delay data;

[0029] For the network transmission delay data, a data sharding calculator is used to obtain the optimal sharding scale parameter.

[0030] Preferably, in step S106, node resource monitoring data is obtained, and the occupancy rate of the node processor and memory is counted using a resource load calculator to obtain a resource load value;

[0031] Next, the data processor calculates the processing rate and the transmission rate of the node according to the resource load value, and obtains the transmission sample of the corresponding tool chain through the data sampler;

[0032] Generate a test data set using the transmission sample, and record the amount of data backlog to obtain backlog status data;

[0033] The dependency validator checks the dependency relationship of the tool nodes according to the backlog status data, and calculates the priority value to obtain the tool execution sequence;

[0034] A scheduling order table is obtained from the tool execution sequence, and a resource allocator divides the computing resource quota according to the table, while a resource monitor dynamically adjusts the concurrent number of the tool.

[0035] The cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method described in the present application has the advantage that, in order to monitor the interface protocol change problem caused by the version upgrade of the core tool in the tool chain, the present invention uses semantic analysis technology to determine the interface difference before and after the upgrade, automatically adjusts the data mapping relationship, and utilizes a rule-based reasoning method to generate data mapping scripts between tools of different versions according to preset adaptation rules to achieve dynamic data transfer and conversion. At the same time, the present invention continuously optimizes the data field matching model, analyzes the correlation between the newly added fields and the existing fields, constructs a complete data mapping relationship, and after deleting the abandoned interface, uses a graph algorithm to re-plan the data transfer path between tools.

[0036] The present invention also constructs a tool chain performance model to analyze the processing delay and throughput before and after the interface change, optimizes data flow efficiency by adjusting the data shard size and transmission batch parameters, and dynamically adjusts the tool execution order and concurrency, thereby achieving intelligent adaptation and performance optimization of the tool chain. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a process of a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method described in this application Figure 1 ;

[0038] Figure 2 This is a process of a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method described in this application Figure 2 . DETAILED DESCRIPTION

[0039] like Figure 1-Figure 2 As shown, the present application describes a cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method, comprising the following steps:

[0040] like Figure 1-Figure 2 As shown, in step S101, the version changes of the core tools in the tool chain are monitored, the newly added data fields and abandoned interface information after the upgrade are obtained, the differences in the interface protocols before and after the upgrade are determined by a semantic analysis method, the data mapping relationship is adjusted according to the differences, the adjustment content is obtained, and a data mapping relationship change list is generated;

[0041] Among them, a field change set is obtained from the version management warehouse according to the patch package serial number, and a field change index table is established through the field change set;

[0042] Regular matching is used to identify the name and type of the newly added field in the field change index table, and a semantic feature value of the newly added field is obtained through a feature vector calculator;

[0043] For the semantic feature value, select the closest field pair whose similarity is greater than the similarity threshold, and determine the type compatibility of the field pair through a data type checker;

[0044] Generate data migration rules using a data conversion compensator according to the type compatibility of the field pair, and obtain data consistency verification results of the data migration rules through a rule verifier;

[0045] Mark the reliability level of the mapping rules according to the verification results, and generate a mapping relationship list with reliability level marks;

[0046] For the mapping rules whose reliability level is lower than the threshold in the mapping relationship list with reliability level mark, the mapping relationship is reviewed, the reliability level of the mapping rule is updated according to the review result, and a data mapping change list is generated.

[0047] Specifically, in step S101, a data collection trigger is used to collect the upgrade version number, upgrade date, and upgrade patch package serial number for the monitoring tool chain, a field change set in the patch package is obtained from the version management warehouse, and a field change index table is established according to the patch package serial number;

[0048] Retrieve the interface change records before and after the upgrade from the version management warehouse, use regular matching to identify the names and types of the newly added fields, extract the semantic feature values ​​of the fields through the feature vector calculator, and calculate the similarity matrix of the new and old fields based on the semantic feature values;

[0049] A mapping matching threshold is set for the field similarity matrix, and the maximum similarity matching algorithm is used to select the field pairs with similarity greater than the threshold and the closest similarity. The field type compatibility is verified through the data type verifier, and the initial mapping rule table is generated according to the matching results.

[0050] Obtain the field mapping rule set from the initial mapping rule table, use the data conversion compensator to generate data migration rules for the abandoned fields, verify data consistency through the rule validator, and generate a data migration rule set based on the verification results;

[0051] Obtain a complete mapping solution from the field mapping rule set and the data migration rule set, use a data sample validator to verify the mapping correctness of each mapping rule, mark the reliability level of the mapping rule according to the verification result, and generate a mapping relationship list with reliability level marks;

[0052] For the mapping rules with reliability levels lower than the threshold in the mapping relationship list with reliability level marks, the mapping relationship is reviewed using the manual confirmation interface, and the mapping rule reliability level is updated based on the review results to generate a data mapping change list;

[0053] During the data collection process of the monitoring tool chain, when the data collection trigger collects the upgrade patch package information, an index relationship is established for the patch package serial number corresponding rules. The patch package serial number is a unique identifier generated by combining the year, month, day, hour, minute, second and a random number, including the patch package serial number 20240102143022899, where 14:30:22 on January 2, 2024 is the timestamp and 899 is a random number. The patch package serial number index can be used to quickly locate the version change record;

[0054] In the version management warehouse, the interface change record contains the field name, field type, and default value attribute information. For the character user name field, the original interface definition is username, which is changed to user_name after the upgrade. When extracting the field semantic feature value through the feature vector calculator, the word segmentation technology is used to split the field into independent morphemes, and the similarity between morphemes is calculated to obtain the field similarity matrix. The similarity value is between 0 and 1. The larger the value, the closer the field semantics.

[0055] In the mapping match threshold setting, select appropriate thresholds based on business scenarios, including setting the field full match threshold to 0.9 and the partial match threshold to 0.7. When the field similarity is greater than the threshold, a mapping relationship is established. For the user_name and username fields, the similarity is 0.95, and a field mapping rule is established.

[0056] The data type validator verifies whether the old and new field types are compatible. If the original field type is varchar(50) and the new field type is varchar(100), the types are determined to be compatible and a mapping can be established.

[0057] In the data migration rule generation, for the abandoned field phone_no, new fields mobile_phone and telephone are added. The original phone number data is split and migrated to the corresponding new fields according to the mobile phone and landline rules through the data conversion compensator. The mobile phone number with 1 starting with 11 digits is stored in the mobile_phone field, and the landline phone is stored in the telephone field.

[0058] During the mapping rule verification phase, the data sample verifier extracts samples of different data distributions to verify the mapping rules. The verification data accounts for no less than 30%. Mapping rules with a verification pass rate higher than 95% are marked as high reliability, those with a pass rate between 80% and 95% are marked as medium reliability, and those with a pass rate lower than 80% are marked as low reliability.

[0059] For low-reliability mapping rules, including mapping user_addr to two candidate fields, user_address and user_location, manual review and confirmation of the mapping relationship is required to generate a data mapping change list, which includes the field mapping rule identifier, source field name, target field name, field type, default value, whether to abandon it, data conversion rules, and mapping reliability level attributes. The change list is used to guide subsequent data migration work to ensure field mapping accuracy and data integrity during the data migration process.

[0060] like Figure 1-Figure 2 As shown, in step S102, according to the data mapping relationship change list, a rule-based reasoning method is used to analyze the changes in the interface protocol and data fields, and according to the pre-configured interface protocol adaptation rules, a data mapping script between different versions of tools is generated;

[0061] Wherein, a rule parser is used to read the interface change rule set from the data mapping relationship change list, and the rule parser parses the field mapping relationship through a rule interpreter to obtain a rule syntax tree;

[0062] Generate an initial rule relationship graph according to the rule syntax tree, and obtain a node connection integrity verification result by identifying the relationship between rule nodes through an integrity checker on the initial rule relationship graph;

[0063] According to the node connection integrity verification result, a matching rule is selected from a rule template library to complete the mapping relationship between rule nodes, and a completed rule relationship diagram is obtained;

[0064] The completed rule relationship diagram is converted into a data processing instruction sequence by a mapping template generator, and the data processing instruction sequence is verified for data conversion correctness by a data sample tester to obtain a data mapping script.

[0065] Specifically, in step S102, a mapping rule parser is used to read an interface change rule set from a data mapping relationship change list, a rule interpreter is used to parse field mapping relationships and data conversion rules, a rule syntax tree is generated for the change rule set, and an initial rule relationship graph is generated according to the syntax tree node relationship;

[0066] For the initial rule relationship graph, an integrity checker is used to identify the relationship between rule nodes, and the node association validator is used to determine the integrity of the rule node connection. A rule completer is used to select matching rules from the rule template library, and the mapping relationship between rule nodes is completed according to the selected rule template.

[0067] Obtain a complete mapping rule set from the completed rule relationship graph, convert the rules into a data processing instruction sequence through a mapping template generator, generate a data mapping code snippet for the instruction sequence, and generate a complete data mapping script according to a code assembler;

[0068] Use the version compatibility checker to verify the tool version compatibility of the data mapping script, use the data sample tester to verify the correctness of data conversion, generate a deployment configuration file for the verified script, and load the data mapping script according to the configuration file;

[0069] Get the deployed mapping script from the data mapping script loader, establish the inter-tool communication pipeline through the data receiving adapter, use the current limiting controller to adjust the data processing speed for the data transmission process, and save the data conversion process according to the data processing log recorder;

[0070] Use an exception monitor to monitor the data processing log in real time, use a data quality checker to verify the integrity of the converted data, use a data reconciliation device to check the consistency between the source data and the target data, and judge the correctness of the data conversion based on the verification results;

[0071] When the mapping rule parser processes the data mapping relationship change list, it parses the field mapping relationship and data conversion rules and builds a rule syntax tree. Each node contains the corresponding relationship between the conversion rule and the data field. In the customer information change scenario, the original interface field customer_info contains three subfields: name, age, and address. After the upgrade, it is split into two field groups: basic_info and detail_info. The rule syntax tree records the field split relationship.

[0072] During the rule integrity check, the node association validator identifies the integrity of the relationship between the rule nodes. Taking order processing as an example, the order creation node is associated with the order payment node, and the payment node is associated with the order delivery node to form a complete business chain. When it is found that the payment result processing rule is missing between the order payment node and the order delivery node, the rule completer selects the payment result processing template from the rule template library to supplement the node relationship;

[0073] In the data mapping script generation stage, the mapping template generator converts the complete rules into data processing instructions. For the product information synchronization scenario, the original interface product description field product_desc is limited to 500 characters in length, and the upgraded field description is limited to 1000 characters in length. The instruction sequence includes field length extension and data truncation processing operations. The code assembler assembles multiple processing instructions into a complete mapping script.

[0074] The version compatibility check process uses a data sample tester to verify the correctness of data conversion. In the sales data processing scenario, the precision of the amount field in the original interface is 2 decimal places, and the precision of the sale_amount field after the upgrade is 4 decimal places. The accuracy of the amount data conversion is verified through test data. After verification, a script deployment configuration is generated;

[0075] During data transmission, the current limiting controller adjusts the data processing speed according to the tool processing capacity, including limiting the processing speed of the log collection tool to 1,000 records per second. When the transmission speed of the upstream data source reaches 2,000 records per second, the current limiting controller starts the queue cache mechanism to temporarily store the data that exceeds the processing capacity in the queue to avoid data loss.

[0076] In the data processing process monitoring, the exception monitor detects data processing anomalies in real time. In the inventory synchronization scenario, the data quality checker verifies the correctness of the inventory quantity. The original interface inventory field stock_qty is an integer, and the upgraded field inventory supports decimals. The data reconciler checks the consistency of the inventory quantity before and after the conversion. When inventory differences are found, the exception log is recorded and the data correction mechanism is triggered.

[0077] like Figure 1-Figure 2As shown, in step S103, when the tool version upgrade causes the interface protocol to change, the data mapping relationship is adjusted. During the adjustment process, the newly added fields are analyzed for similarity with the existing fields, the relevance between the newly added fields and the existing fields is inferred, the newly added fields are included in the data field matcher, a complete data mapping relationship is constructed, and the abandoned interface is deleted;

[0078] Among them, the new and old versions of the interface definition files are read through the protocol change parser, the field attribute analyzer is used to extract the field name and type, and the field feature vector is generated according to the text vectorization tool;

[0079] For the field feature vector, a vector similarity calculator is used to calculate the cosine similarity between the newly added field and the existing field, and if the similarity is higher than a similarity threshold, a field mapping rule set is generated;

[0080] According to the field mapping rule set, a data sample of the newly added field is obtained through a data sampler, and a feature extractor is used to calculate a data distribution feature vector to determine the accuracy of the field mapping rule;

[0081] According to the field mapping rules, the abandoned interface dependencies are identified by calling the link analyzer, and the topological sorting algorithm is used to generate the cleanup order table. The deactivated status of the cleanup interface is confirmed according to the call validator.

[0082] Specifically, in step S103, the interface protocol change notification information is obtained from the upgrade tool, the protocol change parser is used to read the new and old version interface definition files, the field name, type, length, and default value are extracted through the field attribute analyzer, the text vectorization tool is used to calculate the semantic features of the field name, and the field feature vector is generated according to the field attribute features and semantic features;

[0083] Get all newly added field data from the field feature vector, calculate the cosine similarity between the newly added field and the existing field feature vector through the vector similarity calculator, generate mapping associations for field pairs with similarity higher than the similarity threshold, and determine the field type compatibility based on the field mapping rule validator to obtain the initial mapping solution;

[0084] Obtain a mapping rule set from the initial mapping scheme, obtain a new field data sample set through a data sampler, use a feature extractor to calculate the data distribution feature vector, calculate the data feature similarity based on a deep neural network, and determine the accuracy of the field mapping rules;

[0085] For field pairs whose mapping rule accuracy is lower than the threshold, an incremental learning algorithm is used to update the field matcher based on the newly added data samples. The matching accuracy is evaluated through a cross-validator. The accuracy threshold is used to determine whether to continue optimization, and the optimized mapping solution is obtained.

[0086] Obtain the discarded interface identifier from the interface call record, identify the discarded interface dependency by calling the link analyzer, generate a cleanup sequence table for the dependency link using a topological sorting algorithm, and generate an interface cleanup instruction based on the sequence table;

[0087] For interface cleanup instructions, use the configuration updater to delete the abandoned interface configuration information, call the validator to confirm that the cleanup interface has been completely disabled, determine the validity of the cleanup result based on the verification result, and generate an interface cleanup report;

[0088] When the protocol change parser processes the interface definition files of the old and new versions, the field attribute analyzer extracts the characteristic information of each field, including the product name field in the product information synchronization interface. The original interface field name is product_name, the type is varchar, the length is 50 characters, and the default value is empty. After the upgrade, the field name is changed to item_name, the type is varchar, the length is 100 characters, and the default value is empty. The text vectorization tool analyzes the semantics of the field name and calculates the characteristic vector value;

[0089] During the vector similarity calculation process, cosine similarity is used to calculate the similarity between field feature vectors. For the order processing interface, the original interface order status field order_status has values ​​including created, paid, and shipped. After the upgrade, the status field is split into payment status payment_status and logistics status shipping_status. The mapping relationship between fields is determined by feature vector calculation. If the similarity is higher than 0.8, it is considered that there is an association;

[0090] In the data feature extraction phase, the data sampler obtains sample data of the newly added fields. Taking the user information synchronization scenario as an example, the original interface user address field user_address is split into three fields after the upgrade: province province, city city, and detailed address detail_address. The feature extractor analyzes the distribution law of the original address and the split address data, and the deep neural network determines the accuracy of field mapping based on the similarity of data features.

[0091] During the incremental learning process, for the mobile phone number field, the original interface field was mobile_phone, which was changed to phone_number after the upgrade. The initial mapping accuracy was 0.75. The incremental learning algorithm was trained based on 1,000 new data samples. Cross-validation showed that the accuracy was improved to 0.92, exceeding the 0.9 threshold, and the optimization was completed.

[0092] In the abandoned interface processing stage, call link analysis identifies interface dependencies. Taking the payment scenario as an example, the payment result query interface depends on the payment status query interface, and the status query interface depends on the payment creation interface. The topological sorting generates a cleanup order of first cleaning the result query interface, then cleaning the status query interface, and finally cleaning the creation interface.

[0093] During the interface cleanup verification, after the configuration updater deletes the interface configuration, the call verifier continuously monitors for 30 minutes to confirm that the number of abandoned interface calls is 0 and there are no abnormal errors, indicating that the interface cleanup is successful, and a cleanup report is generated to record the processing process;

[0094] After the mapping plan was implemented, mapping relationships were established for 95% of the fields in the original interface, and 5% of the fields could not be mapped due to business changes. They were deleted from the list after manual confirmation, completing the interface protocol upgrade.

[0095] like Figure 1-Figure 2 As shown, in step S104, after deleting the abandoned interface, a graph-based shortest path search algorithm is used to bypass the deleted abandoned interface, re-plan the data transfer path between tools, and re-form a complete tool chain data transfer path;

[0096] Among them, a detection request is sent to the node in the tool chain through the node detector, and the tool chain topology structure is obtained according to the comparison result of the response time of the node and the response time threshold;

[0097] According to the tool link topology, a connectivity detector is used to calculate a data flow path between nodes, and a weighted shortest path is obtained according to a transmission rate and a processing rate of the data flow path;

[0098] For the weighted shortest path, the transmission delay score, packet loss rate score and load balancing score are calculated by the path scorer, and the optimal transmission path is determined according to the weight of each dimension score;

[0099] A configuration transmitter is used to update the routing configuration table of the node according to the optimal transmission path, and a transmission performance indicator is obtained through a performance collector. If an abnormal performance indicator is detected, the alternative path is switched.

[0100] Specifically, in step S104, a tool chain scanner is used to obtain a tool chain topology structure, a detection request is sent to each node through a node detector, a node whose response time exceeds a preset threshold is marked as an abnormal state, and a tool chain topology graph is generated according to the calling relationship between nodes;

[0101] The node connection relationship is obtained from the tool link topology diagram, the data flow path between nodes is calculated through the connectivity detector, the load calculator is used to calculate the data packet transmission rate and node processing rate for each data transmission path, and the weighted shortest path is calculated according to the Dijkstra algorithm;

[0102] Obtain candidate paths from the weighted shortest path set, and use the path scorer to perform multi-dimensional scoring on the candidate paths. Scores are calculated for the three dimensions of transmission delay, packet loss rate, and load balancing, and the comprehensive score of the path is calculated based on the weights of each dimension.

[0103] Select the best transmission path from the path comprehensive score list, verify the path connectivity by sending a detection data packet through a data packet detector, obtain the transmission quality index for the connected path using a data sampler, and generate a path availability report based on the transmission quality verification result;

[0104] To verify the transmission path, the path configurator is used to generate routing configuration information, the node routing configuration table is updated through the configuration sender, the configuration verifier is used to check the configuration consistency based on the configuration delivery result, and the path status is updated according to the verification result;

[0105] Obtain transmission path information from the updated routing configuration, monitor data transmission performance indicators through the performance collector, use the path switcher to switch to the alternative path when the performance indicators are abnormal, and update the routing configuration table according to the switching results;

[0106] In the tool chain topology scan, the node detector detects the node status by sending heartbeat packets. The normal node response time is controlled within 200 milliseconds. If the detection response exceeds 500 milliseconds for three consecutive times, it is marked as an abnormal state. In the order processing tool chain, data is transmitted between the order creation node, payment node, and inventory node through the message queue;

[0107] When calculating the data flow path, the connectivity detector identifies the data transmission channel between nodes and collects the performance indicators of each node through the load calculator. The order creation node processing rate is 1,000 orders per second, the message queue transmission rate is 2,000 messages per second, and the payment node processing rate is 800 transactions per second. The Dijkstra algorithm uses the node processing delay and transmission delay as the path weight to calculate the shortest path;

[0108] During the path scoring process, candidate paths are scored in multiple dimensions. In the transmission delay dimension, an end-to-end delay of less than 500 milliseconds is scored 90 points, a delay of 500 to 800 milliseconds is scored 70 points, and a delay of more than 800 milliseconds is scored 50 points.

[0109] In the packet loss rate dimension, a packet loss rate below 0.1% scores 90 points, 0.1% to 0.5% scores 70 points, and more than 0.5% scores 50 points; in the load balance dimension, a node load rate below 60% scores 90 points, 60% to 80% scores 70 points, and more than 80% scores 50 points;

[0110] During the path verification phase, the data packet detector sends a detection packet every 60 seconds to detect path connectivity. The data sampler samples and analyzes the transmission data at a sampling ratio of one thousandth, and records data transmission quality indicators, including average transmission delay, packet loss rate, and data error rate. When the transmission delay is less than 800 milliseconds and the packet loss rate is less than 0.5%, the path is considered available. During the routing configuration update, the configuration sender sends the routing table to the node. The routing table contains the target node identifier, the next hop node, and the transmission channel information. The configuration verifier verifies the configuration delivery result by comparing the node's local routing table with the delivered configuration. The routing update is completed when the configuration consistency reaches 100%;

[0111] During the transmission performance monitoring phase, the performance collector collects performance indicators every 30 seconds. When it is found that the node processing delay exceeds the preset threshold or the number of queue backlog messages exceeds 80% of the queue capacity, the path switcher activates the alternative path and switches the data traffic to the alternative channel. During the switching process, the message sequence is ensured to avoid message duplication or loss. After the switching is completed, the routing configuration is updated and the switching event is recorded.

[0112] In the commodity inventory synchronization scenario, the commodity service node, inventory service node, and order service node form a tool chain to transmit inventory change messages through message queues. When a performance bottleneck occurs in the inventory service node, causing the message processing delay to exceed 1 second, the path switcher switches the message traffic to the backup inventory node to ensure real-time synchronization of inventory data.

[0113] like Figure 1-Figure 2 As shown, in step S105, the data processing delay and resource consumption of the tool nodes in the re-formed tool chain data transmission path are collected, the average processing delay and throughput before and after the interface protocol is changed are analyzed, and the existence of network delay and bandwidth limitation is determined. If so, the data segment size and transmission batch parameters in the data processing process are obtained;

[0114] Wherein, a performance collector is used to obtain processing delay data and resource consumption data from a tool chain node, wherein the processing delay data is generated by a tool chain node processing process;

[0115] According to the processing delay data and the resource consumption data, obtaining performance indicator data of a unified scale through a data normalization processor, wherein the performance indicator data includes a processing delay indicator and a resource consumption indicator;

[0116] Acquire network transmission delay data by sending a detection packet through a network detector, wherein the network transmission delay data is calculated based on the performance indicator data;

[0117] With respect to the network transmission delay data, a data sharding calculator is used to generate an optimal sharding scale parameter, and the optimal sharding scale parameter is calculated based on the network bandwidth and the node processing capability.

[0118] Specifically, in step S105, a performance collector is used to obtain processing delay data and resource consumption data from tool chain nodes, and the mean and standard deviation of processing delay of each node are calculated through a data statistics device. Periodic sampling is performed on the CPU occupancy rate, memory occupancy rate, and disk read and write rate, and resource consumption trend data is generated according to the time series processor;

[0119] The performance indicators before and after the interface protocol change are obtained from the resource consumption trend data. The indicators of different dimensions are converted into a unified scale through the data normalization processor. The Bayesian inference algorithm is used to identify performance anomalies for processing latency and throughput. The tool chain performance prediction function is built based on the performance predictor.

[0120] Get the performance prediction results from the tool chain performance prediction function, calculate the network transmission delay by sending the detection packet through the network detector, use the traffic monitor to record the real-time traffic data according to the network bandwidth usage, and generate the network transmission quality score according to the network quality scorer;

[0121] The network status evaluation result is obtained from the network transmission quality score, the transmission bottleneck node is identified through the network bottleneck locator, the resource load calculator is used to calculate the processing capacity threshold of the bottleneck node, and the network resource allocation plan is generated according to the load balancer;

[0122] For the network resource allocation scheme, a data sharding calculator is used to calculate the optimal sharding scale according to the network bandwidth and node processing capacity, the data batch transmission parameters are set through the transmission batch generator, and the sharding transmission effect is verified according to the data transmission simulator;

[0123] Obtain optimization parameters from the fragment transmission verification results, verify the rationality of the parameter value range through the parameter constraint checker, use the parameter applicator to update the data transmission configuration for the parameters that pass the verification, and record the parameter adjustment effect according to the transmission performance monitor;

[0124] When the performance collector collects running data at the tool chain node, taking the data synchronization scenario as an example, the node processing delay is sampled 100 times per minute. The sampled data shows that the average processing delay is 200 milliseconds and the standard deviation is 50 milliseconds. At the same time, the node resource consumption data is recorded, with an average CPU occupancy rate of 75%, a memory occupancy rate of 65%, and a disk read and write rate of 120MB / second.

[0125] During the data normalization process, all performance indicators were uniformly converted to the range of 0-1. The performance comparison before and after the interface protocol change showed that the processing delay increased from 200 milliseconds to 350 milliseconds, and the data throughput decreased from 1,000 records per second to 600 records per second. The Bayesian inference algorithm identified the processing delay exceeding 300 milliseconds as an abnormal point, and the constructed performance prediction function showed that the processing capacity was on a downward trend.

[0126] In network status monitoring, the network detector sends a detection packet every 30 seconds to calculate the network delay. The detection results show that the average transmission delay is 150 milliseconds, the network jitter is within 50 milliseconds, and the bandwidth utilization rate recorded by the traffic monitor reaches 85%. The network quality score is based on a percentage system, and the current network transmission quality score is 75 points;

[0127] The network bottleneck identification process shows that insufficient processing capacity of the data receiving node leads to data backlog. The node processing capacity threshold is 800 messages per second, while the upstream data source generates 1,200 messages per second. The load balancer distributes data traffic to the primary and standby nodes in a ratio of 3:7 based on the processing capacity difference.

[0128] The data sharding calculation results show that based on 100MB bandwidth and the node's ability to process 800 messages per second, the optimal shard size is set to 512KB, and the number of messages transmitted in a single batch is controlled within 100. The data transmission simulation test shows that after adopting sharding transmission, the node processing delay is reduced to 180 milliseconds, and the data backlog phenomenon is alleviated;

[0129] During the parameter optimization phase, the parameter constraint checker verifies that the shard size ranges from 64KB to 1024KB, and the batch size ranges from 50 to 200. After the parameters are applied, the transmission performance monitoring shows that the node processing delay is stable within 200 milliseconds, the CPU utilization rate drops to 65%, the memory utilization rate drops to 55%, and the network bandwidth utilization rate drops to 70%;

[0130] In the order processing tool chain, data is transmitted between the order query service, order statistics service, and order export service through message queues. During peak hours, the order export service has insufficient processing capacity, resulting in a backlog of message queues. After the above optimization, the order processing link has resumed normal operation, and the order export delay has been shortened from the original 10 minutes to less than 3 minutes.

[0131] like Figure 1-Figure 2 As shown, in step S106, the data shard size and transmission batch parameters are adjusted to optimize the data flow efficiency in the tool chain, and the execution order and concurrency of each tool in the tool chain are dynamically adjusted according to the optimized data flow plan;

[0132] The node resource monitoring data is obtained, and a resource load calculator is used to calculate the node processor occupancy rate and memory occupancy rate according to the monitoring data to obtain a resource load value;

[0133] A data processor is used to calculate a node processing rate and a transmission rate according to the resource load value, and a tool chain transmission sample corresponding to the transmission rate is obtained through a data sampler;

[0134] Generate a test data set according to the transmission sample, record the amount of data backlog according to the test data set, and obtain backlog status data;

[0135] Checking tool node dependencies according to the backlog status data by a dependency validator, calculating priority values ​​according to the dependencies, and generating a tool execution sequence according to the priority values;

[0136] Obtain the scheduling order table from the tool execution sequence, use the resource allocator to divide the computing resource quota, and dynamically adjust the number of concurrent tools based on the resource monitor.

[0137] Specifically, in step S106, resource monitoring data is obtained from the tool node, the CPU occupancy rate and memory occupancy rate of each node are counted using a resource load calculator, the node data processing rate and transmission rate are calculated using a data processor, and the initial sharding parameters and batch parameters are calculated using a genetic algorithm for the rate data, and a parameter value interval is generated according to the resource capacity threshold;

[0138] Obtain the data shard size and transmission batch value from the parameter value range, obtain the tool chain transmission sample through the data sampler, use the performance calculator to calculate the transmission delay and throughput for the transmission sample, and adjust the data shard size and transmission batch value according to the preset performance indicators;

[0139] The transmission verifier is used to verify the parameters after adjustment, and the test data set is constructed through the data generator. The queue monitor is used to record the number of data backlogs for test data transmission, and the rationality of the parameters is judged based on the backlog situation;

[0140] Obtain node dependency data from the tool dependency graph, check the correctness of the dependency through the dependency verifier, calculate the tool priority value using the task priority generator based on the verified dependency, and generate the tool execution sequence based on the topology sorter;

[0141] Obtain the scheduling order table from the tool execution sequence, set the tool concurrency threshold through the concurrency controller, use the resource allocator to divide the computing resource quota for tool operation, and dynamically adjust the number of tool concurrency based on the resource monitor;

[0142] According to the tool running status, the execution monitor is used to track the task execution status, the performance collector is used to record the tool processing capacity indicators, the load predictor is used to calculate the resource usage trend, and the tool resource configuration is updated according to the prediction results;

[0143] During the tool node resource monitoring process, taking the order processing tool chain as an example, the CPU occupancy rate of the order creation node is 75%, the memory occupancy rate is 65%, the data processing rate is 1000 orders per second, and the transmission rate is 1500 messages per second. The genetic algorithm iteratively calculates that the initial data shard size is 256KB, the number of messages transmitted in a single batch is 50, and the resource capacity threshold limits the shard size to an upper limit of 1024KB;

[0144] During the performance calculation phase, the data sampler collects transmission samples every 10 seconds. The sampling results show that the average transmission delay is 180 milliseconds and the throughput is 800 orders per second. Compared with the preset performance indicators, the transmission delay is less than 150 milliseconds and the throughput is not less than 1,000 orders per second. Based on the differences, the shard size is adjusted to 128KB and the batch size is increased to 80.

[0145] In the parameter verification phase, the transmission verifier generates a test data set containing 10,000 order data and performs transmission tests in a high-concurrency scenario. The queue monitoring shows that the maximum backlog of the message queue is 500, which is lower than the backlog threshold of 1,000, verifying that the parameter settings are reasonable.

[0146] In the tool dependency analysis, the order processing tool chain includes four nodes: order creation, inventory check, payment processing, and logistics delivery. The dependency validator confirms that the order creation node depends on the inventory check node, the payment processing node depends on the order creation node, and the logistics delivery node depends on the payment processing node. The task priority generator calculates the priority according to the dependency depth, and the topological sorting generates the execution order.

[0147] During resource allocation, the concurrency controller sets the concurrency threshold according to the node processing capacity. The concurrency of the order creation node is set to 4, the concurrency of the inventory check node is set to 6, the concurrency of the payment processing node is set to 8, and the concurrency of the logistics delivery node is set to 4. The resource allocator allocates computing resources in the ratio of 4:6:8:4.

[0148] During the execution monitoring phase, the performance collector records the processing indicators of each node. The average processing time of the order creation node is 150 milliseconds, with a processing success rate of 99.5%. The processing time of the inventory check node is 100 milliseconds, with a processing success rate of 99.8%. The load predictor predicts that the peak processing volume will increase by 30% based on historical data, and increases the node resource quota in advance.

[0149] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0150] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of this application.

Claims

1. A cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method, characterized in that: The following steps are involved: S101, monitoring version changes in the tool chain, obtaining newly added data fields and abandoned interface information after the upgrade, determining the difference between the interface protocols before and after the upgrade through a semantic analysis method, adjusting the data mapping relationship according to the difference, obtaining the adjustment content, and generating a data mapping relationship change list; S102, according to the data mapping relationship change list in step S101, using a rule-based reasoning method, analyzing the changes in the interface protocol and data fields, and generating data mapping scripts between different versions of tools according to pre-configured interface protocol adaptation rules; S103. When the tool version upgrade causes the interface protocol in step S102 to be changed, the data mapping relationship is adjusted. During the adjustment process, a similarity analysis is performed on the newly added fields and the existing fields, the relevance between the newly added fields and the existing fields is inferred, the newly added fields are included in the data field matcher, the data mapping relationship is constructed, and the abandoned interface is deleted; S104, after deleting the abandoned interface in step S103, using a graph-based shortest path search algorithm to bypass the deleted abandoned interface, re-plan the data transfer path between tools, and re-form the tool chain data transfer path; S105. Collect the data processing delay and resource consumption of the tool nodes in the re-formed tool chain data transmission path in step S104, analyze the average processing delay and throughput before and after the interface protocol change, and determine whether there is network delay and bandwidth limitation. If so, obtain the data slicing size and transmission batch parameters in the data processing process; if not, do not obtain the data slicing size and transmission batch parameters in the data processing process; S106. According to the adjustment of the data shard size and the transmission batch parameters in step S105, the data flow efficiency in the tool chain is optimized, and according to the optimized data flow plan, the execution order and concurrency of each tool in the tool chain are dynamically adjusted.

2. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 1, it is characterized by: In step S101, it includes: A field change set is extracted from the version management warehouse according to the patch package serial number, and a field change index table is established. The name and type of the newly added field in the index table are identified through regular matching, and the semantic feature value is obtained using the feature vector calculator. Field pairs with a similarity greater than a threshold are selected, and the type compatibility of the field pairs is determined by the data type verifier. Based on the type compatibility, a data conversion compensator is used to generate data migration rules, and data consistency verification is performed through a rule validator to mark the reliability level of the mapping rules, and a review is performed. The reliability level of the mapping rules is updated according to the review results, and a list of data mapping relationship changes is generated.

3. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 1, it is characterized by: In step S102, it includes: Use the rule parser to read the interface change rule set from the data mapping relationship change list, and parse the field mapping relationship through the rule interpreter to build a rule syntax tree; Generate an initial rule relationship graph based on the rule syntax tree, and verify the integrity of node connections through an integrity checker; Based on the verification results of the integrity checker, matching rules are selected from the rule template library to complete the mapping relationship between rule nodes to obtain a completed rule relationship graph. The completed rule relationship graph is converted into a data processing instruction sequence using a mapping template generator. The data processing instruction sequence is verified for the correctness of data conversion by a data sample tester to obtain a data mapping script.

4. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 1, it is characterized by: In step S103, it includes: reading the new and old version interface definition files through the protocol change parser, extracting the field name and type, and using the text vectorization tool to generate the field feature vector, and calculating the cosine similarity between the newly added field and the existing field; If the similarity exceeds the threshold, a set of field mapping rules is generated; If the similarity does not exceed the threshold, no field mapping rule set is generated; Use the data sampler to obtain data samples of the newly added fields, and calculate the data distribution feature vector to determine the accuracy of the field mapping rules. Identify the dependencies of the abandoned interfaces by calling the link analyzer, and use the topological sorting algorithm to generate a cleanup order table. Confirm the deactivated status of the cleanup interface by calling the validator.

5. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 1, it is characterized in that: In step S104, it includes: Sending a detection request to a node in the tool chain through a node detector, and obtaining a topological structure of the tool chain according to a comparison between a response time of the node and a preset response time threshold; Calculating the data flow path between the nodes using a connectivity detector, and determining a weighted shortest path based on the transmission rate and processing rate of the data flow path; The path scorer calculates the scores of transmission delay, packet loss rate and load balancing according to the weighted shortest path, and determines the optimal transmission path according to the weights of the scores; The configuration sender updates the routing configuration table of the node according to the optimal transmission path, and the performance collector collects transmission performance indicators. If an abnormal performance indicator is detected, it switches to an alternative path.

6. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 1, it is characterized by: In step S105, it includes: Use a performance collector to obtain processing delay data and resource consumption data from the tool chain node, and use a data normalization processor to obtain performance indicator data of a unified scale based on the processing delay data and the resource consumption data, and the performance indicator data includes a processing delay indicator and a resource consumption indicator; send a detection packet through a network detector to obtain network transmission delay data; use a data sharding calculator to obtain the optimal sharding scale parameter for the network transmission delay data.

7. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 1, it is characterized by: In step S106, it includes: acquiring node resource monitoring data, and using a resource load calculator to count the occupancy rates of the node processor and memory to obtain a resource load value.

8. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 7, it is characterized in that: The data processor calculates the processing rate and the transmission rate of the node according to the resource load value, and obtains the transmission sample of the corresponding tool chain through the data sampler.

9. According to the cloud computing-based enterprise digital tool chain dynamic collaboration and deployment method of claim 8, it is characterized by: Generate a test data set using the transmission sample, and record the amount of data backlog to obtain backlog status data; The dependency verifier checks the dependency relationship of the tool nodes according to the backlog status data, and calculates the priority value to obtain the tool execution sequence.

10. The method for dynamic collaboration and deployment of enterprise digital tool chains based on cloud computing according to claim 9, characterized in that: The tool execution sequence obtains a scheduling order table, and the resource allocator divides the computing resource quota according to the table. At the same time, the resource monitor dynamically adjusts the concurrent number of the tool.

Images