Engineering information management method and system based on distributed technology

Through twin neural networks, the parameter configuration model is constructed and distributed systems are automatically configured, which solves the problem of human subjectivity of distributed system configuration in the existing technology, achieves high accuracy and automation, and improves the degree of automation and migration of management systems.

CN119003159AInactive Publication Date: 2024-11-22TIBET HONGJIAN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411023281.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, it is necessary to manually set the configuration parameters of the distributed management system, resulting in the management system configuration being artificially subjective, so that the accuracy and automation of the distributed system configuration cannot be guaranteed.

Method used

Using twin neural networks based on the data attributes and resource attributes of engineering information, a parameter configuration model is built to automatically calculate the configuration parameters of the distributed system, and the distributed system is automatically configured through the parameter configuration model to adapt to the data attributes and resources of engineering information. attributes and add migration coefficients if necessary to improve migration.

Benefits of technology

The automation and accuracy of distributed system configuration is realized, the objectivity and automation of the management system are improved, the accuracy of distributed system configuration is ensured, and the migration of engineering information management is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003159A_ABST
    Figure CN119003159A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of information management, in particular to an engineering information management method and system based on a distributed technology, and the method comprises the following steps: collecting data attributes of engineering information and resource attributes of a distributed system for managing the engineering information; obtaining a parameter configuration model for automatically measuring and calculating configuration parameters of the distributed system based on the data attribute and the resource attribute of the engineering information by utilizing a twin neural network; and automatically configuring the configuration parameters of the distributed system through a parameter configuration model so as to adapt to the data attribute and the resource attribute of the engineering information. According to the method, the parameter configuration model is constructed, the distributed system for managing engineering information is configured, configuration automation is achieved, meanwhile, the management system configuration parameters are set through the unified parameter configuration model, objectivity is high, and the accuracy of distributed system configuration is guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of information management, and in particular to an engineering information management method and system based on distributed technology. Background Art

[0002] Information management method is one of the management methods, which refers to the method of using information principles to achieve scientific management. Its core and essence is to manage information well based on the characteristics of information and its movement laws, and to use scientific methods and means to give full play to the role of information in organizational management. Scientific and effective management of information is the key to the use of this management method. The general principle of its management is to seek truth from facts. The general requirements are timeliness, accuracy, applicability and economy. Its turnover process generally has four basic links: information acquisition, information transmission, information processing and information storage. Effective management of information requires that each link can operate effectively and form an organic whole that is coordinated and closely coordinated.

[0003] The existing technology for managing engineering information requires manual setting of configuration parameters of the distributed management system, which leads to human subjectivity in the configuration of the management system, making it impossible to ensure the accuracy and degree of automation of the distributed system configuration. Summary of the invention

[0004] The purpose of the present invention is to provide an engineering information management method and system based on distributed technology to solve the technical problem that the configuration parameters of the distributed management system need to be set manually in the prior art, resulting in human subjectivity in the management system configuration, making it impossible to guarantee the accuracy and degree of automation of the distributed system configuration.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] An engineering information management method based on distributed technology includes the following steps:

[0007] Collect data attributes of engineering information and resource attributes of distributed systems used to manage engineering information;

[0008] Using twin neural networks, based on the data attributes and resource attributes of engineering information, a parameter configuration model for automatically calculating the configuration parameters of distributed systems is obtained;

[0009] Through the parameter configuration model, the configuration parameters of the distributed system are automatically configured to adapt the data attributes and resource attributes of the engineering information.

[0010] As a preferred solution of the present invention, the data attributes of the engineering information include the data structure, data volume and computing services of the engineering information.

[0011] As a preferred solution of the present invention, the resource attributes of the distributed system include computing resources, storage resources and network bandwidth resources.

[0012] As a preferred solution of the present invention, the configuration parameters of the distributed system include basic environment configuration parameters, storage configuration parameters, Hadoop cluster configuration parameters and server configuration parameters.

[0013] As a preferred solution of the present invention, the method for constructing the parameter configuration model includes:

[0014] Select multiple types of engineering information, multiple types of distributed systems, and optimal configuration parameters of distributed systems;

[0015] The data attributes of various types of engineering information and the optimal configuration parameters of the distributed system are used as the input and output of the first network in the twin neural network respectively;

[0016] The resource attributes of various distributed systems and the optimal configuration parameters of the distributed systems are used as the input and output of the second network in the twin neural network respectively;

[0017] Taking the difference between the output of the first network and the second network in the twin neural network and the performance index of the distributed system as the loss function, the twin network is trained to obtain the parameter configuration model;

[0018] The parameter configuration model is:

[0019]

[0020] In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, and TwinBP2 is the second network of the twin network.

[0021] As a preferred solution of the present invention, the difference between the outputs of the first network and the second network in the twin neural network is quantified by the L1 norm, and the performance indicators of the distributed system include performance, availability, scalability, consistency and partition tolerance. The loss function is:

[0022] L=min||TwinBP1(X)-TwinBP2(Y)|| 1 +max[F1(K best )+F2(K best )+F3(K best )+

[0023] F4(K best )+F5(K best )];

[0024] Where L is the loss function, TwinBP1(X) and TwinBP2(Y) are the outputs between the first and second networks in the twin neural network, respectively, ||TwinBP1(X)-TwinBP2(Y)|| 1 is the L1 norm between TwinBP1(X) and TwinBP2(Y), F1(K best )、F2(K best )、F3(K best )、F4(K best ) and F5(K best ) are K best Corresponding performance, availability, scalability, consistency, and partition tolerance, min is the minimization operator and max is the maximization operator.

[0025] As a preferred solution of the present invention, it also includes adding a migration coefficient in the parameter configuration model to realize the classification migration of engineering information management, wherein:

[0026] When the data attribute of the project information changes and the resource attribute of the distributed system does not change, a migration coefficient for data attribute migration is added to the parameter configuration model to obtain a data migration parameter configuration model for configuring the current distributed system to manage the changed project information;

[0027] The data migration parameter configuration model is:

[0028] In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, TwinBP2 is the second network of the twin network, and Wx is the migration coefficient for data attribute migration;

[0029] When the resource attributes of the distributed system change and the data attributes of the project information do not change, a migration coefficient for resource attribute migration is added to the parameter configuration model to obtain a system migration parameter configuration model for configuring the current project information to be managed by the distributed system after the change;

[0030] The data migration parameter configuration model is:

[0031] In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, TwinBP2 is the second network of the twin network, and Wy is the migration coefficient for data attribute migration.

[0032] As a preferred solution of the present invention, the migration coefficient for data attribute migration is quantified by the data attribute change status of the engineering information, and the migration coefficient for resource attribute migration is quantified by the resource attribute change status of the distributed system;

[0033] Wx = -(1-simx)log(simx);

[0034] Wy=-(1-simy)log(simy);

[0035] Where Wx is the migration coefficient for data attribute migration, Wy is the migration coefficient for data attribute migration, simx is the similarity between the data attributes of the current engineering information and the data attributes of the historical engineering information, and simy is the similarity between the resource attributes of the current distributed system and the resource attributes of the historical distributed system.

[0036] As a preferred solution of the present invention, the similarity is calculated by a similarity function.

[0037] As a preferred solution of the present invention, the present invention provides an engineering information management system based on distributed technology, which is applied to an engineering information management method based on distributed technology. The system includes:

[0038] A data collection unit for collecting data attributes of engineering information and resource attributes of a distributed system for managing engineering information;

[0039] A deep learning unit is used to obtain a parameter configuration model for automatically calculating configuration parameters of a distributed system based on data attributes and resource attributes of engineering information using a twin neural network;

[0040] The automatic configuration unit automatically configures the configuration parameters of the distributed system through a parameter configuration model to adapt the data attributes and resource attributes of the engineering information.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention configures the distributed system for managing engineering information by constructing a parameter configuration model to achieve configuration automation. At the same time, the management system configuration parameters are set by a unified parameter configuration model with strong objectivity, thereby ensuring the accuracy of the distributed system configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0044] Figure 1 A flow chart of an engineering information management method based on distributed technology provided by an embodiment of the present invention;

[0045] Figure 2 A block diagram of an engineering information management system based on distributed technology provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, the present invention provides an engineering information management method based on distributed technology, comprising the following steps:

[0048] Collect data attributes of engineering information and resource attributes of distributed systems used to manage engineering information;

[0049] Using twin neural networks, based on the data attributes and resource attributes of engineering information, a parameter configuration model for automatically calculating the configuration parameters of distributed systems is obtained;

[0050] Through the parameter configuration model, the configuration parameters of the distributed system are automatically configured to adapt the data attributes and resource attributes of the engineering information.

[0051] In order to improve the automation level of engineering information management, the present invention constructs a parameter configuration model to automatically configure the configuration parameters of the distributed system to adapt the data attributes and resource attributes of the engineering information, thereby realizing the automatic configuration of the engineering information management system.

[0052] In order to make the parameter configuration model adapt to the data attributes of the engineering information and the resource attributes of the distributed system, the present invention uses a twin neural network to calculate the configuration parameters of the distributed system based on the data attributes of the engineering information and the resource attributes of the distributed system, wherein:

[0053] The difference between the outputs of the first network and the second network in the twin neural network is used as part of the loss function, so that the configuration parameters of the distributed system calculated based on the data attributes of the engineering information and the resource attributes of the distributed system are highly consistent. That is, the configuration parameters of the distributed system output by the twin neural network can adapt to the data attributes of the engineering information, and can also adapt to the resource attributes of the distributed system. In other words, the configuration parameters of the distributed system output by the twin neural network can execute all its computing operations on the engineering information within the allowed range of the distributed system's resources.

[0054] The performance indicators of the distributed system are used as another part of the loss function, so that the configuration parameters that adapt to the data attributes of the engineering information and the resource attributes of the distributed system can be in a higher performance state when managing the engineering information. In other words, the configuration parameters of the distributed system output by the twin neural network can optimize the execution of all computing operations of the engineering information within the scope allowed by the resources of the distributed system.

[0055] Furthermore, in order to improve the portability of engineering information management, the present invention adds a migration coefficient to the parameter configuration model, so that when the engineering information to be managed changes while the distributed system to be managed remains unchanged (corresponding to the situation of using the distributed system to manage multiple categories of engineering information), the model can sense the change of engineering information during the management process, and adaptively perform migration calculations as the engineering information changes, and output configuration parameters that can manage the changed engineering information; and when the engineering information to be managed remains unchanged while the distributed system to be managed changes (corresponding to the situation of moving the managed engineering information from one distributed system to another distributed system for management), the model can sense the change of the distributed system during the management process, and adaptively perform migration calculations as the distributed system changes, and output configuration parameters of the changed distributed system.

[0056] Moreover, the migration coefficient added by the present invention is related to the degree of change of the engineering information and the degree of change of the distributed system, so that when the degree of change is higher, the corresponding network structure in the parameter configuration model is given a higher weight, so that the parameter configuration model pays more attention to the measurement of the network structure, and the output configuration parameter prediction result is more adapted to the degree of change of the engineering information or the degree of change of the distributed system, that is, it is possible to output the configuration parameters of the engineering information after the change, or adapt the configuration parameters of the distributed system after the change.

[0057] Therefore, the parameter configuration model constructed by the present invention can realize that the configuration parameters of the distributed system can optimize the execution of all computing services of the engineering information within the permitted range of the distributed system's resources, while improving the portability of engineering information management.

[0058] The data attributes of engineering information include the data structure, data volume and computing operations of the engineering information.

[0059] The resource attributes of a distributed system include computing resources, storage resources, and network bandwidth resources. Computing resources include CPU, memory, and storage resources include disk space.

[0060] The configuration parameters of the distributed system include basic environment configuration parameters, storage configuration parameters, Hadoop cluster configuration parameters, and server configuration parameters.

[0061] Basic configuration parameters:

[0062] Set the address server address of each Region (setEndpoint).

[0063] Set the namespace ID (setNamespace).

[0064] Set the application ID (setGroupId).

[0065] Set the AccessKey ID and AccessKey Secret of the cloud account.

[0066] Whether to enable grid computing (setEnableBatchWork).

[0067] Customize the client's listening port (setPort).

[0068] If the client has multiple network cards or a VPN environment, you can customize the host IP (setHost).

[0069] Distributed storage parameters:

[0070] Distributed storage provides a variety of parameter options, which can be flexibly configured according to user needs and budget. For example, the cloud server's HDFS provides two capacity options: 32MB and 256MB, and supports a variety of disaster recovery and fault tolerance mechanisms, including horizontal expansion, vertical expansion and hybrid cloud, to ensure high data availability.

[0071] Hadoop cluster configuration parameters:

[0072] Edit the core-site.xml file to set the basic parameters of the Hadoop cluster, such as the file system and Hadoop temporary directory.

[0073] Edit the hdfs-site.xml file to set HDFS-related parameters, such as data block size, number of replicas, etc. Edit the mapred-site.xml file (if used) to set MapReduce-related parameters.

[0074] Edit the yarn-site.xml file (if used) to set YARN related parameters.

[0075] Distributed system server configuration:

[0076] Deploy multiple single applications (the same application is deployed on each server), and then distribute client requests through load balancing (such as nginx or other software). Even if one or more servers have problems, the remaining servers can continue to process requests.

[0077] In order to improve the automation level of engineering information management, the present invention constructs a parameter configuration model to automatically configure the configuration parameters of the distributed system to adapt the data attributes and resource attributes of the engineering information, thereby realizing the automatic configuration of the engineering information management system, as follows:

[0078] The construction method of the parameter configuration model includes:

[0079] Select multiple types of engineering information, multiple types of distributed systems, and optimal configuration parameters of distributed systems;

[0080] The data attributes of various types of engineering information and the optimal configuration parameters of the distributed system are used as the input and output of the first network in the twin neural network respectively;

[0081] The resource attributes of various distributed systems and the optimal configuration parameters of the distributed systems are used as the input and output of the second network in the twin neural network respectively;

[0082] The difference between the outputs of the first network and the second network in the twin neural network and the performance index of the distributed system are used as the loss function to train the twin network and obtain a parameter configuration model;

[0083] The parameter configuration model is:

[0084]

[0085] In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, and TwinBP2 is the second network of the twin network.

[0086] The difference between the outputs of the first and second networks in the twin neural network is quantified by the L1 norm. The performance indicators of the distributed system include performance, availability, scalability, consistency, and partition tolerance. The loss function is:

[0087] L=min||TwinBP1(X)-TwinBP2(Y)|| 1 +max[F1(K best )+F2(K best )+F3(K best )+

[0088] F4(K best )+F5(K best )];

[0089] Where L is the loss function, TwinBP1(X) and TwinBP2(Y) are the outputs between the first and second networks in the twin neural network, respectively, ||TwinBP1(X)-TwinBP2(Y)|| 1 is the L1 norm between TwinBP1(X) and TwinBP2(Y), F1(K best )、F2(K best )、F3(K best )、F4(K best ) and F5(K best ) are K best Corresponding performance, availability, scalability, consistency, and partition tolerance, min is the minimization operator and max is the maximization operator.

[0090] The performance indicators of distributed systems mainly include performance, availability, scalability, consistency and partition tolerance (P).

[0091] Performance: Measures the system's ability to handle various tasks, including throughput, response time, and completion time. Throughput refers to the number of tasks that a system can handle within a certain period of time. Common throughput metrics include QPS (Queries Per Second), TPS (Transactions Per Second), and BPS (Bits Per Second). QPS is used to measure the number of queries processed by the system per second, TPS is used to measure the number of transactions processed by the system per second, and BPS is used to measure the amount of data processed by the system per second. These metrics directly reflect the performance of the system. When designing a distributed system, focus on how to improve the corresponding performance metrics based on the main operation type of the application (read or write operation).

[0092] Availability: focuses on the ability of the system to correctly provide services when faced with exceptions. Availability means that the system must respond as long as it receives a request, which can be understood as 7*24 continuous service. It needs to be distinguished from system reliability. Reliability refers to the correctness of the system's output, such as the need to query the correct information.

[0093] Scalability: refers to the characteristics of distributed systems that improve system performance (throughput, latency, concurrency), storage capacity, and computing power by expanding the scale of cluster machines.

[0094] Consistency: In a distributed system, the use of replicas may cause replica consistency issues. Data consistency is an inevitable problem derived from multi-partition scenarios. There are many ways to handle data consistency, such as data serialization or using standardized timestamps.

[0095] Partition tolerance (P): It is an important indicator of distributed systems, ensuring that the system can maintain availability and data consistency in the face of failures such as network partitions.

[0096] In order to make the parameter configuration model adapt to the data attributes of the engineering information and the resource attributes of the distributed system, the present invention uses a twin neural network to calculate the configuration parameters of the distributed system based on the data attributes of the engineering information and the resource attributes of the distributed system, wherein:

[0097] The difference between the outputs of the first network and the second network in the twin neural network is used as part of the loss function, so that the configuration parameters of the distributed system calculated based on the data attributes of the engineering information and the resource attributes of the distributed system are highly consistent. That is, the configuration parameters of the distributed system output by the twin neural network can adapt to the data attributes of the engineering information, and can also adapt to the resource attributes of the distributed system. In other words, the configuration parameters of the distributed system output by the twin neural network can execute all its computing operations on the engineering information within the allowed range of the distributed system's resources.

[0098] The performance indicators of the distributed system are used as another part of the loss function, so that the configuration parameters that adapt to the data attributes of the engineering information and the resource attributes of the distributed system can be in a higher performance state when managing the engineering information. In other words, the configuration parameters of the distributed system output by the twin neural network can optimize the execution of all computing operations of the engineering information within the scope allowed by the resources of the distributed system.

[0099] In order to improve the portability of engineering information management, the present invention also includes adding a migration coefficient in the parameter configuration model to realize the classification migration of engineering information management, wherein:

[0100] When the data attribute of the project information changes and the resource attribute of the distributed system does not change, a migration coefficient for data attribute migration is added to the parameter configuration model to obtain a data migration parameter configuration model for configuring the current distributed system to manage the changed project information;

[0101] The data migration parameter configuration model is:

[0102] In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, TwinBP2 is the second network of the twin network, and Wx is the migration coefficient for data attribute migration;

[0103] When the resource attributes of the distributed system change and the data attributes of the project information do not change, a migration coefficient for resource attribute migration is added to the parameter configuration model to obtain a system migration parameter configuration model for configuring the current project information to be managed by the distributed system after the change;

[0104] The data migration parameter configuration model is:

[0105] In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, TwinBP2 is the second network of the twin network, and Wy is the migration coefficient for data attribute migration.

[0106] The migration coefficient for data attribute migration is quantified by the data attribute change status of the engineering information, and the migration coefficient for resource attribute migration is quantified by the resource attribute change status of the distributed system;

[0107] Wx = -(1-simx)log(simx);

[0108] Wy=-(1-simy)log(simy);

[0109] Where Wx is the migration coefficient for data attribute migration, Wy is the migration coefficient for data attribute migration, simx is the similarity between the data attributes of the current engineering information and the data attributes of the historical engineering information, and simy is the similarity between the resource attributes of the current distributed system and the resource attributes of the historical distributed system.

[0110] The similarity is calculated using a similarity function.

[0111] The present invention uses the similarity between the data attributes of the current engineering information and the data attributes of the historical engineering information to construct a migration coefficient for data attribute migration, wherein the lower the similarity between the data attributes of the current engineering information and the data attributes of the historical engineering information, the greater the degree of change of the engineering information, and the higher the migration coefficient for data attribute migration is assigned, so that the network structure TwinBP1 (X) corresponding to the migration coefficient for data attribute migration added in the parameter configuration model has a higher weight, therefore, when the parameter configuration model measures the configuration parameters, it focuses more on adapting the engineering information, thereby realizing the configuration parameters of the engineering information after the output management change. On the other hand, the higher the similarity between the data attributes of the current engineering information and the data attributes of the historical engineering information, the smaller the degree of change of the engineering information, and the smaller the migration coefficient for data attribute migration is assigned, so that the network structure TwinBP1 (X) corresponding to the migration coefficient for data attribute migration added in the parameter configuration model has a smaller weight, that is, the closer to the network structure TwinBP2 (Y), therefore, when the parameter configuration model measures the engineering information unchanged and the distribution system unchanged, there will be no emphasis when configuring the parameters, so that the configuration parameters remain unchanged.

[0112] Similarly, the migration coefficient for resource attribute migration is constructed by using the similarity between the resource attributes of the current distributed system and the resource attributes of the historical distributed system, wherein the lower the similarity between the resource attributes of the current distributed system and the resource attributes of the historical distributed system, the greater the degree of change of the distributed system, and the higher the migration coefficient for resource attribute migration is assigned, so that the network structure TwinBP2(Y) corresponding to the migration coefficient for resource attribute migration added in the parameter configuration model has a higher weight, therefore, when the parameter configuration model calculates the configuration parameters, it focuses more on adapting the distributed system, thereby achieving the configuration parameters of the distributed system after the adaptation change. On the other hand, the higher the similarity between the resource attributes of the current distributed system and the resource attributes of the historical distributed system, the smaller the degree of change of the distributed system, and the smaller the migration coefficient for resource attribute migration is assigned, so that the network structure TwinBP2(Y) corresponding to the migration coefficient for resource attribute migration added in the parameter configuration model has a smaller weight, that is, the closer to the network structure TwinBP1(X), therefore, when the parameter configuration model calculates the engineering information unchanged and the distributed system unchanged, there will be no emphasis when configuring the parameters, so that the configuration parameters remain unchanged.

[0113] When both the engineering information and the distributed system change, the original parameter configuration model is used to calculate the configuration parameters, which is the initialization configuration.

[0114] The present invention adds a migration coefficient to the parameter configuration model, so that when the engineering information to be managed changes while the distributed system to be managed remains unchanged (corresponding to the situation where the distributed system is used to manage multiple categories of engineering information), the model can sense the change of engineering information during the management process, and adaptively perform migration calculations as the engineering information changes, and output configuration parameters that can manage the changed engineering information. Also, when the engineering information to be managed remains unchanged while the distributed system to be managed changes (corresponding to the situation where the managed engineering information needs to be moved from one distributed system to another distributed system for management), the model can sense the change of the distributed system during the management process, and adaptively perform migration calculations as the distributed system changes, and output configuration parameters of the changed distributed system.

[0115] The initial configurations when the managed distributed system remains unchanged (corresponding to the situation where the distributed system is used to manage multiple categories of engineering information) and when the managed distributed system changes (corresponding to the need to move the managed engineering information from one distributed system to another distributed system for management) both use the original parameter configuration model to calculate their configuration parameters.

[0116] Moreover, the migration coefficient added by the present invention is related to the degree of change of the engineering information and the degree of change of the distributed system, so that when the degree of change is higher, the corresponding network structure in the parameter configuration model is given a higher weight, so that the parameter configuration model pays more attention to the measurement of the network structure, and the output configuration parameter prediction result is more adapted to the degree of change of the engineering information or the degree of change of the distributed system, that is, it is possible to output the configuration parameters of the engineering information after the change, or adapt the configuration parameters of the distributed system after the change.

[0117] like Figure 2 As shown, the present invention provides an engineering information management system based on distributed technology, which is applied to an engineering information management method based on distributed technology. The system includes:

[0118] A data collection unit for collecting data attributes of engineering information and resource attributes of a distributed system for managing engineering information;

[0119] A deep learning unit is used to obtain a parameter configuration model for automatically calculating configuration parameters of a distributed system based on data attributes and resource attributes of engineering information using a twin neural network;

[0120] The automatic configuration unit automatically configures the configuration parameters of the distributed system through the parameter configuration model to adapt the data attributes and resource attributes of the engineering information.

[0121] The present invention configures the distributed system for managing engineering information by constructing a parameter configuration model to achieve configuration automation. At the same time, the management system configuration parameters are set by a unified parameter configuration model with strong objectivity, thereby ensuring the accuracy of the distributed system configuration.

[0122] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. An engineering information management method based on distributed technology, Features: The following steps are involved: Collect data attributes of engineering information and resource attributes of distributed systems used to manage engineering information; Using twin neural networks, based on the data attributes and resource attributes of engineering information, a parameter configuration model for automatically calculating the configuration parameters of distributed systems is obtained; Through the parameter configuration model, the configuration parameters of the distributed system are automatically configured to adapt the data attributes and resource attributes of the engineering information.

2. According to claim 1, a distributed engineering information management method, Features: The data attributes of the engineering information include the data structure, data volume and computing operations of the engineering information.

3. According to claim 1, a distributed engineering information management method, Features: The resource attributes of the distributed system include computing resources, storage resources and network bandwidth resources.

4. The engineering information management method based on distributed technology according to claim 1, Features: The configuration parameters of the distributed system include basic environment configuration parameters, storage configuration parameters, Hadoop cluster configuration parameters and server configuration parameters.

5. The engineering information management method based on distributed technology according to claim 1, Features: The method for constructing the parameter configuration model includes: Select multiple types of engineering information, multiple types of distributed systems, and optimal configuration parameters of distributed systems; The data attributes of various types of engineering information and the optimal configuration parameters of the distributed system are used as the input and output of the first network in the twin neural network respectively; The resource attributes of various distributed systems and the optimal configuration parameters of the distributed systems are used as the input and output of the second network in the twin neural network respectively; Taking the difference between the output of the first network and the second network in the twin neural network and the performance index of the distributed system as the loss function, the twin network is trained to obtain the parameter configuration model; The parameter configuration model is: In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, and TwinBP2 is the second network of the twin network.

6. The engineering information management method based on distributed technology according to claim 5, Features: The difference between the outputs of the first network and the second network in the twin neural network is quantified by the L1 norm. The performance indicators of the distributed system include performance, availability, scalability, consistency, and partition tolerance. The loss function is: <h2 style=";text-align:left;direction:ltr">L=min||TwinBP1(X)-TwinBP2(Y)||<h2 style=";text-align:left;direction:ltr"> 1 <h2 style=";text-align:left;direction:ltr"> +max[F1(K<h2 style=";text-align:left;direction:ltr"> best <h2 style=";text-align:left;direction:ltr"> )+F2(K<h2 style=";text-align:left;direction:ltr"> best <h2 style=";text-align:left;direction:ltr"> )+F3(K<h2 style=";text-align:left;direction:ltr"> best <h2 style=";text-align:left;direction:ltr"> )+ F4(K best )+F5(K best )]; Where L is the loss function, TwinBP1(X) and TwinBP2(Y) are the outputs between the first and second networks in the twin neural network, respectively, ||TwinBP1(X)-TwinBP2(Y)|| 1 is the L1 norm between TwinBP1(X) and TwinBP2(Y), F1(K best )、F2(K best )、F3(K best )、F4(K best ) and F5(K best ) are K best Corresponding performance, availability, scalability, consistency, and partition tolerance, min is the minimization operator and max is the maximization operator.

7. The engineering information management method based on distributed technology according to claim 6, Features: It also includes adding migration coefficients in the parameter configuration model to achieve classified migration of engineering information management, where: When the data attribute of the project information changes and the resource attribute of the distributed system does not change, a migration coefficient for data attribute migration is added to the parameter configuration model to obtain a data migration parameter configuration model for configuring the current distributed system to manage the changed project information; In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, TwinBP2 is the second network of the twin network, and Wx is the migration coefficient for data attribute migration; When the resource attributes of the distributed system change and the data attributes of the project information do not change, a migration coefficient for resource attribute migration is added to the parameter configuration model to obtain a system migration parameter configuration model for configuring the current project information to be managed by the distributed system after the change; The data migration parameter configuration model is: In the formula, K best is the optimal configuration parameter of the distributed system, X is the data attribute of the engineering information, Y is the resource attribute of the distributed system, TwinBP1 is the first network of the twin network, TwinBP2 is the second network of the twin network, and Wy is the migration coefficient for data attribute migration.

8. The engineering information management method based on distributed technology according to claim 7, Features: The migration coefficient for data attribute migration is quantified by the data attribute change status of the engineering information, and the migration coefficient for resource attribute migration is quantified by the resource attribute change status of the distributed system; Wx = -(1-simx)log(simx); Wy=-(1-simy)log(simy); Where Wx is the migration coefficient for data attribute migration, Wy is the migration coefficient for data attribute migration, simx is the similarity between the data attributes of the current engineering information and the data attributes of the historical engineering information, and simy is the similarity between the resource attributes of the current distributed system and the resource attributes of the historical distributed system.

9. The engineering information management method based on distributed technology according to claim 8, Features: The similarity is calculated by using a similarity function.

10. An engineering information management system based on distributed technology, It is characterized in that An engineering information management method based on distributed technology applied to any one of claims 1 to 9, the system comprising: A data collection unit for collecting data attributes of engineering information and resource attributes of a distributed system for managing engineering information; A deep learning unit is used to obtain a parameter configuration model for automatically calculating the configuration parameters of the distributed system based on the data attributes and resource attributes of the engineering information using a twin neural network; The automatic configuration unit automatically configures the configuration parameters of the distributed system through a parameter configuration model to adapt the data attributes and resource attributes of the engineering information.