Operation cost estimation method and system based on machine learning

Abstract

The present invention discloses an operation cost estimation method and system based on machine learning, and relates to the technical field of enterprise operation management. The operation cost estimation method based on machine learning of the present invention adopts a distributed computing architecture, which improves the efficiency of data processing and model training. By dividing the operation data into multiple computing nodes for parallel processing, the time cycle of cost estimation can be significantly shortened, so that the enterprise can obtain accurate cost estimation results in time, thereby quickly responding to market changes and adjusting operation strategies in time. Moreover, the distributed computing architecture also has good scalability. With the growth of the enterprise's business and the increase of data volume, computing nodes can be easily added to adapt to new needs, which effectively reduces the enterprise's hardware upgrade costs and system maintenance complexity, and provides strong technical support for the long-term development of the enterprise.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise operation management, and specifically to an operation cost estimation method and system based on machine learning. Background Art

[0002] With the continuous expansion of enterprise scale and increasingly fierce market competition, accurate cost estimation has become more and more important. Cost estimation not only helps enterprises optimize resource allocation and improve operational efficiency, but also provides key data support in the decision-making process. In recent years, the development of machine learning technology has provided new solutions for cost estimation. Through automated data analysis and model training, future cost changes can be predicted more accurately.

[0003] Then, the existing machine learning-based cost estimation system adopts a centralized computing architecture, which is inefficient when processing large-scale data sets, resulting in a long cost estimation cycle and an inability to meet the needs of real-time monitoring. In addition, the monolithic architecture design of the traditional system makes the system scalability and maintainability poor, making it difficult to adapt to changing business needs. It lacks the ability to effectively integrate and process multi-source heterogeneous data, resulting in incomplete feature extraction and affecting the accuracy of the model.

[0004] In summary, the existing operating cost estimation technology is unable to respond and process the large-scale, multi-source and dynamic operating data of modern enterprises in a timely manner. An innovative distributed computing operating cost strategy based on machine learning is urgently needed to solve these problems, so as to improve the level of refinement of enterprise operation management and the scientificity and timeliness of decision-making. Summary of the invention

[0005] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide an operating cost estimation method and system based on machine learning. By building a distributed computing architecture, data processing tasks are distributed to multiple computing nodes to achieve parallel processing capabilities. At the same time, the distributed computing architecture also supports real-time data stream processing, so that the system can quickly respond to market changes and internal operations, thereby supporting the company's real-time cost monitoring and rapid decision-making. This efficient processing capability and real-time response capability are particularly important for companies that need to frequently adjust their strategies, and help companies maintain their advantages in a highly competitive market environment.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: On the one hand, an operation cost estimation method based on machine learning, the specific steps of the method are:

[0007] S1. Collecting raw data of operating costs, and using distributed computing technology to distribute computing tasks of the raw operating data to computing nodes, wherein the computing nodes include a central computing node and multiple sub-computing nodes;

[0008] S2. Use data partitioning strategy to divide the data of different computing nodes into multiple data subsets according to business types, and store them in different computing nodes respectively;

[0009] S3. Each computing node loads a locally stored data subset, and re-partitions the data subset according to the characteristics, time series, and business type of the data subset using the data partitioning strategy. The data subset is then subjected to feature extraction to obtain a feature vector. The feature extraction includes filling missing values, detecting and correcting outliers, and selecting and reducing dimensions.

[0010] S4. Build an operation cost estimation model based on machine learning in the central computing node, start the distributed training task, and train the model using the feature vectors extracted from each sub-computing node;

[0011] S5. After the training is completed, the information exchange between nodes is carried out through the distributed communication protocol, the local gradient information of each sub-computing node is transmitted to the central computing node, and the model parameters are updated on the central computing node;

[0012] S6. Integrate and store the final model parameters, and output the estimated operating cost results through the model.

[0013] Furthermore, the S1 distributed computing technology is used to distribute computing tasks of operational data and model training to multiple sub-computing nodes for parallel processing. The sub-computing nodes are server clusters, each of which can independently perform data processing and resource calculations, and are interconnected and work collaboratively through wireless network communication links.

[0014] Furthermore, the S2 data partitioning strategy is used to partition operational data Divide into Data subset ,in ,satisfy , each sub-computation node Store the corresponding data subset , the data subset The data points in , , and the data points Include Attributes , , the communication bandwidth between each sub-computing node is .

[0015] Furthermore, when each computing node in S3 uses the data partitioning strategy to divide the data subsets, the data is divided according to the business type of the data, that is, the number of business types is set to , then the amount of operational data corresponding to each business type is , ,satisfy , for each business type's operational data , and further divided into multiple subsets according to time series , ,satisfy , and when data is missing, when the business type Lower Attributes The missing value ratio is greater than the missing ratio threshold , then other non-missing sets based on the business type are used The mean filling of , the filled data set is .

[0016] Furthermore, when S3 performs outlier detection and correction on a data subset, it calculates the attribute exist The mean in , standard deviation , when the data point The attribute value satisfies , is the outlier determination coefficient, then it is determined to be an outlier, and the correction value , and, for multiple consecutive outliers The correction adopts linear interpolation method, using normal data points and In Properties The values ​​on and , then the corrected outlier ,in , , and obtain the corrected data set .

[0017] Furthermore, when S3 performs feature selection and dimensionality reduction on a data subset, the feature About Categories Information entropy ,in Yes Category Probability, features in the dataset For the dataset The information gain is ,in It is a feature The value set of yes Medium Features The value is A subset of It is a feature The value is The conditional entropy at the time of ,when When , the feature is removed and the final feature vector is obtained ,in Indicates Features.

[0018] Furthermore, the S4 utilizes each computing node The processed feature vector Perform model training for machine learning-based models , set the model parameters to , the loss function is ,in It is a data point The corresponding actual cost value, It is a model Based on the feature vector and parameters The predicted value of Calculate the local gradient , through distributed communication protocols, according to the information interaction cycle , the local gradient information of each sub-computing node is transmitted to the designated central computing node for parameter aggregation to obtain the global gradient , update the model parameters according to the global gradient ,in is the learning rate, is the number of training iterations.

[0019] Furthermore, the S5 distributed communication protocol adopts a communication method based on the message passing interface MPI. When the gradient is summarized, a global gradient calculation operation is performed and the communication overhead is and the number of data subsets , the number of model parameters And communication broadband related ,satisfy .

[0020] On the other hand, the operating cost estimation system based on machine learning includes: data collection module, data segmentation module, distributed computing architecture module, model building module, information transmission module, and cost estimation module;

[0021] The data collection module collects operational data from the internal and external data platforms of the enterprise through the data collection interface. The data collection interface includes: a database connection interface for connecting to the internal database of the enterprise, a web crawler interface for collecting external network data, and a data call interface for data interaction with the external data platform. The collected data is transmitted to the distributed computing architecture module for storage and processing;

[0022] The data segmentation module uses a data partitioning strategy to segment the data of different computing nodes into multiple data subsets according to business types, and stores them on different computing nodes respectively;

[0023] The distributed computing architecture module uses the data subsets of the data segmentation module to build a distributed computing environment, store operational data and perform feature extraction;

[0024] The model building module builds an operating cost estimation model based on machine learning, and uses the feature vector after feature extraction to perform model training;

[0025] The information transmission module performs information exchange between nodes through a distributed communication protocol, and transmits the gradient information and model parameter changes obtained by each sub-computing node to the central computing node for aggregation and integration;

[0026] The cost estimation module receives the operation data to be estimated, processes it through the distributed computing architecture module, and then inputs it into the model trained by the model building module to obtain and output the operation cost estimation results.

[0027] Furthermore, the computing nodes in the distributed computing architecture module With dynamic resource allocation capability, that is, giving computing nodes Initial computing resources , when the load of the model training task Exceeding the threshold When the computing node Automatically request additional resources from the resource pool , the amount of resources requested meets , is the resource allocation coefficient, and the total amount of resources in the resource pool is ,satisfy .

[0028] Compared with the existing technology, the operating cost estimation method and system based on machine learning has the following beneficial effects:

[0029] 1. The operating cost estimation method based on machine learning of the present invention adopts a distributed computing architecture, which improves the efficiency of data processing and model training. By dividing the operating data into multiple computing nodes for parallel processing, the time cycle of cost estimation can be significantly shortened, so that enterprises can obtain accurate cost estimation results in a timely manner, thereby quickly responding to market changes and adjusting operating strategies in a timely manner. In addition, the distributed computing architecture also has good scalability. As the company's business grows and the amount of data increases, computing nodes can be easily added to adapt to new needs, effectively reducing the company's hardware upgrade costs and system maintenance complexity, and providing strong technical support for the company's long-term development.

[0030] 2. The microservice architecture design of the present invention brings high flexibility and maintainability to the operating cost estimation system. It splits the entire system into multiple independent but mutually cooperative microservice modules. Each microservice module is independently developed, deployed and expanded, so that the system can quickly adapt to changes in enterprise business needs and technological upgrades. When the enterprise introduces new cost factors and data sources, it only needs to modify the data acquisition module and feature engineering module in a targeted manner without affecting the stability of the entire system and the normal operation of other functional modules. At the same time, the microservice architecture is easy to integrate with the company's existing information system, and realizes seamless data connection and business process collaboration through standardized interfaces, realizes efficient circulation and sharing of information, and improves the coordination and intelligence level of the company's overall operation management.

[0031] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of the method for estimating operating costs based on machine learning;

[0034] Figure 2 This is a flowchart of outlier judgment for the operating cost estimation method based on machine learning;

[0035] Figure 3 Model training flowchart for the machine learning-based operating cost estimation method. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Embodiment 1

[0038] This embodiment describes in detail the implementation process of the operating cost estimation method based on machine learning under a distributed computing architecture. It collects operating data from inside and outside the enterprise, uses distributed computing technology and data partitioning strategies, and processes data in parallel on multiple computing nodes, including data subset division, feature extraction, model training, and information interaction between nodes. Ultimately, accurate operating cost estimation is achieved. This demonstrates how this method can efficiently process large-scale data, improve estimation efficiency and accuracy, and adapt to dynamic changes in enterprise business.

[0039] In the specific implementation, first, operational data is collected from within the enterprise, including raw material procurement data, working hours and equipment usage data in the production process, and customer order information and sales channel data. Raw material market price fluctuation data and industry labor cost dynamic information are obtained from external data platforms. The collected operational data are of various formats and large scale, including structured data (various transaction amounts and quantities) and unstructured data (customer feedback texts, market trend reports), forming a set of original operational data with a total volume of D. Using distributed computing technology, the computing tasks of the original operational data are assigned to computing nodes. This distributed computing architecture consists of a central computing node and multiple sub-computing nodes. The sub-computing nodes are in the form of server clusters. Each sub-computing node has independent data processing and resource computing capabilities, and is interconnected and collaborative through high-speed wireless network communication links. This architecture lays the foundation for the efficient processing of subsequent large-scale data, avoids the performance bottleneck of traditional centralized computing architectures when facing massive data, and ensures that it can process the continuously influx of operational data in real time to adapt to the dynamic operating environment of the enterprise.

[0040] Then, using the data partitioning strategy, the operational data Precise division into Data subset ( ),satisfy , each sub-computation node Responsible for storing the corresponding data subset , a subset of the data The data points in ( ), and the data point Include Attributes ( ), the communication bandwidth between sub-computing nodes is In the process of division, we divide the business into coarse-grained divisions according to the business type. Operational data , further subdivided into multiple subsets according to time series ( ),satisfy , so that data can be structured and organized in the storage stage, ensuring that data of the same business type and similar time are concentrated in the same and similar sub-computing nodes, improving the pertinence and efficiency of data processing, and reducing unnecessary data transmission and processing overhead.

[0041] Then, each sub-computing node Load a locally stored subset of data After that, according to the characteristics of data subsets, time series and business types, the data partitioning strategy is used again to deeply divide the data subsets, and feature extraction is carried out to obtain feature vectors. This process includes three core links: filling missing values, detecting and correcting outliers, and selecting and reducing dimensions. Filling missing values: for attributes Missing value handling, when the business type Lower Attributes The missing value ratio exceeds the missing ratio threshold When other non-missing sets based on the business type are used The mean filling strategy is In the manufacturing business data subset, the equipment operation time attribute is missing in some records. If the proportion of missing values ​​of this attribute is too high, it is filled according to the average operation time of the same type of equipment in other time periods to generate a filled data set , ensuring data integrity and avoiding analysis bias and model inaccuracy caused by missing data; outlier detection and correction: calculation attributes exist The mean in With standard deviation , when the data point The attribute value satisfies ( is the outlier determination coefficient) when it is determined as an outlier, and the correction value , for multiple consecutive outliers , use linear interpolation to correct, that is, in the marketing cost data, if the advertising cost of the current time period has continuous abnormal fluctuations, linear interpolation adjustment is performed based on the cost data of the adjacent normal time period, that is, using the normal data points and In Properties The values ​​on and , corrected outliers ( ), generate the corrected dataset , improve data quality, enhance model robustness, and avoid abnormal data interfering with model training and cost estimation accuracy; Feature selection and dimensionality reduction: Calculate features About Categories Information entropy , and features For the dataset Information gain ,in It is a feature The value set of yes Medium Features The value is A subset of It is a feature The value is The conditional entropy at the time of ,when When the feature is removed, the feature vector is finally determined , achieve feature space optimization, reduce model complexity, improve model training efficiency and generalization ability, and focus on key factors to improve cost estimation accuracy.

[0042] Next, the central computing node builds an operating cost estimation model based on machine learning , set the model parameters to , the loss function is ,in For data points The corresponding true cost value, It is a model Based on the feature vector and parameters After starting the distributed training task, each child computing node The feature vector extracted using local features Participate in training, on the child computing node Calculate the local gradient With a fixed information exchange period T, the local gradient information of each sub-computing node is transmitted to the central computing node through a distributed communication protocol based on the message passing interface MPI. That is, each sub-computing node calculates the local gradient corresponding to its own data subset and transmits it to the central computing node in time. The central computing node summarizes the global gradient , according to the learning rate Update model parameters ( is the number of training iterations), and the model is optimized through multiple iterations to make the model accurately fit the data characteristics and cost rules, thereby improving the accuracy and reliability of cost estimation. During the entire training process, the distributed communication protocol is used to ensure smooth information exchange between nodes. For the gradient aggregation stage, the communication overhead and the number of data subsets , the number of model parameters and communication bandwidth Closely related, following the formula By strictly controlling communication overhead and interaction processes, we ensure efficient processing and stable operation, timely aggregate local information scattered across each sub-computing node, provide support for accurate updating of model parameters, promote continuous optimization of the model, and ensure that cost estimation performance is always excellent in a distributed environment.

[0043] Finally, after completing model training and parameter updating, the final model parameters are integrated and stored. When the estimated operational data is input, the feature vector is obtained through data collection, distributed computing architecture processing and feature extraction process and then input into the fully trained model. , the operating cost estimate is obtained through the model The results are fed back to the company's management in real time, providing data basis for the company's operational decisions (pricing strategy, resource allocation, budget planning), helping the company to accurately control costs, optimize operations, improve efficiency and competitiveness in a complex and changing market, and realize intelligent cost management and scientific decision-making.

[0044] This embodiment fully demonstrates the powerful performance of the operating cost estimation method based on machine learning under the distributed computing architecture. It controls the integration of multi-source heterogeneous data from the source of data collection, realizes efficient processing and storage through distributed architecture and sophisticated data partitioning strategy, optimizes data feature space through rigorous feature extraction, and iteratively optimizes model parameters based on distributed training and information interaction mechanism. Finally, it outputs accurate cost estimation results and empowers enterprise decision-making. Relying on the advantages of distributed computing architecture, this method successfully overcomes the technical difficulties of traditional cost estimation, greatly improves processing efficiency, estimation accuracy, system scalability and flexibility, and strongly supports enterprises in accurate cost control and sustainable development in the big data era.

[0045] Embodiment 2

[0046] This example focuses on the operating cost estimation system based on the microservice architecture, and elaborates on the complete process from data collection, processing to cost estimation and system integration. The microservice architecture splits the system into multiple independent modules to work together, showing how to efficiently process operating data, accurately estimate costs, and flexibly adapt to the needs of corporate business and technological changes, providing accurate cost insights and decision-making support for corporate operations management, and improving overall operational efficiency and competitiveness.

[0047] The operating cost estimation system based on microservice architecture mainly includes data collection module, data segmentation module, distributed computing architecture module, model building module, information transmission module and cost estimation module. Each module is independently developed, deployed and expanded, and interacts and collaborates through standardized interfaces to build a flexible and efficient cost estimation system. Among them, the data collection module is the data source of the system, responsible for accurately collecting operating data from multiple data sources within the enterprise and external data platforms. It adapts to the characteristics of different data sources by using the data collection interface. For the structured ERP data within the enterprise, the database connection interface is used to accurately extract the data of material procurement costs and production work order hours based on query statements; for the external semi-structured market research report data, the web crawler interface is enabled to extract key information on market trends and price fluctuations of competing products according to specific rules; for the API interface provided by the cooperative data platform, the data call interface is used to obtain dynamic data on industry labor costs, and the number of data sources that can be collected is set to , each data source The corresponding data acquisition interface is The data collection frequency is The data volume growth rate is , then the total data acquisition rate , ensuring that data is collected in a timely, comprehensive and accurate manner, laying a solid foundation for subsequent cost estimates, capturing changes in cost drivers in all aspects of corporate operations in real time, covering multi-dimensional information such as production inputs, market fluctuations, and changes in customer demand, to avoid cost estimate deviations due to missing or delayed data.

[0048] After data collection is completed, the data segmentation module processes and distributes the data according to the data partitioning strategy, first by business type, and then by time series. In this way, the data is reasonably distributed to each sub-computing node of the distributed computing architecture. , each Store the corresponding data subset The sub-computing nodes exchange data through high-speed network communication links, and the communication bandwidth is , ensure smooth data flow, realize parallel processing of large-scale operational data, avoid processing bottlenecks in centralized architecture, and improve data processing timeliness and overall system performance.

[0049] Then, the data segmentation module works together with the distributed computing architecture module to Feature engineering is carried out to enhance data quality and feature representativeness. First, missing values ​​are filled in the data. , when the business type The missing value ratio exceeds the missing ratio threshold When other non-missing sets under this business type are used Mean filling, that is , and generate , ensuring data integrity and reliability, laying the foundation for in-depth analysis, preventing missing values ​​from distorting feature engineering and cost estimation results, and then performing outlier detection and correction, based on attributes exist The mean in With standard deviation ,when ( is the outlier determination coefficient) when it is determined as an outlier, and the correction value , for continuous outliers ( ), using linear interpolation, borrowing normal data points and Property Value and , the correction value is ( ),generate , improve data quality stability, ensure feature extraction accuracy, enhance model anti-interference ability, avoid abnormal data misleading cost estimation model, and finally perform feature selection and dimensionality reduction to calculate features About Categories Information entropy And the data set Information gain ( yes Value set, yes Value A subset of is the conditional entropy), according to the information gain threshold Culling The characteristics of , effectively compress the feature space, reduce model complexity, improve training efficiency and generalization ability, focus on core cost influencing factors to improve estimation accuracy, and filter out irrelevant redundant feature interference.

[0050] Subsequently, through the model building and training modules, the central computing node builds an operating cost estimation model based on machine learning. , set the model parameters to , loss function ( yes True cost value, is the model prediction value. During the training process, the sub-computation node Based on local eigenvector Calculate local gradient , according to the information interaction cycle The global gradient is transmitted to the central computing node through the MPl communication protocol , according to the learning rate Update model parameters ( The model fitting ability and prediction accuracy are optimized through repeated iterations, and the nonlinear relationship between operating costs and multiple factors is accurately portrayed, laying the foundation for reliable cost estimation. The information transmission module uses the distributed communication protocol as a bridge to ensure smooth data and information interaction within the system, which reduces communication overhead. Tightly manage the number of data subsets n, the number of model parameters q, and the communication bandwidth ,follow Regularity, accurately and efficiently coordinate the information flow between each sub-computing node and the central computing node, ensure that the key information of the local gradient is timely gathered at the central node to update the model parameters, maintain the consistency and efficiency of the system's collaborative work, ensure continuous optimization of model training, support the system to stably and accurately output cost estimation results, reduce communication delays and data loss risks, and improve the overall reliability and response speed of the system.

[0051] Finally, after the cost estimation module receives the estimated operation data, it is first collected through the data acquisition module interface, converted into feature vectors through distributed computing architecture and feature engineering, and then input into the mature training model. , generate operating cost estimates based on the model The results are presented to corporate management and business departments in the form of intuitive reports and visual charts, empowering the company's operational decisions in all aspects.

[0052] In summary, this embodiment deeply analyzes the full-process operation mechanism and technical details of the operating cost estimation system based on the microservice architecture, from ensuring the breadth and depth of data collection, taking advantage of the distributed processing architecture, improving the accuracy of feature engineering, optimizing and iterating model training, and efficient and reliable information transmission, to the accurate output of cost estimation and the flexibility of system integration and expansion. The various links of the system work closely together and complement each other. The microservice architecture gives the system strong adaptability and maintainability, which effectively copes with the challenges of enterprise operation complexity and dynamic changes. Through the collaboration of accurate cost estimation and system integration, intelligent power is injected into enterprise operation management, helping enterprises optimize resource allocation, strengthen cost control, accelerate decision-making response, and enhance market competitiveness and sustainable development capabilities.

[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. The operating cost estimation method based on machine learning is characterized by: The specific steps of this method are: S1. Collecting raw data of operating costs, and using distributed computing technology to distribute computing tasks of the raw operating data to computing nodes, wherein the computing nodes include a central computing node and multiple sub-computing nodes; S2. Use data partitioning strategy to divide the data of different computing nodes into multiple data subsets according to business types, and store them in different computing nodes respectively; S3. Each computing node loads a locally stored data subset, and re-partitions the data subset according to the characteristics, time series, and business type of the data subset using the data partitioning strategy. The data subset is then subjected to feature extraction to obtain a feature vector. The feature extraction includes filling missing values, detecting and correcting outliers, and selecting and reducing dimensions. S4. Build an operation cost estimation model based on machine learning in the central computing node, start the distributed training task, and train the model using the feature vectors extracted from each sub-computing node; S5. After the training is completed, the information exchange between nodes is carried out through the distributed communication protocol, the local gradient information of each sub-computing node is transmitted to the central computing node, and the model parameters are updated on the central computing node; S6. Integrate and store the final model parameters, and output the estimated operating cost results through the model.

2. The method for estimating operating costs based on machine learning according to claim 1, characterized in that: The S1 distributed computing technology is used to distribute computing tasks of operational data and model training to multiple sub-computing nodes for parallel processing. The sub-computing nodes are server clusters. Each computing node can independently perform data processing and resource calculations, and are interconnected and work collaboratively through wireless network communication links.

3. The method for estimating operating costs based on machine learning according to claim 1, characterized in that: The S2 data partitioning strategy is used to partition operational data Divide into Data subset ,in ,satisfy , each sub-computation node Store the corresponding data subset , the data subset The data points in , , and the data points Include Attributes , , the communication bandwidth between each sub-computing node is .

4. The method for estimating operating costs based on machine learning according to claim 1, characterized in that: When each computing node in S3 uses the data partitioning strategy to divide the data subsets, it divides the data according to the business type, that is, the number of business types is set to , then the amount of operational data corresponding to each business type is , ,satisfy , for each business type's operational data , and further divided into multiple subsets according to time series , ,satisfy , and when data is missing, when the business type Lower Attributes The missing value ratio is greater than the missing ratio threshold , then other non-missing sets based on the business type are used The mean filling of , the filled data set is .

5. The method for estimating operating costs based on machine learning according to claim 4, characterized in that: When S3 performs outlier detection and correction on a data subset, it calculates the attribute exist The mean in , standard deviation , when the data point The attribute value satisfies , is the outlier determination coefficient, then it is determined to be an outlier, and the correction value , and, for multiple consecutive outliers The correction adopts linear interpolation method, using normal data points and In Properties The values ​​on and , then the corrected outlier ,in , , and obtain the corrected data set .

6. The method for estimating operating costs based on machine learning according to claim 5, characterized in that: When S3 performs feature selection and dimensionality reduction on a data subset, it calculates the features About Categories Information entropy ,in Yes Category Probability, features in the dataset For the dataset The information gain is ,in It is a feature The value set of yes Medium Features The value is A subset of It is a feature The value is The conditional entropy at the time of ,when When , the feature is removed and the final feature vector is obtained ,in Indicates Features.

7. The method for estimating operating costs based on machine learning according to claim 6, characterized in that: The S4 utilizes each computing node The processed feature vector Perform model training for machine learning-based models , set the model parameters to , the loss function is ,in It is a data point The corresponding actual cost value, It is a model Based on the feature vector and parameters The predicted value of Calculate the local gradient , through distributed communication protocols, according to the information interaction cycle , the local gradient information of each sub-computing node is transmitted to the designated central computing node for parameter aggregation to obtain the global gradient , update the model parameters according to the global gradient ,in is the learning rate, is the number of training iterations.

8. The method for estimating operating costs based on machine learning according to claim 1, characterized in that: The S5 distributed communication protocol adopts a communication method based on the message passing interface MPI. When the gradient is summarized, a global gradient calculation operation is performed and the communication overhead and the number of data subsets , the number of model parameters And communication broadband related ,satisfy .

9. The operating cost estimation system based on machine learning is characterized by: The method comprises using the method for estimating operating costs based on machine learning as described in any one of claims 1 to 8, wherein the system comprises: a data acquisition module, a data segmentation module, a distributed computing architecture module, a model building module, an information transmission module, and a cost estimation module; The data collection module collects operational data from the internal and external data platforms of the enterprise through the data collection interface. The data collection interface includes: a database connection interface for connecting to the internal database of the enterprise, a web crawler interface for collecting external network data, and a data call interface for data interaction with the external data platform. The collected data is transmitted to the distributed computing architecture module for storage and processing; The data segmentation module uses a data partitioning strategy to segment the data of different computing nodes into multiple data subsets according to business types, and stores them on different computing nodes respectively; The distributed computing architecture module uses the data subsets of the data segmentation module to build a distributed computing environment, store operational data and perform feature extraction; The model building module builds an operating cost estimation model based on machine learning, and uses the feature vector after feature extraction to perform model training; The information transmission module performs information exchange between nodes through a distributed communication protocol, and transmits the gradient information and model parameter changes obtained by each sub-computing node to the central computing node for aggregation and integration; The cost estimation module receives the operation data to be estimated, processes it through the distributed computing architecture module, and then inputs it into the model trained by the model building module to obtain and output the operation cost estimation results.

10. The operating cost estimation system based on machine learning according to claim 9, characterized in that: The computing nodes in the distributed computing architecture module With dynamic resource allocation capability, that is, giving computing nodes Initial computing resources , when the load of the model training task Exceeding the threshold When the computing node Automatically request additional resources from the resource pool , the amount of resources requested meets , is the resource allocation coefficient, and the total amount of resources in the resource pool is ,satisfy .

Images