Big data processing and analyzing system of data medium station

Abstract

The invention discloses a big data processing and analysis system for a data medium platform, and belongs to the technical field of data analysis. Data acquisition is performed according to each data source to obtain enterprise data; storing and processing the enterprise data; obtaining product information of each user product, setting a product service scheme according to the product information, configuring an intelligent service model according to the product service scheme, identifying a learning data range corresponding to the intelligent service model, collecting learning optimization data according to the learning data range, and performing optimization adjustment on the intelligent service model according to the learning optimization data; performing product service analysis according to the intelligent service model and the product service scheme to obtain a product analysis result; the situation of data islands formed due to the fact that data are dispersed in different systems and platforms in the enterprise operation process is effectively broken through, efficient circulation and sharing of mass structured and unstructured data are achieved, and the potential value of the data is fully released.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data analysis, and specifically relates to a data middle platform big data processing and analysis system. BACKGROUND

[0002] In the current digital era, enterprises accumulate massive amounts of data in the process of operation. These data are widely sourced, including not only structured data generated by enterprise internal business systems such as ERP, CRM, financial systems and the like, but also unstructured data generated by external channels such as social media, Internet of Things devices and the like. However, these data are often scattered in different systems and platforms, forming a data island, which leads to the fact that the data cannot be effectively circulated and shared, and it is difficult to play its potential value. In addition, with the deepening of the digital transformation of enterprises, the demand of business departments for data is increasingly diversified and individualized, and not only basic data reports are needed, but also deep insights and predictive analysis based on data mining and machine learning algorithms are expected to be obtained to support business innovation and strategic decision-making. However, the traditional data processing system is often single in function, lacks flexibility and openness, and is difficult to quickly respond to business changes and meet diversified needs.

[0003] Based on this, in order to solve the above problems, the application provides a data middle platform big data processing and analysis system. SUMMARY

[0004] In order to solve the problems existing in the above scheme, the application provides a data middle platform big data processing and analysis system.

[0005] The object of the application can be achieved by the following technical scheme: A data middle platform big data processing and analysis system, comprising a platform end and a user end; The user end comprises a collection module, a storage module, a data processing module and a product analysis module; The collection module is used for collecting data according to each data source and obtaining enterprise data; The storage module is used for storing the enterprise data; The data processing is used for processing the enterprise data; The product analysis module is used for real-time analysis of user products, obtaining product information of each user product, setting a product service scheme according to the product information, configuring a corresponding intelligent service model from the platform end according to the product service scheme, identifying a learning data range corresponding to the intelligent service model, collecting corresponding enterprise data according to the learning data range and the storage module, marking the enterprise data as learning optimization data, optimizing and adjusting the intelligent service model according to the learning optimization data, and performing product service analysis according to the optimized and adjusted intelligent service model and the product service scheme to obtain a product analysis result.

[0006] Further, setting product service scheme according to product information, including: The interface is preset by the platform party demand scheme library, and the demand scheme library is used for storing various service demands and product service schemes corresponding to the service demands; Identify product information, determine various service demands suitable for product information, and match corresponding product service schemes from the demand scheme library according to the service demands; The service demand and product service scheme are displayed to the user, and the user selects the applied product service scheme.

[0007] Further, according to the product information analysis, various service demands that can be adapted include: The platform party establishes an adaptation analysis model, and the expression of the adaptation analysis model is: ; In the formula: (A, s i ) is the input data, A represents the product information, s i represents the corresponding service demand, i is the subscript, i=1, 2, …, n, n is the number of corresponding service demands; A→s i The corresponding product information is adapted to the corresponding service demand, and the output data is the adaptation value SA(A, s i ), the adaptation value is 0 or 1; The product information and the corresponding service demand in the demand scheme library are analyzed by the adaptation analysis model to obtain the adaptation value of the service demand; The service demand with an adaptation value of 1 is output.

[0008] Further, the product service scheme includes: According to the enterprise data, the real-time sales of the user product are predicted, and the predicted sales of the corresponding time are obtained; the supplier of the user product is identified, and the supply time of the corresponding supplier supplying the corresponding user product at different times in the future is estimated in real time according to the enterprise data; According to the predicted sales and the supply time, the best inventory at different times in the future is calculated; Real-time identification of actual inventory of user product, the best inventory and actual inventory are displayed to the user.

[0009] Further, according to the product service scheme, the corresponding intelligent service model is configured from the platform end, including: Identify the user service demand corresponding to the product service scheme, and send the user service demand to the user service module of the platform end; receive the intelligent service model information sent by the user service module, and the user configures according to the intelligent service model information.

[0010] The platform end includes a reserve analysis module, a reserve library, and a user service module; The reserve analysis module is used to perform reserve service analysis, preset a demand scheme library, identify various product service schemes in the demand schemes, identify intelligent service needs based on the product service schemes, filter intelligent service needs to obtain target service needs, establish intelligent service models based on target service needs, set the learning data range corresponding to the intelligent service models, and store each intelligent service model and learning data range in the reserve library.

[0011] Furthermore, the demand for intelligent services is filtered, including: The intelligent service requirements are deduplicated, and it is estimated whether the corresponding intelligent service model to be built can meet the service requirements at the user's end. Intelligent service requirements that do not meet the service requirements are eliminated. Estimate the expected benefits of intelligent service demand and eliminate intelligent service demand whose expected benefits do not meet the preset value; Mark the remaining intelligent service requirements as target service requirements.

[0012] Furthermore, it is estimated whether the intelligent service model to be built can meet the service requirements corresponding to the anticipated intelligent service needs, including: Establish a service evaluation model, the expression of which is: ; In the formula: g is the input data, representing the candidate establishment method, and the output data is the service evaluation value FG(g), which is 1 or 0; g meeting the service requirements means that the intelligent service model established according to the corresponding candidate establishment method can meet the service requirements; The system analyzes the intelligent service requirements and establishes the candidate establishment methods for the intelligent service model. The service evaluation model analyzes the candidate establishment methods to obtain the service evaluation value of the corresponding candidate establishment methods. The candidate establishment methods with a service evaluation value of 0 are eliminated. The assessment indicates that intelligent service requirements without alternative establishment methods are not met.

[0013] The reserve is used to store intelligent service models and learning data ranges; The user service module is used to perform user service analysis, identify user service needs, match corresponding intelligent service models and learning data ranges from the reserve based on user service needs, generate corresponding intelligent service model information based on the intelligent service model and learning data range, and send the intelligent service model information to the product analysis module of the corresponding user terminal.

[0014] Compared with the prior art, the beneficial effects of the present invention are: The data platform big data processing and analysis system proposed in this invention can effectively break down the data silos formed by data being scattered across different systems and platforms during enterprise operations, realize the efficient circulation and sharing of massive amounts of structured and unstructured data, and fully release the potential value of data. At the same time, the system has strong flexibility and openness, and can quickly respond to the increasingly diversified and personalized data needs of enterprises in the process of digital transformation, meet the needs of different enterprises for different professional services, and help enterprises enhance their core competitiveness in the digital wave. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a block diagram illustrating the principle of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, a data platform big data processing and analysis system includes a platform side and a user side. The user terminal includes a data acquisition module, a storage module, a data processing module, and a product analysis module; The data acquisition module is used to collect data from various data sources and obtain enterprise data.

[0019] Data sources refer to various business systems (such as ERP, CRM, and financial systems) and external data sources (such as social media and IoT devices) that are distributed within an enterprise.

[0020] The storage module is used to store enterprise data.

[0021] For example, enterprise data can be allocated to appropriate repositories based on its characteristics and usage requirements. Relational data requiring frequent queries and updates can be stored in relational databases; high-concurrency non-relational data can be stored in NoSQL databases; and massive historical data can be stored in distributed file systems or data lakes. Data indexing and partitioning mechanisms can be established to improve query efficiency and storage performance. For instance, indexes can be created for frequently queried fields in relational databases, and data can be partitioned and stored in distributed file systems according to time or business dimensions.

[0022] The data processing is used to process enterprise data.

[0023] For example, data cleaning involves using data cleaning algorithms and tools to fill in missing values ​​in the data (such as using methods like mean, median, and mode to fill in numerical missing values, and using mode or specific rules to fill in categorical missing values), to detect and process outliers (such as using the 3σ principle, box plots, etc. to detect outliers, and correcting or removing them according to business rules), and to merge or delete duplicate values.

[0024] Data conversion: Based on business needs, perform data format conversion (such as converting date formats to standard formats), data type conversion (such as converting string types to numeric types), encoding conversion (such as converting character encoding to UTF-8), and other operations.

[0025] Data integration: This involves combining data from different sources through methods such as association and merging to establish relationships between the data. For example, linking basic customer information data with customer transaction data to create a complete customer profile.

[0026] In one embodiment, the acquisition module, storage module, and data processing module can all be implemented using existing technologies.

[0027] The product analysis module is used to perform real-time analysis of user products, which refer to the products sold by the user. It acquires product information for each user product, sets product service plans based on the product information, configures corresponding intelligent service models from the platform based on the product service plans, identifies the learning data range corresponding to the intelligent service models, collects relevant enterprise data from the storage module based on the learning data range, marks it as learning optimization data, optimizes and adjusts the corresponding intelligent service models based on the learning optimization data, and performs product service analysis based on the optimized and adjusted intelligent service models and product service plans to obtain the corresponding product analysis results.

[0028] In one embodiment, a product service plan is set up based on product information. The product service plan is set up by the user according to their own needs, such as sales forecasting, supply chain management, and other service needs. An implementation plan is set up according to the service needs.

[0029] In one embodiment, setting a product service plan based on product information includes: The platform provides a pre-set demand solution library, which stores various service requirements of various products in various industries, as well as various product and service solutions corresponding to those service requirements. Identify product information, analyze the various service needs that can be adapted based on the product information, and match the corresponding product service solutions from the demand solution library based on the service needs; The service requirements and product service solutions are presented to users, who then choose the appropriate product service solution.

[0030] In one embodiment, various service requirements that can be adapted based on product information analysis can be matched in a requirement solution library using existing methods.

[0031] In one embodiment, various service requirements that can be adapted based on product information analysis include: The platform establishes an adaptation analysis model, which is used to evaluate whether each service requirement in the requirement solution library is suitable for the user's product. The platform trains the model using various service requirements stored in the requirement solution library and an existing product-labeled training set. The expression for the adaptation analysis model is: ; In the formula: (A, s i ) represents the input data, where A represents product information, and s i This represents the corresponding service demand, where i is the subscript, i = 1, 2, ..., n, and n is the quantity of the corresponding service demand; A → s i This indicates that the product information is adapted to the corresponding service requirements, and the output data is the adaptation value SA(A, s). i The adaptation value is 0 or 1. The adaptation analysis model is used to analyze the corresponding service requirements in the product information and demand solution library to obtain the adaptation value of the corresponding service requirements. Output the service requirements with an adaptation value of 1.

[0032] In one embodiment, n represents the total number of service requirements in the requirement solution library.

[0033] In one embodiment, service requirements in the requirement solution library are filtered according to industry, field, etc., where n represents the number of service requirements filtered in the requirement solution library; this is used to avoid unnecessary analysis and improve analysis efficiency.

[0034] In one embodiment, the product service plan includes: Based on enterprise data, real-time sales forecasts for user products are performed to obtain the predicted sales volume for the corresponding time period. Suppliers for user products are identified; suppliers can include the user's franchisees, centralized warehouses, etc., and any entity supplying the products is considered a supplier. The supply duration of the corresponding user products from each supplier is estimated in real-time based on enterprise data, i.e., the time required from order placement to delivery to the user. For example, estimates are made based on order time, weather, order quantity, etc., combined with accumulated historical supply data. In other words, enterprise data includes the data needed to estimate supply duration. The optimal inventory level for each subsequent time period is calculated based on the predicted sales volume and supply duration. That is, the optimal inventory level is first determined based on the predicted sales volume, the user's preset inventory sales days, and the supply duration. This optimal level can be further expanded by a certain margin. Once the predicted sales volume and supply duration are clearly defined, the optimal inventory level can be determined using existing technology and the user's inventory requirements, avoiding insufficient inventory and product backlog, and improving the user's capital utilization rate.

[0035] The system identifies the actual inventory levels of user products in real time and displays both the optimal and actual inventory levels to the user, allowing the user to adjust their product inventory based on the optimal level.

[0036] In the above embodiments, the sales forecasting model and supply duration forecasting model can be determined according to the product service plan, and both can be quickly analyzed using existing methods; an optimal inventory level analysis model can also be included; the specific model is determined based on user needs and the actual situation of the platform, i.e., whether the platform has the corresponding intelligent model.

[0037] In one embodiment, configuring a corresponding intelligent service model from the platform based on the product service plan includes: Identify the user service requirements corresponding to the product service plan, send the user service requirements to the user service module on the platform, and receive the intelligent service model information sent by the user service module, so that the user can configure according to the intelligent service model information.

[0038] In one embodiment, identifying user service needs in a product service plan can be done by having the user determine the service needs that require the platform to provide an intelligent service model, and marking them as user service needs; alternatively, the platform can determine various service needs that can be implemented based on the product service plan, and the user can select applications from these service needs to determine the user service needs.

[0039] In one embodiment, the intelligent service model can also be configured according to the existing product service plan.

[0040] The platform includes a reserve analysis module, a reserve warehouse, and a user service module. The reserve analysis module is used to perform reserve service analysis. It has a pre-set demand solution library, which stores various service demands and their corresponding product service solutions, managed and updated by the platform. The module identifies various product service solutions within the demand solutions, and identifies corresponding intelligent service demands based on these solutions, such as sales forecasting, supply duration forecasting, employee rating, and customer behavior analysis. It filters these intelligent service demands to obtain target service demands, establishes corresponding intelligent service models based on these demands, and sets the learning data range for each model. This learning data range indicates which data will be used to subsequently learn and adjust the intelligent service model. Finally, it stores each intelligent service model and its learning data range in the reserve library.

[0041] In one embodiment, intelligent service requirements are screened to eliminate those that the platform does not need to provide services for, such as intelligent service requirements with low efficiency.

[0042] In one embodiment, the screening of intelligent service requirements includes: The intelligent service requirements are deduplicated, and it is estimated whether the intelligent service model to be built for each intelligent service requirement can meet the service requirements at the user's site. That is, the intelligent service model after being built should be able to learn from the relevant enterprise data accumulated at the user's site, and the adjusted intelligent service model should be able to meet the user's analysis needs and sales forecasting needs. If it can meet the user's needs, it is considered to meet the service requirements. Intelligent service requirements that do not meet the service requirements are eliminated. Estimate the expected revenue from intelligent service demand based on factors such as market share and cost; eliminate intelligent service demand whose expected revenue does not meet the preset value. Mark the remaining intelligent service requirements as target service requirements.

[0043] In one embodiment, the expected revenue from intelligent service demand is estimated, the implementation cost is estimated based on the remaining alternative establishment methods for intelligent service demand, and sales volume is estimated, thereby calculating the expected revenue.

[0044] In one embodiment, the system can also directly filter based on the estimated implementation cost and propose intelligent service requests that exceed the estimated implementation cost.

[0045] In one embodiment, whether the intelligent service model to be built corresponding to the predicted intelligent service demand can meet the service requirements can be predicted and judged based on existing methods, such as predicting according to the existing methods of building intelligent service models to realize the intelligent service demand, and determining whether the intelligent service model corresponding to the building method can meet the service requirements. As long as one of them can meet the requirements, the intelligent service demand is considered to meet the service requirements. Based on this, the platform can build a corresponding service prediction model based on machine learning, deep learning algorithms, etc. for prediction.

[0046] In one embodiment, assessing whether the intelligent service model to be built can meet the service requirements corresponding to the anticipated intelligent service needs includes: Establish a service evaluation model, the expression of which is: ; In the formula: g is the input data, representing the candidate establishment method, and the output data is the service evaluation value FG(g), which is 1 or 0; g meeting the service requirements means that the intelligent service model established according to the corresponding candidate establishment method can meet the service requirements; the corresponding training set is labeled using the corresponding historical data for training; The system analyzes the intelligent service requirements and establishes the candidate establishment methods for the intelligent service model. The service evaluation model analyzes the candidate establishment methods to obtain the service evaluation value of the corresponding candidate establishment methods. The candidate establishment methods with a service evaluation value of 0 are eliminated. The assessment indicates that intelligent service requirements without alternative establishment methods are not met.

[0047] In one embodiment, a corresponding intelligent service model is established based on the target service requirements, utilizing existing methods.

[0048] For example, choose a model type such as LSTM (for processing time series), Prophet (Facebook's open-source time series model), or XGBoost based on user features; Use general data aggregated by the platform (such as historical industry sales and seasonal factors) to train the initial model; for example, use national retail data to train an LSTM model, with the input being sales, promotional activities, and holiday markings for the past 30 days, and the output being the prediction for the next 7 days. Compress model size (e.g., quantization, pruning) to fit user devices (e.g., mobile phones, IoT devices); encapsulate model inference and training interfaces to support local data loading and incremental learning.

[0049] The reserve is used to store intelligent service models and learning data ranges.

[0050] The user service module is used to perform user service analysis, identify user service needs, match corresponding intelligent service models and learning data ranges from the reserve based on user service needs, generate corresponding intelligent service model information based on the intelligent service model and learning data range, such as configuration links, configuration requirements, performance parameters, learning data ranges and other relevant user-concerned information; and send the intelligent service model information to the product analysis module of the corresponding user terminal.

[0051] The above formulas are all numerical calculations after removing dimensions. The formulas are obtained by software simulation based on a large amount of data and are closest to the real situation. The preset parameters and preset thresholds in the formulas are set by those skilled in the art according to the actual situation or obtained by simulation based on a large amount of data.

[0052] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A data middle platform big data processing and analysis system, characterized in that, The platform end and the user end are included; The user end includes a collection module, a storage module, a data processing module, and a product analysis module; the platform end includes a storage analysis module, a storage library, and a user service module; The collection module is used for collecting data from various data sources to obtain enterprise data; The storage module is used for storing enterprise data; The data processing module is used for processing enterprise data; The product analysis module is used for real-time analysis of user products, obtaining product information of each user product, setting a product service scheme according to the product information, configuring a corresponding intelligent service model from the platform end according to the product service scheme, identifying the learning data range corresponding to the intelligent service model, collecting corresponding enterprise data according to the learning data range and the storage module, marking the enterprise data as learning optimization data, and optimizing and adjusting the intelligent service model according to the learning optimization data; Product service analysis is performed according to the optimized and adjusted intelligent service model and the product service scheme, and product analysis results are obtained; The storage analysis module is used for storage service analysis, and a demand scheme library is preset; various product service schemes in the demand scheme are identified, and intelligent service demand is identified according to the product service scheme; the intelligent service demand is filtered to obtain target service demand, and an intelligent service model is established according to the target service demand; the learning data range corresponding to the intelligent service model is set, and each intelligent service model and learning data range is stored in the storage library; The storage library is used for storing intelligent service models and learning data ranges; The user service module is used for user service analysis, identifying user service demand, matching corresponding intelligent service models and learning data ranges from the storage library according to the user service demand, generating corresponding intelligent service model information according to the intelligent service model and the learning data range, and sending the intelligent service model information to the product analysis module of the corresponding user end.

2. The data center big data processing and analysis system of claim 1, wherein, Setting a product service scheme according to product information includes: Connecting a demand scheme library preset by a platform party, the demand scheme library is used for storing various service demands and product service schemes corresponding to the service demands; Identifying product information, determining various service demands suitable for the product information, and matching corresponding product service schemes from the demand scheme library according to the service demand; The service demand and the product service scheme are displayed to the user, and the user selects the product service scheme to be applied.

3. The data hub big data processing and analysis system of claim 2, wherein, Determining various service demands suitable for product information includes: The platform party establishes an adaptation analysis model, and the expression of the adaptation analysis model is: ； In the formula: (A, s i ) is input data, A represents product information, s i represents corresponding service demand, i is an index, i=1, 2, …, n, and n is the number of corresponding service demands; A→s i represents that corresponding product information is adapted to the corresponding service demand, and output data is an adaptation value SA(A, s i ), and the adaptation value is 0 or 1. Through the adaptation analysis model, the product information and the corresponding service demand in the demand scheme library are analyzed to obtain the adaptation value of the service demand; The service demand with an adaptation value of 1 is output.

4. The data center big data processing and analysis system of claim 1 or 2, wherein, Product service scheme includes: Real-time sales prediction of user products according to enterprise data to obtain predicted sales at corresponding time; identifying the suppliers of user products, and real-time estimation of the supply duration of corresponding suppliers supplying corresponding user products at different times in the future according to enterprise data; According to the predicted sales and the supply duration, the optimal inventory at different times in the future is calculated; Real-time recognition of actual inventory of user products, showing the best inventory and actual inventory to the user.

5. The data hub big data processing and analyzing system of claim 1, wherein, According to the product service scheme, configure the corresponding intelligent service model from the platform side, including: Identify the user service demand corresponding to the product service scheme, and send the user service demand to the user service module of the platform side; receive the intelligent service model information sent by the user service module, and the user configures according to the intelligent service model information.

6. The data hub big data processing and analyzing system of claim 1, wherein, Screening intelligent service demand, including: De-duplicate the intelligent service demand, estimate whether the intelligent service demand corresponding to the to-be-built intelligent service model can meet the service requirements at the user, and eliminate the intelligent service demand that does not meet the service requirements; Estimate the expected income of the intelligent service demand, and eliminate the intelligent service demand whose expected income does not meet the preset value; Mark the remaining intelligent service demand as a target service demand.

7. The data hub big data processing and analysis system of claim 6, wherein, Estimate whether the intelligent service demand corresponding to the to-be-built intelligent service model can meet the service requirements, including: Establish a service evaluation model to analyze the to-be-selected establishment method of the intelligent service demand, analyze the to-be-selected establishment method by the service evaluation model, obtain the service evaluation value of the corresponding to-be-selected establishment method, and the service evaluation value is 1 or 0; eliminate the to-be-selected establishment method with a service evaluation value of 0; Evaluate the intelligent service demand without to-be-selected establishment method that does not meet the service requirements.

8. The data hub big data processing and analysis system of claim 7, wherein, The expression of the service evaluation model is: ； In the formula: g is the input data, indicating the to-be-selected establishment method, and the output data is the service evaluation value FG(g). If g meets the service requirements, it means that the intelligent service model established according to the corresponding to-be-selected establishment method can meet the service requirements.

Images