Management service platform system based on big data

Abstract

The invention discloses a management service platform system based on big data. The management service platform system comprises a dynamic data lake construction module, a semantic map generation module, a self-adaptive label engine, a reinforcement learning push module and a closed-loop optimization center. The dynamic data lake module realizes hierarchical storage of multi-source heterogeneous data by calculating a data value coefficient; the semantic graph module constructs a multi-dimensional semantic network based on a knowledge graph and a BERT model, and excavates implicit topic association; the self-adaptive label engine dynamically generates and updates a user label by using a graph neural network and a time decay factor; the reinforcement learning pushing module is combined with a multi-arm machine model and an A / B test optimization pushing strategy; and the closed-loop optimization center realizes global model updating and system dynamic expansion through federated learning and a micro-service architecture. According to the method, the problems of data layering and stiffness, label generation staticization, strategy optimization lag and the like of a traditional central station system are solved, and a closed-loop system of data intelligent management, accurate service pushing and system self-adaptive optimization is realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and big data technology, and in particular relates to a management service middle platform system based on big data. Background Art

[0002] Currently, enterprises face the challenge of integrating heterogeneous data from multiple sources (such as IoT device data, social media data, and business system logs). Existing mid-tier systems often rely on static rules or single algorithms for data processing, resulting in low data classification accuracy and lagging service strategies. For example, traditional systems struggle to dynamically capture changes in user behavior patterns and lack the ability to optimize closed-loop feedback in real time, resulting in a gradual decline in the match between pushed content and user needs.

[0003] The existing technology has the following defects:

[0004] 1. Data stratification is based solely on fixed rules and cannot adapt to dynamic changes in business scenarios;

[0005] 2. Label generation relies on manually preset dimensions and lacks the ability to mine implicit associations;

[0006] 3. Service strategy optimization lacks an adaptive mechanism driven by real-time feedback.

[0007] Therefore, a management service middle platform system based on big data is proposed. Summary of the Invention

[0008] The purpose of the present invention is to provide a management service middle-office system based on big data, which realizes intelligent data stratification, precise label generation and adaptive adjustment of service strategies through dynamic weight allocation, implicit association mining and reinforcement learning optimization, forming a "data-decision-feedback" closed loop, and improving service response speed and user satisfaction.

[0009] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0010] The present invention is a management service mid-stage system based on big data, which includes the following modules:

[0011] Dynamic data lake construction module, semantic graph generation module, adaptive labeling engine, reinforcement learning push module and closed-loop optimization center; the dynamic data lake construction module is used to access multi-source heterogeneous data in real time and calculate the data value coefficient (DVC) to achieve data tiered storage;

[0012] The semantic graph generation module builds a multi-dimensional semantic network based on knowledge graph technology and BERT model;

[0013] The adaptive tag engine generates and updates user tags through a graph neural network (GNN) and a time decay factor;

[0014] The reinforcement learning push module uses a multi-armed bandit model and A / B testing framework to optimize push strategies;

[0015] The closed-loop optimization hub realizes global model updates and dynamic system expansion through federated learning and microservice architecture.

[0016] In one embodiment, in the dynamic data lake construction module, the calculation formula for the data value coefficient is:

[0017]

[0018] Among them, S i is the data freshness, F i is the cross-business call frequency, c i is the context association strength, λ is the dynamic adjustment weight;

[0019] Data is divided into the original layer, feature layer, and application layer according to DVC, and allocated to the tiered storage engine.

[0020] In one embodiment, the semantic graph generation module further includes:

[0021] Entity relationship extraction unit, used to identify entities and their relationships from multi-source data;

[0022] The semantic enhancement unit uses the BERT model to perform semantic analysis on the context and mine implicit topic associations;

[0023] Dynamic update unit, which expands the semantic network in real time based on the new data in the data lake.

[0024] In one embodiment, the adaptive tag engine includes:

[0025] Basic tag generation unit, which generates user basic tags based on semantic graph;

[0026] High-order association mining unit uses graph neural network (GNN) to analyze complex relationships between tags and generate composite tags;

[0027] The weight update unit introduces a time decay factor to dynamically adjust the label weight. The formula is:

[0028] w(t)=w0×e -αt

[0029] Among them, w0 is the initial weight, e is the decay coefficient, and t is the time interval.

[0030] In one embodiment, the reinforcement learning push module further includes:

[0031] Strategy decision unit, which selects push strategies based on the multi-armed bandit model;

[0032] Feedback analysis unit, which calculates strategy benefits based on user click-through rate and stay time;

[0033] The strategy optimization unit compares the effectiveness of strategies through A / B testing and prioritizes the deployment of strategies with the highest returns.

[0034] In one embodiment, the closed-loop optimization hub includes:

[0035] Federated learning unit, which aggregates multi-node feedback data and updates global model parameters;

[0036] The service orchestration unit achieves decoupling between modules through the microservice architecture and supports dynamic expansion;

[0037] Grayscale release unit, which conducts small-scale verification of new strategies before gradually deploying them in full.

[0038] In one embodiment, the dynamic data lake construction module further includes:

[0039] The data cleaning unit uses streaming processing technology to perform real-time denoising and formatting of multi-source heterogeneous data;

[0040] The storage optimization unit allocates storage resources based on data tiering results and enables cache acceleration for frequently accessed data.

[0041] In one embodiment, in the semantic graph generation module, the method for mining implicit topic associations includes:

[0042] Extract contextual semantic features through the BERT model;

[0043] Identify potential relationships between entities based on the attention mechanism;

[0044] The association results are stored in the graph database in the form of triples to form a scalable semantic network.

[0045] The present invention has the following beneficial effects:

[0046] The present invention uses a dynamic weighting algorithm (DVC) to evaluate data value in real time, realizes hierarchical storage and resource allocation, improves the access efficiency of high-frequency data, and reduces storage costs.

[0047] Based on the BERT model and knowledge graph technology, this invention constructs a multidimensional semantic network, accurately identifies explicit and implicit associations between entities, and enhances data comprehension capabilities; uses GNN to analyze complex relationships between labels, and dynamically adjusts label weights in combination with time decay factors to ensure the timeliness and accuracy of user portraits.

[0048] This invention adopts a reinforcement learning framework (multi-armed bandit model) to respond to user feedback in real time, and quickly iterates push strategies through A / B testing, significantly improving user click-through rate and satisfaction.

[0049] This invention is based on the global model update and microservice architecture of federated learning, supports dynamic system expansion and grayscale release, and ensures high service availability and business continuity; through closed-loop optimization of data, decision-making and feedback, it realizes full-link automation from data collection to service deployment, reduces the cost of manual intervention, and improves overall operational efficiency.

[0050] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 This is a system block diagram of a management service middle-office system based on big data. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be understood that the terms "upper", "middle", "outer", "inner" and the like indicating directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore should not be understood as limiting the present invention.

[0055] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "provided with," "connected," etc., should be understood in a broad sense. For example, "connected" may refer to a fixed connection, a detachable connection, or an integral connection; it may refer to a mechanical connection or an electrical connection; it may refer to a direct connection or an indirect connection through an intermediate medium; it may refer to internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0056] See also Figure 1 As shown, the present invention is a management service mid-stage system based on big data, including the following modules:

[0057] Dynamic data lake construction module, semantic graph generation module, adaptive labeling engine, reinforcement learning push module and closed-loop optimization center; the dynamic data lake construction module is used to access multi-source heterogeneous data in real time and calculate the data value coefficient (DVC) to achieve data tiered storage;

[0058] The semantic graph generation module builds a multi-dimensional semantic network based on knowledge graph technology and the BERT model;

[0059] The adaptive tagging engine generates and updates user tags through graph neural networks (GNN) and time decay factors;

[0060] The reinforcement learning push module uses a multi-armed bandit model and A / B testing framework to optimize push strategies;

[0061] The closed-loop optimization hub achieves global model updates and dynamic system expansion through federated learning and microservice architecture.

[0062] In the dynamic data lake construction module, the calculation formula for the data value coefficient is:

[0063]

[0064] Among them, S i is the data freshness, F i is the cross-business call frequency, c i is the context association strength, λ is the dynamic adjustment weight;

[0065] Data is divided into the original layer, feature layer, and application layer according to DVC, and allocated to the tiered storage engine.

[0066] Among them, the semantic graph generation module further includes:

[0067] Entity relationship extraction unit, used to identify entities and their relationships from multi-source data;

[0068] The semantic enhancement unit uses the BERT model to perform semantic analysis on the context and mine implicit topic associations;

[0069] Dynamic update unit, which expands the semantic network in real time based on the new data in the data lake.

[0070] The adaptive tag engine includes:

[0071] Basic tag generation unit, which generates user basic tags based on semantic graph;

[0072] High-order association mining unit uses graph neural network (GNN) to analyze complex relationships between tags and generate composite tags;

[0073] The weight update unit introduces a time decay factor to dynamically adjust the label weight. The formula is:

[0074] w(t)=w0×e -αt

[0075] Among them, w0 is the initial weight, e is the decay coefficient, and t is the time interval.

[0076] Among them, the reinforcement learning push module further includes:

[0077] Strategy decision unit, which selects push strategies based on the multi-armed bandit model;

[0078] Feedback analysis unit, which calculates strategy benefits based on user click-through rate and stay time;

[0079] The strategy optimization unit compares the effectiveness of strategies through A / B testing and prioritizes the deployment of strategies with the highest returns.

[0080] The closed-loop optimization hub includes:

[0081] Federated learning unit, which aggregates multi-node feedback data and updates global model parameters;

[0082] The service orchestration unit achieves decoupling between modules through the microservice architecture and supports dynamic expansion;

[0083] Grayscale release unit, which conducts small-scale verification of new strategies before gradually deploying them in full.

[0084] The dynamic data lake building module also includes:

[0085] The data cleaning unit uses streaming processing technology to perform real-time denoising and formatting of multi-source heterogeneous data;

[0086] The storage optimization unit allocates storage resources based on data tiering results and enables cache acceleration for frequently accessed data.

[0087] Among them, in the semantic graph generation module, the mining methods of implicit topic associations include:

[0088] Extract contextual semantic features through the BERT model;

[0089] Identify potential relationships between entities based on the attention mechanism;

[0090] The association results are stored in the graph database in the form of triples to form a scalable semantic network.

[0091] This embodiment is a management service mid-stage system based on big data, which includes the following modules:

[0092] Dynamic Data Lake Building Blocks:

[0093] Supports real-time access and cleaning of multi-source heterogeneous data (structured, semi-structured, and unstructured);

[0094] The data value coefficient (DVC) is calculated based on the dynamic weight algorithm. The formula is:

[0095]

[0096] Among them, S i is the data freshness, F i is the cross-business call frequency, c i is the context association strength, λ is the dynamic adjustment weight;

[0097] Data is divided into the original layer, feature layer, and application layer according to DVC, supporting hierarchical storage and access optimization.

[0098] Semantic graph generation module:

[0099] Use knowledge graph technology to extract data entities and relationships and build a multi-dimensional semantic network;

[0100] Combined with the BERT model for contextual semantic enhancement, it can identify implicit topic associations (e.g., "user A purchases product B" implies "preferred brand C").

[0101] Adaptive Tag Engine:

[0102] Generate basic tags (such as "high net worth users") based on semantic graphs;

[0103] Graph Neural Networks (GNNs) are used to mine high-order associations between tags and generate composite tags (e.g., “nighttime active – maternal and infant preference – short video user”).

[0104] A time decay factor is introduced to dynamically update label weights.

[0105] Reinforcement learning push module:

[0106] Build a multi-armed bandit model and dynamically adjust push strategies based on real-time user feedback (click-through rate, dwell time);

[0107] Integrate the A / B testing framework to compare the effectiveness of different strategies and prioritize the strategy with the highest return.

[0108] Closed-loop optimization hub:

[0109] Aggregate multi-node feedback data based on federated learning technology and update global model parameters;

[0110] Decoupling between modules is achieved through microservice architecture, supporting dynamic expansion and phased release.

[0111] Throughout this specification, references to terms such as "one embodiment," "example," and "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0112] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A management service middle platform system based on big data, characterized by: Includes the following modules: Dynamic data lake construction module, semantic graph generation module, adaptive labeling engine, reinforcement learning push module and closed-loop optimization center; the dynamic data lake construction module is used to access multi-source heterogeneous data in real time and calculate the data value coefficient (DVC) to achieve data tiered storage; The semantic graph generation module builds a multi-dimensional semantic network based on knowledge graph technology and BERT model; The adaptive tag engine generates and updates user tags through a graph neural network (GNN) and a time decay factor; The reinforcement learning push module uses a multi-armed bandit model and A / B testing framework to optimize push strategies; The closed-loop optimization hub realizes global model updates and dynamic system expansion through federated learning and microservice architecture.

2. A management service mid-stage system based on big data according to claim 1, characterized in that: In the dynamic data lake construction module, the calculation formula for the data value coefficient is: Among them, S i is the data freshness, F i is the cross-business call frequency, c i is the context association strength, λ is the dynamic adjustment weight; Data is divided into the original layer, feature layer, and application layer according to DVC, and allocated to the tiered storage engine.

3. A management service mid-stage system based on big data according to claim 1, characterized in that: The semantic graph generation module further includes: Entity relationship extraction unit, used to identify entities and their relationships from multi-source data; The semantic enhancement unit uses the BERT model to perform semantic analysis on the context and mine implicit topic associations; Dynamic update unit, which expands the semantic network in real time based on the new data in the data lake.

4. A management service mid-stage system based on big data according to claim 1, characterized in that: The adaptive tag engine includes: Basic tag generation unit, which generates user basic tags based on semantic graph; High-order association mining unit uses graph neural network (GNN) to analyze complex relationships between tags and generate composite tags; The weight update unit introduces a time decay factor to dynamically adjust the label weight. The formula is: w(t)=w0×e -αt Among them, w0 is the initial weight, e is the decay coefficient, and t is the time interval.

5. A management service mid-stage system based on big data according to claim 1, characterized in that: The reinforcement learning push module further includes: Strategy decision unit, which selects push strategies based on the multi-armed bandit model; Feedback analysis unit, which calculates strategy benefits based on user click-through rate and stay time; The strategy optimization unit compares the effectiveness of strategies through A / B testing and prioritizes the deployment of strategies with the highest returns.

6. A management service mid-stage system based on big data according to claim 1, characterized in that: The closed-loop optimization hub includes: Federated learning unit, which aggregates multi-node feedback data and updates global model parameters; The service orchestration unit achieves decoupling between modules through the microservice architecture and supports dynamic expansion; Grayscale release unit, which conducts small-scale verification of new strategies before gradually deploying them in full.

7. The management service mid-stage system based on big data according to claim 1, characterized in that: The dynamic data lake construction module also includes: The data cleaning unit uses streaming processing technology to perform real-time denoising and formatting of multi-source heterogeneous data; The storage optimization unit allocates storage resources based on data tiering results and enables cache acceleration for frequently accessed data.

8. A management service mid-stage system based on big data according to claim 1, characterized in that: In the semantic graph generation module, the implicit topic association mining method includes: Extract contextual semantic features through the BERT model; Identify potential relationships between entities based on the attention mechanism; The association results are stored in the graph database in the form of triples to form a scalable semantic network.

Images