Distributed mining and integrating method and framework for intermediate variables

Abstract

The method comprises the following steps: aiming at intermediate variables constructed in the field of simulation deduction, dividing the intermediate variables into three categories according to professional knowledge and different types of used simulation data, namely window type intermediate variables which use entity change data and instruction data; an event link class intermediate variable using the event data; and the task dimension class intermediate variables use entities to count data and construct a unified framework to carry out distributed mining, and the framework supports, expands and integrates newly added intermediate variables. According to the method, distributed mining and integration of large-scale intermediate variables can be effectively supported, and powerful technical support is provided for causal relationship analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of simulation deduction, in particular to a distributed mining and integration method and framework of intermediate variables. BACKGROUND

[0002] Intermediate variables, also known as mediating variables, are variables that bridge the gap between independent variables and dependent variables, and are used to explain how independent variables affect dependent variables. It reveals the internal mechanism of the relationship between independent variables and dependent variables, helping researchers to better understand complex phenomena. Intermediate variables help to reveal the relationship between variables in depth and need to be selected based on theory and professional knowledge. However, due to the diversity of intermediate variables, the dispersion of data sources, and the complexity of calculation, a systematic framework is needed to support the mining and integration of intermediate variables. SUMMARY

[0003] Therefore, it is necessary to provide a distributed mining and integration method and framework of intermediate variables in view of the above technical problems.

[0004] A distributed mining and integration method of intermediate variables is applied to a simulation deduction scene, and the method comprises the following steps: In response to a target scene analysis request initiated by an external system, preset scene configuration information corresponding to a target scene identifier carried in the request is obtained; Based on the scene configuration information, the type of intermediate variable required to be calculated is parsed; wherein the type of intermediate variable includes window type intermediate variable, event link type intermediate variable and task dimension type intermediate variable; According to the type of each intermediate variable and the access address of the corresponding original simulation data, a corresponding mining task is generated, and each mining task is distributed to a corresponding distributed mining service; wherein the original simulation data is simulation running data related to the target scene; the mining task of the window type intermediate variable is distributed to the window mining service, the mining task of the event link type intermediate variable is distributed to the event link mining service, and the mining task of the task dimension type intermediate variable is distributed to the task dimension mining service; Each mining service extracts simulation running data of the corresponding type from the access address of the original simulation data according to the received mining task, performs intermediate variable calculation in parallel, and outputs window type intermediate variable, event link type intermediate variable and task dimension type intermediate variable respectively; The intermediate variables output by each mining service are integrated, and the integrated intermediate variable set is returned to the external system.

[0005] In one of the embodiments, in response to a target scene analysis request initiated by an external system, the preset scene configuration information corresponding to the target scene identifier carried in the request is obtained, specifically comprising: The SceneOrchestrator class receives and processes target scene analysis requests initiated by external systems to obtain target scene identifiers. The SceneOrchestrator class calls the SceneConfigurationManager class to query the local or remote configuration storage based on the target scene identifier, and obtains scene configuration information containing the required list of intermediate variables and the mapping relationship of variable types. The step of parsing the required intermediate variable types based on the scenario configuration information specifically includes: The SceneOrchestrator class calls the SceneConfigurationManager class to parse the types of intermediate variables that need to be calculated based on the scene configuration information. The process of distributing each mining task to the corresponding distributed mining service specifically includes: The SceneOrchestrator class calls the TaskDispatcher class, which triggers task dispatch strategies for window class, event chain class, and task dimension class respectively, based on the parsed intermediate variable type. The TaskDispatcher class calls the ServiceRegistry class to query the currently available endpoints of the corresponding mining service and sends mining tasks only to mining services that are in a healthy and available state.

[0006] In one embodiment, the method further includes: When a target scene analysis request that is not registered or contains an unregistered intermediate variable type is received, the ExtensionManager class is called through the SceneOrchestrator class to verify the validity of the extension configuration. If the verification passes, register the new scenario or new intermediate variable type through the ExtensionManager class and associate it with its corresponding mining service identifier; In the subsequent task distribution process, the TaskDispatcher class distributes the new type of mining task to the corresponding mining service based on the registration information.

[0007] In one embodiment, calculating intermediate variables of the window class specifically includes: The WindowMiningService class receives the mining tasks for intermediate variables of the window class. The WindowMiningService class selects the corresponding processor from multiple window processors based on the intermediate variable type in the intermediate variable mining task of the window class, and calls the WindowDataClient class to extract entity change data and instruction data from the access address of the corresponding original simulation data. The WindowMiningService class calls the unified mining interface mine from the WindowHandler abstract class to enable the selected window processor to obtain the entity change data and instruction data and perform windowed calculations to generate intermediate variables of the window class.

[0008] In one embodiment, calculating intermediate variables of the event chain class specifically includes: The KeyEventMiningService class receives mining tasks from intermediate variables of the event chain class. The KeyEventMiningService class selects the corresponding detector from multiple event detectors based on the intermediate variable type in the mining task of the intermediate variable of the event chain class, and calls the EventDataClient class to extract event data from the access address of the corresponding original simulation data. The KeyEventMiningService class calls the unified detection interface detect of the selected event detector from the EventDetector abstract class, so that the selected event detector can obtain the event data and perform event detection, generating intermediate variables of the event chain class.

[0009] In one embodiment, calculating intermediate variables for the task dimension class specifically includes: The IntermediateMetricMiningService class receives mining tasks from intermediate variables of the task dimension class. The IntermediateMetricMiningService class selects the corresponding calculator from multiple indicator calculators based on the type of intermediate variable in the task dimension class's intermediate variable mining task, and calls the MetricDataClient class to extract entity statistics data from the access address of the corresponding original simulation data. The IntermediateMetricMiningService class calls the unified detection interface calculate of the selected indicator calculator from the MetricCalculator abstract class, so that the selected indicator calculator can obtain the entity statistical data and perform indicator calculation to generate intermediate variables of the task dimension class.

[0010] In one embodiment, the process of integrating the intermediate variables output by each mining service and returning the integrated set of intermediate variables to the external system specifically includes: Call the ResultReceiver class through the ResultIntegrationModule class to register callback functions to receive intermediate variables of the window class, the event chain class, and the task dimension class; After all three types of intermediate variables have been calculated, the ResultIntegrator class is called through the ResultIntegrationModule class to integrate all the intermediate variables; The UnifiedSender class is called through the ResultIntegrationModule class to return the integrated set of intermediate variables to the external system.

[0011] In one embodiment, the target scene analysis request is represented by a SceneRequest data model; The mining task is represented by the MiningTask data model; The intermediate variables of the window class are represented by the WindowResult data model; The intermediate variables of the event chain class are represented by the EventResult data model; The intermediate variables of the task dimension class are represented by the MetricResult data model; The integrated set of intermediate variables is represented by the IntegratedResult data model.

[0012] In one embodiment, the method further performs dynamic expansion through a predefined expansion point mechanism; wherein, The extension point mechanism includes the abstract class ExtensionPoint; The abstract class ExtensionPoint includes the SceneExtensionPoint class, VariableExtensionPoint class, and AlgorithmExtensionPoint class; The ExtensionPoint abstract class defines the basic operations of extension points, including verifying configurations, registering extensions, and listing extensions. The SceneExtensionPoint class manages the extension of scene adapters, allowing for the dynamic registration of new scene adapters; The VariableExtensionPoint class manages the extension of variable handlers, allowing for the dynamic registration of new intermediate variable type handlers; The AlgorithmExtensionPoint class manages algorithm extensions, allowing for the dynamic registration of new mining algorithms.

[0013] A distributed mining and integration framework for intermediate variables, applied to simulation scenarios, the framework includes: The feature engineering construction and coordination control module is used to respond to the target scene analysis request initiated by the external system and obtain the preset scene configuration information corresponding to the target scene identifier carried in the request. The feature engineering construction and coordination control module is also used to parse out the types of intermediate variables to be calculated based on the scenario configuration information; wherein, the types of intermediate variables include window-type intermediate variables, event link-type intermediate variables, and task dimension-type intermediate variables; The feature engineering construction and coordination control module is also used to generate corresponding mining tasks based on the type of each intermediate variable and the access address of its corresponding original simulation data, and to distribute each mining task to the corresponding distributed mining module; wherein, the original simulation data is simulation running data related to the target scene; the mining tasks of window-type intermediate variables are distributed to the window mining module, the mining tasks of event link-type intermediate variables are distributed to the event link mining module, and the mining tasks of task dimension-type intermediate variables are distributed to the task dimension mining module. The window mining module, the event link mining module, and the task dimension mining module are respectively used to extract simulation running data of the corresponding type from the access address of the original simulation data according to the received mining task, perform intermediate variable calculation in parallel, and output window class intermediate variables, event link class intermediate variables, and task dimension class intermediate variables. The results integration module is used to integrate the intermediate variables output by each mining service and return the integrated set of intermediate variables to the external system. This application provides a distributed mining and integration method and framework for intermediate variables, which can effectively support the distributed mining and integration of large-scale intermediate variables and provide strong technical support for causal relationship analysis. Attached Figure Description

[0014] Figure 1 This is a structural diagram of a distributed mining and integration framework for intermediate variables in one embodiment; Figure 2 This is a class diagram of the feature engineering construction and coordination control module in one embodiment; Figure 3 Here is a class diagram of the window mining module in one embodiment; Figure 4 Here is a class diagram of the event chain mining module in one embodiment; Figure 5 Here is a class diagram of the task dimension mining module in one embodiment; Figure 6 Here is a class diagram of the result integration module in one embodiment; Figure 7 This is a class diagram of the core data model in one embodiment; Figure 8 This is an example of an extension point class diagram. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] This application proposes a distributed mining and integration framework for intermediate variables, specifically designed for simulation scenarios. The framework employs a distributed microservice architecture, using a feature engineering construction and coordination control module as a unified scheduling hub to distribute computational tasks for three types of intermediate variables to corresponding dedicated mining services. The result integration module then fuses and outputs the results from multiple sources. The overall architecture emphasizes loose coupling, high cohesion, and scalability, supporting the dynamic addition of new scenarios, variable types, and algorithm strategies.

[0017] Specifically, such as Figure 1 As shown, the framework includes the following core modules: The feature engineering construction and coordination control module is used to respond to the target scene analysis request initiated by the external system and obtain the preset scene configuration information corresponding to the target scene identifier carried in the request. The feature engineering construction and coordination control module is also used to parse out the types of intermediate variables to be calculated based on the scenario configuration information; among them, the types of intermediate variables include window-type intermediate variables, event link-type intermediate variables, and task dimension-type intermediate variables. The feature engineering construction and coordination control module is also used to generate corresponding mining tasks based on the type of each intermediate variable and the access address of its corresponding original simulation data, and to distribute each mining task to the corresponding distributed mining module; wherein, the original simulation data is simulation running data related to the target scene; the mining tasks of window-type intermediate variables are distributed to the window mining module, the mining tasks of event link-type intermediate variables are distributed to the event link mining module, and the mining tasks of task dimension-type intermediate variables are distributed to the task dimension mining module. The window mining module, event link mining module, and task dimension mining module are used to extract simulation running data of the corresponding type from the access address of the original simulation data according to the received mining task, perform intermediate variable calculation in parallel, and output window class intermediate variables, event link class intermediate variables, and task dimension class intermediate variables. The results integration module is used to integrate the intermediate variables output by each mining service and return the integrated set of intermediate variables to an external system. Optionally, data concatenation or data fusion can be used to integrate the intermediate variables output by each mining service.

[0018] In one embodiment, the typical operation flow of the above framework is as follows: An external system initiates a scenario analysis request; The feature engineering construction and coordination control module receives requests and parses the scenario configuration; The feature engineering construction and coordination control module sends the window class intermediate variables to the window mining module; The feature engineering construction and coordination control module sends event-type intermediate variables to the event link mining module; The feature engineering construction and coordination control module sends intermediate variable indicators to the task dimension mining module; The three services perform the mining calculations in parallel; Each service sends its results to the results integration module; The results integration module integrates all results; The results integration module sends the unified results to the external system.

[0019] In one embodiment, the extended scenario process is as follows: Register a new scene configuration; The feature engineering construction and coordination control module loads new scene configurations; Register new scenario adapters for each service; New scenario requests are executed according to the normal process.

[0020] In one embodiment, the process of expanding intermediate variables is as follows: Register new intermediate variable processing logic in the corresponding service; Update the scene configuration to include new intermediate variables; The feature engineering construction and coordination control module automatically identifies and distributes new intermediate variables.

[0021] In one embodiment, such as Figure 2 As shown, the Feature Engineering Construction and Coordination Control Module (SceneOrchestrator) is used to receive external scene analysis requests, manage all scene configurations, parse the intermediate variables required by the scene, distribute the intermediate variables to the corresponding mining services, and support the dynamic expansion of new scenes and intermediate variables.

[0022] Specifically, the feature engineering construction and coordination control module is centered on the SceneOrchestrator class, which relies on four classes—SceneConfigurationManager, TaskDispatcher, ExtensionManager, and ServiceRegistry—to fulfill its responsibilities.

[0023] The SceneOrchestrator class serves as the entry point for the feature engineering construction and coordination control module. It receives external requests and coordinates internal components to complete scene parsing and task distribution. Its `process_scene_request` method handles scene requests, retrieves the scene configuration by calling `SceneConfigurationManager`, parses out the necessary intermediate variables, and then uses `TaskDispatcher` to distribute the task to the corresponding mining service. Simultaneously, it supports extending new scene and intermediate variable types through `ExtensionManager` and obtains available service endpoints through `ServiceRegistry`.

[0024] The SceneConfigurationManager class manages the configuration information for all scenes, including scene name, description, list of required intermediate variables, and access address for raw simulation data. It provides CRUD (Create, Read, Update, Delete) functionality for scene configurations, ensuring accuracy and consistency.

[0025] The TaskDispatcher class is responsible for distributing mining tasks to the corresponding mining services. It calls different dispatch methods based on the type of intermediate variables (window, event, metric) and maintains the task's state. It obtains the endpoints of each service through the ServiceRegistry to ensure tasks are sent correctly.

[0026] The ExtensionManager class manages the system's extension functionality, including the registration of new scenarios and new intermediate variable types. It verifies the validity of extension configurations and provides query functionality for extension points.

[0027] The ServiceRegistry class maintains registration information for all services being discovered, including service endpoints and health status. It provides service discovery and health checks, ensuring that tasks are only distributed to healthy services.

[0028] It should be noted that the names of the SceneOrchestrator, SceneConfigurationManager, TaskDispatcher, ExtensionManager, and ServiceRegistry classes mentioned above are merely exemplary identifiers introduced for the purpose of facilitating the description of the relevant content, and are not intended to limit the actual functionality of these classes. In actual implementation, any other suitable naming convention can be used according to actual needs, as long as the functionality implemented by each class remains unchanged.

[0029] Through the collaboration of these classes, the feature engineering construction and coordination control module can receive scenario requests, parse scenario configurations, distribute intermediate variables to the corresponding mining services, and support dynamic expansion.

[0030] In one embodiment, the Window Mining Service is used to receive window class intermediate variable mining tasks sent by the Feature Engineering Construction and Coordination Control Module, execute window pattern mining algorithms, support multiple window types and mining strategies, send the mining results to the integration module, and support dynamic expansion of new window types.

[0031] Specifically, such as Figure 3 As shown, the window mining module is based on the WindowMiningService class, which manages a group of WindowHandlers (window processors), uses WindowDataClient to obtain data, and uses ResultSender to send results.

[0032] The WindowMiningService class receives mining tasks from the Feature Engineering Construction and Coordination Control module, selects the corresponding window handler (such as SlidingWindowHandler, ExpandingWindowHandler, etc.) based on the variable type in the task, calls the handler to perform mining, and sends the results to the integration module. It supports dynamically extending new window types through the register_window_handler method.

[0033] The WindowHandler abstract class defines a unified interface, mine, for window mining. Each specific window handler (such as SlidingWindowHandler and ExpandingWindowHandler) inherits from and implements this interface to complete the mining of specific types of windows.

[0034] The WindowDataClient class is responsible for obtaining entity change data required by window mining from the data source. It encapsulates the data access logic and supports multiple data sources.

[0035] The ResultSender class is responsible for sending the mining results to the integration module. It handles the formatting and network transmission of the results, including retry mechanisms.

[0036] It should be noted that the names of the WindowMiningService, WindowHandler abstract class, WindowDataClient, and ResultSender classes mentioned above are merely exemplary identifiers introduced for the purpose of conveniently describing the relevant content, and are not intended to limit the actual functionality of these classes. In actual implementation, any other suitable naming convention can be used according to actual needs, as long as the functionality implemented by each class remains unchanged.

[0037] Through the collaboration of the above classes, the window mining module can handle mining tasks for various window types and supports dynamic expansion of new window types.

[0038] In one embodiment, the window class intermediate variable is calculated using entity change data and instruction data. Entity change data describes the current information of an entity and includes at least five attributes: entity name, entity ID, entity location, category, damage level, and accessories carried by the entity. For example, the entity change data for a car might have the entity name "Car," entity ID "EN0000," damage level "0%," category "FAW," and accessories including one spare tire and one jack. Instruction data describes different action instructions given to the issuing entity and includes at least the above five attributes: action ID, information about the executing entity (executing entity ID, category, location), action type, start time, and end time. For example, the car movement instruction data might have the action ID "AC0000," car ID "EN0000," category "FAW," action type "movement," start time "December 19, 2025, 21:00," and end time "December 19, 2025, 22:00." In one embodiment, the Key Event Mining Service is used to receive event-type intermediate variable mining tasks sent by the Feature Engineering Construction and Coordination Control Module, execute event link detection algorithms, support multiple event types and detection strategies, send the detection results to the integration module, and support dynamic expansion of new event types.

[0039] Specifically, such as Figure 4As shown, the event chain mining module is based on the KeyEventMiningService class, which manages a group of EventDetectors, uses EventDataClient to obtain event data, and uses ResultSender to send the results.

[0040] The KeyEventMiningService class receives mining tasks, selects the corresponding event detector (such as ThresholdEventDetector, PatternEventDetector, etc.) based on the variable type, calls the detector to perform event detection, and sends the results. It supports the extension of new event types through the register_event_detector method.

[0041] The EventDetector abstract class defines a unified interface, detect, for event detection. Specific event detectors implement this interface to perform event detection for specific types.

[0042] The EventDataClient class is responsible for retrieving data from event data sources and supports functions such as filtering and statistics.

[0043] The ResultSender class is shared with the window mining module and is responsible for sending results.

[0044] It should be noted that the names of the KeyEventMiningService, EventDetector (abstract class), EventDataClient, and ResultSender classes mentioned above are merely exemplary identifiers introduced for the purpose of conveniently describing the relevant content, and are not intended to limit the actual functionality of these classes. In actual implementation, any other suitable naming convention can be used according to actual needs, as long as the functionality implemented by each class remains unchanged.

[0045] Through the collaboration of the above classes, the event chain mining module can handle detection tasks for various event types and supports the dynamic expansion of new event types.

[0046] In one embodiment, the intermediate variables of the event chain are calculated using event data. Event data refers to the data recording events that occur during the simulation process, containing at least five attributes: event name, event importance, event code, name of the entity where the event occurred, location of the entity where the event occurred, and event occurrence time. For example, in the car start event data, the event name is "Car Start," the event importance is 0, the event code is EV0000, the name of the entity where the event occurred is "Car," the event location is (0, 0), and the event occurrence time is 21:00 on December 19, 2025.

[0047] In one embodiment, the task dimension mining module (Intermediate Metric Mining Service) is used to receive task dimension variable calculation tasks sent by the feature engineering construction and coordination control module, execute indicator calculation and derivation algorithms, support multiple indicator types and calculation strategies, send the calculation results to the integration module, and support dynamic expansion of new indicator types.

[0048] Specifically, such as Figure 5 As shown, the task dimension mining module is centered on the IntermediateMetricMiningService class, which manages a set of MetricCalculators, uses MetricDataClient to obtain statistical data, and uses ResultSender to send the results.

[0049] The `IntermediateMetricMiningService` class receives mining tasks, selects the corresponding metric calculator (such as `RatioMetricCalculator`, `CompositeMetricCalculator`, etc.) based on the variable type, calls the calculator to calculate the metric, and sends the results. It supports the extension of new metric types through the `register_metric_calculator` method.

[0050] The abstract class MetricCalculator defines a unified interface for calculating metrics, called Calculate. Specific metric calculators implement this interface to perform calculations for specific types of metrics.

[0051] The MetricDataClient class is responsible for obtaining basic indicator data from statistical data sources and supports data aggregation and metadata querying.

[0052] The ResultSender class is shared with the previous two services and is responsible for sending results.

[0053] It should be noted that the names of the IntermediateMetricMiningService, MetricCalculator (abstract class), MetricDataClient, and ResultSender classes mentioned above are merely exemplary identifiers introduced for the purpose of conveniently describing the relevant content, and are not intended to limit the actual functionality of these classes. In actual implementation, any other suitable naming convention can be used according to actual needs, as long as the functionality implemented by each class remains unchanged.

[0054] Through the collaboration of the above classes, the task dimension mining module can handle calculation tasks of various indicator types and supports the dynamic expansion of new indicator types.

[0055] In one embodiment, the task dimension class intermediate variable is calculated using entity statistics. Entity statistics are derived from the entity change data of each frame, and the entity statistics have different formats depending on the statistical dimension. For example, vehicle damage statistics include information about vehicle 1 with a damaged number of vehicles: {vehicle ID is EN0001, vehicle location is (0,0), and category is FAW}.

[0056] In one embodiment, the Result Integration Module receives the computation results from three mining services; integrates the intermediate variable results from different services; performs unified formatting and data validation; and sends the integrated results to an external system.

[0057] Specifically, such as Figure 6 As shown, the result integration module is centered around the ResultIntegrationModule class. It uses ResultReceiver to receive the results from each service, ResultIntegrator to integrate the results, UnifiedSender to send the integrated results, and QualityAssessor to evaluate the quality of the results.

[0058] The ResultIntegrationModule class starts the integration service and registers callback functions through ResultReceiver to receive results from various services. When all results for the same task arrive, it calls ResultIntegrator to integrate them and then uses UnifiedSender to send them to external systems.

[0059] The ResultReceiver class is responsible for receiving and caching the results from various mining services, providing a callback registration mechanism, and notifying the ResultIntegrationModule when a new result is received. It is also responsible for clearing the cache and providing reception statistics.

[0060] The ResultIntegrator class is responsible for integrating the results from three types of intermediate variables, generating cross-variable insights and summaries, and resolving conflicts between results. It uses QualityAssessor to evaluate the quality of the results.

[0061] The UnifiedSender class is responsible for sending the integrated results to external systems, supporting various external systems and data formats.

[0062] The QualityAssessor class is responsible for evaluating the quality of results, including metrics such as completeness, consistency, and timeliness.

[0063] It should be noted that the names of the ResultIntegrationModule, ResultReceiver, ResultIntegrator, UnifiedSender, and QualityAssessor classes mentioned above are merely exemplary identifiers introduced for the purpose of conveniently describing the relevant content, and are not intended to limit the actual functionality of these classes. In actual implementation, any other suitable naming convention can be used according to actual needs, as long as the functionality implemented by each class remains unchanged.

[0064] Through collaboration among the above classes, the results integration module can receive, integrate, and send the mining results of the three types of intermediate variables, and provide quality assessment.

[0065] In one embodiment, such as Figure 7 As shown, the core data model is as follows: MiningTask: The data model for mining tasks, including task ID, scenario name, variable list, etc.

[0066] SceneRequest: The data model for scene requests, which includes request parameters and metadata.

[0067] WindowResult: The data model for the window mining results.

[0068] EventResult: The data model for event detection results.

[0069] MetricResult: The data model for the metric calculation results.

[0070] IntegratedResult: The data model for integrated results, which includes all types of intermediate variables.

[0071] In one embodiment, such as Figure 8 As shown, the extension mechanism consists of the abstract class ExtensionPoint and its concrete subclasses SceneExtensionPoint, VariableExtensionPoint, and AlgorithmExtensionPoint.

[0072] The ExtensionPoint abstract class defines the basic operations for extension points, including validating configurations, registering extensions, and listing extensions. The SceneExtensionPoint class manages the extensions of scene adapters, allowing for the dynamic registration of new scene adapters.

[0073] The VariableExtensionPoint class manages extensions for variable handlers, allowing for the dynamic registration of new variable type handlers.

[0074] The AlgorithmExtensionPoint class manages algorithm extensions, allowing for the dynamic registration of new mining algorithms.

[0075] It should be noted that the names of the above-mentioned abstract classes ExtensionPoint, SceneExtensionPoint, VariableExtensionPoint, and AlgorithmExtensionPoint are merely exemplary identifiers introduced for the purpose of facilitating the description of the relevant content, and are not intended to limit the actual functionality of these classes. In actual implementation, any other suitable naming convention can be adopted according to actual needs, as long as the functionality implemented by each class remains unchanged.

[0076] These extension points are integrated into the feature engineering construction and coordination control module and various mining services, enabling the entire framework to flexibly extend to new scenarios, variable types and algorithms.

[0077] Through the class diagram and its description above, we have clearly demonstrated how each module fulfills its responsibilities through the collaboration of multiple classes, and explained how extensibility is supported.

[0078] This framework design has the following characteristics: Distributed architecture: The intermediate variables are systematically divided into three categories: event chain, window, and task dimension, which correspond to different data foundations and mining methods. The three types of mining services are deployed independently and can be horizontally scaled, solving the problem that the data scale is too large to be calculated because the three types of variables share a complete set of data.

[0079] Unified scheduling: The feature engineering construction and coordination control module centrally manages task distribution and scheduling. Modular design: Each module has clearly defined responsibilities and interfaces.

[0080] Strong extensibility: Supports dynamic expansion to new scenarios and new intermediate variable types.

[0081] Standardized interface: The modules communicate with each other through a standardized interface.

[0082] Results Integration: The integration module provides unified results integration and output. This framework effectively supports distributed mining and integration of large-scale intermediate variables, providing strong technical support for causal relationship analysis.

[0083] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A distributed mining and integration method for intermediate variables, applied to simulation scenarios, characterized in that, The method includes: In response to a target scene analysis request initiated by an external system, obtain preset scene configuration information corresponding to the target scene identifier carried in the request; Based on the scenario configuration information, the types of intermediate variables to be calculated are parsed; wherein, the types of intermediate variables include window-type intermediate variables, event chain-type intermediate variables, and task dimension-type intermediate variables; Based on the type of each intermediate variable and the access address of its corresponding original simulation data, a corresponding mining task is generated and distributed to the corresponding distributed mining service; wherein, the original simulation data is simulation running data related to the target scene; the mining task of window-type intermediate variables is distributed to the window mining service, the mining task of event link-type intermediate variables is distributed to the event link mining service, and the mining task of task dimension-type intermediate variables is distributed to the task dimension mining service. Each mining service extracts the corresponding type of simulation running data from the access address of the original simulation data according to the received mining task, performs intermediate variable calculation in parallel, and outputs window class intermediate variables, event link class intermediate variables and task dimension class intermediate variables respectively. The intermediate variables output by each mining service are integrated, and the integrated set of intermediate variables is returned to the external system.

2. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, The process of responding to a target scene analysis request initiated by an external system and obtaining preset scene configuration information corresponding to the target scene identifier carried in the request specifically includes: The SceneOrchestrator class receives and processes target scene analysis requests initiated by external systems to obtain target scene identifiers. The SceneOrchestrator class calls the SceneConfigurationManager class to query the local or remote configuration storage based on the target scene identifier, and obtains scene configuration information containing the required list of intermediate variables and the mapping relationship of variable types. The step of parsing the required intermediate variable types based on the scenario configuration information specifically includes: The SceneOrchestrator class calls the SceneConfigurationManager class to parse the types of intermediate variables that need to be calculated based on the scene configuration information. The process of distributing each mining task to the corresponding distributed mining service specifically includes: The SceneOrchestrator class calls the TaskDispatcher class, which triggers task dispatch strategies for window class, event chain class, and task dimension class respectively, based on the parsed intermediate variable type. The TaskDispatcher class calls the ServiceRegistry class to query the currently available endpoints of the corresponding mining service and sends mining tasks only to mining services that are in a healthy and available state.

3. The distributed mining and integration method for intermediate variables according to claim 2, characterized in that, The method further includes: When a target scene analysis request that is not registered or contains an unregistered intermediate variable type is received, the ExtensionManager class is called through the SceneOrchestrator class to verify the validity of the extension configuration. If the verification passes, register the new scenario or new intermediate variable type through the ExtensionManager class and associate it with its corresponding mining service identifier; In the subsequent task distribution process, the TaskDispatcher class distributes the new type of mining task to the corresponding mining service based on the registration information.

4. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, Calculate intermediate variables for the window class, specifically including: The WindowMiningService class receives the mining tasks for intermediate variables of the window class. The WindowMiningService class selects the corresponding processor from multiple window processors based on the intermediate variable type in the intermediate variable mining task of the window class, and calls the WindowDataClient class to extract entity change data and instruction data from the access address of the corresponding original simulation data. The WindowMiningService class calls the unified mining interface mine from the WindowHandler abstract class to enable the selected window processor to obtain the entity change data and instruction data and perform windowed calculations to generate intermediate variables of the window class.

5. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, Calculate intermediate variables for the event chain class, specifically including: The KeyEventMiningService class receives mining tasks from intermediate variables of the event chain class. The KeyEventMiningService class selects the corresponding detector from multiple event detectors based on the intermediate variable type in the mining task of the intermediate variable of the event chain class, and calls the EventDataClient class to extract event data from the access address of the corresponding original simulation data. The KeyEventMiningService class calls the unified detection interface detect of the selected event detector from the EventDetector abstract class, so that the selected event detector can obtain the event data and perform event detection, generating intermediate variables of the event chain class.

6. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, The intermediate variables for calculating task dimension classes specifically include: The IntermediateMetricMiningService class receives mining tasks from intermediate variables of the task dimension class. The IntermediateMetricMiningService class selects the corresponding calculator from multiple indicator calculators based on the type of intermediate variable in the task dimension class's intermediate variable mining task, and calls the MetricDataClient class to extract entity statistics data from the access address of the corresponding original simulation data. The IntermediateMetricMiningService class calls the unified detection interface calculate of the selected indicator calculator from the MetricCalculator abstract class, so that the selected indicator calculator can obtain the entity statistical data and perform indicator calculation to generate intermediate variables of the task dimension class.

7. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, The process of integrating the intermediate variables output by each mining service and returning the integrated set of intermediate variables to the external system specifically includes: Call the ResultReceiver class through the ResultIntegrationModule class to register callback functions to receive intermediate variables of the window class, the event chain class, and the task dimension class; After all three types of intermediate variables have been calculated, the ResultIntegrator class is called through the ResultIntegrationModule class to integrate all the intermediate variables; The UnifiedSender class is called through the ResultIntegrationModule class to return the integrated set of intermediate variables to the external system.

8. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, The target scene analysis request is represented by the SceneRequest data model; The mining task is represented by the MiningTask data model; The intermediate variables of the window class are represented by the WindowResult data model; The intermediate variables of the event chain class are represented by the EventResult data model; The intermediate variables of the task dimension class are represented by the MetricResult data model; The integrated set of intermediate variables is represented by the IntegratedResult data model.

9. The distributed mining and integration method for intermediate variables according to claim 1, characterized in that, The method also dynamically extends through a predefined extension point mechanism; wherein... The extension point mechanism includes the abstract class ExtensionPoint; The abstract class ExtensionPoint includes the SceneExtensionPoint class, VariableExtensionPoint class, and AlgorithmExtensionPoint class; The ExtensionPoint abstract class defines the basic operations of extension points, including verifying configurations, registering extensions, and listing extensions. The SceneExtensionPoint class manages the extension of scene adapters, allowing for the dynamic registration of new scene adapters; The VariableExtensionPoint class manages the extension of variable handlers, allowing for the dynamic registration of new intermediate variable type handlers; The AlgorithmExtensionPoint class manages algorithm extensions, allowing for the dynamic registration of new mining algorithms.

10. A distributed mining and integration framework for intermediate variables, applied to simulation and deduction scenarios, characterized in that, The framework includes: The feature engineering construction and coordination control module is used to respond to the target scene analysis request initiated by the external system and obtain the preset scene configuration information corresponding to the target scene identifier carried in the request. The feature engineering construction and coordination control module is also used to parse out the types of intermediate variables to be calculated based on the scenario configuration information; wherein, the types of intermediate variables include window-type intermediate variables, event link-type intermediate variables, and task dimension-type intermediate variables; The feature engineering construction and coordination control module is also used to generate corresponding mining tasks based on the type of each intermediate variable and the access address of its corresponding original simulation data, and to distribute each mining task to the corresponding distributed mining module; wherein, the original simulation data is simulation running data related to the target scene; the mining tasks of window-type intermediate variables are distributed to the window mining module, the mining tasks of event link-type intermediate variables are distributed to the event link mining module, and the mining tasks of task dimension-type intermediate variables are distributed to the task dimension mining module. The window mining module, the event link mining module, and the task dimension mining module are respectively used to extract simulation running data of the corresponding type from the access address of the original simulation data according to the received mining task, perform intermediate variable calculation in parallel, and output window class intermediate variables, event link class intermediate variables, and task dimension class intermediate variables; the result integration module is used to integrate the intermediate variables output by each mining service and return the integrated intermediate variable set to the external system.

Images