Multi-source data configuration parameter management method and platform

Abstract

The invention discloses a multi-source data configuration parameter management method and platform, and relates to the data processing related field, and the method comprises the steps: traversing a plurality of data sources to carry out type identification, obtaining a plurality of data classes, determining a plurality of configuration targets of multi-source data, and determining a target parameter set based on the plurality of data sources; data scene information is introduced to decompose and optimize a plurality of configuration targets, a plurality of configuration optimization targets are obtained and combined with the plurality of data classes to analyze the target parameter set, a parameter configuration matrix is constructed to perform collaborative filtering, a configuration scheme is formulated, and the configuration scheme is executed to dynamically manage the target parameter set. The technical problems of resource waste and insufficient data processing performance caused by lack of an intelligent data configuration scheme and difficulty in automatic parameter adjustment are solved, and the technical effects of dynamically adjusting data configuration, improving data processing efficiency and optimizing configuration parameter management are achieved.

Description

Technical Field

[0001] The present application relates to data processing related fields, and in particular to a configuration parameter management method and platform for multi-source data. Background Art

[0002] In existing multi-source data management technologies, enterprises and organizations face the challenge of integrating data from different sources, formats, and structures. With the development of information technology, data from various sources, such as sensor data, social media data, internal business system data, etc., flow into enterprise systems at different frequencies and formats. In order to effectively utilize this data, the system needs to perform unified configuration parameter management. This process involves configuring parameters for the data source, such as data collection frequency, data cleaning rules, storage location, and security level. In traditional multi-source data management systems, parameter configuration is usually performed manually or semi-automatically, and system managers need to adjust various parameters according to different data sources so that the data can be processed, stored, and analyzed. Summary of the invention

[0003] This application solves the technical problems of lack of intelligent data configuration scheme, difficulty in automatically adjusting parameters, resulting in waste of resources and insufficient data processing performance by providing a configuration parameter management method and platform for multi-source data, and achieves the technical effect of dynamically adjusting data configuration to improve data processing efficiency and optimize configuration parameter management.

[0004] The present application provides a configuration parameter management method for multi-source data, which is applied to a configuration parameter management platform for multi-source data, including: traversing multiple data sources to perform type identification, obtaining multiple data classes, and determining multiple configuration targets for the multi-source data according to the multiple data classes; determining a target parameter set based on the multiple data sources in combination with the multiple data classes; introducing data scenario information, decomposing and optimizing the multiple configuration targets, and obtaining multiple configuration optimization targets; analyzing the target parameter set according to the multiple configuration optimization targets in combination with the multiple data classes, and constructing a parameter configuration matrix; performing collaborative filtering according to the parameter configuration matrix, formulating a configuration plan, and executing the configuration plan to dynamically manage the target parameter set.

[0005] In a possible implementation, the data scenario information performs the following processing: traverse the multiple data sources to perform data collection and identification, determine the data collection chain, and obtain the data flow path according to the data collection chain; perform business flow analysis according to the data flow path to obtain business process information; parse the multiple data sources to obtain multiple internal data sources and multiple external data sources; analyze the parameter collection frequency of the multiple internal data sources in combination with the business process information to determine the first business dynamic demand information; analyze the parameter collection frequency of the multiple external data sources in combination with the business process information to determine the second business dynamic demand information; perform data analysis based on the first business dynamic demand information and the second business dynamic demand information to determine internal data usage scenario information and external data usage scenario information; add the internal data usage scenario information and the external data usage scenario information to the data scenario information.

[0006] In a possible implementation, data scenario information is introduced, the multiple configuration targets are decomposed and optimized to obtain multiple configuration optimization targets, and the following processing is performed: weights are assigned to the multiple configuration targets based on the internal data usage scenario information to obtain a first weight coefficient; the multiple configuration targets are decomposed and sorted according to the first weight coefficient to obtain an internal configuration target sequence; weights are assigned to the multiple configuration targets based on the external data usage scenario information to obtain a second weight coefficient; the multiple configuration targets are decomposed and sorted according to the second weight coefficient to obtain an external configuration target sequence; configuration impact analysis is performed according to a parameter association network to generate parameter configuration impact factors, and the internal configuration target sequence is optimized in combination with the external configuration target sequence based on the parameter configuration impact factors to obtain the multiple configuration optimization targets.

[0007] In a possible implementation, the target parameter set is analyzed according to the multiple configuration optimization objectives in combination with the multiple data classes, a parameter configuration matrix is ​​constructed, and the following processing is performed: the target parameter set is matched by traversing the multiple data classes to determine multiple target parameter classes; the target parameter set is calculated based on the multiple target parameter classes in combination with the multiple configuration optimization objectives to generate multiple configuration contribution degrees; the parameter configuration influencing factors are updated according to the multiple configuration contribution degrees to generate a parameter configuration priority list; and the parameter configuration matrix is ​​constructed according to the parameter configuration priority list in combination with the target parameter set.

[0008] In a possible implementation, the parameter configuration impact factor is updated according to the multiple configuration contributions, a parameter configuration priority list is generated, and the following processing is performed: based on the first business dynamic demand information and the second business dynamic demand information, it is determined whether the multiple configuration contributions are greater than a preset contribution threshold; if there is a configuration contribution greater than the preset contribution threshold among the multiple configuration contributions, an update queue is generated, and the configuration contribution greater than the preset contribution threshold is added to the update queue; the configuration contributions of the update queue are traversed to update the parameter configuration impact factor to generate a parameter configuration impact update factor; the multiple configuration targets are sorted according to the parameter configuration impact update factor to generate the parameter configuration priority list.

[0009] In a possible implementation, a parameter configuration matrix is ​​constructed according to the parameter configuration priority list in combination with the target parameter set, and the following processing is performed: row data is determined based on the data scenario information in combination with the parameter configuration priority list; column data is determined based on the target parameter set in combination with the parameter configuration priority list; matrix calculations are performed according to the parameter configuration priority list in combination with the multiple configuration optimization objectives to generate matrix element values; the row data and the column data are mapped and filled in according to the matrix element values ​​to construct the parameter configuration matrix.

[0010] In a possible implementation, collaborative filtering is performed according to the parameter configuration matrix, a configuration plan is formulated, and the following processing is performed: based on the parameter configuration matrix combined with the first weight coefficient, a similarity analysis is performed on the multiple data classes to generate a first similarity coefficient; based on the parameter configuration matrix combined with the second weight coefficient, a similarity analysis is performed on the multiple data classes to generate a second similarity coefficient; according to the first similarity coefficient, a configuration collaborative analysis is performed on the target parameter set to generate a first parameter configuration collaborative result; according to the second similarity coefficient, a configuration collaborative analysis is performed on the target parameter set to generate a second parameter configuration collaborative result; the first parameter configuration collaborative result is combined with the second parameter configuration collaborative result to be processed according to the expected filtering threshold to generate a recommended configuration result, and the configuration plan is formulated according to the recommended configuration result.

[0011] The present application also provides a configuration parameter management platform for multi-source data, including: a data identification module, the data identification module is used to traverse multiple data sources for type identification, obtain multiple data classes, and determine multiple configuration targets of the multi-source data according to the multiple data classes; a parameter determination module, the parameter determination module is used to determine a target parameter set based on the multiple data sources in combination with the multiple data classes; a decomposition and optimization module, the decomposition and optimization module is used to introduce data scenario information, decompose and optimize the multiple configuration targets, and obtain multiple configuration optimization targets; a matrix construction module, the matrix construction module is used to analyze the target parameter set according to the multiple configuration optimization targets in combination with the multiple data classes, and construct a parameter configuration matrix; a collaborative filtering module, the collaborative filtering module is used to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

[0012] One or more technical solutions provided in this application have at least the following technical effects or advantages: The present application provides a configuration parameter management method and platform for multi-source data, which relates to the field of data processing technology. It solves the technical problems of lack of intelligent data configuration scheme, difficulty in automatically adjusting parameters, resulting in waste of resources and insufficient data processing performance, and achieves the technical effect of dynamically adjusting data configuration to improve data processing efficiency and optimize configuration parameter management. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solution of the embodiment of the present application, the accompanying drawings of the embodiment of the present application will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the platform according to the embodiment of the present application. It should be understood that the previous or following operations are not necessarily performed precisely in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0014] Figure 1 A flowchart of a configuration parameter management method for multi-source data provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a configuration parameter management platform for multi-source data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0016] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0017] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, platform, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0018] The present application embodiment provides a method for managing configuration parameters of multi-source data, and the method is applied to a configuration parameter management platform for multi-source data, such as Figure 1 As shown, the method includes: Step A100, traverse multiple data sources to perform type identification, obtain multiple data classes, and determine multiple configuration targets of multi-source data based on the multiple data classes; first, traverse all connected data sources, each data source can come from different sources, such as sensor data, API interface data, internal database, third-party service data, etc., collect relevant metadata based on each data source, the metadata can be the format, frequency, size, source, etc. of the data, and classify the data source using a data type identification algorithm based on the metadata of the data source, the data type can include structured data types, unstructured data types, semi-structured data types, real-time data types, etc. After type identification, the system classifies multiple data sources according to their data types. Data types are clustered to form multiple data classes. Through the aggregation of data classes, common configuration goals can be set for data sources of similar types, which means determining the corresponding configuration goals based on the characteristics of each data class. That is, for real-time data classes, the data processing speed and response time can be prioritized, so performance optimization goals can be configured. For unstructured data classes, storage efficiency can be optimized and space occupancy can be reduced, so data quality optimization goals can be configured. For data classes involving sensitive information, data security can be improved to ensure encryption and access control, so security optimization goals can be configured. Each configuration goal will be set according to the specific data class requirements, so that multiple configuration goals can be determined to ensure the efficiency, stability and security of data processing.

[0019] Execute step A200 to determine a target parameter set based on the multiple data sources in combination with the multiple data classes; first, extract basic attributes of each data source, such as data type, data size, data source, update frequency and other information, and assign multiple data sources to corresponding data classes according to their data characteristics through type identification, such as structured data, unstructured data, real-time data, etc. According to the characteristics of different data classes, extract key parameters that affect system performance, storage, data quality and security. Exemplarily, key parameters of structured data classes may include: data consistency, index efficiency, query response time, etc., key parameters of unstructured data classes may include: storage space, file reading speed, compression ratio, etc., and key parameters of real-time data classes may include: data delay, throughput, update frequency, etc.

[0020] Further, according to the classification of multiple data classes, the parameters of the same data source in each data class are aggregated to form an overall target parameter set. For example, the data consistency, query response time and other parameters of the data sources of all structured data classes will be aggregated into a parameter set, so as to determine the target parameter set and ensure that subsequent optimization can make decisions based on comprehensive parameter data.

[0021] Execute step A300, introduce data scenario information, decompose and optimize the multiple configuration targets, and obtain multiple configuration optimization targets; in a possible implementation, step A300 further includes step A310, traverse the multiple data sources to perform data collection and identification, determine the data collection chain, and obtain the data flow path according to the data collection chain; execute step A320, perform business flow analysis according to the data flow path, and obtain business process information; execute step A330, parse the multiple data sources, obtain multiple internal data sources and multiple external data sources; execute step A340, according to the multiple Analyze the parameter collection frequency of the multiple internal data sources in combination with the business process information to determine the first business dynamic demand information; execute step A350, analyze the parameter collection frequency of the multiple external data sources in combination with the business process information to determine the second business dynamic demand information; execute step A360, perform data analysis based on the first business dynamic demand information and the second business dynamic demand information to determine the internal data usage scenario information and the external data usage scenario information; execute step A370, add the internal data usage scenario information and the external data usage scenario information to the data scenario information.

[0022] First, traverse all connected data sources to perform data collection and identification on each data source, extract the type, format, collection frequency, data update method and other information of the data source, and establish a data collection chain based on data collection and identification. The data collection chain is used to characterize the complete process from data collection to processing. Based on the determined data collection chain, analyze the data flow path between different nodes. The data flow path shows the process of data flowing from the collection point to each processing node, including storage, analysis, cleaning and output. According to the data flow path, further business flow analysis is performed, which means combining data flow with actual business processes to analyze the flow of data in different business processes and application scenarios. Through business flow analysis, the system can identify the specific role of each data source in business activities, and then extract specific business process information, including data requirements, data usage frequency, data output nodes and other information of each business process, thereby obtaining business process information.

[0023] All traversed data sources are further parsed and divided into internal data sources and external data sources. Internal data sources are usually business systems or databases from within the company, while external data sources include third-party services, API interfaces, social media, etc. from outside. At the same time, according to the parameter collection frequency of each internal data source, combined with the business process information, analysis is performed to determine the first business dynamic demand information, which is used to reflect the real-time requirements, data processing frequency, etc. of the internal data source in the business process. Then, according to the parameter collection frequency of each external data source, combined with the business process information, analysis is performed to determine the second business dynamic demand information, which is used to reflect the data synchronization frequency, real-time and dependence on external data.

[0024] Further, comprehensive data analysis is performed based on the first business dynamic demand information and the second business dynamic demand information to determine the actual usage scenario of each data source in the system. Exemplary internal data sources may be used for financial report generation, and external data sources may be used for market trend analysis, thereby determining internal data usage scenario information and external data usage scenario information. Internal data usage scenario information and external data usage scenario information are used to reflect the application of different data sources in different business processes, such as data for decision support, real-time monitoring, etc., and finally the generated internal data usage scenario information and external data usage scenario information are integrated and added to the system's data scenario information library to ensure that the scenario information library contains the latest business dynamic demand and data flow information, and to ensure that the platform can perform configuration optimization, decision support and dynamic adjustment based on complete data scenario information.

[0025] In one possible implementation, step A300 further includes step A380, assigning weights to the multiple configuration targets based on the internal data usage scenario information to obtain a first weight coefficient; executing step A390, decomposing and sorting the multiple configuration targets according to the first weight coefficient to obtain an internal configuration target sequence; executing step A3100, assigning weights to the multiple configuration targets based on the external data usage scenario information to obtain a second weight coefficient; executing step A3110, decomposing and sorting the multiple configuration targets according to the second weight coefficient to obtain an external configuration target sequence; executing step A3120, performing configuration impact analysis according to a parameter association network to generate a parameter configuration impact factor, and optimizing the internal configuration target sequence in combination with the external configuration target sequence based on the parameter configuration impact factor to obtain the multiple configuration optimization targets.

[0026] Since some scenarios have higher requirements for performance, while other scenarios may focus more on storage efficiency, the role of each configuration target in the internal data usage scenario is extracted based on the internal data usage scenario information generated above, and weights are allocated according to the importance of each configuration target in the internal data scenario. The first weight coefficient of each configuration target can be allocated by analyzing factors such as data processing requirements, real-time requirements, and data volume in the internal business process. Furthermore, multiple configuration targets are decomposed and sorted from large to small according to the first weight coefficient to form an internal configuration target sequence. The basis for sorting is the importance and priority of each configuration target, and targets with high weights will be ranked first. The internal configuration target sequence is used to characterize the priority of each configuration target in the internal data scenario. Through weight allocation and sorting, the system clarifies the optimal configuration target sequence in the internal scenario.

[0027] Furthermore, since some external data sources may require higher security, while others may have higher requirements for the real-time performance of data transmission, the role of external data sources in the platform is analyzed based on the external data usage scenario information obtained above. On this basis, weights are allocated according to the usage requirements of the external data sources to obtain a second weight coefficient. Further, according to the second weight coefficient, multiple configuration targets are decomposed and sorted from large to small. The configuration targets in the external scenario may include performance, storage, data quality and security. Corresponding weights will be assigned according to the real-time performance, transmission frequency, dependency, etc. of the external data, and configuration targets with higher weights have higher priorities in the sequence, thereby generating an external configuration target sequence to display the configuration priority in the external data scenario.

[0028] Further, by analyzing the correlation between internal and external data sources and configuration targets, a parameter association network is constructed. The parameter association network is used to reflect the relationship between different data sources and configuration targets. Since some parameters have an important impact on both performance and security targets, an impact analysis is performed through the parameter association network to generate a parameter configuration impact factor for each configuration target. The higher the impact factor, the greater the role of the configuration target in the overall system configuration. Based on the parameter configuration impact factor, the internal configuration target sequence and the external configuration target sequence are optimized. It means that some high-impact factor targets (such as performance and security) may appear in both internal and external configurations. These targets will be optimized first to ensure that the final configuration optimization target can achieve the best effect in internal and external data scenarios. Finally, through the above analysis, multiple configuration optimization targets are generated. The multiple configuration optimization targets balance the needs in different scenarios through the combination of internal and external data usage scenarios, and are optimized according to the parameter configuration impact factor.

[0029] Execute step A400, analyze the target parameter set according to the multiple configuration optimization objectives and the multiple data types, and construct a parameter configuration matrix; In a possible implementation, step A400 further includes step A410, traversing the multiple data classes to match the target parameter set and determine multiple target parameter classes; executing step A420, calculating the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization goals, and generating multiple configuration contribution degrees; first traversing all data classes, including structured data, unstructured data, real-time data, semi-structured data, etc., and matching them with the parameters in the target parameter set, for each data class, selecting suitable target parameters for matching according to its characteristics, illustratively, structured data classes may match performance parameters such as query response time and data consistency, by traversing data classes and matching target parameter sets, identifying parameters that have an important impact on specific data classes, and according to the matching results of data classes and target parameter sets, the system classifies the parameters and determines multiple target parameter classes. The multiple target parameter classes can cover different dimensions such as performance, storage, data quality and security.

[0030] For each target parameter class, its contribution in the system is calculated in combination with multiple configuration optimization objectives. This is done through weighted average or similar algorithms. The contribution of each target parameter class to each configuration optimization objective is calculated in combination with the characteristics of the data class and the impact of the parameter class on the optimization objective. The target parameter class with a higher contribution has a greater role in the optimization objective. Multiple configuration contributions are thus obtained, providing a basis for updating the parameter configuration influencing factors.

[0031] Execute step A430, update the parameter configuration influence factor according to the multiple configuration contributions, and generate a parameter configuration priority list; in a possible implementation, step A430 further includes step A431, determine whether the multiple configuration contributions are greater than a preset contribution threshold according to the first business dynamic demand information and the second business dynamic demand information; execute step A432, if there is a configuration contribution greater than the preset contribution threshold among the multiple configuration contributions, generate an update queue, and add the configuration contribution greater than the preset contribution threshold to the update queue; execute step A433, traverse the configuration contributions of the update queue to update the parameter configuration influence factor, and generate a parameter configuration influence update factor; execute step A434, sort the multiple configuration targets according to the parameter configuration influence update factor, and generate the parameter configuration priority list.

[0032] First, the configuration contribution of each configuration target is analyzed by combining the first business dynamic demand information derived from the internal data usage scenario and the second business dynamic demand information derived from the external data usage scenario. The contribution judgment is made based on the business dynamic demand information to ensure that the system can identify the configuration targets that are critical to system optimization. The configuration contribution of each configuration target is compared with a preset contribution threshold. The contribution represents the contribution of the configuration target to the overall optimization of the system, and the contribution threshold is a preset reference standard for screening out key configuration targets. If there is a configuration target whose contribution is greater than the threshold, the configuration target has a higher optimization value. When there is a configuration target whose configuration contribution is greater than the preset contribution threshold, the system will generate an update queue, which is used to save all configuration targets whose contributions exceed the threshold, traverse all configuration targets, and compare their contributions with the threshold. Configuration targets whose contributions are greater than the threshold are added to the update queue to ensure that subsequent impact factor updates are only focused on key configuration targets, thereby improving update efficiency.

[0033] Traverse each configuration target and its contribution in the update queue, process them one by one, and the contribution of each configuration target will be used as the basis for updating the influence factor in the update process. The corresponding parameter configuration influence factor is updated by the contribution of each configuration target in the update queue. The update rule of the parameter configuration influence factor is based on the influence of the configuration target on the overall optimization of the system. The configuration target with a higher contribution has a larger update amplitude of its influence factor, thereby generating a new parameter configuration influence update factor, i.e., the parameter configuration influence update factor. The parameter configuration influence update factor is used to identify the optimization influence of each configuration target after the update. Finally, all configuration targets are sorted according to the parameter configuration influence update factor of each configuration target. The target with a larger influence update factor will be prioritized in the sorting, and the sorted configuration targets will generate a parameter configuration priority list, i.e., the configuration targets with a high priority will be given priority for optimization, ensuring that the key configuration targets can be processed first in the subsequent optimization process.

[0034] Execute step A440 to construct the parameter configuration matrix according to the parameter configuration priority list and the target parameter set.

[0035] In one possible implementation, step A440 further includes step A441, determining row data based on the data scenario information in combination with the parameter configuration priority list; executing step A442, determining column data based on the target parameter set in combination with the parameter configuration priority list; executing step A443, performing matrix calculations according to the parameter configuration priority list in combination with the multiple configuration optimization objectives to generate matrix element values; executing step A444, mapping and filling the row data and the column data according to the matrix element values ​​to construct the parameter configuration matrix.

[0036] First, various data sources and related information are extracted from the data scenario information. The data scenario information may include internal and external data sources, data types, business dynamic requirements, etc. The row data in the matrix is ​​determined according to the data scenario information, and each row in the row data represents a different data source or data class. Exemplarily, the row data may include structured data, unstructured data, real-time data, etc. All target parameters that need to be configured are further extracted from the target parameter set. The extracted parameters may cover different aspects such as performance, storage, data quality and security, and the target parameters are sorted in combination with the parameter configuration priority list to determine the column data in the matrix. The column data represents different configuration target parameters, and each column corresponds to a parameter.

[0037] Further, the optimization target to be calculated in the matrix is ​​extracted from multiple configuration optimization targets. The weight of each optimization target will affect the calculation of the matrix element value. By combining the parameter configuration priority list and the configuration optimization target, the calculation is performed for each combination of row data and column data to generate the element value in the matrix. The matrix element value represents the contribution or importance of a certain data class to a certain configuration target. The matrix element value is used to characterize the role of each data class in each configuration target. The row data and column data determined above are further mapped according to the calculated matrix element value and filled into the matrix. By completing the mapping of row data and column data, a parameter configuration matrix is ​​constructed. The rows of the parameter configuration matrix represent data classes, and the columns represent configuration targets. Each cell in the matrix reflects the contribution of the data class to the configuration target.

[0038] Next, step A500 is executed to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

[0039] In one possible implementation, step A500 further includes step A510, performing a similarity analysis on the multiple data classes based on the parameter configuration matrix combined with the first weight coefficient to generate a first similarity coefficient; executing step A520, performing a similarity analysis on the multiple data classes based on the parameter configuration matrix combined with the second weight coefficient to generate a second similarity coefficient; executing step A530, performing a configuration collaborative analysis on the target parameter set according to the first similarity coefficient to generate a first parameter configuration collaborative result; executing step A540, performing a configuration collaborative analysis on the target parameter set according to the second similarity coefficient to generate a second parameter configuration collaborative result; executing step A550, processing the first parameter configuration collaborative result in combination with the second parameter configuration collaborative result according to the expected filtering threshold to generate a recommended configuration result, and formulating the configuration plan based on the recommended configuration result.

[0040] First, based on the parameter configuration matrix combined with the first weight coefficient, a similarity analysis is performed on multiple data classes, which means that by comparing each data class, the similarity of each data class under different configuration targets is calculated, and the similarity under different configuration targets is weighted based on the first weight coefficient to generate a first similarity coefficient. The first similarity coefficient is used to reflect the similarity between different data classes in the internal scenario by combining the calculation of the first weight coefficient and the parameter configuration matrix. Then, based on the parameter configuration matrix combined with the second weight coefficient, a similarity analysis is performed on the same data class, which means that the similarity of each pair of data classes under different configuration targets is generated through weighted calculation, and a second similarity coefficient is generated to reflect the similarity of the data class in the external scenario.

[0041] Furthermore, by performing configuration collaboration analysis on the target parameter set based on the first similarity coefficient, data classes with similar optimization requirements can be identified according to the first similarity coefficient, and relevant target parameters can be collaboratively configured to obtain the first parameter configuration collaboration result. The first parameter configuration collaboration result is used to reflect which target parameter configurations can be collaboratively optimized through similarity in internal scenarios, thereby ensuring that data classes can share configuration optimization strategies under similar requirements.

[0042] Furthermore, by performing configuration collaborative analysis on the target parameter set based on the second similarity coefficient, the second parameter configuration collaborative result can be obtained by identifying data classes with similar optimization requirements in external scenarios and collaboratively configuring relevant parameters. The second parameter configuration collaborative result is used to reflect which target parameter configurations in external scenarios can be collaboratively optimized through similarity, thereby ensuring that the data classes can achieve collaborative optimization of configuration under external optimization requirements.

[0043] Finally, the first parameter configuration coordination result is combined with the second parameter configuration coordination result. Through the similarity analysis results, the internal and external configuration coordination requirements are uniformly processed, which means that the expected filtering threshold can be set based on retaining high contribution, and the combined coordination results can be screened at the same time to eliminate the coordination results that do not meet the threshold, and the purpose of retaining configuration results with higher contribution can be achieved, thereby improving the effectiveness of configuration optimization and generating recommended configuration results. According to the recommended configuration results, specific configuration optimization steps, resource allocation strategies, and priority processing methods are defined to generate configuration plans, thereby improving the overall resource utilization efficiency and performance of the system.

[0044] The embodiments of the present application solve the technical problems of lack of intelligent data configuration scheme, difficulty in automatically adjusting parameters, resulting in waste of resources and insufficient data processing performance, and achieve the technical effect of dynamically adjusting data configuration to improve data processing efficiency and optimize configuration parameter management.

[0045] In the above, refer to Figure 1A configuration parameter management method for multi-source data according to an embodiment of the present application is described in detail. Figure 2 A configuration parameter management platform for multi-source data according to an embodiment of the present application is described.

[0046] A configuration parameter management platform for multi-source data according to an embodiment of the present application is used to solve the technical problems of lack of intelligent data configuration scheme, difficulty in automatically adjusting parameters, waste of resources and insufficient data processing performance, and achieve the technical effect of dynamically adjusting data configuration to improve data processing efficiency and optimize configuration parameter management. A configuration parameter management platform for multi-source data includes: a data identification module 10, a parameter determination module 20, a decomposition optimization module 30, a matrix construction module 40, and a collaborative filtering module 50.

[0047] A data identification module 10, the data identification module 10 is used to traverse multiple data sources to perform type identification, obtain multiple data classes, and determine multiple configuration targets of multi-source data according to the multiple data classes; A parameter determination module 20, the parameter determination module 20 is used to determine a target parameter set based on the multiple data sources in combination with the multiple data classes; A decomposition and optimization module 30, wherein the decomposition and optimization module 30 is used to introduce data scenario information, decompose and optimize the multiple configuration targets, and obtain multiple configuration optimization targets; A matrix construction module 40, the matrix construction module 40 is used to analyze the target parameter set according to the multiple configuration optimization objectives in combination with the multiple data types to construct a parameter configuration matrix; The collaborative filtering module 50 is used to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

[0048] The specific configuration of the decomposition and optimization module 30 will be described in detail below. As described above, the data scenario information, the decomposition and optimization module 30 may further include: traversing the multiple data sources to perform data collection and identification, determine the data collection chain, and obtain the data flow path according to the data collection chain; perform business flow analysis according to the data flow path to obtain business process information; parse the multiple data sources to obtain multiple internal data sources and multiple external data sources; analyze the first business dynamic demand information according to the parameter collection frequency of the multiple internal data sources combined with the business process information; analyze the second business dynamic demand information according to the parameter collection frequency of the multiple external data sources combined with the business process information; perform data analysis according to the first business dynamic demand information and the second business dynamic demand information to determine the internal data usage scenario information and the external data usage scenario information; add the internal data usage scenario information and the external data usage scenario information to the data scenario information.

[0049] The specific configuration of the decomposition optimization module 30 will be described in detail below. As described above, the data scenario information is introduced, the multiple configuration targets are decomposed and optimized, and multiple configuration optimization targets are obtained. The decomposition optimization module 30 may further include: weighting the multiple configuration targets based on the internal data usage scenario information to obtain a first weight coefficient; decomposing and sorting the multiple configuration targets according to the first weight coefficient to obtain an internal configuration target sequence; weighting the multiple configuration targets based on the external data usage scenario information to obtain a second weight coefficient; decomposing and sorting the multiple configuration targets according to the second weight coefficient to obtain an external configuration target sequence; performing configuration impact analysis according to a parameter association network to generate a parameter configuration impact factor, optimizing the internal configuration target sequence in combination with the external configuration target sequence based on the parameter configuration impact factor to obtain the multiple configuration optimization targets.

[0050] The specific configuration of the matrix construction module 40 will be described in detail below. As described above, the target parameter set is analyzed according to the multiple configuration optimization objectives in combination with the multiple data classes to construct a parameter configuration matrix, and the matrix construction module 40 may further include: traversing the multiple data classes to match the target parameter set and determine multiple target parameter classes; calculating the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization objectives to generate multiple configuration contribution degrees; updating the parameter configuration influencing factors according to the multiple configuration contribution degrees to generate a parameter configuration priority list; and constructing the parameter configuration matrix according to the parameter configuration priority list in combination with the target parameter set.

[0051] The specific configuration of the matrix construction module 40 will be described in detail below. As described above, the parameter configuration impact factor is updated according to the multiple configuration contributions to generate a parameter configuration priority list. The matrix construction module 40 may further include: judging whether the multiple configuration contributions are greater than a preset contribution threshold according to the first business dynamic demand information and the second business dynamic demand information; if there is a configuration contribution greater than the preset contribution threshold among the multiple configuration contributions, then generating an update queue, and adding the configuration contribution greater than the preset contribution threshold to the update queue; traversing the configuration contributions of the update queue to update the parameter configuration impact factor to generate a parameter configuration impact update factor; sorting the multiple configuration targets according to the parameter configuration impact update factor to generate the parameter configuration priority list.

[0052] The specific configuration of the matrix construction module 40 will be described in detail below. As described above, the parameter configuration matrix is ​​constructed according to the parameter configuration priority list in combination with the target parameter set, and the matrix construction module 40 may further include: determining row data based on the data scenario information in combination with the parameter configuration priority list; determining column data based on the target parameter set in combination with the parameter configuration priority list; performing matrix calculation according to the parameter configuration priority list in combination with the multiple configuration optimization targets to generate matrix element values; mapping and filling the row data and the column data according to the matrix element values ​​to construct the parameter configuration matrix.

[0053] The specific configuration of the collaborative filtering module 50 will be described in detail below. As described above, collaborative filtering is performed according to the parameter configuration matrix to formulate a configuration plan. The collaborative filtering module 50 may further include: performing similarity analysis on the multiple data classes based on the parameter configuration matrix in combination with the first weight coefficient to generate a first similarity coefficient; performing similarity analysis on the multiple data classes based on the parameter configuration matrix in combination with the second weight coefficient to generate a second similarity coefficient; performing configuration collaborative analysis on the target parameter set according to the first similarity coefficient to generate a first parameter configuration collaborative result; performing configuration collaborative analysis on the target parameter set according to the second similarity coefficient to generate a second parameter configuration collaborative result; processing the first parameter configuration collaborative result in combination with the second parameter configuration collaborative result according to the expected filtering threshold to generate a recommended configuration result, and formulating the configuration plan based on the recommended configuration result.

[0054] A configuration parameter management platform for multi-source data provided in an embodiment of the present application can execute a configuration parameter management method for multi-source data provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0055] Although the present application makes various references to certain modules in the platform according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present application.

[0056] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A configuration parameter management method for multi-source data, characterized in that: The method comprises: Traversing multiple data sources to perform type identification, obtaining multiple data classes, and determining multiple configuration targets of the multi-source data according to the multiple data classes; determining a target parameter set based on the plurality of data sources in combination with the plurality of data classes; Introducing data scenario information, decomposing and optimizing the multiple configuration objectives, and obtaining multiple configuration optimization objectives; Analyze the target parameter set according to the multiple configuration optimization objectives and the multiple data types to construct a parameter configuration matrix; Collaborative filtering is performed according to the parameter configuration matrix, a configuration plan is formulated, and the configuration plan is executed to dynamically manage the target parameter set.

2. A configuration parameter management method for multi-source data according to claim 1, characterized in that: The data scene information method includes: Traversing the multiple data sources to perform data collection and identification, determine a data collection chain, and obtain a data flow path according to the data collection chain; Perform business flow analysis according to the data flow path to obtain business process information; Parsing the multiple data sources to obtain multiple internal data sources and multiple external data sources; Analyze the parameter collection frequency of the multiple internal data sources in combination with the business process information to determine the first business dynamic demand information; Analyze the parameter collection frequencies of the multiple external data sources in combination with the business process information to determine the second business dynamic demand information; Perform data analysis according to the first business dynamic demand information and the second business dynamic demand information to determine internal data usage scenario information and external data usage scenario information; The internal data usage scenario information and the external data usage scenario information are added to the data scenario information.

3. A configuration parameter management method for multi-source data as claimed in claim 2, characterized in that: Introducing data scenario information, decomposing and optimizing the multiple configuration targets, and obtaining multiple configuration optimization targets, the method includes: Based on the internal data usage scenario information, weights are assigned to the multiple configuration targets to obtain a first weight coefficient; Decomposing and sorting the multiple configuration targets according to the first weight coefficient to obtain an internal configuration target sequence; Based on the external data usage scenario information, weights are assigned to the multiple configuration targets to obtain a second weight coefficient; Decomposing and sorting the multiple configuration targets according to the second weight coefficient to obtain an external configuration target sequence; Perform configuration impact analysis according to the parameter association network to generate parameter configuration impact factors, optimize the internal configuration target sequence in combination with the external configuration target sequence based on the parameter configuration impact factors, and obtain the multiple configuration optimization targets.

4. A configuration parameter management method for multi-source data as claimed in claim 3, characterized in that: The target parameter set is analyzed according to the multiple configuration optimization objectives in combination with the multiple data types to construct a parameter configuration matrix, the method comprising: Traversing the multiple data classes to match the target parameter set, and determining multiple target parameter classes; Calculating the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization objectives to generate multiple configuration contribution degrees; Update the parameter configuration influencing factors according to the multiple configuration contribution degrees, and generate a parameter configuration priority list; The parameter configuration matrix is ​​constructed according to the parameter configuration priority list in combination with the target parameter set.

5. A configuration parameter management method for multi-source data as claimed in claim 4, characterized in that: The parameter configuration influencing factor is updated according to the multiple configuration contribution degrees to generate a parameter configuration priority list, the method comprising: Determining whether the multiple configuration contribution degrees are greater than a preset contribution threshold according to the first business dynamic demand information and the second business dynamic demand information; If a configuration contribution degree greater than the preset contribution threshold exists among the multiple configuration contributions, an update queue is generated, and the configuration contribution degree greater than the preset contribution threshold is added to the update queue; Traversing the configuration contribution of the update queue to update the parameter configuration impact factor, and generating a parameter configuration impact update factor; The multiple configuration targets are sorted according to the parameter configuration impact update factors to generate the parameter configuration priority list.

6. A configuration parameter management method for multi-source data as claimed in claim 4, characterized in that: Constructing a parameter configuration matrix according to the parameter configuration priority list and the target parameter set, the method comprising: Determine row data based on the data scenario information in combination with the parameter configuration priority list; Determining column data based on the target parameter set in combination with the parameter configuration priority list; Perform matrix calculation according to the parameter configuration priority list and the multiple configuration optimization objectives to generate matrix element values; The row data and the column data are mapped and filled in according to the matrix element values ​​to construct the parameter configuration matrix.

7. A configuration parameter management method for multi-source data as claimed in claim 3, characterized in that: Perform collaborative filtering according to the parameter configuration matrix and formulate a configuration plan, the method comprising: Perform similarity analysis on the multiple data classes based on the parameter configuration matrix in combination with the first weight coefficient to generate a first similarity coefficient; Perform similarity analysis on the multiple data classes based on the parameter configuration matrix combined with the second weight coefficient to generate a second similarity coefficient; Performing configuration coordination analysis on the target parameter set according to the first similarity coefficient to generate a first parameter configuration coordination result; Performing configuration coordination analysis on the target parameter set according to the second similarity coefficient to generate a second parameter configuration coordination result; The first parameter configuration collaboration result is combined with the second parameter configuration collaboration result and processed according to the expected filtering threshold to generate a recommended configuration result, and the configuration plan is formulated according to the recommended configuration result.

8. A configuration parameter management platform for multi-source data, characterized in that: The platform is used to implement a configuration parameter management method for multi-source data according to any one of claims 1 to 7, and the platform includes: A data identification module, the data identification module is used to traverse multiple data sources to perform type identification, obtain multiple data classes, and determine multiple configuration targets of multi-source data according to the multiple data classes; a parameter determination module, the parameter determination module being used to determine a target parameter set based on the multiple data sources in combination with the multiple data classes; A decomposition and optimization module, wherein the decomposition and optimization module is used to introduce data scenario information, decompose and optimize the multiple configuration targets, and obtain multiple configuration optimization targets; A matrix construction module, the matrix construction module is used to analyze the target parameter set according to the multiple configuration optimization objectives in combination with the multiple data types to construct a parameter configuration matrix; A collaborative filtering module is used to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

Images