A method and platform for managing configuration parameters of multi-source data

Abstract

The present application discloses a method and platform for managing configuration parameters of multi-source data, relating to the field of data processing. The method includes: traversing multiple data sources for type identification to obtain multiple data classes; determining multiple configuration targets of the multi-source data; combining to determine a target parameter set based on the multiple data sources; introducing data scenario information to decompose and optimize the multiple configuration targets to obtain multiple configuration optimization targets; analyzing the target parameter set in combination with the multiple data classes; constructing a parameter configuration matrix for collaborative filtering; formulating a configuration plan; and executing the configuration plan to dynamically manage the target parameter set, solving the technical problem of lacking an intelligent data configuration plan and being difficult to automatically adjust parameters, resulting in resource waste and insufficient data processing performance, and achieving the technical effect of dynamically adjusting data configuration to improve data processing efficiency and optimizing configuration parameter management.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a method and platform for managing configuration parameters of multi-source data. Background Art

[0002] In existing multi-source data management technologies, enterprises and organizations face the challenge of integrating data from different sources, formats, and structures. With the development of information technology, data from various sources, such as sensor data, social media data, and internal business system data, flows into enterprise systems at different frequencies and in different formats. To effectively utilize this data, the system needs to perform unified configuration parameter management. This process involves configuring parameters for data sources, such as data collection frequency, data cleaning rules, storage location, and security level. In traditional multi-source data management systems, parameter configuration is usually carried out manually or semi-automatically, and system administrators need to adjust various parameters according to different data sources so that the data can be processed, stored, and analyzed. Summary of the Invention

[0003] This application provides a method and platform for managing configuration parameters of multi-source data, which solves the technical problems of the lack of intelligent data configuration solutions and the difficulty in automatically adjusting parameters, resulting in resource waste and insufficient data processing performance, and achieves the technical effects of dynamically adjusting data configuration to improve data processing efficiency and optimizing configuration parameter management.

[0004] This application provides a method for managing configuration parameters of multi-source data. The method is applied to a platform for managing configuration parameters of multi-source data and includes: traversing multiple data sources for type recognition to obtain multiple data classes, and determining multiple configuration objectives of multi-source data according to the multiple data classes; determining a target parameter set based on the multiple data sources in combination with the multiple data classes; introducing data scenario information, decomposing and optimizing the multiple configuration objectives to obtain multiple configuration optimization objectives; analyzing the target parameter set according to the multiple configuration optimization objectives in combination with the multiple data classes, and constructing a parameter configuration matrix; performing collaborative filtering according to the parameter configuration matrix, formulating a configuration plan, and executing the configuration plan to dynamically manage the target parameter set.

[0005] In a possible implementation, for the data scenario information, the following processing is performed: traverse the multiple data sources for data collection and identification, determine a data collection chain, and obtain a data flow path according to the data collection chain; perform business flow analysis according to the data flow path to obtain business process information; parse the multiple data sources to obtain multiple internal data sources and multiple external data sources; analyze according to the parameter collection frequencies of the multiple internal data sources in combination with the business process information to determine first business dynamic requirement information; analyze according to the parameter collection frequencies of the multiple external data sources in combination with the business process information to determine second business dynamic requirement information; perform data analysis according to the first business dynamic requirement information and the second business dynamic requirement information to determine internal data usage scenario information and external data usage scenario information; add the internal data usage scenario information and the external data usage scenario information to the data scenario information.

[0006] In a possible implementation, introduce the data scenario information, decompose and optimize the multiple configuration targets to obtain multiple configuration optimization targets, and perform the following processing: perform weight assignment on the multiple configuration targets based on the internal data usage scenario information to obtain a first weight coefficient; decompose and sort the multiple configuration targets according to the first weight coefficient to obtain an internal configuration target sequence; perform weight assignment on the multiple configuration targets based on the external data usage scenario information to obtain a second weight coefficient; decompose and sort the multiple configuration targets according to the second weight coefficient to obtain an external configuration target sequence; perform configuration impact analysis according to a parameter association network to generate a parameter configuration impact factor, and optimize the internal configuration target sequence in combination with the external configuration target sequence based on the parameter configuration impact factor to obtain the multiple configuration optimization targets.

[0007] In a possible implementation, analyze the target parameter set according to the multiple configuration optimization targets in combination with the multiple data classes, construct a parameter configuration matrix, and perform the following processing: traverse the multiple data classes to match the target parameter set to determine multiple target parameter classes; calculate the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization targets to generate multiple configuration contribution degrees; update the parameter configuration impact factor according to the multiple configuration contribution degrees to generate a parameter configuration priority list; construct the parameter configuration matrix according to the parameter configuration priority list in combination with the target parameter set.

[0008] In a possible implementation, the parameter configuration impact factor is updated according to the multiple configuration contribution degrees, and a parameter configuration priority list is generated, and the following processing is performed: It is determined whether the multiple configuration contribution degrees are greater than a preset contribution threshold according to the first service dynamic requirement information and the second service dynamic requirement information; if there is a configuration contribution degree greater than the preset contribution threshold among the multiple configuration contribution degrees, an update queue is generated, and the configuration contribution degrees greater than the preset contribution threshold are added to the update queue; the configuration contribution degrees in the update queue are traversed to update the parameter configuration impact factor, and a parameter configuration impact update factor is generated; the multiple configuration targets are sorted according to the parameter configuration impact update factor to generate the parameter configuration priority list.

[0009] In a possible implementation, a parameter configuration matrix is constructed according to the parameter configuration priority list in combination with the target parameter set, and the following processing is performed: Row data is determined based on the data scenario information in combination with the parameter configuration priority list; column data is determined based on the target parameter set in combination with the parameter configuration priority list; matrix calculation is performed according to the parameter configuration priority list in combination with the multiple configuration optimization targets to generate matrix element values; the row data and the column data are mapped and filled in according to the matrix element values to construct the parameter configuration matrix.

[0010] In a possible implementation, collaborative filtering is performed according to the parameter configuration matrix to formulate a configuration plan, and the following processing is performed: Similarity analysis is performed on the multiple data classes based on the parameter configuration matrix in combination with the first weight coefficient to generate a first similarity coefficient; similarity analysis is performed on the multiple data classes based on the parameter configuration matrix in combination with the second weight coefficient to generate a second similarity coefficient; configuration collaborative analysis is performed on the target parameter set according to the first similarity coefficient to generate a first parameter configuration collaborative result; configuration collaborative analysis is performed on the target parameter set according to the second similarity coefficient to generate a second parameter configuration collaborative result; the first parameter configuration collaborative result is combined with the second parameter configuration collaborative result and processed according to an expected filtering threshold to generate a recommended configuration result, and the configuration plan is formulated according to the recommended configuration result.

[0011] The present application also provides a configuration parameter management platform for multi-source data, including: a data recognition module, which is used to traverse multiple data sources for type recognition to obtain multiple data classes, and determine multiple configuration targets for the multi-source data according to the multiple data classes; a parameter determination module, which is used to determine a target parameter set based on the multiple data sources in combination with the multiple data classes; a decomposition and optimization module, which is used to introduce data scenario information, decompose and optimize the multiple configuration targets to obtain multiple configuration optimization targets; a matrix construction module, which is used to analyze the target parameter set according to the multiple configuration optimization targets in combination with the multiple data classes to construct a parameter configuration matrix; a collaborative filtering module, which is used to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

[0012] One or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0013] A configuration parameter management method and platform for multi-source data provided by the present application relate to the technical field of data processing, solve the technical problems of lack of intelligent data configuration schemes and difficulty in automatically adjusting parameters, resulting in resource waste and insufficient data processing performance, and achieve the technical effects of dynamically adjusting data configuration to improve data processing efficiency and optimizing configuration parameter management. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings of the embodiments of the present application will be briefly introduced below. Flowcharts are used in the present application to illustrate the operations performed by the platform according to the embodiments of the present application. It should be understood that the operations in the front or below do not necessarily need to be executed precisely in sequence. On the contrary, according to needs, they can be executed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several operations can be removed from these processes.

[0015] Figure 1 It is a schematic flowchart of a configuration parameter management method for multi-source data provided by an embodiment of the present application;

[0016] Figure 2 It is a schematic structural diagram of a configuration parameter management platform for multi-source data provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The above description is only an overview of the technical solutions of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the description. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the following specifically illustrates the specific embodiments of the present application.

[0018] In order to make the objectives, technical solutions, and advantages of this application clearer, the following will further describe this application in detail with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

[0019] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict. The terms "first" and "second" involved are only used to distinguish similar objects and do not represent a specific order for the objects. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, platform, product, or server that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or are inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application.

[0020] An embodiment of this application provides a method for managing configuration parameters of multi-source data. The method is applied to a platform for managing configuration parameters of multi-source data, as Figure 1 shown, and the method includes:

[0021] Step A100: Traverse multiple data sources for type identification to obtain multiple data classes, and determine multiple configuration targets for multi-source data according to the multiple data classes. First, traverse all the accessed data sources. Each data source can come from different sources, such as sensor data, API interface data, internal database, third-party service data, etc. Collect relevant metadata based on each data source. The metadata can be the format, frequency, size, source, etc. of the data. According to the metadata of the data source, use a data type identification algorithm to classify the data source. The data types can include structured data types, unstructured data types, semi-structured data types, real-time data types, etc. After type identification, the system clusters multiple data sources according to their data types to form multiple data classes. By aggregating the data classes, common configuration targets can be set for data sources of similar types. It means determining the corresponding configuration targets according to the characteristics of each data class. That is, for the real-time data class, the data processing speed and response time can be preferentially improved, so performance optimization targets can be configured. For the unstructured data class, the storage efficiency can be optimized to reduce space occupancy, so data quality optimization targets can be configured. For the data class involving sensitive information, the data security can be enhanced to ensure encryption and access control, so security optimization targets can be configured. Each configuration target will be set according to the specific requirements of the data class, so as to determine multiple configuration targets and ensure the efficiency, stability and security of data processing.

[0022] Execute step A200: Determine a set of target parameters based on the multiple data sources in combination with the multiple data classes. First, extract basic attributes from each data source, such as information on data type, data size, data source, update frequency, etc. Through type identification, allocate multiple data sources to the corresponding data classes according to their data characteristics, such as structured data, unstructured data, real-time data, etc. According to the characteristics of different data classes, extract the key parameters that affect system performance, storage, data quality and security. Exemplarily, the key parameters of the structured data class may include: data consistency, index efficiency, query response time, etc. The key parameters of the unstructured data class may include: storage space, file reading speed, compression ratio, etc. The key parameters of the real-time data class may include: data latency, throughput, update frequency, etc.

[0023] Furthermore, according to the classification of multiple data classes, aggregate the parameters of the same data source in each data class to form an overall set of target parameters. For example, the parameters such as data consistency and query response time of the data sources in all structured data classes will be aggregated into a parameter set, so as to determine the set of target parameters and ensure that subsequent optimizations can make decisions based on comprehensive parameter data.

[0024] Execute step A300 to introduce data scenario information, decompose and optimize the multiple configuration targets to obtain multiple configuration optimization targets; in a possible implementation, step A300 further includes step A310, traverse the multiple data sources for data collection and identification, determine the data collection chain, and obtain the data flow path according to the data collection chain; execute step A320, perform business flow analysis according to the data flow path to obtain business process information; execute step A330, parse the multiple data sources to obtain multiple internal data sources and multiple external data sources; execute step A340, analyze according to the parameter collection frequencies of the multiple internal data sources in combination with the business process information to determine the first business dynamic requirement information; execute step A350, analyze according to the parameter collection frequencies of the multiple external data sources in combination with the business process information to determine the second business dynamic requirement information; execute step A360, perform data analysis according to the first business dynamic requirement information and the second business dynamic requirement information to determine the internal data usage scenario information and the external data usage scenario information; execute step A370, add the internal data usage scenario information and the external data usage scenario information to the data scenario information.

[0025] First, traverse all the connected data sources to identify data for each data source, extract information such as the type, format, collection frequency, and data update method of the data source, and establish a data collection chain on the basis of data collection and identification. The data collection chain is used to represent the complete process of data from collection to processing. Based on the determined data collection chain, analyze the data flow path between different nodes. The data flow path shows the process of data flowing from the collection point to each processing node, including all links of storage, analysis, cleaning, and output. According to the data flow path, further perform business flow analysis, which means combining the data flow with the actual business process to analyze the flow situation and application scenarios of data in different business processes. Through business flow analysis, the system can identify the specific role of each data source in business activities, and then extract specific business process information, including data requirements, data usage frequency, data output nodes, etc. of each business process, so as to obtain business process information.

[0026] Further parse all the data sources traversed and classify them into internal data sources and external data sources. Internal data sources are usually business systems or databases within the company, while external data sources include external third-party services, API interfaces, social media, etc. At the same time, according to the parameter collection frequency of each internal data source and combined with business process information for analysis, determine the first business dynamic demand information, which is used to reflect the real-time requirements, data processing frequency, etc. of the internal data source in the business process. Then, according to the parameter collection frequency of each external data source and combined with business process information for analysis, determine the second business dynamic demand information, which is used to reflect the synchronization frequency, real-time nature, and dependence on external data of the data.

[0027] Further, based on the first business dynamic demand information and the second business dynamic demand information, conduct comprehensive data analysis to determine the actual usage scenarios of each data source in the system. Exemplarily, internal data sources may be used for financial statement generation, and external data sources may be used for market trend analysis, so as to determine the internal data usage scenario information and the external data usage scenario information. The internal data usage scenario information and the external data usage scenario information are used to reflect the applications of different data sources in different business processes, such as data for decision support, real-time monitoring, etc. Finally, integrate and add the generated internal data usage scenario information and external data usage scenario information to the data scenario information library of the system to ensure that the scenario information library contains the latest business dynamic requirements and data flow information, and ensure that the platform can perform configuration optimization, decision support, and dynamic adjustment based on the complete data scenario information.

[0028] In a possible implementation manner, step A300 further includes step A380, allocate weights to the multiple configuration targets based on the internal data usage scenario information to obtain the first weight coefficient; execute step A390, decompose and sort the multiple configuration targets according to the first weight coefficient to obtain the internal configuration target sequence; execute step A3100, allocate weights to the multiple configuration targets based on the external data usage scenario information to obtain the second weight coefficient; execute step A3110, decompose and sort the multiple configuration targets according to the second weight coefficient to obtain the external configuration target sequence; execute step A3120, conduct configuration impact analysis according to the parameter association network to generate parameter configuration impact factors, and optimize the internal configuration target sequence combined with the external configuration target sequence based on the parameter configuration impact factors to obtain the multiple configuration optimization targets.

[0029] Since the performance requirements are high in some scenarios, while storage efficiency may be more emphasized in other scenarios, the role of each configuration target in the internal data usage scenario is extracted based on the above-generated internal data usage scenario information, and weight distribution is performed according to the importance of each configuration target in the internal data scenario. The first weight coefficient of each configuration target can be allocated by analyzing factors such as data processing requirements, real-time requirements, and data volume in the internal business process. Further, according to the first weight coefficient, multiple configuration targets are decomposed and sorted from largest to smallest to form an internal configuration target sequence. The basis for sorting is the importance and priority of each configuration target, and the target with a higher weight will be ranked at the front. The internal configuration target sequence is used to represent the priority order of each configuration target in the internal data scenario. Through weight distribution and sorting, the system determines the optimal configuration target sequence in the internal scenario.

[0030] Further, since some external data sources may require high security, while others may have high real-time requirements for data transmission, based on the above-obtained external data usage scenario information, the role of external data sources in the platform is analyzed, and weight distribution is performed according to the usage requirements of external data sources on this basis to obtain the second weight coefficient. Further, according to the second weight coefficient, multiple configuration targets are decomposed and sorted from largest to smallest. The configuration targets in the external scenario can include performance, storage, data quality, and security, and corresponding weights will be allocated according to the real-time nature, transmission frequency, dependence, etc. of external data, and the configuration target with a higher weight will have a higher priority in the sequence, thus generating an external configuration target sequence to show the configuration priority in the external data scenario.

[0031] Further, by analyzing the relevance between internal and external data sources and configuration targets, a parameter association network is constructed. The parameter association network is used to reflect the relationship between different data sources and configuration targets. Since some parameters have important impacts on both performance and security targets, impact analysis is performed through the parameter association network to generate the parameter configuration impact factor of each configuration target. The higher the impact factor, the greater the role of the configuration target in the overall system configuration. Based on the parameter configuration impact factor, the internal configuration target sequence and the external configuration target sequence are optimized, which means that some targets with high impact factors (such as performance and security) may appear in both internal and external configurations at the same time, and these targets will be optimized first to ensure that the final configuration optimization target can achieve the best effect in both internal and external data scenarios. Finally, through the above analysis, multiple configuration optimization targets are generated. The multiple configuration optimization targets balance the requirements in different scenarios through the combination of internal and external data usage scenarios and are optimized based on the parameter configuration impact factor.

[0032] Execute step A400, analyze the target parameter set according to the multiple configuration optimization objectives in combination with the multiple data classes, and construct a parameter configuration matrix;

[0033] In a possible implementation, step A400 further includes step A410, traverse the multiple data classes to match the target parameter set to determine multiple target parameter classes; execute step A420, calculate the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization objectives to generate multiple configuration contribution degrees. First, traverse all data classes, including structured data, unstructured data, real-time data, semi-structured data, etc., and match them with the parameters in the target parameter set. For each data class, select suitable target parameters for matching according to its characteristics. Exemplarily, the structured data class may match performance parameters such as query response time and data consistency. By traversing the data classes and matching the target parameter set, the parameters that have an important impact on a specific data class are identified. According to the matching results of the data classes and the target parameter set, the system classifies the parameters to determine multiple target parameter classes. The multiple target parameter classes may cover different dimensions such as performance, storage, data quality, and security.

[0034] For each target parameter class, calculate its contribution degree in the system in combination with multiple configuration optimization objectives, through weighted average or similar algorithms. Calculate the contribution degree of each target parameter class to each configuration optimization objective in combination with the characteristics of the data class and the influence of the parameter class on the optimization objective. The higher the contribution degree of a target parameter class, the greater its role in this optimization objective, so as to obtain multiple configuration contribution degrees, providing a basis for updating the parameter configuration influence factor.

[0035] Execute step A430, update the parameter configuration influence factor according to the multiple configuration contribution degrees to generate a parameter configuration priority list; in a possible implementation, step A430 further includes step A431, judge whether the multiple configuration contribution degrees are greater than a preset contribution threshold according to the first service dynamic requirement information and the second service dynamic requirement information; execute step A432, if there are configuration contribution degrees greater than the preset contribution threshold among the multiple configuration contribution degrees, generate an update queue and add the configuration contribution degrees greater than the preset contribution threshold to the update queue; execute step A433, traverse the configuration contribution degrees in the update queue to update the parameter configuration influence factor to generate a parameter configuration influence update factor; execute step A434, sort the multiple configuration objectives according to the parameter configuration influence update factor to generate the parameter configuration priority list.

[0036] First, combine the first business dynamic requirement information derived from the internal data usage scenario and the second business dynamic requirement information derived from the external data usage scenario, analyze the configuration contribution degree of each configuration target, and make a contribution degree judgment based on the business dynamic requirement information to ensure that the system can identify the configuration targets crucial for system optimization. Compare the configuration contribution degree of each configuration target with a preset contribution threshold. The contribution degree represents the contribution degree of this configuration target to the overall system optimization, and the contribution threshold is a preset reference standard used to screen out key configuration targets. If the contribution degree of a certain configuration target is greater than the threshold, then this configuration target has a high optimization value. When there are configuration targets with a configuration contribution degree greater than the preset contribution threshold, the system will generate an update queue, which is used to save all configuration targets with a contribution degree exceeding the threshold. Traverse all configuration targets, compare their contribution degrees with the threshold, and add the configuration targets with a contribution degree greater than the threshold to the update queue to ensure that subsequent impact factor updates are only concentrated on key configuration targets, improving the update efficiency.

[0037] Traverse each configuration target and its contribution degree in the update queue, process them one by one, and the contribution degree of each configuration target will be used as the basis for updating the impact factor during the update process. Update the corresponding parameter configuration impact factor through the contribution degree of the configuration target in each update queue. The update rule of the parameter configuration impact factor is based on the impact of the configuration target on the overall system optimization. The higher the contribution degree of the configuration target, the greater the update amplitude of its impact factor, thereby generating a new parameter configuration impact update factor, that is, the parameter configuration impact update factor, which is used to identify the optimization impact of each configuration target after the update. Finally, sort all configuration targets according to the parameter configuration impact update factor of each configuration target. The target with a larger impact update factor will be prioritized in the sorting, and the sorted configuration targets will generate a parameter configuration priority list, that is, the configuration targets with a higher priority will be given priority consideration for optimization to ensure that key configuration targets can be processed preferentially in the subsequent optimization process.

[0038] Execute step A440, and construct the parameter configuration matrix according to the parameter configuration priority list in combination with the target parameter set.

[0039] In a possible implementation manner, step A440 further includes step A441, determining row data based on the data scenario information in combination with the parameter configuration priority list; executing step A442, determining column data based on the target parameter set in combination with the parameter configuration priority list; executing step A443, performing matrix calculation according to the parameter configuration priority list in combination with the multiple configuration optimization targets to generate matrix element values; executing step A444, mapping and filling in the row data and the column data according to the matrix element values to construct the parameter configuration matrix.

[0040] First, extract various data sources and their related information from the data scenario information. The data scenario information may include internal and external data sources, data types, business dynamic requirements, etc. Determine the row data in the matrix according to the data scenario information, where each row in the row data represents a different data source or data category. Exemplarily, the row data may include structured data, unstructured data, real-time data, etc. Further, extract all target parameters that need to be configured from the target parameter set. The extracted parameters can cover different aspects such as performance, storage, data quality, and security. Combine the parameter configuration priority list to sort the target parameters and determine the column data in the matrix. The column data represents different configured target parameters, and each column corresponds to one parameter.

[0041] Further, extract the optimization objectives to be subjected to matrix calculation from multiple configuration optimization objectives. The weight of each optimization objective will affect the calculation of the matrix element values. By combining the parameter configuration priority list and the configuration optimization objectives, calculate for each combination of row data and column data to generate the element values in the matrix. The matrix element values represent the contribution degree or importance of a certain data category to a certain configuration objective. The matrix element values are used to characterize the role of each data category in each configuration objective. Further, map the determined row data and column data according to the calculated matrix element values and fill them into the matrix. By completing the mapping of the row data and column data, construct a parameter configuration matrix. The rows of the parameter configuration matrix represent data categories, the columns represent configuration objectives, and each cell in the matrix reflects the contribution degree of the data category to the configuration objective.

[0042] Next, execute step A500, perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

[0043] In a possible implementation manner, step A500 further includes step A510, perform similarity analysis on the multiple data categories based on the parameter configuration matrix in combination with the first weight coefficient to generate a first similarity coefficient; execute step A520, perform similarity analysis on the multiple data categories based on the parameter configuration matrix in combination with the second weight coefficient to generate a second similarity coefficient; execute step A530, perform configuration collaborative analysis on the target parameter set according to the first similarity coefficient to generate a first parameter configuration collaborative result; execute step A540, perform configuration collaborative analysis on the target parameter set according to the second similarity coefficient to generate a second parameter configuration collaborative result; execute step A550, process the first parameter configuration collaborative result in combination with the second parameter configuration collaborative result according to the expected filtering threshold to generate a recommended configuration result, and formulate the configuration plan according to the recommended configuration result.

[0044] First, perform similarity analysis on multiple data classes by combining the parameter configuration matrix with the first weight coefficient. This refers to calculating the similarity of each data class on different configuration targets by comparing each data class, calculating the similarity of each data class on different configuration targets, and generating a first similarity coefficient by weighted calculation of the similarity under different configuration targets based on the first weight coefficient. The first similarity coefficient is used to reflect the similarity between different data classes in the internal scenario through the calculation combining the first weight coefficient and the parameter configuration matrix. Subsequently, perform similarity analysis on the same data classes by combining the parameter configuration matrix with the second weight coefficient. This refers to generating the similarity of each pair of data classes under different configuration targets through weighted calculation and generating a second similarity coefficient to reflect the similarity of the data classes in the external scenario.

[0045] Furthermore, based on the first similarity coefficient, perform configuration collaboration analysis on the target parameter set. It is possible to identify data classes with similar optimization requirements according to the first similarity coefficient and collaboratively configure relevant target parameters, thereby obtaining the first parameter configuration collaboration result. The first parameter configuration collaboration result is used to reflect which target parameter configurations can be optimized collaboratively through similarity in the internal scenario, ensuring that data classes can share configuration optimization strategies under similar requirements.

[0046] Furthermore, based on the second similarity coefficient, perform configuration collaboration analysis on the target parameter set. It is possible to identify data classes with similar optimization requirements in the external scenario and collaboratively configure relevant parameters, thereby obtaining the second parameter configuration collaboration result. The second parameter configuration collaboration result is used to reflect which target parameter configurations can be collaboratively optimized through similarity in the external scenario, ensuring that data classes can achieve collaborative optimization of configurations under external optimization requirements.

[0047] Finally, combine the first parameter configuration collaboration result with the second parameter configuration collaboration result, and through the similarity analysis result, uniformly process the internal and external configuration collaboration requirements. This means that it is possible to set the expected filtering threshold based on retaining high contribution degrees, and at the same time screen the combined collaboration result, eliminate the collaboration results that do not meet the threshold, and achieve the purpose of retaining the configuration results with higher contribution degrees, improving the effectiveness of configuration optimization, generating a recommended configuration result, and defining specific configuration optimization steps, resource allocation strategies, and priority processing methods according to the recommended configuration result to generate a configuration plan, improving the overall resource utilization efficiency and performance of the system.

[0048] The embodiment of the present application solves the technical problems of lacking an intelligent data configuration scheme, being difficult to automatically adjust parameters, resulting in resource waste and insufficient data processing performance, and achieves the technical effects of dynamically adjusting data configuration to improve data processing efficiency and optimizing configuration parameter management.

[0049] In the above text, refer to Figure 1A method for managing configuration parameters of multi-source data according to an embodiment of the present application is described in detail. Next, a platform for managing configuration parameters of multi-source data according to an embodiment of the present application will be described with reference to Figure 2 A platform for managing configuration parameters of multi-source data according to an embodiment of the present application will be described.

[0050] A platform for managing configuration parameters of multi-source data according to an embodiment of the present application is used to solve the technical problems of lack of intelligent data configuration solutions, difficulty in automatically adjusting parameters, resulting in resource waste and insufficient data processing performance, and achieves the technical effects of dynamically adjusting data configuration to improve data processing efficiency and optimizing configuration parameter management. A platform for managing configuration parameters of multi-source data includes: a data identification module 10, a parameter determination module 20, a decomposition and optimization module 30, a matrix construction module 40, and a collaborative filtering module 50.

[0051] The data identification module 10 is used to traverse multiple data sources for type identification to obtain multiple data classes, and determine multiple configuration targets of multi-source data according to the multiple data classes;

[0052] The parameter determination module 20 is used to determine a target parameter set based on the multiple data sources in combination with the multiple data classes;

[0053] The decomposition and optimization module 30 is used to introduce data scenario information, decompose and optimize the multiple configuration targets to obtain multiple configuration optimization targets;

[0054] The matrix construction module 40 is used to analyze the target parameter set according to the multiple configuration optimization targets in combination with the multiple data classes to construct a parameter configuration matrix;

[0055] The collaborative filtering module 50 is used to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

[0056] Next, the specific configuration of the decomposition and optimization module 30 will be described in detail. As described above, for the data scenario information, the decomposition and optimization module 30 may further include: traversing the multiple data sources for data collection and identification to determine a data collection chain, and obtaining a data flow path according to the data collection chain; performing business flow analysis according to the data flow path to obtain business process information; parsing the multiple data sources to obtain multiple internal data sources and multiple external data sources; analyzing according to the parameter collection frequencies of the multiple internal data sources in combination with the business process information to determine first business dynamic requirement information; analyzing according to the parameter collection frequencies of the multiple external data sources in combination with the business process information to determine second business dynamic requirement information; performing data analysis according to the first business dynamic requirement information and the second business dynamic requirement information to determine internal data usage scenario information and external data usage scenario information; and adding the internal data usage scenario information and the external data usage scenario information to the data scenario information.

[0057] Next, the specific configuration of the decomposition and optimization module 30 will be described in detail. As described above, introducing the data scenario information to decompose and optimize the multiple configuration targets to obtain multiple configuration optimization targets, the decomposition and optimization module 30 may further include: performing weight assignment on the multiple configuration targets based on the internal data usage scenario information to obtain a first weight coefficient; decomposing and sorting the multiple configuration targets according to the first weight coefficient to obtain an internal configuration target sequence; performing weight assignment on the multiple configuration targets based on the external data usage scenario information to obtain a second weight coefficient; decomposing and sorting the multiple configuration targets according to the second weight coefficient to obtain an external configuration target sequence; performing configuration impact analysis according to a parameter association network to generate a parameter configuration impact factor, and optimizing the internal configuration target sequence in combination with the external configuration target sequence based on the parameter configuration impact factor to obtain the multiple configuration optimization targets.

[0058] Next, the specific configuration of the matrix construction module 40 will be described in detail. As described above, analyzing the target parameter set according to the multiple configuration optimization targets in combination with the multiple data classes to construct a parameter configuration matrix, the matrix construction module 40 may further include: traversing the multiple data classes to match the target parameter set to determine multiple target parameter classes; calculating the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization targets to generate multiple configuration contribution degrees; updating the parameter configuration impact factor according to the multiple configuration contribution degrees to generate a parameter configuration priority list; and constructing the parameter configuration matrix according to the parameter configuration priority list in combination with the target parameter set.

[0059] Next, the specific configuration of the matrix construction module 40 will be described in detail. As described above, the parameter configuration influence factor is updated according to the multiple configuration contribution degrees, and a parameter configuration priority list is generated. The matrix construction module 40 may further include: determining whether the multiple configuration contribution degrees are greater than a preset contribution threshold according to the first service dynamic requirement information and the second service dynamic requirement information; if there is a configuration contribution degree greater than the preset contribution threshold among the multiple configuration contribution degrees, generating an update queue, and adding the configuration contribution degrees greater than the preset contribution threshold to the update queue; traversing the configuration contribution degrees in the update queue to update the parameter configuration influence factor, generating a parameter configuration influence update factor; sorting the multiple configuration targets according to the parameter configuration influence update factor to generate the parameter configuration priority list.

[0060] Next, the specific configuration of the matrix construction module 40 will be described in detail. As described above, a parameter configuration matrix is constructed according to the parameter configuration priority list in combination with the target parameter set. The matrix construction module 40 may further include: determining row data based on the data scenario information in combination with the parameter configuration priority list; determining column data based on the target parameter set in combination with the parameter configuration priority list; performing matrix calculation according to the parameter configuration priority list in combination with the multiple configuration optimization targets to generate matrix element values; mapping and filling the row data and the column data according to the matrix element values to construct the parameter configuration matrix.

[0061] Next, the specific configuration of the collaborative filtering module 50 will be described in detail. As described above, collaborative filtering is performed according to the parameter configuration matrix to formulate a configuration plan. The collaborative filtering module 50 may further include: performing similarity analysis on the multiple data classes based on the parameter configuration matrix in combination with the first weight coefficient to generate a first similarity coefficient; performing similarity analysis on the multiple data classes based on the parameter configuration matrix in combination with the second weight coefficient to generate a second similarity coefficient; performing configuration collaborative analysis on the target parameter set according to the first similarity coefficient to generate a first parameter configuration collaborative result; performing configuration collaborative analysis on the target parameter set according to the second similarity coefficient to generate a second parameter configuration collaborative result; processing the first parameter configuration collaborative result in combination with the second parameter configuration collaborative result according to an expected filtering threshold to generate a recommended configuration result, and formulating the configuration plan according to the recommended configuration result.

[0062] A configuration parameter management platform for multi-source data provided by an embodiment of the present application can execute a configuration parameter management method for multi-source data provided by any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method.

[0063] Although this application makes various references to certain modules in the platform according to embodiments of this application, however, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and are not used to limit the protection scope of this application.

[0064] The above specific implementation manners do not constitute a limitation on the protection scope of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims

1. A configuration parameter management method for multi-source data, characterized in that: The method comprises: Traversing multiple data sources to perform type identification, obtaining multiple data classes, and determining multiple configuration targets of the multi-source data according to the multiple data classes; determining a target parameter set based on the plurality of data sources in combination with the plurality of data classes; Introducing data scenario information, decomposing and optimizing the multiple configuration targets, and obtaining multiple configuration optimization targets, wherein the multiple configuration optimization targets are obtained, including: Based on the internal data usage scenario information, weights are assigned to the multiple configuration targets to obtain a first weight coefficient; Decomposing and sorting the multiple configuration targets according to the first weight coefficient to obtain an internal configuration target sequence; Based on the external data usage scenario information, weights are assigned to the multiple configuration targets to obtain a second weight coefficient; Decomposing and sorting the multiple configuration targets according to the second weight coefficient to obtain an external configuration target sequence; Performing configuration impact analysis according to the parameter association network to generate parameter configuration impact factors, optimizing the internal configuration target sequence in combination with the external configuration target sequence based on the parameter configuration impact factors to obtain the multiple configuration optimization targets; Analyzing the target parameter set according to the multiple configuration optimization objectives in combination with the multiple data classes to construct a parameter configuration matrix, wherein constructing the parameter configuration matrix includes: Traversing the multiple data classes to match the target parameter set, and determining multiple target parameter classes; Calculating the target parameter set based on the multiple target parameter classes in combination with the multiple configuration optimization objectives to generate multiple configuration contribution degrees; Update the parameter configuration influencing factors according to the multiple configuration contribution degrees, and generate a parameter configuration priority list; Constructing the parameter configuration matrix according to the parameter configuration priority list in combination with the target parameter set; Perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set, wherein the formulating of the configuration plan includes: Perform similarity analysis on the multiple data classes based on the parameter configuration matrix in combination with the first weight coefficient to generate a first similarity coefficient; Perform similarity analysis on the multiple data classes based on the parameter configuration matrix combined with the second weight coefficient to generate a second similarity coefficient; Performing configuration coordination analysis on the target parameter set according to the first similarity coefficient to generate a first parameter configuration coordination result; Performing configuration coordination analysis on the target parameter set according to the second similarity coefficient to generate a second parameter configuration coordination result; The first parameter configuration collaboration result is combined with the second parameter configuration collaboration result and processed according to the expected filtering threshold to generate a recommended configuration result, and the configuration plan is formulated according to the recommended configuration result.

2. A configuration parameter management method for multi-source data according to claim 1, characterized in that: The data scene information method includes: Traversing the multiple data sources to perform data collection and identification, determine a data collection chain, and obtain a data flow path according to the data collection chain; Perform business flow analysis according to the data flow path to obtain business process information; Parsing the multiple data sources to obtain multiple internal data sources and multiple external data sources; Analyze the parameter collection frequency of the multiple internal data sources in combination with the business process information to determine the first business dynamic demand information; Analyze the parameter collection frequencies of the multiple external data sources in combination with the business process information to determine the second business dynamic demand information; Perform data analysis according to the first business dynamic demand information and the second business dynamic demand information to determine internal data usage scenario information and external data usage scenario information; The internal data usage scenario information and the external data usage scenario information are added to the data scenario information.

3. A configuration parameter management method for multi-source data according to claim 1, characterized in that: The parameter configuration influencing factor is updated according to the multiple configuration contribution degrees to generate a parameter configuration priority list, the method comprising: Determining whether the multiple configuration contribution degrees are greater than a preset contribution threshold according to the first business dynamic demand information and the second business dynamic demand information; If a configuration contribution degree greater than the preset contribution threshold exists among the multiple configuration contributions, an update queue is generated, and the configuration contribution degree greater than the preset contribution threshold is added to the update queue; Traversing the configuration contribution of the update queue to update the parameter configuration impact factor, and generating a parameter configuration impact update factor; The multiple configuration targets are sorted according to the parameter configuration impact update factors to generate the parameter configuration priority list.

4. A configuration parameter management method for multi-source data according to claim 1, characterized in that: Constructing a parameter configuration matrix according to the parameter configuration priority list and the target parameter set, the method comprising: Determine row data based on the data scenario information in combination with the parameter configuration priority list; Determining column data based on the target parameter set in combination with the parameter configuration priority list; Perform matrix calculation according to the parameter configuration priority list and the multiple configuration optimization objectives to generate matrix element values; The row data and the column data are mapped and filled in according to the matrix element values ​​to construct the parameter configuration matrix.

5. A configuration parameter management platform for multi-source data, characterized in that: The platform is used to implement a configuration parameter management method for multi-source data according to any one of claims 1 to 4, and the platform includes: A data identification module, the data identification module is used to traverse multiple data sources to perform type identification, obtain multiple data classes, and determine multiple configuration targets of multi-source data according to the multiple data classes; a parameter determination module, the parameter determination module being used to determine a target parameter set based on the multiple data sources in combination with the multiple data classes; A decomposition and optimization module, wherein the decomposition and optimization module is used to introduce data scenario information, decompose and optimize the multiple configuration targets, and obtain multiple configuration optimization targets; A matrix construction module, the matrix construction module is used to analyze the target parameter set according to the multiple configuration optimization objectives in combination with the multiple data types to construct a parameter configuration matrix; A collaborative filtering module is used to perform collaborative filtering according to the parameter configuration matrix, formulate a configuration plan, and execute the configuration plan to dynamically manage the target parameter set.

Images