AI intelligent decision-making method for unstructured data processing

Abstract

The present invention discloses an AI intelligent decision-making method for unstructured data processing, which involves data processing related fields. The method includes: collecting various historical unstructured data sets for various unstructured data source ends of target enterprises; performing data dynamics and format consistency analysis and aggregation, and establishing the aggregation results of the first data source end; performing file management process analysis based on the usage scenario, and optimizing the aggregation results of the first data source end according to the analysis results to generate the aggregation results of the second data source end; building an unstructured data processing middle platform, and performing intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source end. The method solves the technical problems of low efficiency, insufficient accuracy, difficulty in adapting to the ever-changing data processing needs, and lack of flexibility in the existing unstructured data processing, and achieves the technical effect of improving the efficiency and accuracy of data processing, and improving the flexibility and scalability of the system.

Description

Technical Field

[0001] The present application relates to data processing related fields, and in particular to an AI intelligent decision-making method for unstructured data processing. Background Art

[0002] With the rapid development of information technology, enterprises generate a large amount of unstructured data every day, such as text, logs, sensor data, videos, audio, images, etc. These data contain rich information and are of great significance to the decision-making, business optimization and risk management of enterprises. However, the complexity, diversity and massiveness of unstructured data pose huge challenges to data processing. Therefore, how to efficiently process unstructured data and tap its value has become an urgent problem for enterprises. Traditionally, enterprises usually rely on manual classification, organization and analysis of data, but this method is inefficient and prone to errors. For specific business needs, enterprises will also develop customized data processing systems, but this method is costly, has a long cycle, and is difficult to adapt to changing data processing needs.

[0003] Among the current related technologies, unstructured data processing has technical problems such as low efficiency, insufficient accuracy, difficulty in adapting to ever-changing data processing needs, and lack of flexibility. Summary of the invention

[0004] The present application provides an AI intelligent decision-making method for unstructured data processing, collects historical unstructured data sets from each unstructured data source of the target enterprise, analyzes the data dynamics and format consistency of the collected historical data, establishes the aggregation result of the first data source, analyzes the file management process based on the usage scenarios of the historical data, optimizes the aggregation result of the first data source according to the analysis result, generates the aggregation result of the second data source, builds an unstructured data processing middle platform based on the aggregation result of the second data source, and performs intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source, and other technical means, thereby achieving the technical effect of improving the efficiency and accuracy of data processing and improving the flexibility and scalability of the system.

[0005] This application provides an AI intelligent decision-making method for unstructured data processing, including:

[0006] Collect various historical unstructured data sets from various unstructured data sources of the target enterprise; perform data dynamics and format consistency analysis and aggregation on the various historical unstructured data sets to establish a first data source aggregation result; analyze the file management process based on the usage scenarios of the various historical unstructured data sets, and optimize the aggregation results of the first data source based on the analysis results to generate a second data source aggregation result; build an unstructured data processing middle platform based on the aggregation results of the second data source, and perform intelligent decision-making processing on the real-time unstructured data generated by the various unstructured data sources.

[0007] In a possible implementation, data dynamics and format consistency analysis and aggregation are performed on each of the historical unstructured data sets to establish a first data source end aggregation result, and the following processing is performed:

[0008] Perform dynamic indicator statistics on each of the historical unstructured data sets according to a preset dynamic indicator set to generate each dynamic indicator value set; obtain a structured processing template for each of the historical unstructured data sets, perform consistency analysis on the structured processing template, and generate a format consistency judgment result; perform similarity analysis on each of the historical unstructured data sets based on the each dynamic indicator value set and the format consistency judgment result to generate an unstructured data source end combination whose similarity is greater than a preset similarity threshold; aggregate each of the unstructured data source ends using the unstructured data source end combination to generate the first data source end aggregation result.

[0009] In a possible implementation, the following processing is performed:

[0010] The preset dynamic indicator set includes at least data type, data volume, data volume growth rate, access frequency and access mode.

[0011] In a possible implementation, the file management process is analyzed based on the usage scenarios of each historical unstructured data set, the aggregation result of the first data source end is aggregated and optimized according to the analysis result, the aggregation result of the second data source end is generated, and the following processing is performed:

[0012] Based on the usage scenarios of each historical unstructured data set, a historical file management process is collected, wherein the historical file management process includes multiple process nodes; according to the historical file management process, a complexity analysis of the file management process is performed on the first data source end combination in the first data source end aggregation result, and a first complexity index is generated; it is determined whether the first complexity index meets the preset complexity index, and if not, a first aggregation optimization instruction is generated; based on the first aggregation optimization instruction, with the goal of reducing the complexity of the file management process, the first data source end combination is aggregated and optimized to generate a first data source end optimization result; the first data source end optimization result is added to the second data source end aggregation result.

[0013] In a possible implementation, a complexity analysis of the file management process is performed on the first data source end combination in the first data source end aggregation result according to the historical file management process to generate a first complexity index, and the following processing is performed:

[0014] Based on the multiple process nodes within the historical file management process, a data lifecycle analysis is performed on the first data source terminal combination, the differences in the lifecycle and the diversity of the cleanup strategies are identified and integrated to generate the lifecycle complexity; based on the multiple process nodes within the historical file management process, a complexity analysis of the permission management is performed on the first data source terminal combination to generate the permission management complexity; the lifecycle complexity and the permission management complexity are weightedly integrated to generate the first complexity index.

[0015] In a possible implementation, based on the first aggregation optimization instruction, with the goal of reducing the complexity of the file management process, the first data source end combination is aggregated and optimized to generate a first data source end optimization result, and the following processing is performed:

[0016] According to the first aggregation optimization instruction, binary classification aggregation is performed based on the similarity of historical file management processes to generate a first optimization aggregation result; whether the complexity of the file management process of the first optimization aggregation result meets the preset complexity index is analyzed; if not, the first optimization aggregation result is continued to be binary classification aggregation, and so on, until the optimization aggregation result that meets the preset complexity index is obtained, and the optimization result of the first data source end is generated.

[0017] In a possible implementation, an unstructured data processing platform is built based on the aggregation result of the second data source, and intelligent decision-making processing is performed on the real-time unstructured data generated by each unstructured data source, and the following processing is performed:

[0018] Extract multiple data source combinations within the aggregation result of the second data source; extract dynamic indicator values ​​for the multiple data source combinations respectively, configure processing resources of the data center according to the dynamic indicator value extraction results, and generate multiple data source resource configuration results; extract file management processes for the multiple data source combinations respectively, build multiple processing nodes according to the file management process, and establish a mapping relationship between the data source and the processing node; construct the unstructured data processing center with the resource configuration results, the multiple processing nodes and the mapping relationship; based on the mapping relationship, call the multiple processing nodes to perform decision processing on the real-time unstructured data generated by each unstructured data source.

[0019] In a possible implementation, the following processing is also performed:

[0020] When adding a new unstructured data source, determine the preset data dynamics, preset data structured format and preset file management process of the new unstructured data source, combine the new unstructured data source with any data source in the aggregation result of the second data source to perform data dynamics, format consistency analysis, and file management process aggregation analysis to determine whether there is a candidate aggregation combination; if so, adjust the unstructured data processing middle station with the candidate aggregation combination; if not, build an independent aggregation result for the new unstructured data source, and then adjust the unstructured data processing middle station.

[0021] The AI ​​intelligent decision-making method for unstructured data processing proposed in this application is to first collect various historical unstructured data sets from various unstructured data sources of the target enterprise, then perform data dynamics and format consistency analysis and aggregation on each historical unstructured data set, establish the aggregation result of the first data source, and then analyze the file management process based on the usage scenarios of each historical unstructured data set, and optimize the aggregation result of the first data source according to the analysis results to generate the aggregation result of the second data source, and finally build an unstructured data processing middle platform based on the aggregation result of the second data source, and perform intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source, thereby achieving the technical effect of improving the efficiency and accuracy of data processing and improving the flexibility and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the method according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0023] Figure 1 A flowchart of an AI intelligent decision-making method for unstructured data processing provided in an embodiment of the present application.

[0024] Figure 2 A schematic diagram of a process for performing aggregation optimization on the aggregation results of the first data source end in the AI ​​intelligent decision-making method for unstructured data processing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0027] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0028] The present application embodiment provides an AI intelligent decision-making method for unstructured data processing, such as Figure 1 As shown, the method includes:

[0029] Step S100, collect each historical unstructured data set for each unstructured data source of the target enterprise. Specifically, determine all unstructured data sources of the target enterprise (such as large manufacturing enterprises, retail enterprises, etc.), which may include equipment sensors, log files, video monitoring systems, audio recording devices, image storage systems, etc. Collect historical unstructured data sets from these data sources, which include past equipment status records, transaction logs, monitoring videos, customer feedback recordings, product pictures, etc. Among them, unstructured data refers to data that has no fixed format or pattern, such as text, images, audio, video, etc., as opposed to structured data (such as tables in a database). The data source refers to the source or system that generates or stores unstructured data.

[0030] Step S200, perform data dynamics and format consistency analysis and aggregation on each of the historical unstructured data sets, and establish a first data source aggregation result. Specifically, data dynamics refers to the characteristics of data changing over time, including the speed at which data is generated, access patterns, etc.; format consistency refers to the similarity in format or structure between different data sets. Analyze the collected historical unstructured data sets, and evaluate the dynamics of the data, including the growth rate of data volume, etc., to determine the activity and usage pattern of the data. Check the format consistency of the data, that is, whether there is a unified structured processing template or format standard between different data sets. Based on these analyses, data sources with similar dynamics and format consistency are aggregated to form a first data source aggregation result.

[0031] In a possible implementation, the data dynamics and format consistency analysis and aggregation are performed on each of the historical unstructured data sets to establish the aggregation result of the first data source end. Step S200 further includes step S210, performing dynamic indicator statistics on each of the historical unstructured data sets according to a preset dynamic indicator set to generate each dynamic indicator value set. Specifically, the preset dynamic indicator set is a set of indicators determined in advance for measuring the dynamic characteristics of data, and these indicators can reflect the dynamic characteristics of data. For each historical unstructured data set, statistics are performed according to the preset dynamic indicator set to calculate the specific value of each indicator. The calculated dynamic indicator values ​​are combined into a set, which represents the dynamic characteristics of the data set. Step S220, obtain the structured processing templates of each of the historical unstructured data sets, perform consistency analysis on the structured processing templates, and generate a format consistency judgment result. Specifically, obtain the structured processing template of each historical unstructured data set, which defines the format and structure of the data. Perform consistency analysis on these structured processing templates to check whether there are differences between them, such as whether the field name, data type, data format, etc. are consistent. According to the results of consistency analysis, a format consistency judgment result is generated, which indicates the consistency degree of each data set in terms of format. Among them, the structured processing template is a template that defines how unstructured data is converted into structured data (such as database records).

[0032] Step S230, based on the respective dynamic indicator value sets and the format consistency judgment results, similarity analysis is performed on the respective historical unstructured data sets, and a combination of unstructured data sources with a similarity greater than a preset similarity threshold is generated. Specifically, in combination with the dynamic indicator value set and the format consistency judgment results, a similarity analysis is performed on the respective historical unstructured data sets. Similarity analysis includes calculating the similarity between data sets (such as cosine similarity, Euclidean distance, etc.), and comparing these similarities with a preset similarity threshold (a threshold used to judge the similarity between data sets, and data sets exceeding this threshold are considered to have sufficient similarity). Data sets with similarities greater than the preset similarity threshold are combined together to form an unstructured data source end combination. Step S240, the respective unstructured data source ends are aggregated with the unstructured data source end combination to generate the first data source end aggregation result. Specifically, according to the combination of unstructured data sources obtained in step S230, each unstructured data source is aggregated, including operations such as merging, classifying, and labeling data sets to form a data source set that is easier to manage and analyze, and finally generate a first data source aggregation result, which includes a combination of data sources with similar dynamics and format consistency. This implementation method reduces the complexity and redundancy of data processing and improves the efficiency of data processing by aggregating data sources with similar dynamics and format consistency. The aggregated data sources are easier to manage and maintain in a unified manner, reducing the difficulty and cost of data management.

[0033] In a possible implementation, step S210 further includes step S211, wherein the preset dynamic indicator set includes at least data type, data volume, data volume growth rate, access frequency and access mode. Specifically, for each historical unstructured data set, its data type, i.e., the essential attribute of the data, such as text, image, audio, video, log, sensor data, etc., is identified. The data volume of each data set is counted, which can be the number of data files or the total size of the data content (such as MB, GB, etc.), and the data volume is used to determine the scale of the data and the resources required to store and process the data. The data volume growth rate of each data set within a certain time range is calculated, which reflects the growth trend and speed of the data. For application scenarios such as predictive maintenance, the growth of data volume means an increase in the activity of the device or system, which requires more frequent monitoring and analysis. The number and frequency of accesses to each data set are recorded, which reflects the usage and importance of the data. For frequently accessed data sets, a more efficient storage and retrieval mechanism is required. The access mode of each data set is analyzed, such as whether it is through API, database query, file system, etc., to determine the access mode of the data and possible performance bottlenecks. This implementation method clarifies the specific content of the preset dynamic indicator set. By counting and analyzing dynamic indicators, we can have a deeper understanding of the characteristics of the data, including the type, scale, growth trend and usage pattern of the data, so that we can formulate more reasonable data management strategies, such as data backup, storage optimization, access control, etc., to improve the efficiency and security of data management.

[0034] Step S300, analyze the file management process based on the usage scenarios of each historical unstructured data set, and optimize the aggregation results of the first data source end according to the analysis results to generate the aggregation results of the second data source end. Specifically, the usage scenario refers to the specific business or decision-making scenario in which the unstructured data is used, such as predictive maintenance, market strategy optimization, etc., and the file management process in these scenarios is analyzed, including process nodes such as data storage, access, backup, and deletion. Based on these analyses, the aggregation results of the first data source end are further optimized to reduce the complexity of the file management process and improve the efficiency of data processing. The optimized result is the aggregation result of the second data source end.

[0035] like Figure 2As shown, in a possible implementation, the file management process is analyzed based on the usage scenarios of each historical unstructured data set, and the aggregation result of the first data source end is aggregated and optimized according to the analysis result to generate the aggregation result of the second data source end. Step S300 further includes step S310, collecting the historical file management process based on the usage scenarios of each historical unstructured data set, wherein the historical file management process includes multiple process nodes. Specifically, according to the actual usage scenarios of each historical unstructured data set (such as predictive maintenance, market analysis, customer feedback, etc.), the demand and usage of data in these scenarios are determined. For each usage scenario, the file management process used by the enterprise in the past when processing similar data is collected, that is, a series of operation steps formulated by the enterprise for managing and processing data, including multiple links such as data reception, classification, labeling, encryption, archiving, and deletion. The collected file management process is sorted out to determine the specific content and operation steps of each process node.

[0036] Step S320, performing a complexity analysis of the file management process for the first data source end combination in the first data source end aggregation result according to the historical file management process, and generating a first complexity index. Specifically, a data source end combination is selected from the first data source end aggregation result as an analysis object. According to the historical file management process, a complexity analysis is performed on the file management process of each data source end combination, including evaluating the number of process nodes, the difficulty of operation of each node, the dependencies between processes, etc. Based on the analysis results, an index (first complexity index) representing the complexity of the file management process is generated for each data source end combination.

[0037] Step S330, determine whether the first complexity index meets the preset complexity index, if not, generate a first aggregation optimization instruction. Specifically, the preset complexity index is a reference value for evaluating the complexity of the file management process. Compare the first complexity index of each data source end combination with the preset complexity index. If the first complexity index does not meet the preset complexity index, it indicates that the file management process complexity of the first data source end combination is large, and then generate an instruction (first aggregation optimization instruction) to instruct the data source end combination to perform aggregation optimization. Step S340, based on the first aggregation optimization instruction, with the goal of reducing the complexity of the file management process, perform aggregation optimization on the first data source end combination, and generate a first data source end optimization result. Specifically, with the goal of reducing the complexity of the file management process, determine specific optimization measures according to the first aggregation optimization instruction. Perform optimization measures such as merging, reclassifying, and simplifying operations on the data sources in the first data source end combination to reduce process nodes, reduce operation difficulty, and reduce dependencies between processes. After optimization, generate a new data source end combination as the first data source end optimization result. Step S350, add the first data source end optimization result to the second data source end aggregation result. Specifically, the optimized result of the first data source end after aggregation optimization is added to the aggregated result of the second data source end to replace the original data source end combination with higher complexity. While adding the optimized results, the integrity and consistency of the data are ensured to avoid data loss or damage. This implementation method improves data management efficiency by reducing the complexity of the file management process, reducing unnecessary operation steps and process nodes. The optimized data management process can respond to business needs more quickly, improve data availability and utilization, and thus enhance the value of data.

[0038] In a possible implementation, the complexity of the file management process is analyzed for the first data source end combination in the first data source end aggregation result according to the historical file management process to generate a first complexity index. Step S320 further includes step S321, performing data life cycle analysis on the first data source end combination based on multiple process nodes in the historical file management process, identifying the differentiation of the life cycle and the diversity of the cleaning strategy and integrating them to generate the life cycle complexity. Specifically, according to multiple process nodes in the historical file management process, the life cycle stages of each data set in the first data source end combination are identified, and these stages include data generation, storage, use, archiving and deletion. The differences between different data sets in different life cycle stages are analyzed, including differences in the life cycle length, storage requirements, access frequency, etc. of the data sets. For example, some data sets need to be stored for a long time, while other data sets only need to be retained for a short time. The cleaning strategies adopted by each data set at the end of the life cycle are identified, including deletion, archiving, migration, etc. The diversity of cleaning strategies for different data sets is analyzed, including the applicability, execution frequency, cost-effectiveness, etc. of the strategies. The differentiation of the life cycle and the diversity of the cleaning strategy are integrated to evaluate the complexity of the data life cycle management of the first data source end combination.

[0039] Step S322, based on the multiple process nodes in the historical file management process, the complexity of the permission management of the first data source end combination is analyzed to generate the complexity of the permission management. Specifically, the permission settings of each data set in the first data source end combination are analyzed, including the access rights and modification rights of the data. The management rules applicable to these permission settings are identified, including the permission allocation of different users or user groups, the permission change process, etc. According to the diversity of permission settings and management rules, the complexity of permission management is evaluated, including the number of rules, the mutual dependence of rules, etc. Step S323, the life cycle complexity and the permission management complexity are weighted and integrated to generate the first complexity index. Specifically, the weights of the life cycle complexity and the permission management complexity are determined, and the life cycle complexity and the permission management complexity are weighted and calculated according to the determined weights to generate the first complexity index, which is used to measure the complexity of the file management process of the first data source end combination. This implementation method comprehensively evaluates the complexity of the data management process by performing complexity analysis on the data life cycle and permission management respectively, and integrating the complexity of the two aspects, providing an accurate basis for subsequent aggregation optimization.

[0040] In a possible implementation, based on the first aggregation optimization instruction, with the goal of reducing the complexity of the file management process, the first data source end combination is aggregated and optimized to generate a first data source end optimization result, and step S340 further includes step S341, according to the first aggregation optimization instruction, binary classification aggregation is performed based on the similarity of the historical file management process to generate a first optimization aggregation result. Specifically, in the historical file management process, by comparing the functions, operations, input and output of the process nodes, similar process nodes or steps are identified. Based on the identified similarity, the data sources in the first data source end combination are binary classified and aggregated according to the similarity of the historical file management process, that is, the data sources with similar file management processes are classified into the same category to form a first optimization aggregation result. Step S342, analyze whether the complexity of the file management process of the first optimization aggregation result meets the preset complexity index. Specifically, according to step S320, the file management process of each data source end combination in the first optimization aggregation result is evaluated for complexity, and the complexity of the file management process obtained by the evaluation is compared with the preset complexity index. According to the comparison result, determine whether the complexity of the file management process of the first optimization aggregation result meets the preset complexity index. Step S343, if not, continue to perform binary classification aggregation on the first optimization aggregation result, and so on, until the optimization aggregation result that meets the preset complexity index is obtained, and the optimization result of the first data source is generated. Specifically, if the complexity of the file management process of the first optimization aggregation result does not meet the preset complexity index, continue to perform binary classification aggregation on the first optimization aggregation result, that is, divide the data source end into two categories again according to the similarity of the file management process to form a new optimization aggregation result. Repeat the above binary classification aggregation and complexity evaluation process until the optimization aggregation result that meets the preset complexity index is obtained. The optimization aggregation result that meets the preset complexity index is used as the optimization result of the first data source. This implementation method continuously optimizes the combination method of the data source end through binary classification aggregation and iterative process, reduces the complexity of the file management process, improves the efficiency and accuracy of data management, and reduces unnecessary management costs.

[0041] Step S400, build an unstructured data processing middle platform based on the aggregation result of the second data source, and perform intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source. Specifically, based on the aggregation result of the second data source, build an unstructured data processing middle platform, which is responsible for receiving, storing, processing and analyzing real-time data from each unstructured data source. Intelligent decision-making processing of unstructured data is achieved by configuring processing resources, building processing nodes, etc. For example, for data from device sensors, the middle platform can perform real-time analysis to predict equipment failures; for social media content, the middle platform can perform sentiment analysis to optimize marketing strategies. The embodiment of the present application adopts technical means such as collecting historical unstructured data sets from each unstructured data source of the target enterprise, analyzing the data dynamics and format consistency of the collected historical data, establishing the aggregation result of the first data source, analyzing the file management process based on the usage scenarios of the historical data, optimizing the aggregation result of the first data source according to the analysis results, generating the aggregation result of the second data source, building an unstructured data processing middle platform based on the aggregation result of the second data source, and performing intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source, thereby achieving the technical effect of improving the efficiency and accuracy of data processing and improving the flexibility and scalability of the system.

[0042] In a possible implementation, an unstructured data processing middle platform is built based on the aggregation result of the second data source end, and intelligent decision-making processing is performed on the real-time unstructured data generated by each unstructured data source end. Step S400 further includes step S410, extracting multiple data source end combinations in the aggregation result of the second data source end. Specifically, the aggregation result of the second data source end includes each data source end contained therein and their mutual relationships. In the aggregation result of the second data source end, multiple data source end combinations are identified and extracted. Step S420, dynamic indicator values ​​are extracted for each of the multiple data source end combinations, and processing resource configuration of the data middle platform is performed according to the dynamic indicator value extraction result to generate a multi-data source end resource configuration result. Specifically, for each data source end combination, its corresponding dynamic indicator value is extracted, and the extracted dynamic indicator value is analyzed to determine the data characteristics and processing requirements of each data source end combination. According to the analysis results, appropriate processing resources, such as computing power, storage space, network bandwidth, etc., are allocated to the middle platform to meet the processing requirements of each data source end combination.

[0043] Step S430, extract the file management process for the multiple data source end combinations respectively, build multiple processing nodes according to the file management process, and establish a mapping relationship between the data source end and the processing node. Specifically, analyze and determine the file management process of each data source end combination, including data collection, storage, processing, distribution and other links. According to the file management process, design and build multiple processing nodes, each processing node is responsible for specific data processing tasks, such as data cleaning, conversion, analysis, etc. Establish appropriate connections and dependencies between processing nodes to optimize the data processing process and improve processing efficiency. Determine one or more processing nodes for each data source end combination, and establish a mapping relationship between the data source end and the processing node to ensure that the data can be processed and distributed correctly. Step S440, construct the unstructured data processing middle station with the resource configuration results, the multiple processing nodes and the mapping relationship. Specifically, integrate the resource configuration results, the multiple processing nodes and the mapping relationship to form a complete unstructured data processing middle station architecture. After the construction is completed, verify and test the architecture to ensure that it can operate normally and meet the expected processing requirements.

[0044] Step S450, based on the mapping relationship, call the multiple processing nodes to perform decision processing on the real-time unstructured data generated by each unstructured data source. Specifically, the data generated by each unstructured data source is received in real time. According to the mapping relationship, the received data is assigned to the corresponding processing node for processing. The processing node performs intelligent decision processing on the data according to the preset algorithms and models, such as classification, identification, prediction, etc. The processing results are output to the required applications or services to support the decision-making and operation of the enterprise. This implementation method ensures that the middle station can efficiently process various types of data by extracting the dynamic indicator values ​​of the data source combination and allocating resources accordingly. By establishing a mapping relationship between the data source and the processing node, the data processing process can be flexibly adjusted to meet different data processing needs. By building an unstructured data processing middle station, the speed and accuracy of data processing are improved, providing real-time and accurate data support for enterprises.

[0045] In a possible implementation, step S400 further includes step S460, when adding a new unstructured data source, determining the preset data dynamics, preset data structured format and preset file management process of the new unstructured data source, combining the new unstructured data source with any data source in the aggregation result of the second data source to perform data dynamics, format consistency analysis, and aggregation analysis of the file management process, and judging whether there is a candidate aggregation combination. Specifically, when an enterprise adds a new unstructured data source, first collect relevant information of the data source, including its data dynamics (such as data generation frequency, data volume change, etc.), data structured format (such as data type, data format, etc.) and file management process (such as data collection, storage, processing, etc.). Compare the information of the new data source with any data source combination in the aggregation result of the second data source to check whether there are similarities or compatibility points in their data dynamics, format consistency and file management process. Based on the comparison results, perform aggregation analysis to determine whether the new data source can be aggregated with the existing data source combination. If so, the corresponding existing data source combination is the candidate aggregation combination.

[0046] Step S470, if yes, adjust the unstructured data processing center with the candidate aggregation combination. Specifically, if there is a candidate aggregation combination, adjust the processing resource configuration of the data center according to the data dynamics and format consistency of the new data source, including increasing computing resources, storage space or adjusting the data processing flow. Update the mapping relationship between the data source and the processing node, and include the new data source in the processing flow corresponding to the candidate aggregation combination. According to the adjusted resource configuration and mapping relationship, make corresponding adjustments to the structure of the unstructured data processing center to ensure that it can efficiently process the data from the new data source.

[0047] Step S480, if not, construct an independent aggregation result for the new unstructured data source, and then adjust the unstructured data processing center. Specifically, if there is no candidate aggregation combination, construct an independent aggregation result for the new data source, including dynamic and format consistency analysis of the data at the new data source, designing a special file management process, and allocating appropriate processing resources for it. Integrate the newly constructed independent aggregation results into the unstructured data processing center to expand the processing capabilities and coverage of the center. Establish a mapping relationship with the processing node for the new data source to ensure that the data can be processed and distributed correctly. This implementation method can flexibly adjust the structure and resource configuration of the unstructured data processing center by judging whether the new data source can be combined with the existing data source to form an aggregation, which helps enterprises quickly adapt to new data processing needs and improve the flexibility and scalability of the center.

[0048] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. An AI intelligent decision-making method for unstructured data processing, characterized in that: include: Collect various historical unstructured data sets from various unstructured data sources of the target enterprise; Performing data dynamics and format consistency analysis and aggregation on each of the historical unstructured data sets to establish a first data source end aggregation result; Analyze the file management process based on the usage scenarios of each of the historical unstructured data sets, and optimize the aggregation results of the first data source end according to the analysis results to generate the aggregation results of the second data source end; Building an unstructured data processing platform based on the aggregation result of the second data source, and performing intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source; The data dynamics and format consistency analysis and aggregation are performed on each of the historical unstructured data sets to establish the aggregation result of the first data source end, including: Performing dynamic indicator statistics on each of the historical unstructured data sets according to a preset dynamic indicator set to generate each dynamic indicator value set; Acquire structured processing templates for each of the historical unstructured data sets, perform consistency analysis on the structured processing templates, and generate format consistency judgment results; Based on the dynamic indicator value sets and the format consistency judgment results, similarity analysis is performed on the historical unstructured data sets to generate an unstructured data source end combination whose similarity is greater than a preset similarity threshold; Aggregating the respective unstructured data sources with the unstructured data source combination to generate the first data source aggregation result; The process of analyzing the file management process based on the usage scenarios of each historical unstructured data set, and optimizing the aggregation result of the first data source end according to the analysis result to generate the aggregation result of the second data source end includes: Collecting a historical file management process based on the usage scenarios of each of the historical unstructured data sets, wherein the historical file management process includes a plurality of process nodes; Performing a complexity analysis of the file management process on the first data source end combination in the first data source end aggregation result according to the historical file management process to generate a first complexity index; Determine whether the first complexity index meets a preset complexity index, and if not, generate a first aggregation optimization instruction; Based on the first aggregation optimization instruction, with the goal of reducing the complexity of the file management process, the first data source end combination is aggregated and optimized to generate a first data source end optimization result; The optimization result of the first data source is added to the aggregation result of the second data source.

2. The AI ​​intelligent decision-making method for unstructured data processing according to claim 1, characterized in that: The preset dynamic indicator set includes at least data type, data volume, data volume growth rate, access frequency and access mode.

3. The AI ​​intelligent decision-making method for unstructured data processing according to claim 1, characterized in that: Performing a complexity analysis of the file management process on the first data source end combination in the first data source end aggregation result according to the historical file management process to generate a first complexity index includes: Based on multiple process nodes in the historical file management process, data lifecycle analysis is performed on the first data source terminal combination, the differences in lifecycles and the diversity of cleanup strategies are identified and integrated, and lifecycle complexity is generated; Performing a complexity analysis of the permission management of the first data source terminal combination based on multiple process nodes in the historical file management process to generate a permission management complexity; The lifecycle complexity and the authority management complexity are weighted and integrated to generate the first complexity index.

4. The AI ​​intelligent decision-making method for unstructured data processing according to claim 1, characterized in that: Based on the first aggregation optimization instruction, with the goal of reducing the complexity of the file management process, the first data source end combination is aggregated and optimized to generate a first data source end optimization result, including: According to the first aggregation optimization instruction, binary classification aggregation is performed based on the similarity of the historical file management process to generate a first optimization aggregation result; Analyzing whether the complexity of the file management process of the first optimization aggregation result meets the preset complexity index; If not, continue to perform binary classification aggregation on the first optimization aggregation result, and so on, until an optimization aggregation result that meets the preset complexity index is obtained, and the first data source end optimization result is generated.

5. The AI ​​intelligent decision-making method for unstructured data processing according to claim 1, characterized in that: Based on the aggregation result of the second data source, an unstructured data processing platform is built to perform intelligent decision-making processing on the real-time unstructured data generated by each unstructured data source, including: Extracting multiple data source end combinations in the second data source end aggregation result; Extracting dynamic indicator values ​​for the multiple data source end combinations respectively, configuring processing resources of the data center according to the dynamic indicator value extraction results, and generating multi-data source end resource configuration results; Extracting the file management process for each of the plurality of data source combinations, building a plurality of processing nodes according to the file management process, and establishing a mapping relationship between the data source and the processing node; Constructing the unstructured data processing platform based on the resource configuration result, the multiple processing nodes and the mapping relationship; Based on the mapping relationship, the multiple processing nodes are called to perform decision processing on the real-time unstructured data generated by each unstructured data source end.

6. The AI ​​intelligent decision-making method for unstructured data processing according to claim 5, characterized in that: Also includes: When a new unstructured data source is added, the preset data dynamics, preset data structured format, and preset file management process of the new unstructured data source are determined, and the new unstructured data source is combined with any data source in the aggregation result of the second data source to perform data dynamics, format consistency analysis, and file management process aggregation analysis to determine whether there is a candidate aggregation combination; If so, adjusting the unstructured data processing platform with the candidate aggregation combination; If not, build an independent aggregation result for the new unstructured data source, and then adjust the unstructured data processing center.

Images