Intelligent fusion processing method, system and device for coal mine multi-source heterogeneous data and storage medium

Abstract

The invention discloses an intelligent fusion processing method, system and device for coal mine multi-source heterogeneous data and a storage medium, and relates to the technical field of data processing and distributed calculation scheduling, and the method comprises the steps: collecting multi-source heterogeneous data, constructing a feature signature, and generating a quality score; and generating a fusion result according to the feature signature and the quality score, collecting a node state through the fusion result, calculating a reward and punishment factor, and executing dynamic scheduling by using the reward and punishment factor. According to the method, format recognition is performed on collected multi-source heterogeneous data, and the multi-source heterogeneous data subjected to format recognition is output, so that unified input of the multi-source heterogeneous data in the same processing link is realized; constructing a feature signature object and generating a quality score based on the multi-source heterogeneous data subjected to format recognition, so that quality information is transmitted along with the data in a computable form; and the fusion processing has clear input, calculation paths and output.

Description

Technical Field

[0001] This invention relates to the field of data processing and distributed computing scheduling technology, specifically to an intelligent fusion processing method, system, device, and storage medium for multi-source heterogeneous data in coal mines. Background Technology

[0002] Multi-source heterogeneous data processing typically employs a pipeline-style process of acquisition, parsing, cleaning, and fusion, or it can be achieved by establishing separate adaptation processing links for different data sources before aggregation.

[0003] In this type of scheme, on the one hand, due to the lack of clear identification and unified internal representation of data structure types, it is difficult to align the parsing results of different data sources under the same fusion mechanism; on the other hand, the evaluation of data quality is mostly limited to local operations such as missing value imputation and anomaly removal, lacking quality quantification output combined with subsequent fusion decisions, making it difficult to use data quality as a transferable input element in the calculation during fusion processing, thus easily leading to problems such as unclear fusion basis and difficulty in reproducing the fusion process.

[0004] After fusion processing, the fusion results often require further computation, storage, analysis, or push tasks, typically relying on distributed computing nodes or virtualized resources. Existing scheduling methods mostly rely on static thresholds, fixed weights, or simple polling, making it difficult to establish scheduling criteria related to task content when task inputs change. Furthermore, the lack of a clear input-output link between node load status collection, metric normalization, comprehensive evaluation, and scheduling selection makes the entire process of fusion results, node status, and scheduling execution prone to breakpoints in engineering implementation, hindering the formation of a continuously running processing loop. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is that existing multi-source heterogeneous data fusion processing and task scheduling methods have problems such as inconsistent data format recognition and internal representation, lack of transferable feature signatures and quality scores leading to unclear fusion basis, unclear input-output links between fusion results and node status acquisition and scheduling decisions, low connectivity of processing flow, and how to achieve a continuous and executable interactive process of multi-source heterogeneous data after format recognition, feature signature objects and quality scores, consistency factors, fusion results, node status records, reward and punishment factors, and dynamic scheduling in the same processing link.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: an intelligent fusion processing method for multi-source heterogeneous data in coal mines, comprising the following steps: Collect heterogeneous data from multiple sources to construct feature signatures and generate quality scores.

[0008] The fusion result is generated based on the feature signature and quality score. The node status is collected and the reward and punishment factor is calculated based on the fusion result. Dynamic scheduling is then performed using the reward and punishment factor.

[0009] In this process, format recognition is performed on the collected multi-source heterogeneous data. The format-recognized multi-source heterogeneous data is then used as input to construct feature signatures and generate quality scores.

[0010] The process of generating the fusion result includes taking the feature signature and quality score as input, calculating the consistency factor, and performing fusion rule selection.

[0011] As a preferred embodiment of the intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in this invention, the step of collecting multi-source heterogeneous data includes accessing structured data sources, semi-structured data sources, and unstructured data sources.

[0012] An input channel is established for each data source according to the corresponding access protocol to receive raw data records.

[0013] Add a collection timestamp and data source identifier to the received raw data records.

[0014] Write to the internal buffer and establish queue relationships according to the receiving order.

[0015] The steps for performing format recognition include reading the original data record and selecting the parsing rule corresponding to the current record based on the data source identifier.

[0016] Output format-recognized multi-source heterogeneous data.

[0017] As a preferred embodiment of the intelligent fusion processing method for multi-source heterogeneous data in coal mines described in this invention, the step of constructing feature signatures and generating quality scores includes extracting data source identifiers, structure type identifiers, parsed field sets, timestamp information, and semantic features from the multi-source heterogeneous data after format recognition.

[0018] The data source identifier, structure type identifier, parsed field set, timestamp information, and semantic features are combined into a feature signature object.

[0019] Establish a signature index.

[0020] Perform a field integrity check on the set of fields associated with each feature signature.

[0021] Calculate multiple quality sub-items based on the field integrity verification results.

[0022] The quality sub-items are converted into quantifiable scores, and the quantifiable scores are used as the quality scores.

[0023] As a preferred embodiment of the intelligent fusion processing method for multi-source heterogeneous data in coal mines described in this invention, the calculation of the consistency factor includes reading the feature signature objects and their corresponding quality scores in chronological order.

[0024] The set of feature signature objects belonging to the same monitoring object and the same time window is determined by feature signature, and the set is used as the input for the consistency factor calculation.

[0025] Read the structure type identifier, parsed field set, and semantic features of two feature signature objects in the set.

[0026] The parsed field set is compared to ensure field correspondence, and the degree of field name matching and field type consistency are recorded.

[0027] The semantic features are compared by vector to obtain the degree of semantic association.

[0028] Read the quality score corresponding to the feature signature object and use the quality score as the weight.

[0029] The structural alignment relationship, semantic alignment relationship, and quality score are combined to generate a consistency factor to guide subsequent fusion processing.

[0030] Establish a binding relationship between the consistency factor and the set of feature signature objects.

[0031] As a preferred embodiment of the intelligent fusion processing method for multi-source heterogeneous data in coal mines described in this invention, the generation of fusion results includes obtaining a set of feature signature objects participating in the fusion based on a consistency factor bound to a set of feature signature objects.

[0032] Read the data source identifier, parsed field set, timestamp information, semantic features and corresponding quality score of each feature signature object in the set.

[0033] Divide the quality score within the set by the sum of the quality scores within the set to obtain the fusion weight of each feature signature object.

[0034] Perform field alignment processing on the parsed field set.

[0035] Perform field composition processing on each field item.

[0036] Perform vector synthesis processing on semantic features.

[0037] The fusion result is formed by combining the parsed field set obtained from the fusion, the fusion semantic features, the list of data source identifiers participating in the fusion, the corresponding consistency factors, and the corresponding timestamp information.

[0038] As a preferred embodiment of the intelligent fusion processing method for multi-source heterogeneous data in coal mines described in this invention, the status of the acquisition node includes generating a task identifier based on the fusion result.

[0039] Read the timestamp information and data source identifier list from the fusion result corresponding to the task identifier.

[0040] Send a status acquisition request to each computing node in the computing node list.

[0041] Each status acquisition request carries a task identifier.

[0042] Receive the node status returned by each computing node and establish a node status record.

[0043] The calculation of reward and penalty factors includes normalizing the processor, memory usage, task queue length, and node power data utilization of all computing nodes within the same data collection period.

[0044] Calculate the sum of the variances of processor usage, memory utilization, task queue length, node power data utilization, and data usage during the acquisition period.

[0045] Calculate the weights of processor, memory usage, task queue length, and node power data utilization, and then perform a weighted sum.

[0046] Output the node's overall value, and calculate the reward / penalty factor based on the node's overall value.

[0047] As a preferred embodiment of the intelligent fusion processing method for multi-source heterogeneous data in coal mines described in this invention, the dynamic scheduling includes reading the reward and punishment factors corresponding to the task identifier and obtaining the corresponding list of computing nodes.

[0048] Sort the computing nodes in the computing node list from largest to smallest according to the reward / penalty factor.

[0049] Select the first-ranked computing node as the target computing node.

[0050] Send a task distribution request to the target computing node.

[0051] Start the confirmation receipt timer.

[0052] Dynamic scheduling is performed based on the received confirmation timing results.

[0053] Another objective of this invention is to provide an intelligent fusion processing system for multi-source heterogeneous data in coal mines. This system can perform format recognition on the collected multi-source heterogeneous data and output the format-recognized multi-source heterogeneous data; construct feature signature objects based on the format-recognized multi-source heterogeneous data and generate quality scores; use the feature signature objects and quality scores as inputs to calculate consistency factors and generate fusion results; and use the fusion results to drive the collection of node status records and calculate reward and punishment factors to perform dynamic scheduling. This solves the problems of current multi-source heterogeneous data fusion processing and distributed task scheduling technologies, such as fragmented data input / output links, incoherent fusion and scheduling criteria, and unclear process connections that make it difficult to form a continuous and executable processing chain.

[0054] As a preferred embodiment of the intelligent fusion processing system for multi-source heterogeneous data in coal mines as described in this invention, it includes a feature signature and quality score generation module and a fusion result-driven scheduling module.

[0055] The feature signature and quality score generation module is used to collect multi-source heterogeneous data to construct feature signatures and generate quality scores.

[0056] The fusion result-driven scheduling module is used to generate fusion results based on feature signatures and quality scores, collect node status through the fusion results and calculate reward and penalty factors, and use the reward and penalty factors to perform dynamic scheduling.

[0057] Another object of the present invention is to provide an intelligent fusion processing device for multi-source heterogeneous data in coal mines, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of an intelligent fusion processing method for multi-source heterogeneous data in coal mines.

[0058] Another objective of this invention is to provide an intelligent fusion processing storage medium for multi-source heterogeneous data in coal mines, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of an intelligent fusion processing method for multi-source heterogeneous data in coal mines.

[0059] The beneficial effects of this invention are: The intelligent fusion processing method for multi-source heterogeneous data in coal mines provided by this invention achieves unified input of multi-source heterogeneous data in the same processing link by performing format recognition on the collected multi-source heterogeneous data and outputting the format-recognized multi-source heterogeneous data; by constructing feature signature objects and generating quality scores based on the format-recognized multi-source heterogeneous data, quality information is transmitted along with the data in a computable form; by using the feature signature objects and quality scores as inputs to calculate consistency factors and generate fusion results, the fusion processing has clear input, calculation paths, and outputs; by generating task identifiers from the fusion results, collecting node status records, calculating reward and punishment factors, and performing dynamic scheduling accordingly, the fusion results and resource scheduling form a connectable execution link. This invention achieves better results in terms of multi-source data access consistency, the completeness of the executable link in the fusion process, and the process continuity of task distribution after fusion. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is an overall flowchart of an intelligent fusion processing method for multi-source heterogeneous data in coal mines, provided in Embodiment 1 of the present invention. Detailed Implementation

[0062] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0063] Example 1, referring to Figure 1 As an embodiment of the present invention, an intelligent fusion processing method for multi-source heterogeneous data in coal mines is provided, comprising: S1: Collect multi-source heterogeneous data to construct feature signatures and generate quality scores.

[0064] Access structured data sources, semi-structured data sources, and unstructured data sources.

[0065] An input channel is established for each data source according to the corresponding access protocol to receive raw data records.

[0066] Add a collection timestamp and data source identifier to the received raw data records.

[0067] Write to the internal buffer and establish queue relationships according to the receiving order.

[0068] The steps for performing format recognition include reading the original data record and selecting the parsing rule corresponding to the current record based on the data source identifier.

[0069] Selecting the parsing rule corresponding to the current record includes parsing structured data based on field name, field length, and field type.

[0070] Semi-structured data is parsed based on its key-value structure.

[0071] Unstructured data is parsed based on its accompanying metadata.

[0072] Output format-recognized multi-source heterogeneous data.

[0073] Furthermore, data source identifiers, structure type identifiers, parsed field sets, timestamp information, and semantic features are extracted from the multi-source heterogeneous data after format recognition.

[0074] The data source identifier, structure type identifier, parsed field set, timestamp information, and semantic features are combined into a feature signature object.

[0075] Establish a feature signature index; one preferred approach is: Parse , , , , , Hash s , in, Indicates the original data record sequence number. Represents the original data record. This indicates the data source identifier, and Parse indicates the format recognition and parsing function. This represents multi-source heterogeneous data after format recognition. This represents the semantic feature extraction function. Represents semantic features, Indicates the structure type identifier. This represents the set of parsed fields. Indicates the collection timestamp. Represents the signature object, This represents the signature index, and Hash represents the hash function.

[0076] Perform a field integrity check on the set of fields associated with each feature signature.

[0077] Field integrity verification includes validity checks, recording field missing information, parsing anomaly markers, and the difference between the data timestamp and the current system time.

[0078] A preferred approach for field integrity verification is: , , in, Indicates field integrity verification. Indicates whether a field is missing. Indicates the parsing of exception markers. This represents the difference between the data timestamp and the current system time. Indicates the current system time.

[0079] Calculate multiple quality sub-items based on the field integrity verification results.

[0080] The quality sub-items are converted into quantifiable scores, and the quantifiable scores are used as the quality scores.

[0081] A preferred approach to generating quality scores is: , , in, Indicates quality score. This indicates the number of fields in the parsed field set. Represents the time decay scale parameter, and , , , These represent the weights of three different quality sub-items.

[0082] S2: Generate fusion results based on feature signatures and quality scores, collect node status and calculate reward / penalty factors through fusion results, and perform dynamic scheduling using reward / penalty factors.

[0083] Read the signature objects and their corresponding quality scores in chronological order.

[0084] The set of feature signature objects belonging to the same monitoring object and the same time window is determined by feature signature, and the set is used as the input for the consistency factor calculation.

[0085] Read the structure type identifier, parsed field set, and semantic features of two feature signature objects in the set.

[0086] The parsed field set is compared to ensure field correspondence, and the degree of field name matching and field type consistency are recorded.

[0087] The semantic features are compared by vector to obtain the degree of semantic association.

[0088] Read the quality score corresponding to the feature signature object and use the quality score as the weight.

[0089] The structural alignment relationship, semantic alignment relationship, and quality score are combined to generate a consistency factor to guide subsequent fusion processing.

[0090] Establish a binding relationship between the consistency factor and the set of feature signature objects.

[0091] A preferred approach to establishing a binding relationship is: , , , , in, This represents the index of the signature object within the same set. Indicates and Indices of different signature objects within the same set The consistency factor is represented by the following: express The corresponding quality score, express The corresponding quality score, Indicates by The set of extracted field names, Indicates field name exist Field types in Indicates by The set of extracted field names, Indicates field name exist Field types in express Corresponding semantic features express Corresponding semantic features Indicates the degree of matching of field names. This indicates that the field types are consistent. Indicates the degree of semantic relevance. Indicates a small quantity to prevent division by zero. Indicates a field name index. This indicates the weight of the degree of matching between field names. Indicates the field type consistency weight. The weight represents the degree of semantic relevance.

[0092] Furthermore, the set of feature signature objects participating in the fusion is obtained based on the consistency factor bound to the set of feature signature objects.

[0093] Read the data source identifier, parsed field set, timestamp information, semantic features and corresponding quality score of each feature signature object in the set.

[0094] Divide the quality score within the set by the sum of the quality scores within the set to obtain the fusion weight of each feature signature object.

[0095] Perform field alignment processing on the parsed field set.

[0096] Field alignment processing includes grouping fields with the same field name and field type into the same field item.

[0097] Fields with the same field name but different field types will retain their original field type identifiers to form field conflict items.

[0098] Perform field composition processing on each field item.

[0099] The field fusion process includes, when the field item is a numeric field, performing a weighted summation of each numeric value using fusion weights to obtain a fused value.

[0100] When the field is a category field, the frequency of occurrence of each category is weighted and accumulated using the fusion weight, and the category with the largest accumulated value is selected as the fusion category.

[0101] When a field is a conflicting field, the field value of the record with the highest quality score among the conflicting fields will be used as the merged field value.

[0102] Perform vector synthesis processing on semantic features.

[0103] The vector synthesis process includes linearly weighting and summing the semantic features within the set according to the fusion weights, and then normalizing the weighted result to obtain the fused semantic features.

[0104] The fusion result is formed by combining the parsed field set obtained from the fusion, the fusion semantic features, the list of data source identifiers participating in the fusion, the corresponding consistency factors, and the corresponding timestamp information.

[0105] A preferred approach to outputting the fusion result is: , , , , in, This represents the set of feature signature objects participating in the fusion. Indicates the fusion weight. Indicates the first Field names in a signature object field values, This indicates the candidate category values ​​for the category field. This represents the index of the record with the highest quality score under the field conflict item. This indicates the fusion of semantic features. Indicates the fusion result. This represents the set of parsed fields obtained after fusion. This represents a list of data source identifiers participating in the fusion. This indicates the corresponding consistency factor binding information. This indicates the corresponding timestamp information. Indicates the value of the merged field. Indicates an indicator function.

[0106] Furthermore, task identifiers are generated based on the fusion results.

[0107] Read the timestamp information and data source identifier list from the fusion result corresponding to the task identifier.

[0108] Send a status acquisition request to each computing node in the computing node list.

[0109] Each status acquisition request carries a task identifier.

[0110] Receive the node status returned by each computing node and establish a node status record.

[0111] The node status records include processor utilization, memory usage, task queue length, and node power data.

[0112] The calculation of reward and penalty factors includes normalizing the processor, memory usage, task queue length, and node power data utilization of all computing nodes within the same data collection period.

[0113] Calculate the sum of the variances of processor usage, memory utilization, task queue length, node power data utilization, and data usage during the acquisition period.

[0114] Calculate the weights of processor, memory usage, task queue length, and node power data utilization, and then perform a weighted sum.

[0115] Output the node's overall value, and calculate the reward / penalty factor based on the node's overall value.

[0116] Read the reward / penalty factor corresponding to the task identifier and obtain the corresponding list of computing nodes.

[0117] Sort the computing nodes in the computing node list from largest to smallest according to the reward / penalty factor.

[0118] Select the first-ranked computing node as the target computing node.

[0119] Send a task distribution request to the target computing node.

[0120] The task issuance request includes the task identifier and the corresponding fusion result of the task identifier.

[0121] Start the confirmation receipt timer.

[0122] Dynamic scheduling is performed based on the received confirmation timing results.

[0123] The receiving confirmation timing includes ending the receiving confirmation timing and recording the delivery completion status corresponding to the task identifier when receiving the receiving confirmation information returned by the target computing node within the receiving time.

[0124] If the receiving confirmation timer exceeds the receiving time and no receiving confirmation information is received, select the next ranked computing node as the new target computing node and repeat the sending task to send the request and receive confirmation timer.

[0125] When a rejection flag is received from the target computing node, the target computing node is removed from the list of computing nodes corresponding to the task identifier, and the next computing node in the sorted order is selected as the new target computing node to continue executing the task distribution request.

[0126] An optimal solution for task identification, reward / penalty factors, and dynamic scheduling is as follows:

[0127] , , , , in, Indicates the task identifier. Indicates the compute node index. Indicates processor utilization. Indicates memory usage. Indicates the task queue length. Indicates the node power data utilization rate. Indicates the index of indicator categories. This represents the normalization result. This represents the traversal index of "all computing nodes" within the same collection period. Represents variance. The weights represent the variance normalization results. Indicates reward and punishment factors. This represents a list of compute nodes. Indicates the receiving time. This function represents the result of the reception confirmation timer. If a reception confirmation message is received within the reception time, the value is 1; otherwise, the value is 0 if no confirmation is received within the time limit or if a rejection flag is received. Indicates the target computing node.

[0128] Example 2 is an embodiment of the present invention, which provides a specific application of an intelligent fusion processing method for multi-source heterogeneous data in coal mine scenarios.

[0129] In a coal mine scenario, access can be made to structured data sources, semi-structured data sources, and unstructured data sources.

[0130] Specifically, the structured data sources include time-series data tables of sensors such as gas concentration, CO concentration, temperature and humidity, wind speed, and wind pressure; monitoring tables of fan current, voltage, power, and speed; monitoring tables of belt speed and motor temperature rise; and positioning tables of personnel location coordinates.

[0131] The semi-structured data sources are JSON messages uploaded by the device gateway, event logs generated by the scheduling system, and device alarm logs.

[0132] Unstructured data sources include keyframes from tunnel video streams, voice-to-text transcriptions of equipment inspection records, or image inspection records, along with metadata such as camera number, acquisition location, and time.

[0133] An input channel is established for each data source according to the corresponding access protocol. For structured data sources, a database connection protocol is used to establish the input channel and timestamp-based polling is used to retrieve data.

[0134] For semi-structured data sources, a message queue subscription protocol is used to establish an input channel and the subscription topic is identified by the data source identifier.

[0135] For unstructured data sources, file or object storage access protocols are used to establish input channels and keyframes are retrieved by metadata index or received via streaming interfaces.

[0136] Add a collection timestamp and data source identifier to the received raw data records and write them to the internal cache. Establish a queue relationship in the internal cache according to the receiving order to ensure that records from different sources can be read sequentially within the same collection period.

[0137] When performing format recognition, the original data record is read, and the parsing rule corresponding to the current record is selected based on the data source identifier. Structured data is parsed according to field name, field length and field type, semi-structured data is parsed according to key-value structure and unstructured data is parsed according to its attached metadata.

[0138] The image content is preserved as the original payload for subsequent semantic feature extraction, and the output is multi-source heterogeneous data after format recognition.

[0139] Extract data source identifiers, structure type identifiers, parsed field sets, timestamp information, and semantic features from the multi-source heterogeneous data after format recognition.

[0140] In coal mine applications, semantic feature extraction includes: executing targets on video keyframes, identifying regions to obtain personnel density, smoke and dust visibility, belt misalignment area proportion, and equipment appearance anomaly confidence semantic vectors.

[0141] The speech-to-text is processed by semantic encoding of keywords and instructions to obtain a semantic vector of fault description category, location, and urgency.

[0142] The data source identifier, structure type identifier, parsed field set, timestamp information, and semantic features are combined into a feature signature object, and a feature signature index is established to support subsequent retrieval by monitoring object and time window.

[0143] Perform field integrity checks on the set of fields associated with each feature signature. Field integrity checks include validity checks, checking for missing fields, parsing anomaly markers, and checking the difference between the data timestamp and the current system time.

[0144] In coal mine scenarios, missing key fields such as gas and wind speed, missing unit fields, missing sensor numbers, parsed values ​​exceeding the physical allowable range, and timestamp bounce or excessive delay can be used as sources of anomalies.

[0145] Based on the field integrity verification results, multiple quality sub-items are calculated and converted into quantifiable scores. The quantifiable scores are then stored as quality scores, enabling the quality scores to be directly used as input in subsequent consistency factor and fusion weight calculations.

[0146] The signature objects and their corresponding quality scores are read in chronological order. In coal mine applications, signature objects belonging to the same monitoring object and the same time window are identified through signature analysis.

[0147] The monitoring targets can be a certain roadway section, a certain mining face, a certain ventilation circuit node, a certain belt conveyor, a transfer point, or the area corresponding to a certain sensor cluster.

[0148] The time window can be a sliding window based on the collection period, and the set of feature signature objects belonging to the same monitoring object and the same time window can be used as the input for the consistency factor calculation.

[0149] Read the structure type identifier, parsed field set, and semantic features of any two feature signature objects in the set.

[0150] The parsed field set is compared to ensure field correspondence, and the degree of field name matching and field type consistency are recorded.

[0151] By using a field alias mapping table, field names are normalized to a unified field name during the parsing phase, improving the degree of field name matching.

[0152] The semantic features are vector-based to obtain the degree of semantic association. The quality score corresponding to the feature signature object is read. The quality score is used as a weight to generate a consistency factor by combining the structural alignment relationship, semantic alignment relationship and quality score. The consistency factor is then bound to the set of feature signature objects.

[0153] Furthermore, based on the consistency factor bound to the feature signature object set, the set of feature signature objects participating in the fusion is obtained. The data source identifier, parsed field set, timestamp information, semantic features and corresponding quality scores of each feature signature object in the set are read. The fusion weight of each feature signature object is obtained by dividing the quality score in the set by the sum of the quality scores in the set.

[0154] The parsed field set is then aligned by performing field alignment processing: fields with the same field name and field type are merged into the same field item, while field items with the same field name but different field types retain their original field type identifiers to form field conflict items.

[0155] For each field item, a field synthesis process is performed. Specifically, when the field item is a numeric field, the numeric values ​​are weighted and summed to obtain the synthesized value.

[0156] When the field is a category field, the frequency of occurrence of each category is weighted and accumulated through fusion weights, and the category with the largest accumulated value is selected as the fusion category.

[0157] When a field is a conflicting field, the field value of the record with the highest quality score among the conflicting fields will be used as the merged field value.

[0158] The semantic features are processed by vector synthesis, specifically by linearly weighting and summing each semantic feature in the set according to the fusion weights and normalizing it to obtain the fused semantic features.

[0159] The fusion result is formed by combining the parsed field set obtained from the fusion, the fusion semantic features, the list of data source identifiers participating in the fusion, the corresponding consistency factors, and the corresponding timestamp information.

[0160] In coal mine applications, the fusion results can form working face safety situation fusion results and ventilation circuit situation fusion results. The parsed field set includes the fused gas concentration, CO concentration, wind speed, wind pressure, equipment electrical parameters, and personnel density.

[0161] The fused semantic features include the fused confidence scores for dust and smoke anomalies, belt misalignment, equipment appearance anomalies, and voice fault description category vectors. The fusion results can be directly used as input for subsequent tasks.

[0162] Furthermore, a task identifier is generated based on the fusion result, the timestamp information and data source identifier list in the fusion result corresponding to the task identifier are read, and a status collection request is sent to each computing node in the computing node list. Each status collection request carries the task identifier.

[0163] Receive node status returned by each computing node and establish node status records. The node status records include processor utilization, memory usage, task queue length, and node power data.

[0164] When calculating the reward and penalty factor, the processor, memory usage, task queue length, and node power data utilization of all computing nodes within the same collection period are normalized. The variance of the above four indicators within the collection period is calculated, the weights of the four indicators are calculated and weighted summation is performed, the node comprehensive value is output, and the reward and penalty factor is calculated based on the node comprehensive value.

[0165] Read the reward / penalty factor corresponding to the task identifier, obtain the corresponding list of computing nodes, sort the computing nodes in the list from largest to smallest according to the reward / penalty factor, select the computing node ranked first as the target computing node, send a task distribution request to the target computing node, the task distribution request includes the task identifier and the corresponding fusion result of the task identifier, and start the receiving confirmation timer.

[0166] Dynamic scheduling is performed based on the received confirmation timing result. Specifically, when the received confirmation information is received from the target computing node within the receiving time, the received confirmation timing ends and the delivery completion status corresponding to the task identifier is recorded.

[0167] If the confirmation timer exceeds the reception time and no confirmation message is received, select the next ranked computing node as the new target computing node and repeat the task sending request and confirmation timer.

[0168] When a rejection flag is received from the target computing node, the target computing node is removed from the list of computing nodes corresponding to the task identifier, and the next computing node in the sorted order is selected as the new target computing node to continue executing the task distribution request.

[0169] After receiving the task, the target computing node can perform fusion situation judgment and linkage tasks in the coal mine scenario. Specifically, it performs joint calculations on fields such as gas concentration and wind speed changes, video dust anomaly confidence, and personnel density in the fusion results to generate processing results such as working face risk level, ventilation adjustment suggestions, and inspection tasks. The processing results are then written into the result queue of the scheduling system or alarm system for subsequent business system calls.

[0170] Example 3 is an embodiment of the present invention, which provides an intelligent fusion processing system for multi-source heterogeneous data in coal mines, including a feature signature and quality score generation module and a fusion result-driven scheduling module.

[0171] The feature signature and quality score generation module is used to collect multi-source heterogeneous data to construct feature signatures and generate quality scores.

[0172] The fusion result-driven scheduling module is used to generate fusion results based on feature signatures and quality scores, collect node status through fusion results and calculate reward and penalty factors, and use reward and penalty factors to perform dynamic scheduling.

[0173] This embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the intelligent fusion processing method for multi-source heterogeneous data in coal mines as proposed in the above embodiment.

[0174] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the intelligent fusion processing method for multi-source heterogeneous data in coal mines as proposed in the above embodiment.

[0175] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0177] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0178] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0179] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent fusion processing of multi-source heterogeneous data in coal mines, characterized in that, include: Collect multi-source heterogeneous data to construct feature signatures and generate quality scores; A fusion result is generated based on the feature signature and quality score. The node status is collected using the fusion result, and a reward / penalty factor is calculated. Dynamic scheduling is then performed using this reward / penalty factor. Format recognition is performed on the collected multi-source heterogeneous data. The format-recognized multi-source heterogeneous data is used as input to construct feature signatures and generate quality scores. The process of generating the fusion result includes taking the feature signature and quality score as input, calculating the consistency factor, and performing fusion rule selection.

2. The intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in claim 1, characterized in that: The steps for collecting multi-source heterogeneous data include: Access structured data sources, semi-structured data sources, and unstructured data sources; Establish an input channel for each data source according to the corresponding access protocol to receive raw data records; Add a collection timestamp and data source identifier to the received raw data records; Write to the internal buffer and establish queue relationships according to the receiving order; The steps for performing format recognition include reading the original data record and selecting the parsing rule corresponding to the current record based on the data source identifier; Output format-recognized multi-source heterogeneous data.

3. The intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in claim 1 or 2, characterized in that: The construction of feature signatures and generation of quality scores includes, Extract data source identifiers, structure type identifiers, parsed field sets, timestamp information, and semantic features from the multi-source heterogeneous data after format recognition; The data source identifier, structure type identifier, parsed field set, timestamp information, and semantic features are combined into a feature signature object; Establish a signature index; Perform a field integrity check on the set of fields associated with each feature signature; Calculate multiple quality sub-items based on the field integrity verification results; The quality sub-items are converted into quantifiable scores, and the quantifiable scores are used as the quality scores.

4. The intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in claim 3, characterized in that: The calculated consistency factor includes, Read the signature objects and their corresponding quality scores in chronological order; The set of feature signature objects belonging to the same monitoring object and the same time window is determined by feature signature, and the set is used as the input for the consistency factor calculation; Read the structure type identifier, parsed field set, and semantic features of two feature signature objects in the set; The parsed field set is compared to ensure field correspondence, and the degree of field name matching and field type consistency are recorded. Semantic features are compared using vectors to obtain the degree of semantic association; Read the quality score corresponding to the feature signature object and use the quality score as the weight; The structural alignment relationship, semantic alignment relationship, and quality score are combined to generate a consistency factor to guide subsequent fusion processing; Establish a binding relationship between the consistency factor and the set of feature signature objects.

5. The intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in claim 1, 2, or 4, characterized in that: The generated fusion result includes, The set of feature signature objects participating in the fusion is obtained based on the consistency factor bound to the set of feature signature objects; Read the data source identifier, parsed field set, timestamp information, semantic features and corresponding quality scores of each feature signature object in the set; Divide the quality score within the set by the sum of the quality scores within the set to obtain the fusion weight of each feature signature object; Perform field alignment processing on the parsed field set; Perform field composition processing on each field item; Perform vector synthesis processing on semantic features; The fusion result is formed by combining the parsed field set obtained from the fusion, the fusion semantic features, the list of data source identifiers participating in the fusion, the corresponding consistency factors, and the corresponding timestamp information.

6. The intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in claim 5, characterized in that: The status of the data acquisition node includes, Generate task identifiers based on the fusion results; Read the timestamp information and data source identifier list from the fusion result corresponding to the task identifier; Send a status acquisition request to each computing node in the computing node list; Each status acquisition request carries a task identifier; Receive the node status returned by each computing node and establish a node status record; The calculation of reward and penalty factors includes normalizing the processor, memory usage, task queue length, and node power data utilization of all computing nodes within the same data collection period. Calculate the sum of the variances of processor, memory usage, task queue length, and node power data utilization during the acquisition period; Calculate the weights of processor, memory usage, task queue length, and node power data utilization, and then perform a weighted sum. Output the node's overall value, and calculate the reward / penalty factor based on the node's overall value.

7. The intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in claims 1, 2, 4, or 6, characterized in that: The execution dynamic scheduling includes, Read the reward / penalty factor corresponding to the task identifier and obtain the corresponding list of computing nodes; Sort the computing nodes in the computing node list from largest to smallest according to the reward / penalty factor; Select the first-ranked computing node as the target computing node; Send a task distribution request to the target computing node; Start the confirmation receipt timer; Dynamic scheduling is performed based on the received confirmation timing results.

8. An intelligent fusion processing system for multi-source heterogeneous data in coal mines, employing the intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in any one of claims 1 to 7, characterized in that: Includes a feature signature and quality score generation module, and a fusion result-driven scheduling module; The feature signature and quality score generation module is used to collect multi-source heterogeneous data to construct feature signatures and generate quality scores. The fusion result-driven scheduling module is used to generate fusion results based on feature signatures and quality scores, collect node status through the fusion results and calculate reward and penalty factors, and use the reward and penalty factors to perform dynamic scheduling.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent fusion processing method for multi-source heterogeneous data in coal mines as described in any one of claims 1 to 7.

Images