Multi-source heterogeneous big data fusion preprocessing method and intelligent system

Abstract

The application discloses a multi-source heterogeneous big data fusion preprocessing method and an intelligent system, and particularly relates to the field of data fusion preprocessing, and comprises the following steps: acquiring multi-source original data in an edge data transmission link; writing the multi-source original data under the same device identifier into a transmission sequence according to the edge gateway receiving sequence; and identifying a data segment with the collection sequence opposite to the receiving sequence in the transmission sequence to generate a disorder segment; the application identifies the disorder segment with the collection sequence opposite to the receiving sequence in the edge data transmission link, reorganizes the measurement data, image abstracts and supplementary data according to the event starting point, and generates full-item input, missing-item input or blocked input by combining the supplementary operation, transmission state splicing and event boundary rewriting, so as to solve the problem that the multi-source heterogeneous data of the same on-site event is split or miscombined in the preprocessing stage in the prior art.

Description

Technical Field

[0001] This invention relates to the field of data fusion and preprocessing technology, and more specifically, to a method and intelligent system for multi-source heterogeneous big data fusion and preprocessing. Background Technology

[0002] In the preprocessing stage of multi-source heterogeneous big data fusion, existing systems generally first solve the problems of protocol inconsistency, field inconsistency, and time inconsistency. That is, the edge gateway or central platform completes protocol parsing, field mapping, timestamp correction, missing data completion, and formatted writing for sensor measurements, device status, control logs, alarm records, and image summaries, respectively, and then sends the processed data into subsequent fusion calculations. During the equipment status monitoring process at the production site, the same equipment will simultaneously generate current, temperature, vibration, PLC status, alarm logs, and on-site image summaries during heating, start-stop switching, or load surges. These data come from different acquisition terminals and enter the edge data transmission link through different gateways. The on-site network bandwidth, gateway buffer capacity, and alarm data priority will cause alarm logs to be uploaded first, ordinary measurement values ​​to be retransmitted later, and image summaries to arrive late. As a result, the order of data received at the central end is no longer the same as the order of occurrence at the site. Multiple types of data for the same on-site event will be scattered in different upload batches. Data from two consecutive on-site events will fall into the same preprocessing window due to their close timestamps. Specifically, alarm records have entered the central end but the corresponding measurement values ​​have not yet arrived, retransmitted measurement values ​​are registered as new status changes, image summaries cannot be attached to the original abnormal process, and the same equipment will generate multiple incomplete data fragments in a short period of time. The root cause is that the existing preprocessing mainly focuses on correcting data content, field format, and time fields, without establishing a determination relationship between on-site event attribution, edge upload batches, and retransmission missing items at the fusion entry point. Therefore, the problem this application aims to solve is: how to prevent multi-source heterogeneous data of the same field event from being split or mistakenly merged during the preprocessing stage when edge data transmission involves caching, priority uploading, and breakpoint retransmission. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multi-source heterogeneous big data fusion preprocessing method and intelligent system. By identifying out-of-order segments in the edge data transmission link where the acquisition order is reversed from the reception order, and recombining measured data, image summaries, and supplementary transmission data with the event starting point, and then combining completion operation, transmission state splicing, and event boundary rewriting to generate full-item input, missing-item input, or blocked input, the problems mentioned in the background art are solved.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a multi-source heterogeneous big data fusion preprocessing method, comprising: S1. Obtain multi-source raw data in the edge data transmission link, write multi-source raw data under the same device identifier into the transmission sequence according to the receiving order of the edge gateway, and identify data segments in the transmission sequence whose acquisition order is opposite to the receiving order, and generate out-of-order segments. S2. Using the first alarm data or the first state switching data in the disordered segment as the event starting point, read forward the measurement data that has not yet been attached under the same device identifier, and read backward the image summary and supplementary data under the same device identifier. Reorganize the read data into a candidate event group. S3. Perform a batch posterior gain greedy algorithm on the candidate event group, test the changes in the number of missing items, batch breaks, and order reversals caused by adding unattached data to the candidate event group, and write the unattached data with the largest reduction in quantity into the candidate event group until the remaining unattached data cannot be reduced by any quantity, and generate a completion group. S4. Perform the transmission state post-hoc splicing algorithm on the supplementary group, rearrange the data in the group according to the transmission state order of first upload, cache upload and breakpoint retransmission, and determine the data path in which the source collection order, gateway receiving order and retransmission order are all true at the same time as the event path. S5. Rewrite the event boundaries of the completion group according to the event path. Output the completion group with closed path and zero source missing items as full item input. Output the completion group with closed path but still with source missing items as missing item input. Output the data with open path or reversed order that has not been eliminated as blocking input.

[0005] In a preferred embodiment, S1 includes: S1-1. Read the raw data from multiple sources under the same device identifier, assign the acquisition bits in ascending order of acquisition time, assign the receiving bits in ascending order of edge gateway reception, and write the acquisition bits and receiving bits into the transmission sequence. S1-2. Scan the transmission sequence along the increasing direction of the received bits, mark two multi-source raw data with adjacent received bits and the latter collected bit being less than the former collected bit as reverse pairs, and generate the out-of-order start bit and the out-of-order end bit in the former received bit of the reverse pair. S1-3. Limit the interception range by the starting and ending positions of the out-of-order sequence, intercept the original data from multiple sources in the transmission sequence, and generate out-of-order segments of data whose acquisition bits are not arranged in ascending order in the interception result.

[0006] In a preferred embodiment, S2 includes: S2-1. Read the alarm data and state switching data in the out-of-order segment, select the first alarm data or the first state switching data in ascending order of received bits as the event starting point, and record the device identifier, acquisition bit and receiving bit of the event starting point; S2-2. Starting with the receiving bit at the event origin, read the measurement data that has not yet been attached under the same device identifier along the decreasing direction of the receiving bit. Read until the beginning of the transmission sequence, the attached data, or the position where the device identifier changes, and generate a forward measurement group.

[0007] In a preferred embodiment, S2 further includes: S2-3. Starting with the receiving bit at the beginning of the event, read the image summary and supplementary data under the same device identifier along the increasing direction of the receiving bit. Read until the end of the transmission sequence, the next alarm data, or the next state switching data, and generate a backward supplementary evidence group. S2-4. Write the forward measurement group, event start point and backward supplementary verification group into the same group record in ascending order of received bits, and generate the candidate event group with the device identifier of the event start point and the received bit generation group identifier.

[0008] In a preferred embodiment, S3 includes: S3-1. Using candidate event groups, event start points, and unattached data as input, calculate the first acquisition bit, the last acquisition bit, the first reception bit, and the last reception bit according to the acquisition bit and reception bit of the data in the candidate event group, and generate a table containing the source and a table containing the batch according to the data source type and upload batch identifier. S3-2. Based on the first and last bits of data acquisition, the first and last bits of data reception, the included source table, and the included batch table, write the unattached data whose received bit is less than the event start receiving bit and whose data source type is measured data as the pre-test chip; write the unattached data whose received bit is between the first and last bits of reception and whose uploaded batch identifier has not entered the included batch table as the clipped chip; write the unattached data with a supplementary transmission identifier and whose data source type has not entered the included source table as the back-insertion chip; write the unattached data whose order reversal number is increased after being written into the candidate event group as the erroneous chip; otherwise, write it as the chip to be checked. S3-3. For pre-test chips, clipped chips, reinserted chips, erroneously merged chips, and chips under investigation, limit the pre-test chip to be written before the event start point receiving bit, limit the clipped chip to be written between adjacent upload batch identifiers, limit the reinserted chip to be written at the location of the source missing item, write the erroneously merged chip to the rejection table, and write the chips under investigation to the test table, and generate a limit table.

[0009] In a preferred embodiment, S3 further includes: S3-4. Through greedy operation of the post-test sub-module, write the pre-test piece, clip piece, back-insertion piece and retained piece in the limit table and the test table one by one. Calculate the reduction of source missing items, batch breakage, and sequence reversal after the trial writing. Write the data that generate new reduction and do not reuse the same missing item, the same breakage, or the same reversal into the candidate event group. Otherwise, keep it in the test table and generate the return group. S3-5. Based on the D-optimal supplementary verification operation, write the eliminated source missing items, batch breaks and order reversals in the repositioning group into the supplementary verification matrix. Calculate the determinant increment of each data in the test table after adding it to the supplementary verification matrix. Write the data with the first determinant increment that does not increase the number of source missing items, batch breaks and order reversals into the repositioning group. Otherwise, keep it in the test table and generate the expansion group. S3-6. Insert the data from the test table and the elimination table back into the expansion group one by one and recalculate the number of missing items from the source, the number of batch breaks, and the number of sequence reversals. Merge the data whose three quantities have not increased and whose at least one quantity has decreased into the expansion group; otherwise, delete the attachment mark and output the expansion group as the completion group.

[0010] In a preferred embodiment, S4 includes: S4-1. Taking the supplementary group as input, write the data in the group into the transmission state according to the supplementary transmission identifier and the upload batch identifier. Data without supplementary transmission identifier and whose upload batch identifier is the first batch in the group is written into the first upload state. Data without supplementary transmission identifier and whose upload batch identifier is not the first batch in the group is written into the cached upload state. Data with supplementary transmission identifier is written into the breakpoint supplementary transmission state. Generate the transmission state table. S4-2. Read the transmission state table using the transmission state post-hoc splicing algorithm. Generate candidate edges based on two adjacent data within the same group under the same data source type. Write the data into the source bit when the data acquisition bit of the later bit is greater than the data acquisition bit of the earlier bit. Write the data into the gateway bit when the data reception bit of the later bit is greater than the data reception bit of the earlier bit. Write the data into the supplementary transmission bit when the data of the later bit is in the breakpoint retransmission state and the data upload batch identifier of the later bit is greater than the data upload batch identifier of the earlier bit. Otherwise, write the empty bit and generate the edge bit table.

[0011] In a preferred embodiment, S4 further includes: S4-3. Generate edge posterior values ​​using the number of bits in the source internal bit, gateway bit, and supplementary transmission bit in the edge bit table. Starting from the first uploaded state data, concatenate edges one by one according to the decreasing edge posterior values. When the edge posterior values ​​are the same, select candidate edges in ascending order of received bits to generate the initial path. S4-4. Perform D-optimization bridge operation on the candidate edges where empty positions are located in the initial path. Insert the data that has not entered the initial path into the candidate edges where empty positions are located. Form a three-dimensional row according to the source internal bit, gateway bit, and supplementary bit after insertion. Calculate the determinant increment of the three-dimensional row after merging it into the path bit matrix. Merge the data with the first determinant increment and no empty positions in the three-dimensional row into the initial path. Otherwise, move it back to the completion group and generate the event path.

[0012] In a preferred embodiment, S5 includes: S5-1. Taking the event path and padding group as input, read the first and last received bits in the event path, take the first to last received bits as the event boundary of the padding group, and move the data outside the event boundary with the same group identifier out of the padding group to generate the delimitation group. S5-2. Based on the delimited group, recalculate the number of missing items and the number of order reversals. When adjacent data in the event path have source bits and gateway bits and the breakpoint supplementary transmission data have supplementary transmission bits, generate a path closure bit; otherwise, generate a path opening bit. S5-3. Output the delimited group with path closure bit and zero source missing item number as full item input, output the delimited group with path closure bit and non-zero source missing item number as missing item input, and output the data with path break bit or non-zero sequence reversal number as blocking input.

[0013] In a preferred embodiment, the multi-source heterogeneous big data fusion preprocessing intelligent system includes: The out-of-order identification module is used to acquire multi-source raw data in the edge data transmission link, write multi-source raw data under the same device identifier into the transmission sequence according to the receiving order of the edge gateway, and identify data segments in the transmission sequence whose acquisition order is opposite to the receiving order to generate out-of-order segments. The event reorganization module is used to take the first alarm data or the first state switching data in the out-of-order segment as the event starting point, read forward the measurement data that has not yet been attached under the same device identifier, read backward the image summary and supplementary data under the same device identifier, and reorganize the read data into a candidate event group. The greedy completion module is used to perform a batch posterior gain greedy algorithm on the candidate event group. It tests the changes in the number of missing items, batch breaks, and order reversals caused by adding unattached data to the candidate event group one by one, and writes the unattached data with the largest reduction into the candidate event group until the remaining unattached data cannot be reduced by any amount, thus generating a completion group. The post-transmission stitching module is used to perform a transmission-state post-transmission stitching algorithm on the completion group, rearrange the data in the group according to the transmission state order of first upload, cache upload and breakpoint retransmission, and determine the data path that simultaneously meets the source acquisition order, gateway reception order and retransmission order as the event path; The boundary rewriting module rewrites the event boundaries of the completion group according to the event path. It outputs the completion group with closed path and zero source missing items as full input, the completion group with closed path but still with source missing items as missing item input, and the data with open path or reversed order that has not been eliminated as blocking input.

[0014] The technical effects and advantages of this invention are as follows: 1. By identifying out-of-order segments through acquisition and reception bits, and then reconstructing forward measurement values ​​and backward supplementary evidence data based on the event starting point, the same on-site event is less likely to be split or mistakenly merged due to buffering, retransmission and priority uploading in edge data transmission. 2. By classifying and processing pre-tested pieces, clipped pieces, back-inserted pieces, misaligned pieces, and retained pieces, unconnected data can be returned to its correct position according to the cause of transmission distortion, which can relatively reduce the risk of incorrect connection caused by ordinary completion rules. 3. By using the greedy operation of the post-test sub-model and the D-optimal supplementary evidence operation, data is supplemented according to the source of missing items, batch breaks and sequence reversal changes, which improves the adaptability of the supplementary group to the incomplete evidence of multiple sources and batch breaks. 4. The data within the group is rearranged through the initial upload state, cached upload state, and breakpoint retransmission state, and the event path is spliced ​​with source bits, gateway bits, and retransmission bits to make the event path more closely match the actual arrival relationship in the edge transmission process. 5. By redrawing event boundaries, determining path closure, and splitting three types of inputs, the full input, missing input, and blocked input are output separately, which can relatively suppress the entry of data with broken paths or reversed order into subsequent fusion calculations. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method steps of the present invention.

[0016] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Refer to the instruction manual appendix Figure 1-2 The multi-source heterogeneous big data fusion preprocessing method of the present invention includes: S1. Obtain multi-source raw data in the edge data transmission link, write multi-source raw data under the same device identifier into the transmission sequence according to the receiving order of the edge gateway, and identify data segments in the transmission sequence whose acquisition order is opposite to the receiving order, and generate out-of-order segments. This implementation focuses on the out-of-order segment generation process. First, multi-source raw data under the same device identifier is converted into a transmission sequence with acquisition bits and reception bits. Then, the inverse relationship between acquisition bits and reception bits is used to identify out-of-order regions formed in edge data transmission. Finally, data segments whose acquisition bits are not arranged in ascending order are extracted from the out-of-order regions to provide input for subsequent event start point selection and candidate event group reorganization. The multi-source raw data includes at least the device identifier, data source type, acquisition time, edge gateway reception time, edge gateway write sequence number, upload batch identifier, retransmission identifier, and raw data payload. Acquisition bits and reception bits are generated and written into the transmission sequence in this step. The implementation process includes the following steps: In S1-1, multi-source raw data under the same device identifier is read and first sorted in ascending order by acquisition time; if the acquisition time is the same, it is sorted in a fixed order by data source type; if the data source type is the same, it is sorted by edge gateway reception time; if the edge gateway reception time is the same, it is sorted by edge gateway writing sequence number, and the acquisition bits are assigned sequentially from the beginning according to the sorting result; then it is sorted in ascending order by edge gateway reception time; if the edge gateway reception time is the same, it is sorted by edge gateway writing sequence number, and the reception bits are assigned sequentially from the beginning according to the sorting result; after the bit assignment is completed, the device identifier, data source type, acquisition time, edge gateway reception time, upload batch identifier, retransmission identifier, acquisition bits and reception bits are written into the same transmission sequence, so that each multi-source raw data retains both the on-site acquisition order and the edge gateway reception order; In S1-2, the transmission sequence is scanned along the increasing direction of the received bits. Each time, two adjacent multi-source raw data are read, and the acquisition bits of the two multi-source raw data are compared. If the acquisition bit of the later multi-source raw data is less than the acquisition bit of the earlier multi-source raw data, it means that the edge gateway's receiving order is reversed from the field acquisition order. These two multi-source raw data are marked as a reverse pair, and the received bits of the earlier multi-source raw data in the reverse pair are written as the out-of-order start bit, and the received bits of the later multi-source raw data are written as the out-of-order end bit. If the acquisition bit of the later multi-source raw data is greater than or equal to the acquisition bit of the earlier multi-source raw data, no reverse pair is generated, and the scan moves to the next group of adjacent received bit data. In S1-3, the generated reverse pairs in the transmission sequence are read, and the truncation range is defined by the out-of-order start and end bits. The truncation range includes the multi-source original data where the out-of-order start and end bits are located. In the truncation result, the previous multi-source original data is taken as the segment head from the position where the next acquisition bit is less than the previous acquisition bit. The data is scanned backward along the increasing direction of the received bits until the acquisition bits are restored to the ascending order. The previous multi-source original data is used as the segment tail, and the data segment between the segment head and the segment tail is generated as an out-of-order segment. When the truncation ranges of multiple reverse pairs overlap or are adjacent on the received bits, the overlapping or adjacent truncation ranges are merged to generate an out-of-order segment. When there is a received bit interval between the truncation ranges of multiple reverse pairs, different out-of-order segments are generated respectively. Through the above processing, the abnormal receiving order caused by caching, retransmission or priority uploading of the edge gateway is converted into an out-of-order segment jointly defined by the acquisition bit and the receiving bit. Subsequent steps no longer determine the event attribution based solely on the receiving time, but instead use the reverse relationship between the acquisition order and the receiving order as the starting point for event reassembly. In practical applications: When the same motor equipment generates temperature measurement, current measurement, PLC status switching and alarm logs in the field, the edge gateway receives the alarm log first because alarm data is uploaded first, and then receives the temperature measurement and current measurement. In this step, the temperature measurement, current measurement, PLC status switching and alarm log are written to the acquisition bit and the receiving bit respectively. When the receiving bit of the alarm log is in front but the acquisition bit is behind, the system generates a reverse pair accordingly and extracts the data segment containing the alarm log, the supplementary measurement value and the status switching data as the out-of-order segment.

[0019] S2. Using the first alarm data or the first state switching data in the disordered segment as the event starting point, read forward the measurement data that has not yet been attached under the same device identifier, and read backward the image summary and supplementary data under the same device identifier. Reorganize the read data into a candidate event group. This implementation method revolves around the process of generating candidate event groups. Based on the already formed out-of-order segments, the starting point of the event, representing the location of the change in the field, is first determined from alarm data and state transition data. Then, centered on the event starting point, measured data is retrieved backward, and image summaries and supplementary data are added backward. Finally, the retrieved data is written into the same group of records. Alarm data is data sourced from alarm records; state transition data is data sourced from equipment start / stop, operating state change, or control state change; measured data is numerical data such as temperature, current, vibration, pressure, and flow; image summaries are image recognition results or keyframe summaries uploaded by edge vision devices; and supplementary data is data with a supplementary identifier. When the multi-source raw data is initially written into the transmission sequence, the tag is empty; after being written into the candidate event group, the tag is written into the group identifier. The implementation process includes the following steps: In S2-1, alarm data and state transition data in the out-of-order segment are read and sorted in ascending order by received bits. The alarm data appearing first in the sorting result is taken as the event starting point, and the state transition data appearing first is taken as the event starting point. If alarm data and state transition data have the same received bit, the data with the earlier sequence number written by the edge gateway is taken as the event starting point. After determining the event starting point, the device identifier, acquisition bit, and receiving bit of the event starting point are recorded. The device identifier of the event starting point is used to limit the data range of subsequent forward and backward readings. The receiving bit of the event starting point is used to determine the reading direction, and the acquisition bit of the event starting point is used to determine whether the read data is related to the same on-site change. If there is no alarm data or state transition data in the out-of-order segment, no event starting point is generated, and the data in the out-of-order segment remains in an empty connection state, waiting for subsequent data retransmission or a new out-of-order segment to trigger grouping. In S2-2, the receiving bit at the event start point is used as the starting bit for reading. The transmission sequence is scanned in the decreasing direction of the receiving bit, and only multi-source raw data with the same device identifier as the event start point device identifier, empty attachment mark, and data source type of measurement data are read. When the first bit of the transmission sequence is encountered during the reading process, if the first bit data meets the above reading conditions, the first bit data is included in the forward measurement value group. When the data that has been written to other group identifiers is encountered during the reading process, reading is stopped and the attached data is not included. When the device identifier changes position during the reading process, reading is stopped and the data at the device identifier change position is not included. After reading is completed, the read measurement data is arranged in ascending order of receiving bit to generate a forward measurement value group. This step is used to retrieve ordinary measurement data that fell before the event start point due to alarm priority uploading, so that temperature, current, vibration data before the event start point can participate in subsequent reconstruction. In S2-3, the receiving bit at the event start point is used as the starting bit for reading. The transmission sequence is scanned along the increasing direction of the receiving bits, and data with the same device identifier as the event start point device identifier is read. Among them, data with image summary as the data source type is directly included in the backward verification group, and data with retransmission identifier is not restricted by data source type and is included in the backward verification group. When the last bit of the transmission sequence is encountered during the reading process, if the last bit of data meets the above reading conditions, the last bit of data is included in the backward verification group. When the next alarm data or the next state change data is encountered during the reading process, the reading is stopped and the alarm data or state change data is not included. After the reading is completed, the read image summary and retransmission data are arranged in ascending order of receiving bits to generate the backward verification group. This step is used to access data that falls after the event start point due to large image data volume, interrupted retransmission, or batch delay, to prevent the late image summary or retransmission measurement value from being mistakenly regarded as a new field change. In S2-4, the forward measurement group, event origin, and backward verification group are written into the same group record and arranged in ascending order of received bits. Each data entry in the group record is written with the same group identifier, which is generated by combining the device identifier of the event origin and the received bits of the event origin. After writing, the data attachment markers in the forward measurement group, event origin, and backward verification group are all written into this group identifier, and unread data continues to maintain its original attachment state. This yields candidate event groups, which at least include a group identifier, event origin, forward measurement group, backward verification group, data acquisition bits and receiving bits within the group, providing objects for subsequent calculations of source missing items, batch breakage numbers, and sequence reversal numbers. Through the above processing, out-of-order segments are no longer directly entered into the fusion preprocessing as a whole. Instead, alarm data or state switching data are used to locate the starting point of the on-site change, and then the front-side measured data and the back-side supplementary evidence data are retrieved separately, thereby reducing the probability that the same on-site change is split into multiple data segments. This process also restricts repeated readings by attaching tags, so that a multi-source raw data only enters one candidate event group, avoiding duplicate attribution in the subsequent completion process. In practical applications: After a sudden load change occurs in a motor, the current and temperature rise first. Then, the PLC state switches and triggers an alarm. The edge gateway first uploads the alarm data, then uploads the current and temperature measurements in the cache. The image summary arrives in the next batch. This step uses the alarm data as the starting point of the event, reads the current and temperature measurements that have not yet been connected under the same device identifier to generate a forward measurement group, reads the image summary and the supplementary measurement values ​​to generate a backward supplementary verification group, and then writes the three parts of data into the same group identifier to obtain the candidate event group.

[0020] S3. Perform a batch posterior gain greedy algorithm on the candidate event group, test the changes in the number of missing items, batch breaks, and order reversals caused by adding unattached data to the candidate event group, and write the unattached data with the largest reduction in quantity into the candidate event group until the remaining unattached data cannot be reduced by any quantity, and generate a completion group. This implementation method revolves around the process of completing candidate event groups. After the candidate event groups have been formed by forward measurement groups, event starting points, and backward verification groups, the collection range, reception range, types of data sources, and batches of data already included in the candidate event groups are calculated first. Then, the unattached data is divided into forward measurement pieces, clipped pieces, back-insertion pieces, erroneously merged pieces, and retained pieces, and their allowed writing positions are restricted respectively. Subsequently, the greedy operation of the posterior submodulus prioritizes writing data that can eliminate missing source items, batch breaks, or sequence reversals. Then, the D-optimal verification operation is used to supplement data that contributes more to the verification matrix. Finally, back-insertion verification is performed on the data in the test table and the elimination table to obtain the complete group. The implementation process includes the following steps: In S3-1, the candidate event group, event starting point, and unattached data are taken as input. The acquisition bits and reception bits of each data in the candidate event group are read. The smallest acquisition bit in the group is taken as the first acquisition bit, and the largest acquisition bit in the group is taken as the last acquisition bit. The smallest reception bit in the group is taken as the first reception bit, and the largest reception bit in the group is taken as the last reception bit. When there is only one data in the candidate event group, the acquisition bit of that data is taken as both the first and last acquisition bit, and the reception bit of that data is taken as both the first and last reception bit. Then, the data source type of each data in the candidate event group is read, and duplicates are written to the included source table according to the data source type. The upload batch identifier of each data in the candidate event group is read, and duplicates are written to the included batch table according to the upload batch identifier. The first and last reception bits of each upload batch are recorded in the candidate event group for each upload batch identifier. Unattached data is data with an empty attachment mark. An empty attachment mark indicates that the data has not yet been written to the candidate event group, the return group, the expansion group, or the completion group. In S3-2, based on the first and last data points acquired, the first and last data points received, the included source table, and the included batch table, unattached data is categorized. The categorization proceeds in the following order: erroneously merged data, re-inserted data, clipped data, pre-test data, and data retained for review. First, the unattached data is inserted into the candidate event group according to its received bit order, and the order reversal number is recalculated. If the order reversal number increases after insertion, the unattached data is written as an erroneously merged data point. Among the data not written as erroneously merged data, those with a retransmission identifier and whose data source type is not included in the included source table are... For data not specified as erroneous or reconnected, if the received bit is between the first and last received bit and the uploaded batch identifier is not included in the batch table, it is written as a clipped bit; for data not specified as erroneous, reconnected, or clipped, if the received bit is less than the event start received bit and the data source type is measured data, it is written as a pre-test bit; the remaining unattached data is written as a check bit; the number of adjacent relationships where the next data acquisition bit is less than the previous data acquisition bit is calculated after the candidate event groups are arranged in ascending order by the received bit. In S3-3, write-allowed positions are generated for the pre-test chip, clipped chip, reinserted chip, erroneously connected chip, and retained chip, respectively. The pre-test chip is only allowed to be written before the event start-up receive bit, used to supplement measurement data that is stuck before the event start-up due to alarm priority uploading. The clipped chip is only allowed to be written between adjacent upload batch identifiers, which are determined in ascending order by the first receive bit of each upload batch within the candidate event group, used to connect data within the group that was split by the uploaded batch. The reinserted chip is only allowed to be written at the location of the missing source item. The missing source item is generated using the group identifier and the missing data source type. If the reinserted chip receive bit falls within the candidate event group... The interval between two adjacent data receiving bits is used as the location of the missing source item. If the received bit of the inserted chip is less than the first received bit, the position before the event start receiving bit is used as the location of the missing source item. If the received bit of the inserted chip is greater than the last received bit, the position after the last received bit is used as the location of the missing source item. Erroneously connected chips are written to the rejection table, and chips to be checked are written to the test list. The data identifier, chip type, allowed write position, missing item identifier, break identifier, reversal identifier, and group identifier are written to the limit table. The break identifier is generated by the group identifier, the identifier of the previous upload batch, and the identifier of the next upload batch. The reversal identifier is generated by the group identifier, the previous data receiving bit, and the next data receiving bit. In S3-4, the limit table and the test table's pre-test piece, clipped piece, reinserted piece, and retained piece are written one by one through a greedy operation of the posterior sub-module. In each round of operation, a test data is temporarily written to the candidate event group according to the allowed write position in the limit table. The number of missing source items, batch breakage, and sequence reversal before and after the trial writing are calculated, and the reduction in the number of missing source items, batch breakage, and sequence reversal is obtained. The number of missing source items is the number of data source types that do not appear in the candidate event group in the target source table. The target source table is generated from the data source types that have actually appeared under the same device identifier in the out-of-order segment. The number of batch breakage is the number of adjacent data upload batches after the candidate event group is sorted in ascending order by received bits. The number of breaks where the same device identifier is not attached to data exists between different identifiers; each round selects data that generates new or reduced amounts and does not reuse the same missing item identifier, the same break identifier, or the same reversal identifier and writes them into the candidate event group; if multiple data meet the writing requirements, they are sorted by the total of the three reduction amounts, and if the totals are the same, they are sorted in the order of the reduction amount of the source missing items, the reduction amount of the batch break items, and the reduction amount of the sequence reversal items, and if they are still the same, they are sorted in ascending order of the received bits; data that does not meet the writing requirements are kept in the test list; after each data is written, the number of source missing items, the number of batch break items, and the number of sequence reversals are recalculated, and the limit table and the test list are updated until there are no new or reduced amounts in this round, and a return group is generated; In S3-5, based on the D-optimal supplementary verification operation, the eliminated source missing items, batch breaks, and sequence reversals within the repositioning group are written into the supplementary verification matrix. The supplementary verification matrix uses the eliminated missing item identifier, break identifier, and reversal identifier as columns, and the data already written within the repositioning group as rows. When a certain data eliminates a certain column item, that column is written as 1; otherwise, it is written as 0. Each data in the test table is tested, and the test data is temporarily added to the repositioning group according to the allowed writing position in the limit table, and a row of supplementary verification vector is generated for the test data. After the supplementary verification vector is added to the supplementary verification matrix, the determinant increment of the square matrix obtained by multiplying the transpose of the supplementary verification matrix by the supplementary verification matrix is ​​calculated. The data with the highest determinant increment and which does not increase the number of source missing items, batch breaks, and sequence reversals after being added is written into the repositioning group. When the determinant increments are parallel, they are sorted by the total reduction of the three items, and then sorted in ascending order of the received bits. Data with a determinant increment not greater than zero are retained in the test table. After writing is completed, an expansion group is generated. In S3-6, data from the test list and the rejection list are inserted back into the expansion group one by one, and the number of missing items, batch breaks, and sequence reversals are recalculated. The insertion order is to process the test list first, then the rejection list. The test list is inserted back in ascending order of received bits, and the rejection list is inserted back in ascending order of the increase in sequence reversals. When inserting each data, the order of insertion position in the expansion group is determined by the received bit, and the changes in the three items before and after insertion are calculated. If the number of missing items, batch breaks, and sequence reversals have not increased, and at least one of them has decreased, then the data is merged into the expansion group and written into the group identifier; otherwise, the attachment mark is deleted. Deleting the attachment mark only indicates that the data will not enter the completion group; the original data is still retained in the transmission sequence and can be used as blocking input or unattached data of the next candidate event group. After the test list and the rejection list are processed, the expansion group is output as the completion group. Through the above processing, unattached data is no longer directly merged into the candidate event group according to the reception time. Instead, it is first divided according to its fragment attributes in edge data transmission, and then the decision on whether to enter the completion group is made step by step according to the allowed write position, the reduction of the three items, the increment of the supplementary matrix, and the back-insertion review. This process can put the measured data before the event starting point, the interrupted data between batches, and the late supplementary data back into their appropriate positions in the candidate event group, while eliminating erroneous data that would introduce order reversal. In practical applications: alarm data from the same device has been included in the candidate event group as the event starting point; temperature measurements are uploaded before the event starting point due to delays in the upload of ordinary data; vibration measurements are located between two upload batches; and image summaries have a supplementary upload identifier and the image summary data has not yet entered the source table. This step will write the temperature measurements as the previous measurement piece, the vibration measurements as the clipping piece, and the image summary as the back-insertion piece. Then, through a greedy operation of the posterior sub-modulus, data that can reduce source missing items and batch breaks will be added first. Then, through the D-optimal supplementary verification operation, data that contributes more to the supplementary verification matrix will be added. Finally, back-insertion verification will be performed on the remaining data to generate a complete group for subsequent transmission state splicing.

[0021] S4. Perform the transmission state post-hoc splicing algorithm on the supplementary group, rearrange the data in the group according to the transmission state order of first upload, cache upload and breakpoint retransmission, and determine the data path in which the source collection order, gateway receiving order and retransmission order are all true at the same time as the event path. This implementation focuses on the process of generating event paths for the completion group. After the completion group has completed the placement of the pre-test chip, clip chip, and back-insertion chip, the group data is first written into the transmission state according to the supplementary transmission identifier and the upload batch identifier. Then, candidate edges are generated according to the adjacent relationship of the received bits. The source bit, gateway bit, and supplementary transmission bit are used to determine whether the candidate edges can carry over the field acquisition order, the edge gateway receiving order, and the breakpoint supplementary transmission order. Subsequently, the initial path is spliced ​​according to the edge posterior value, and the D-optimization bridge operation is performed on the empty bits in the initial path. Data that has not yet entered the path but can fill the breakpoint is incorporated into the path to obtain the event path. The implementation process includes the following steps: In S4-1, the data in the data completion group is read as input. The data completion identifier, upload batch identifier, and receive bits are read. The first data in the data completion group is determined in ascending order of the receive bits, and the upload batch identifier of the first data is taken as the first batch in the group. Data without a completion identifier and whose upload batch identifier is the first batch in the group is written as the first upload state. Data without a completion identifier and whose upload batch identifier is different from the first batch in the group is written as the cached upload state. Data with a completion identifier is written as the breakpoint completion state. After the transmission state is written, the data payload and group identifier in the data completion group are not changed. Only the transmission state field is added to each data in the group, and a transmission state table is generated. The transmission state table includes at least the group identifier, data identifier, data source type, acquisition bits, receive bits, upload batch identifier, completion identifier, and transmission state. In S4-2, the transmission state table is read using the transmission state post-hoc splicing algorithm. First, the data within the padded group is sorted in ascending order of received bits. Then, candidate edges are generated using two adjacent data points within the group. Candidate edges include preceding and following data. For adjacent data of the same data source type, if the following data acquisition bit is greater than the preceding data acquisition bit, a source bit is written into the candidate edge; otherwise, an empty bit is written. For candidate edges between different data source types, no source bit judgment is performed. If the following data received bit is greater than the preceding data received bit, a gateway bit is written; otherwise, an empty bit is written. If the following data is in a breakpoint retransmission state and the following data upload batch identifier is greater than the preceding data upload batch identifier, a retransmission bit is written; otherwise, an empty bit is written. For non-breakpoint retransmission state data, the retransmission bit is not a necessary judgment. After candidate edge processing, an edge bit table is generated, recording candidate edges, source bits, gateway bits, retransmission bits, and empty bits. In S4-3, the edge posterior value is generated based on the number of available bits in the source internal bit, gateway bit, and supplementary bit in the edge bit table. For each available bit in the source internal bit, gateway bit, and supplementary bit, the edge posterior value is incremented. Path concatenation starts from the first uploaded data state, reads candidate edges adjacent to the last data in the current path, selects candidate edges in descending order of edge posterior value, and writes them into the initial path. When edge posterior values ​​are the same, candidate edges are selected in ascending order of the received bits of the subsequent data. When the received bits of the subsequent data are the same, candidate edges are selected in the order of first uploaded state, cached uploaded state, and breakpoint supplementary transmission state. Each piece of data within a group is written to the initial path only once; data that has already entered the initial path with the same received bits is not written again. After the above edge concatenation is completed, the initial path is generated. In S4-4, a D-optimization bridging operation is performed on candidate edges with empty slots in the initial path. Data that has not entered the initial path is inserted one by one between the preceding and following data of the candidate edge containing the empty slot. The source internal bit, gateway bit, and supplementary bit are regenerated according to the adjacency relationship after insertion, resulting in a three-digit row. The path bit matrix uses the candidate edges in the initial path as rows and the source internal bit, gateway bit, and supplementary bit as columns, writing 1 for available bits and 0 for empty bits. After merging the three-digit rows into the path bit matrix, the determinant increment of the square matrix obtained by transposing the path bit matrix and multiplying it by the path bit matrix is ​​calculated. Data with the first position of the determinant increment and whose bits participating in the current candidate edge judgment are all available bits are merged into the initial path; otherwise, the data is moved back to the completion group and no longer participates in this round of D-optimization bridging operation. After all candidate edges containing empty slots have been processed, the bridged initial path is generated as the event path. Through the above processing, the data in the group is not simply sorted by the received bits and then entered into the subsequent fusion. Instead, the edge data transmission process is first identified by the initial upload state, the cached upload state, and the breakpoint retransmission state. Then, the source bits, gateway bits, and retransmission bits are used to verify the connection relationship between adjacent data. Data that can fill the gaps is added through D-optimized bridge operation, so that the event path can simultaneously reflect the on-site acquisition order, the edge gateway receiving order, and the breakpoint retransmission order. In practical applications: Temperature and current measurements of the same motor arrive with the first upload batch, PLC status data enters the next upload batch due to gateway caching, and image summaries with a retransmission identifier arrive after network recovery. This step writes temperature and current measurements as the first upload state, PLC status data as the cached upload state, and image summaries as the breakpoint retransmission state. Candidate edges are then generated based on the acquisition bits, receiving bits, and upload batch identifiers. If the source bit and gateway bit between temperature and current measurements are valid, and the retransmission bit between PLC status data and image summaries is valid, the relevant data are sequentially concatenated into the event path. Data that fails to fill gaps is retained in the completion group for subsequent delimitation and blocking judgment.

[0022] S5. Rewrite the event boundary of the completion group according to the event path. Output the completion group with closed path and zero source missing items as full item input. Output the completion group with closed path but still with source missing items as missing item input. Output the data with open path or reversed order that has not been eliminated as blocking input. This implementation focuses on the process of turning the event path output into a fused input. After the event path has been generated by splicing the completion group, the event boundary of the completion group is first redefined using the range of received bits in the event path. Then, the number of missing items and the number of order reversals are recalculated for the data within the delimited group. Path closure bits or path disconnect bits are generated based on the source bits, gateway bits, and supplementary transmission bits. Finally, the delimited group is split into full input, missing item input, or blocked input. The implementation process includes the following steps: In S5-1, taking the event path and the padding group as input, the data with the smallest received bit in the event path is read as the first bit data, and the data with the largest received bit in the event path is read as the last bit data. The received bits from the first bit data to the last bit data are used as the event boundary of the padding group, and the event boundary includes the first bit data and the last bit data. Then, all data with the same group identifier in the padding group are read. Data whose received bits are within the event boundary are retained in the padding group, and data whose received bits are less than the first received bit or greater than the last received bit are removed from the padding group. The removed data has its group identifier deleted and is written to the outside data table. The data in the outside data table is used as blocking input or the source of unattached data for the next candidate event group. After the boundary processing is completed, the data retained within the event boundary is used to generate the delimitation group. In S5-2, the number of missing sources and the number of order reversals are recalculated based on the delimitation group. The number of missing sources is calculated according to the number of data source types that do not appear in the delimitation group in the target source table. The target source table is the same as the table formed in S3 by the data source types that actually appeared in the same device identifier within the out-of-order segment. The number of order reversals is calculated according to the ascending order of the received bits of the data in the delimitation group. When the bit of the next data acquisition is less than the bit of the previous data acquisition, it is counted as one order reversal. Then, the candidate edges in the event path are read. Adjacent data of the same data source type are checked for source bits, all adjacent data are checked for gateway bits, and data in the breakpoint retransmission state are checked for retransmission bits. When all the bits involved in the check exist, a path closure bit is generated. When any bit involved in the check is empty, a path opening bit is generated. When there is no data in the breakpoint retransmission state in the event path, the retransmission bit is not used for closure judgment. In S5-3, the results are output in the order of blocking first, then splitting. When the delimited group has a path break bit or the sequence reversal number is not zero, the data that causes the path break or sequence reversal, the group identifier, the break position, the reversal position, and the original data payload are output as blocking inputs. When the delimited group has a path closure bit and the sequence reversal number is zero, it is split according to the number of missing items from the source. The output of the delimited group with zero missing items from the source is the full input. The full input includes the group identifier, event boundary, event path, data payload of each data source type, number of missing items from the source, number of batch breaks, and number of sequence reversals. The output of the delimited group with a non-zero number of missing items from the source is the missing input. The missing input adds a source missing table to the full input fields. The source missing table records the missing data source type and group identifier. Through the above processing, the missing data group undergoes event boundary shrinkage, path closure judgment, and three types of input diversion before entering the fusion calculation. This can prevent data outside the event path from continuing to participate in the same fusion input, and also prevent data remaining after path breakage or sequence reversal from entering the subsequent fusion calculation. This processing creates a mutual exclusion relationship between full input, missing input, and blocked input. The subsequent fusion calculation can directly read the full input, process the missing input according to the source missing input table, and perform manual review, edge retransmission, or next candidate event group reorganization on blocked input. In practical applications: The event path of a certain motor consists of temperature measurement, current measurement, PLC status data, and supplementary image summary. The first receiving bit of the event path is 15, and the last receiving bit is 22. If there is still log data in the same group with receiving bit 24 in the completion group, this step uses receiving bits 15 to 22 as the event boundary and moves the data with receiving bit 24 out of the completion group. Then, the delimitation group is recalculated. If the temperature measurement to the current measurement has source bits and gateway bits, and the PLC status data to the supplementary image summary has gateway bits and supplementary bits, and the number of missing source items is zero, then the full item input is output. If the image summary is still missing but the path is closed, then the missing item input is output. If any candidate edge is missing a bit that participates in the check, or if there is still a sequence reversal in the delimitation group, then the blocking input is output.

[0023] Furthermore, a multi-source heterogeneous big data fusion preprocessing intelligent system includes: The out-of-order identification module is used to acquire multi-source raw data in the edge data transmission link, write multi-source raw data under the same device identifier into the transmission sequence according to the receiving order of the edge gateway, and identify data segments in the transmission sequence whose acquisition order is opposite to the receiving order to generate out-of-order segments. The event reorganization module is used to take the first alarm data or the first state switching data in the out-of-order segment as the event starting point, read forward the measurement data that has not yet been attached under the same device identifier, read backward the image summary and supplementary data under the same device identifier, and reorganize the read data into a candidate event group. The greedy completion module is used to perform a batch posterior gain greedy algorithm on the candidate event group. It tests the changes in the number of missing items, batch breaks, and order reversals caused by adding unattached data to the candidate event group one by one, and writes the unattached data with the largest reduction into the candidate event group until the remaining unattached data cannot be reduced by any amount, thus generating a completion group. The post-transmission stitching module is used to perform a transmission-state post-transmission stitching algorithm on the completion group, rearrange the data in the group according to the transmission state order of first upload, cache upload and breakpoint retransmission, and determine the data path that simultaneously meets the source acquisition order, gateway reception order and retransmission order as the event path; The boundary rewriting module rewrites the event boundaries of the completion group according to the event path. It outputs the completion group with closed path and zero source missing items as full input, the completion group with closed path but still with source missing items as missing item input, and the data with open path or reversed order that has not been eliminated as blocking input.

[0024] The working principle of this scheme is as follows: First, read the original data from multiple sources of the same device in the edge data transmission link and assign acquisition bits and reception bits to determine whether the on-site acquisition order is reversed with the edge gateway reception order. When an out-of-order segment is detected, take the alarm data or state switching data as the event starting point, trace back the measured data, and then connect the image summary and supplementary data to form a candidate event group. Then, divide the unattached data into pre-test pieces, clipped pieces, back-inserted pieces, erroneously merged pieces, and retained pieces. Through the greedy operation of the post-analysis sub-modulus and the D-optimal supplementation operation, add the data that can reduce source missing items, batch breaks, and order reversal into the event group to obtain the complete group. Finally, splice the event path according to the transmission state of the first upload, cache upload, and breakpoint supplementation, and redefine the event boundary according to the event path. Output the result as a full-item input, missing-item input, or blocked input to avoid the same on-site event being separated or mistakenly merged in the preprocessing stage. In practical applications, for example, when a production device experiences an abnormal temperature rise, temperature measurements, current measurements, PLC status changes, alarm logs, and image summaries will be generated sequentially on-site. However, during edge data transmission, alarm logs may arrive first due to their higher priority, while temperature and current measurements may be cached by the gateway and then retransmitted. Image summaries may also arrive late due to the large amount of data. With this solution, the system will not simply divide these data into multiple events based on arrival time. Instead, it will identify out-of-order data through the acquisition and reception bits, and then, using the alarm log as the starting point, reclassify the previously retransmitted temperature and current measurements and the later-arriving image summaries into the same event. If the data source is complete and the path is closed, a full-item input is generated. If an image summary or a certain type of measurement is still missing, a missing item input is generated. If the sequential relationship cannot be closed, a blocking input is generated, thus allowing subsequent fusion analysis to receive data that better matches the on-site process.

[0025] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-source heterogeneous big data fusion preprocessing method, characterized in that, include: S1. Obtain multi-source raw data in the edge data transmission link, write multi-source raw data under the same device identifier into the transmission sequence according to the receiving order of the edge gateway, and identify data segments in the transmission sequence whose acquisition order is opposite to the receiving order, and generate out-of-order segments. S2. Using the first alarm data or the first state switching data in the disordered segment as the event starting point, read forward the measurement data that has not yet been attached under the same device identifier, and read backward the image summary and supplementary data under the same device identifier. Reorganize the read data into a candidate event group. S3. Perform a batch posterior gain greedy algorithm on the candidate event group, test the changes in the number of missing items, batch breaks, and order reversals caused by adding unattached data to the candidate event group, and write the unattached data with the largest reduction in quantity into the candidate event group until the remaining unattached data cannot be reduced by any quantity, and generate a completion group. Among them, the number of missing sources is the number of data source types that do not appear in the candidate event group in the target source table; the number of batch breaks is the number of breaks where adjacent data upload batch identifiers are different and there is a break between them where the same device identifier is not attached to the data after the candidate event group is sorted in ascending order of receiving bits; the number of order reversals is the number of adjacent relationships where the later data acquisition bit is less than the previous data acquisition bit after the candidate event group is sorted in ascending order of receiving bits. S4. Perform the transmission state post-hoc splicing algorithm on the supplementary group, rearrange the data in the group according to the transmission state order of first upload, cache upload and breakpoint retransmission, and determine the data path in which the source collection order, gateway receiving order and retransmission order are all true at the same time as the event path. S5. Rewrite the event boundaries of the completion group according to the event path. Output the completion group with closed path and zero source missing items as full item input. Output the completion group with closed path but still with source missing items as missing item input. Output the data with open path or reversed order that has not been eliminated as blocking input.

2. The multi-source heterogeneous big data fusion preprocessing method according to claim 1, characterized in that: S1 includes: S1-1. Read the raw data from multiple sources under the same device identifier, assign the acquisition bits in ascending order of acquisition time, assign the receiving bits in ascending order of edge gateway reception, and write the acquisition bits and receiving bits into the transmission sequence. S1-2. Scan the transmission sequence along the increasing direction of the received bits, mark two multi-source raw data with adjacent received bits and the latter collected bit being less than the former collected bit as reverse pairs, and generate the out-of-order start bit and the out-of-order end bit in the former received bit of the reverse pair. S1-3. Limit the interception range by the starting and ending positions of the out-of-order sequence, intercept the original data from multiple sources in the transmission sequence, and generate out-of-order segments of data whose acquisition bits are not arranged in ascending order in the interception result.

3. The multi-source heterogeneous big data fusion preprocessing method according to claim 2, characterized in that: S2 includes: S2-1. Read the alarm data and state switching data in the out-of-order segment, select the first alarm data or the first state switching data in ascending order of received bits as the event starting point, and record the device identifier, acquisition bit and receiving bit of the event starting point; S2-2. Starting with the receiving bit at the event origin, read the measurement data that has not yet been attached under the same device identifier along the decreasing direction of the receiving bit. Read until the beginning of the transmission sequence, the attached data, or the position where the device identifier changes, and generate a forward measurement group.

4. The multi-source heterogeneous big data fusion preprocessing method according to claim 3, characterized in that: S2 further includes: S2-3. Starting with the receiving bit at the beginning of the event, read the image summary and supplementary data under the same device identifier along the increasing direction of the receiving bit. Read until the end of the transmission sequence, the next alarm data, or the next state switching data, and generate a backward supplementary evidence group. S2-4. Write the forward measurement group, event start point and backward supplementary verification group into the same group record in ascending order of received bits, and generate the candidate event group with the device identifier of the event start point and the received bit generation group identifier.

5. The multi-source heterogeneous big data fusion preprocessing method according to claim 4, characterized in that: S3 includes: S3-1. Using candidate event groups, event start points, and unattached data as input, calculate the first acquisition bit, the last acquisition bit, the first reception bit, and the last reception bit according to the acquisition bit and reception bit of the data in the candidate event group, and generate a table containing the source and a table containing the batch according to the data source type and upload batch identifier. S3-2. Based on the first and last bits of data acquisition, the first and last bits of data reception, the included source table, and the included batch table, write the unattached data whose received bit is less than the event start receiving bit and whose data source type is measured data as the pre-test chip; write the unattached data whose received bit is between the first and last bits of reception and whose uploaded batch identifier has not entered the included batch table as the clipped chip; write the unattached data with a supplementary transmission identifier and whose data source type has not entered the included source table as the back-insertion chip; write the unattached data whose order reversal number is increased after being written into the candidate event group as the erroneous chip; otherwise, write it as the chip to be checked. S3-3. For pre-test chips, clipped chips, reinserted chips, erroneously merged chips, and chips under investigation, limit the pre-test chip to be written before the event start point receiving bit, limit the clipped chip to be written between adjacent upload batch identifiers, limit the reinserted chip to be written at the location of the source missing item, write the erroneously merged chip to the rejection table, and write the chips under investigation to the test table, and generate a limit table.

6. The multi-source heterogeneous big data fusion preprocessing method according to claim 5, characterized in that: S3 further includes: S3-4. Through greedy operation of the post-test sub-module, write the pre-test piece, clip piece, back-insertion piece and retained piece in the limit table and the test table one by one. Calculate the reduction of source missing items, batch breakage, and sequence reversal after the trial writing. Write the data that generate new reduction and do not reuse the same missing item, the same breakage, or the same reversal into the candidate event group. Otherwise, keep it in the test table and generate the return group. S3-5. Based on the D-optimal supplementary verification operation, write the eliminated source missing items, batch breaks and order reversals in the repositioning group into the supplementary verification matrix. Calculate the determinant increment of each data in the test table after adding it to the supplementary verification matrix. Write the data with the first determinant increment that does not increase the number of source missing items, batch breaks and order reversals into the repositioning group. Otherwise, keep it in the test table and generate the expansion group. S3-6. Insert the data from the test table and the elimination table back into the expansion group one by one and recalculate the number of missing items from the source, the number of batch breaks, and the number of sequence reversals. Merge the data whose three quantities have not increased and whose at least one quantity has decreased into the expansion group; otherwise, delete the attachment mark and output the expansion group as the completion group.

7. The multi-source heterogeneous big data fusion preprocessing method according to claim 6, characterized in that: S4 includes: S4-1. Taking the supplementary group as input, write the data in the group into the transmission state according to the supplementary transmission identifier and the upload batch identifier. Data without supplementary transmission identifier and whose upload batch identifier is the first batch in the group is written into the first upload state. Data without supplementary transmission identifier and whose upload batch identifier is not the first batch in the group is written into the cached upload state. Data with supplementary transmission identifier is written into the breakpoint supplementary transmission state. Generate the transmission state table. S4-2. Read the transmission state table using the transmission state post-hoc splicing algorithm. Generate candidate edges based on two adjacent data within the same group under the same data source type. Write the data into the source bit when the data acquisition bit of the later bit is greater than the data acquisition bit of the earlier bit. Write the data into the gateway bit when the data reception bit of the later bit is greater than the data reception bit of the earlier bit. Write the data into the supplementary transmission bit when the data of the later bit is in the breakpoint retransmission state and the data upload batch identifier of the later bit is greater than the data upload batch identifier of the earlier bit. Otherwise, write the empty bit and generate the edge bit table.

8. The multi-source heterogeneous big data fusion preprocessing method according to claim 7, characterized in that: S4 further includes: S4-3. Generate edge posterior values ​​using the number of bits in the source internal bit, gateway bit, and supplementary transmission bit in the edge bit table. Starting from the first uploaded state data, concatenate edges one by one according to the decreasing edge posterior values. When the edge posterior values ​​are the same, select candidate edges in ascending order of received bits to generate the initial path. S4-4. Perform D-optimization bridge operation on the candidate edges where empty positions are located in the initial path. Insert the data that has not entered the initial path into the candidate edges where empty positions are located. Form a three-dimensional row according to the source internal bit, gateway bit, and supplementary bit after insertion. Calculate the determinant increment of the three-dimensional row after merging it into the path bit matrix. Merge the data with the first determinant increment and no empty positions in the three-dimensional row into the initial path. Otherwise, move it back to the completion group and generate the event path.

9. The multi-source heterogeneous big data fusion preprocessing method according to claim 8, characterized in that: S5 includes: S5-1. Taking the event path and padding group as input, read the first and last received bits in the event path, take the first to last received bits as the event boundary of the padding group, and move the data outside the event boundary with the same group identifier out of the padding group to generate the delimitation group. S5-2. Based on the delimited group, recalculate the number of missing items and the number of order reversals. When adjacent data in the event path have source bits and gateway bits and the breakpoint supplementary transmission data have supplementary transmission bits, generate a path closure bit; otherwise, generate a path opening bit. S5-3. Output the delimited group with path closure bit and zero source missing item number as full item input, output the delimited group with path closure bit and non-zero source missing item number as missing item input, and output the data with path break bit or non-zero sequence reversal number as blocking input.

10. A multi-source heterogeneous big data fusion preprocessing intelligent system, characterized in that: include: The out-of-order identification module is used to acquire multi-source raw data in the edge data transmission link, write multi-source raw data under the same device identifier into the transmission sequence according to the receiving order of the edge gateway, and identify data segments in the transmission sequence whose acquisition order is opposite to the receiving order to generate out-of-order segments. The event reorganization module is used to take the first alarm data or the first state switching data in the out-of-order segment as the event starting point, read forward the measurement data that has not yet been attached under the same device identifier, read backward the image summary and supplementary data under the same device identifier, and reorganize the read data into a candidate event group. The greedy completion module is used to perform a batch posterior gain greedy algorithm on the candidate event group. It tests the changes in the number of missing items, batch breaks, and order reversals caused by adding unattached data to the candidate event group one by one, and writes the unattached data with the largest reduction into the candidate event group until the remaining unattached data cannot be reduced by any amount, thus generating a completion group. Among them, the number of missing sources is the number of data source types that do not appear in the candidate event group in the target source table; the number of batch breaks is the number of breaks where adjacent data upload batch identifiers are different and there is a break between them where the same device identifier is not attached to the data after the candidate event group is sorted in ascending order of receiving bits; the number of order reversals is the number of adjacent relationships where the later data acquisition bit is less than the previous data acquisition bit after the candidate event group is sorted in ascending order of receiving bits. The post-transmission stitching module is used to perform a transmission-state post-transmission stitching algorithm on the completion group, rearrange the data in the group according to the transmission state order of first upload, cache upload and breakpoint retransmission, and determine the data path that simultaneously meets the source acquisition order, gateway reception order and retransmission order as the event path; The boundary rewriting module rewrites the event boundaries of the completion group according to the event path. It outputs the completion group with closed path and zero source missing items as full input, the completion group with closed path but still with source missing items as missing item input, and the data with open path or reversed order that has not been eliminated as blocking input.

Images