A compression processing method based on multi-source heterogeneous data fusion

Abstract

The present invention discloses a compression processing method based on multi-source heterogeneous data fusion. The method first collects and analyzes multi-source heterogeneous data, and then uses an established data fusion algorithm to perform fusion processing on the multi-source heterogeneous data to obtain a data fusion result and a fusion anomaly ratio. At the same time, a data compression algorithm is established to compress the data fusion result to obtain a first compression result, and then the data compression algorithm is trained according to the first compression result and the second compression result to obtain an updated data compression algorithm for the next compression, and the compression misjudgment ratio is also obtained. By establishing a unified analysis model, the influencing factors and corresponding influencing modes of the problems in the whole process are obtained from multiple angles, the problem situation is judged and processed, and a problem processing plan is designed to optimize the corresponding algorithm to ensure that the compression quality is improved in the data fusion process and the accuracy of the compression result is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a compression processing method based on multi-source heterogeneous data fusion. Background Art

[0002] With the continuous development of information technology, IoT data presents the characteristics of large scale, multiple sources and different structures. In addition to storing the data of related sensors, it also stores a large amount of video surveillance data in real time. In related technologies, data compression algorithms are usually adopted so that the data compression algorithms can be used to compress data in subsequent multiple data fusion processes. However, in the process of collecting or using data, the original data is prone to jitter, resulting in problems with data sequence and data interval, which in turn affects the compression quality in the data fusion process and causes the problem of unreliable compression results;

[0003] In view of the above technical defects, a solution is now proposed. Summary of the invention

[0004] The purpose of the present invention is to propose a compression processing method based on multi-source heterogeneous data fusion, which solves the problem that the compression quality is affected during the data fusion process and the compression result is unreliable.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: a compression processing method based on multi-source heterogeneous data fusion, comprising the following steps:

[0006] Step 1: Collect multi-source heterogeneous data: collect multi-source heterogeneous data through sensors and monitoring equipment, where the multi-source heterogeneous data includes sensor information and module monitoring information;

[0007] Step 2: Analyze multi-source heterogeneous data:

[0008] A pre-established data fusion algorithm is used to fuse multi-source heterogeneous data to obtain a data fusion result; a pre-established data compression algorithm is used to compress the data fusion result to obtain a first compression result, and a second compression result of the data fusion result is obtained by manual participation; the data compression algorithm is then trained according to the first compression result and the second compression result to obtain an updated data compression algorithm for the next compression;

[0009] At the same time, the influencing factors of the collection process include data transmission time and data signal-to-noise ratio; and by comparing and analyzing the data fusion results before and after each compression, the proportion of fusion anomalies and compression misjudgment in the measurement process;

[0010] Step 3: Build a unified analysis model: Set the sampling period, and substitute the influencing factors, compression misjudgment ratio, and fusion anomaly ratio within the same sampling period into the unified analysis model to determine the influencing factors and corresponding influencing patterns that cause data fusion and problems in the compression process;

[0011] Step 4: Design a solution to the problem: set corresponding threshold ranges for influencing factors, monitor the influencing factors in real time and control them within the preset range. If they do not meet the preset range, generate an optimization adjustment signal and optimize the corresponding algorithm immediately after receiving the optimization adjustment signal.

[0012] Furthermore, the specific working process of obtaining the data fusion result is as follows:

[0013] Step A1, marking the time between the start time and the end time of the acquisition process as a time threshold, and then dividing the time threshold into i sub-time periods, where i is a positive integer greater than 0; and then obtaining sensor information and module monitoring information in each sub-time period;

[0014] Step A2, preprocessing the sensor information and the module monitoring information respectively, and obtaining the first target data and the second target data correspondingly, integrating the first target data and the second target data to obtain the target data set; at the same time, by obtaining the control mode of the sensor and the monitoring device, the control mode includes the location, type, quantity and operation frequency of the sensor and the monitoring device, and establishing a common identification tag between the first target data and the second target data based on the control mode, and the common identification tag describes a hash value formed by the data corresponding to the sensor information and the module monitoring information;

[0015] Step A3, bring the target data set into the pre-established data fusion algorithm, and divide it into three parts: training set, validation set and test set; batch train the algorithm model through the training set to obtain multimodal feature vectors, and then classify the control mode used by the data corresponding to the multimodal feature vectors to design a fully connected layer, and use the softmax function as the activation function of the output layer, and use the gradient descent method to optimize the network weights; monitor the performance of the algorithm model through the validation set to adjust the network structure or parameters in the algorithm model to prevent overfitting of the data; evaluate the performance of the algorithm model through the test set to check the stability and generalization ability; and then train to obtain the data fusion result.

[0016] Furthermore, the preprocessing process includes:

[0017] Sensor information is in the form of digital and analog signals, and module monitoring information is in the form of text, pictures, and videos;

[0018] Step Sa, preprocessing the sensor information: first, performing digital-to-analog conversion on the digital signal and the analog signal, and converting them into data of the same type of signal, then performing wavelet transform, filtering and denoising on the data, and integrating them to obtain the first target data;

[0019] Step Sb, preprocessing the module monitoring information: using natural language processing technology to convert text into feature vectors, and marking them as feature vectors X1; using computer vision technology to extract feature vectors of images, and marking them as feature vectors X2; decomposing the video into a series of image frames, and then processing them according to step Sb2, and marking the obtained feature vectors as feature vectors X3;

[0020] Then, the characteristic vector X1, characteristic vector X2 and characteristic vector X3 in any sub-time period are obtained, and a characteristic monitoring vector <characteristic vector X1, characteristic vector X2, characteristic vector X3> is constructed. The monitoring vectors are integrated based on the time series to obtain the second target data.

[0021] Furthermore, the overall missing value Rh1 and the fusion ratio value Rh2 of the data before and after the fusion are obtained during the fusion process, and then the overall missing value Rh1 and the fusion ratio value Rh2 are multiplied to calculate the fusion abnormality ratio RH.

[0022] Further, training the data compression algorithm according to the first compression result and the second compression result includes:

[0023] The first compression result includes a first data set U1 that meets the preset compression standard, a second data set U2 that does not meet the preset compression standard, and a third data set U3 that cannot be determined whether it meets the preset compression standard, obtained by compressing the data fusion result according to the data compression algorithm;

[0024] The second compression result includes a fourth data set U4 that meets the compression standard and a fifth data set U5 that does not meet the preset compression standard, which is obtained by compressing the data fusion result in a manual manner;

[0025] Step B1, compare and analyze the first data set U1 and the fourth data set U4 to obtain the corresponding sixth data set U6 and seventh data set U7;

[0026] Then, the sixth data set U6 and the first identifier, and the seventh data set U7 and the second identifier are used as input information to train the data compression algorithm;

[0027] Step B2, comparing and analyzing the second data set U2 and the fifth data set U5 to obtain the corresponding eighth data set U8 and ninth data set U9;

[0028] Then, the eighth data set U8 and the first identifier, and the ninth data set U9 and the second identifier are used as input information to train the data compression algorithm;

[0029] Step B3, compare and analyze the fourth data set U4 with the first data set U1 and the third data set U3 to obtain the corresponding tenth data set U10, eleventh data set U11 and twelfth data set U12; and compare and analyze the third data set U3 with the fifth data set U5 to obtain the corresponding thirteenth data set U13;

[0030] Then, the tenth data set U10 and the first identifier, the eleventh data set U11 and the second identifier, the twelfth data set U12 and the third identifier, and the thirteenth data set U13 and the fourth identifier are used as input information to train the data compression algorithm.

[0031] Furthermore, the first identifier is used to identify data that meets the preset compression standard obtained according to the data compression algorithm and is mistakenly judged as data that does not meet the preset compression standard; the second identifier is used to identify data that does not meet the preset compression standard obtained according to the data compression algorithm and is mistakenly judged as data that meets the preset compression standard; the third identifier is used to indicate data that cannot be identified according to the data compression algorithm but actually meets the preset compression standard; the fourth identifier is used to indicate data that cannot be identified according to the data compression algorithm but actually does not meet the preset compression standard.

[0032] Furthermore, the number of data corresponding to the first identifier, the second identifier, the third identifier and the fourth identifier is accumulated to obtain the compression misjudgment ratio YS.

[0033] Furthermore, the specific working process of building a unified model is as follows:

[0034] Step C1, set the sampling period to Tc, set the influencing factors to set Rx, Rx={data transmission time R1, data signal-to-noise ratio R2}, and then accumulate each element in the set Rx within the sampling period to obtain the cumulative amount of each influencing factor within a sampling period, and mark it as a set RWx, and mark it as a set RWx, RWx={data transmission time cumulative amount RW1, data signal-to-noise ratio cumulative amount RW2}, and at the same time determine the problem proportion set FWx according to the fusion anomaly proportion RH and the compression misjudgment proportion YS, FWx={fusion anomaly proportion cumulative amount WRH, compression misjudgment proportion cumulative amount WYS}, and correspondingly set the problem category generated by the influencing factor to set Fx, Fx={fusion anomaly problem F1, compression misjudgment problem F2};

[0035] Step C2, analyzing the influence relationship between the influencing factor Rx and each element in the problem category Fx in the same sampling period, and determining the single influence mode of the single influencing factor at the same time, and then assigning corresponding weight coefficients to each element in the proportion set FWx of each problem category according to the different influence degrees of the single influencing factor on each problem category, and then obtaining the comprehensive influence degree value Mki, and constructing a dynamic relationship diagram S0 of the single influencing factor cumulative amount RWx-comprehensive influence degree value Mki with the single influencing factor cumulative amount RWx as the X-axis and the comprehensive influence degree value Mki as the Y-axis;

[0036] Step C3, first assign corresponding weight correction coefficients to the cumulative amount set RWx of each influencing factor according to the different influence degrees of each element in the set RWx on each problem category, and then obtain the comprehensive influence value Gzx through the formula; then, with the comprehensive influence value Gzx as the X-axis and the comprehensive influence degree value Mki as the Y-axis, establish a dynamic relationship diagram S1 of the comprehensive influence value Gzx-comprehensive influence degree value Mki, and at the same time determine the comprehensive influence mode of the comprehensive influencing factors.

[0037] Furthermore, the specific working process of designing a solution to the problem is as follows:

[0038] Step D1, first according to the dynamic relationship diagram S1 of the comprehensive influence value Gzx-comprehensive influence degree value Mki, at the same time preset a threshold relationship diagram S2 in the coordinate system where the dynamic relationship diagram S1 is located, the dynamic relationship diagram S1 and the threshold relationship diagram are overlapped with each other, and then the line segment corresponding to the dynamic relationship diagram S1 is obtained above the line segment corresponding to the dynamic relationship diagram S2, and it is marked as Xd, and then the line segment Xd is cut and divided into multiple small segments, and then the average value and fluctuation value corresponding to each small segment are obtained to determine the curve evaluation value Zm corresponding to the small segment, and then the horizontal coordinate Xzm of the critical point where the corresponding Zm on the line segment Xd is from 0 to a positive value is marked as the comprehensive factor threshold, and then the preset interval of the comprehensive influence value Gzx is obtained as [0, Xzm), and the comprehensive influence value Gzx is calculated by real-time monitoring of the data transmission time and the data signal-to-noise ratio, and it is limited to the corresponding preset interval;

[0039] Step D2, when the comprehensive impact value Gzx exceeds the preset interval, combined with the dynamic relationship diagram S0 of the single impact factor accumulation amount RWx-comprehensive impact degree value Mki, determine the single impact factor that does not meet the preset interval to optimize and adjust the corresponding algorithm.

[0040] Furthermore, the process of determining a single influencing factor that is not within the preset range is as follows:

[0041] Obtain all points on the dynamic relationship diagram S0 of the cumulative amount of a single influencing factor RWx-the comprehensive influence degree value Mki, and then obtain the critical point where the influence degree value Mki is located from a value of 0 to a positive value and the critical point where the influence degree value Mki is located from a negative value to a value of 0, and set the horizontal coordinate of the position of the critical point as the threshold of the single influencing factor, and then determine the preset interval of the single influencing factor, where the preset interval of the data transmission time is [0, qj1], and the preset interval of the data signal-to-noise ratio is [qj2, 0].

[0042] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0043] The present invention first collects and analyzes multi-source heterogeneous data, and then uses the established data fusion algorithm to fuse the multi-source heterogeneous data to obtain data fusion results and fusion anomaly ratios; at the same time, a data compression algorithm is established to compress the data fusion results to obtain a first compression result, and then the data compression algorithm is trained according to the first compression result and the second compression result to obtain an updated data compression algorithm for the next compression, and the compression misjudgment ratio is also obtained; by establishing a unified analysis model, the influencing factors and corresponding influencing modes of the problems in the whole process are obtained from multiple angles, the problem situation is determined and processed, and a problem processing plan is designed to optimize the corresponding algorithm to ensure that the compression quality is improved in the data fusion process and the accuracy of the compression result is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the process of the present invention is shown. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Embodiment 1:

[0047] like Figure 1 As shown, a compression processing method based on multi-source heterogeneous data fusion includes the following steps:

[0048] S1, step 1, collecting multi-source heterogeneous data: collecting multi-source heterogeneous data through sensors and monitoring equipment, the multi-source heterogeneous data includes sensor information and module monitoring information; wherein the sensor information is in the form of digital signals and analog signals, and the module monitoring information is in the form of text, pictures and videos;

[0049] S2, step 2, analyzing multi-source heterogeneous data:

[0050] S2-1, using a pre-established data fusion algorithm to fuse multi-source heterogeneous data to obtain a data fusion result;

[0051] Step A1, marking the time between the start time and the end time of the acquisition process as a time threshold, and then dividing the time threshold into i sub-time periods, where i is a positive integer greater than 0; and then obtaining sensor information and module monitoring information in each sub-time period;

[0052] Step A2: pre-process the sensor information and module monitoring information respectively:

[0053] Step Sa, preprocessing the sensor information: first, performing digital-to-analog conversion on the digital signal and the analog signal, and converting them into data of the same type of signal, then performing wavelet transform, filtering and denoising on the data, and integrating them to obtain the first target data;

[0054] Step Sb, preprocessing the module monitoring information:

[0055] Step Sb1, using natural language processing technology to convert the text into a feature vector and mark it as feature vector X1;

[0056] Step Sb2, using computer vision technology to extract the feature vector of the image, and mark it as the feature vector X2; it should be noted that both natural language processing technology and computer vision technology apply convolutional neural network models, and the specific process of using natural language processing technology is: first remove special characters, punctuation marks, stop words, etc. in the text, and then segment the text according to words to form a vocabulary, restore the vocabulary and restore it to its basic form to reduce the increase in dimensions caused by vocabulary deformation, and then encode the text according to the vocabulary to form a document-vocabulary matrix, calculate the frequency of the term in the document, and consider the inverse document frequency of the term in the entire corpus, and then obtain the feature vector; the specific process of using computer vision processing technology is: first preprocess the image, such as resizing, cropping, graying, etc., and then use the network model to extract image features;

[0057] Step Sb3, decomposing the video into a series of image frames, and then processing them according to step Sb2, marking the obtained feature vector as feature vector X3;

[0058] Step Sb3, obtaining the feature vector X1, feature vector X2 and feature vector X3 in any sub-time period, and constructing a feature monitoring vector <feature vector X1, feature vector X2, feature vector X3>, integrating the monitoring vectors based on the time series to obtain the second target data;

[0059] Step A2-1, integrating the first target data and the second target data to obtain a target data set;

[0060] Step A2-2, by simultaneously acquiring the control mode of the sensor and the monitoring device, the control mode includes the location, type, quantity and operating frequency of the sensor and the monitoring device, and establishing a common identification tag between the first target data and the second target data based on the control mode, and the common identification tag describes a hash value formed by the data corresponding to the sensor information and the module monitoring information;

[0061] Step A3, bring the target data set into the pre-established data fusion algorithm and divide it into three parts: training set, validation set and test set; batch train the algorithm model through the training set to obtain multimodal feature vectors, and then classify the control mode used by the data corresponding to the multimodal feature vector to design a fully connected layer, and use the softmax function as the activation function of the output layer, and use the gradient descent method to optimize the network weights; monitor the performance of the algorithm model through the validation set to adjust the network structure or parameters in the algorithm model to prevent overfitting of the data; evaluate the performance of the algorithm model through the test set to check the stability and generalization ability; and then train to obtain the data fusion result; at the same time, obtain the overall missing value Rh1 and the fusion ratio value Rh2 of the data before and after the fusion process, and then multiply the overall missing value Rh1 and the fusion ratio value Rh2 to calculate the fusion abnormality ratio RH: ; It should be noted that the overall missing value Rh1 indicates the data loss in the fusion result due to the difference between different data sources or the missing in the data collection process during the fusion process. The measurement indicators are the outliers and error values ​​in the data fusion process; the fusion ratio value Rh2 is expressed as the contribution weight of each data source or feature to the final fusion result during the data fusion process;

[0062] S2-2, compressing the data fusion result using a pre-established data compression algorithm to obtain a first compression result, and obtaining a second compression result of the data fusion result by manually participating in the compression process; and then training the data compression algorithm according to the first compression result and the second compression result to obtain an updated data compression algorithm for the next compression;

[0063] At the same time, the influencing factors of the collection process include data transmission time and data signal-to-noise ratio; and by comparing and analyzing the data fusion results before and after each compression, the proportion of fusion anomalies and compression misjudgment in the measurement process;

[0064] Wherein, training the data compression algorithm according to the first compression result and the second compression result includes:

[0065] The first compression result includes a first data set U1 that meets the preset compression standard, a second data set U2 that does not meet the preset compression standard, and a third data set U3 that cannot be determined whether it meets the preset compression standard, obtained by compressing the data fusion result according to the data compression algorithm;

[0066] The second compression result includes a fourth data set U4 that meets the compression standard and a fifth data set U5 that does not meet the preset compression standard, which is obtained by compressing the data fusion result in a manual manner;

[0067] Step B1, obtaining data not present in the first data set U1 from the fourth data set U4, to obtain a sixth data set U6; obtaining data not present in the fourth data set U4 from the first data set U1, to obtain a seventh data set U7;

[0068] Then, the sixth data set U6 and the first identifier, and the seventh data set U7 and the second identifier are used as input information to train the data compression algorithm;

[0069] Step B2, obtaining data not present in the fifth data set U5 from the second data set U2, to obtain an eighth data set U8; obtaining data not present in the second data set U2 from the fifth data set U5, to obtain a ninth data set U9;

[0070] Then, the eighth data set U8 and the first identifier, and the ninth data set U9 and the second identifier are used as input information to train the data compression algorithm;

[0071] Step B3, obtaining data from the fourth data set U4 that does not exist in the first data set U1 and the third data set U3, to obtain a tenth data set U10; obtaining data from the first data set U1 that does not exist in the fourth data set U4 and the third data set U3, to obtain an eleventh data set U11; obtaining data from the third data set U3 that does not exist in the fourth data set U4, to obtain a twelfth data set U12; obtaining data from the third data set U3 that does not exist in the fifth data set U5, to obtain a thirteenth data set U13;

[0072] Then, the tenth data set U10 and the first identifier, the eleventh data set U11 and the second identifier, the twelfth data set U12 and the third identifier, and the thirteenth data set U13 and the fourth identifier are used as input information to train the data compression algorithm; wherein, the first identifier is used to identify that data obtained according to the data compression algorithm that meets the preset compression standard is mistakenly judged as data that does not meet the preset compression standard; the second identifier is used to identify that data obtained according to the data compression algorithm that does not meet the preset compression standard is mistakenly judged as data that meets the preset compression standard; the third identifier is used to indicate that data obtained according to the data compression algorithm cannot be identified but actually meets the preset compression standard; the fourth identifier is used to indicate that data obtained according to the data compression algorithm cannot be identified but actually does not meet the preset compression standard; and then by The number of data corresponding to the first identifier, the second identifier, the third identifier and the fourth identifier are accumulated to obtain the compression misjudgment ratio YS; it should be noted that the sixth data set U6, the eighth data set U8 and the tenth data set U10 are all sets of data that meet the preset compression standard and are misjudged as data that do not meet the preset compression standard according to the data compression algorithm, and the seventh data set U7, the ninth data set U9 and the eleventh data set U11 are all sets of data that do not meet the preset compression standard and are misjudged as data that meet the preset compression standard according to the data compression algorithm; the twelfth data set U12 is a set of data that cannot be determined but actually meets the preset compression standard; the thirteenth data set U13 is a set of data that cannot be determined but actually does not meet the preset compression standard;

[0073] S3, step 3, building a unified analysis model: setting a sampling period, and substituting the influencing factors, compression misjudgment ratio and fusion anomaly ratio in the same sampling period into the unified analysis model to determine the influencing factors and corresponding influencing patterns that cause data fusion and problems in the compression process;

[0074] The specific process is:

[0075] Step C1, set the sampling period to Tc, set the influencing factors to set Rx, Rx = {data transmission time R1, data signal-to-noise ratio R2}, and then accumulate each element in the set Rx within the sampling period to obtain the cumulative amount of each influencing factor within a sampling period, and mark it as a set RWx, RWx = {data transmission time cumulative amount RW1, data signal-to-noise ratio cumulative amount RW2}, and at the same time determine the problem proportion set FWx according to the fusion anomaly proportion RH and the compression misjudgment proportion YS, FWx = {fusion anomaly proportion cumulative amount WRH, compression misjudgment proportion cumulative amount WYS}, and correspondingly set the problem category generated by the influencing factor to set Fx, Fx = {fusion anomaly problem F1, compression misjudgment problem F2};

[0076] Step C2, analyze the influence relationship between the influencing factor Rx and each element in the problem category Fx in the same sampling period, and determine the single influence mode of the single influencing factor. Then, according to the different influence degrees of the single influencing factor on each problem category, assign corresponding weight coefficients to each element in the proportion set FWx of each problem category, and then obtain the comprehensive influence degree value Mki: ; Among them, α1 and α2 are the weight coefficients of the cumulative fusion anomaly proportion WRH and the cumulative compression misjudgment proportion WYS to the fusion anomaly problem F1 and the compression misjudgment problem F2, respectively, and α1 and α2 are both greater than 0; It represents the functional relationship between the comprehensive impact value Mki and the accumulated data transmission time RW1, It represents the functional relationship between the comprehensive influence degree value Mki and the data signal-to-noise ratio RW2; and with the single influence factor cumulative amount RWx as the X-axis and the comprehensive influence degree value Mki as the Y-axis, a dynamic relationship diagram S0 between the single influence factor cumulative amount RWx and the comprehensive influence degree value Mki is constructed;

[0077] Step C3, firstly, according to the different influence degrees of each element in the set RWx on each problem category, the corresponding weight correction coefficient is assigned to the cumulative amount set RWx of each influencing factor, and then the formula is used: Obtain the comprehensive impact value Gzx; where β1 and β2 are the weight correction coefficients of the fusion abnormality ratio RH and the compression misjudgment ratio YS in the overall process, and both β1 and β2 are greater than 0; is the functional relationship between the comprehensive influence value Gzx and the comprehensive influence degree value Mki; and then, with the comprehensive influence value Gzx as the X-axis and the comprehensive influence degree value Mki as the Y-axis, a dynamic relationship diagram S1 of the comprehensive influence value Gzx-comprehensive influence degree value Mki is established, and at the same time, the comprehensive influence mode of the comprehensive influencing factors is determined;

[0078] S4, step 4, designing a solution to the problem: setting a corresponding threshold interval for the influencing factor, monitoring the influencing factor in real time and controlling it within the preset interval. If it does not meet the preset interval, an optimization adjustment signal is generated, and the corresponding algorithm is optimized immediately after receiving the optimization adjustment signal;

[0079] The specific process is:

[0080] Step D1, first according to the dynamic relationship diagram S1 of the comprehensive influence value Gzx-comprehensive influence degree value Mki, at the same time preset a threshold relationship diagram S2 in the coordinate system where the dynamic relationship diagram S1 is located, the dynamic relationship diagram S1 and the threshold relationship diagram are superimposed on each other, and then the line segment corresponding to the dynamic relationship diagram S1 is obtained above the line segment corresponding to the dynamic relationship diagram S2, and it is marked as Xd, and then the line segment Xd is cut and divided into multiple small segments, and then the average value pj and fluctuation value bd corresponding to each small segment are obtained to determine the curve evaluation value Zm corresponding to the small segment: ;in, is the normalization correction coefficient, which is used to convert the curve evaluation value Zm into the interval [-1, 1];

[0081] Then, the horizontal coordinate Xzm of the critical point where Zm on the line segment Xd is from 0 to a positive value is marked as the comprehensive factor threshold, and then the preset interval of the comprehensive influence value Gzx is obtained as [0, Xzm). By real-time monitoring of the data transmission time and the data signal-to-noise ratio, the comprehensive influence value Gzx is calculated and limited to the corresponding preset interval;

[0082] Step D2, when the comprehensive influence value Gzx exceeds the preset interval, the single influence factor that does not meet the preset interval is determined by combining the dynamic relationship diagram S0 of the single influence factor cumulative amount RWx-comprehensive influence degree value Mki, and the specific process is: obtain all points on the dynamic relationship diagram S0 of the single influence factor cumulative amount RWx-comprehensive influence degree value Mki, and then obtain the critical point where the influence degree value Mki is located from a value of 0 to a positive value and the critical point where the influence degree value Mki is located from a negative value to a value of 0, and set the horizontal coordinate of the critical point as the threshold value of the single influence factor, and then determine the preset interval of the single influence factor, wherein the preset interval of the data transmission time is [0, qj1], and the preset interval of the data signal-to-noise ratio is [qj2, 0];

[0083] When a single influencing factor is not within the corresponding preset range, the corresponding algorithm is immediately optimized and adjusted;

[0084] When the data transmission time is greater than the preset interval [0, qj1], it means that the data transmission time is too long, and the data fusion process is optimized, for example, by adding several blank new connection layers after the training layer in the pre-established data fusion algorithm, and gradually adjusting the full connection layer forward from the new connection layer based on the model input features to achieve model optimization; when the data signal-to-noise ratio is less than the preset interval [qj2, 0], it means that the data signal-to-noise ratio is too low, and the data compression process is optimized, for example, by directly performing the next data compression process in the pre-established data compression algorithm, the efficiency of data compression is improved and the proportion of compression misjudgments of data compression is reduced;

[0085] When the data transmission time and the data signal-to-noise ratio are not within the preset range, the data fusion process and the data compression process are optimized simultaneously.

[0086] In summary of the above technical solutions: the present invention first collects and analyzes multi-source heterogeneous data, and then uses the established data fusion algorithm to fuse the multi-source heterogeneous data to obtain data fusion results and the proportion of fusion anomalies; at the same time, a data compression algorithm is established to compress the data fusion results to obtain a first compression result, and then the data compression algorithm is trained according to the first compression result and the second compression result to obtain an updated data compression algorithm for the next compression, and the proportion of compression misjudgments is also obtained; by establishing a unified analysis model, the influencing factors and corresponding influencing modes of the problems in the whole process are obtained from multiple angles, the problem situation is determined and processed, and a problem processing plan is designed to optimize the corresponding algorithm to ensure that the compression quality is improved during the data fusion process and the accuracy of the compression results is improved.

[0087] The size of the interval and threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by technical personnel in this field for each group of sample data; as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0088] The above formulas are all dimensionless and numerical calculations. The formula is a formula obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0089] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A compression processing method based on multi-source heterogeneous data fusion, characterized by: The following steps are involved: Step 1: Collect multi-source heterogeneous data: collect multi-source heterogeneous data through sensors and monitoring equipment, where the multi-source heterogeneous data includes sensor information and module monitoring information; Step 2: Analyze multi-source heterogeneous data: A pre-established data fusion algorithm is used to fuse multi-source heterogeneous data to obtain a data fusion result; a pre-established data compression algorithm is used to compress the data fusion result to obtain a first compression result, and a second compression result of the data fusion result is obtained by manual participation; the data compression algorithm is then trained according to the first compression result and the second compression result to obtain an updated data compression algorithm for the next compression; At the same time, the influencing factors of the collection process include data transmission time and data signal-to-noise ratio; and by comparing and analyzing the data fusion results before and after each compression, the proportion of fusion anomalies and compression misjudgment in the measurement process; Step 3: Build a unified analysis model: Set the sampling period, and substitute the influencing factors, compression misjudgment ratio, and fusion anomaly ratio within the same sampling period into the unified analysis model to determine the influencing factors and corresponding influencing patterns that cause data fusion and problems in the compression process; Step 4: Design a solution to the problem: set corresponding threshold ranges for influencing factors, monitor the influencing factors in real time and control them within the preset range. If they do not meet the preset range, generate an optimization adjustment signal and optimize the corresponding algorithm immediately after receiving the optimization adjustment signal.

2. The compression processing method based on multi-source heterogeneous data fusion according to claim 1 is characterized in that: The specific working process of obtaining data fusion results is as follows: Step A1, marking the time between the start time and the end time of the acquisition process as a time threshold, and then dividing the time threshold into i sub-time periods, where i is a positive integer greater than 0; and then obtaining sensor information and module monitoring information in each sub-time period; Step A2, preprocessing the sensor information and the module monitoring information respectively, and obtaining the first target data and the second target data correspondingly, integrating the first target data and the second target data to obtain the target data set; at the same time, by obtaining the control mode of the sensor and the monitoring device, the control mode includes the location, type, quantity and operation frequency of the sensor and the monitoring device, and establishing a common identification tag between the first target data and the second target data based on the control mode, and the common identification tag describes a hash value formed by the data corresponding to the sensor information and the module monitoring information; Step A3, bring the target data set into the pre-established data fusion algorithm, and divide it into three parts: training set, validation set and test set; batch train the algorithm model through the training set to obtain multimodal feature vectors, and then classify the control mode used by the data corresponding to the multimodal feature vectors to design a fully connected layer, and use the softmax function as the activation function of the output layer, and use the gradient descent method to optimize the network weights; monitor the performance of the algorithm model through the validation set to adjust the network structure or parameters in the algorithm model to prevent overfitting of the data; evaluate the performance of the algorithm model through the test set to check the stability and generalization ability; and then train to obtain the data fusion result.

3. The compression processing method based on multi-source heterogeneous data fusion according to claim 2 is characterized in that: The preprocessing process includes: Sensor information is in the form of digital signals and analog signals, and module monitoring information is in the form of text, pictures and videos; Step Sa, preprocessing the sensor information: first, performing digital-to-analog conversion on the digital signal and the analog signal, and converting them into data of the same type of signal, then performing wavelet transform, filtering and denoising on the data, and integrating them to obtain the first target data; Step Sb, preprocessing the module monitoring information: using natural language processing technology to convert text into feature vectors, and marking them as feature vectors X1; using computer vision technology to extract feature vectors of images, and marking them as feature vectors X2; decomposing the video into a series of image frames, and then processing them according to step Sb2, and marking the obtained feature vectors as feature vectors X3; Then, the characteristic vector X1, characteristic vector X2 and characteristic vector X3 in any sub-time period are obtained, and a characteristic monitoring vector <characteristic vector X1, characteristic vector X2, characteristic vector X3> is constructed. The monitoring vectors are integrated based on the time series to obtain the second target data.

4. The compression processing method based on multi-source heterogeneous data fusion according to claim 2 is characterized in that: In the fusion process, the overall missing value Rh1 and the fusion ratio value Rh2 of the data before and after fusion are obtained, and then the overall missing value Rh1 and the fusion ratio value Rh2 are multiplied to calculate the fusion abnormality ratio RH.

5. The compression processing method based on multi-source heterogeneous data fusion according to claim 1 is characterized in that: Training the data compression algorithm according to the first compression result and the second compression result includes: The first compression result includes a first data set U1 that meets the preset compression standard, a second data set U2 that does not meet the preset compression standard, and a third data set U3 that cannot be determined whether it meets the preset compression standard, obtained by compressing the data fusion result according to the data compression algorithm; The second compression result includes a fourth data set U4 that meets the compression standard and a fifth data set U5 that does not meet the preset compression standard, which is obtained by compressing the data fusion result in a manual manner; Step B1, compare and analyze the first data set U1 and the fourth data set U4 to obtain the corresponding sixth data set U6 and seventh data set U7; Then, the sixth data set U6 and the first identifier, and the seventh data set U7 and the second identifier are used as input information to train the data compression algorithm; Step B2, comparing and analyzing the second data set U2 and the fifth data set U5 to obtain the corresponding eighth data set U8 and ninth data set U9; Then, the eighth data set U8 and the first identifier, and the ninth data set U9 and the second identifier are used as input information to train the data compression algorithm; Step B3, compare and analyze the fourth data set U4 with the first data set U1 and the third data set U3 to obtain the corresponding tenth data set U10, eleventh data set U11 and twelfth data set U12; and compare and analyze the third data set U3 with the fifth data set U5 to obtain the corresponding thirteenth data set U13; Then, the tenth data set U10 and the first identifier, the eleventh data set U11 and the second identifier, the twelfth data set U12 and the third identifier, and the thirteenth data set U13 and the fourth identifier are used as input information to train the data compression algorithm.

6. The compression processing method based on multi-source heterogeneous data fusion according to claim 5 is characterized in that: The first identifier is used to identify data obtained according to the data compression algorithm that meets the preset compression standard and is mistakenly judged as data that does not meet the preset compression standard; the second identifier is used to identify data obtained according to the data compression algorithm that does not meet the preset compression standard and is mistakenly judged as data that meets the preset compression standard; the third identifier is used to indicate data obtained according to the data compression algorithm that cannot be identified but actually meets the preset compression standard; the fourth identifier is used to indicate data obtained according to the data compression algorithm that cannot be identified but actually does not meet the preset compression standard.

7. The compression processing method based on multi-source heterogeneous data fusion according to claim 6 is characterized in that: The number of data corresponding to the first identifier, the second identifier, the third identifier, and the fourth identifier is accumulated to obtain the compression misjudgment ratio YS.

8. The compression processing method based on multi-source heterogeneous data fusion according to claim 1 is characterized in that: The specific working process of building a unified model is as follows: Step C1, set the sampling period to Tc, set the influencing factors to set Rx, Rx = {data transmission time R1, data signal-to-noise ratio R2}, and then accumulate each element in the set Rx within the sampling period to obtain the cumulative amount of each influencing factor within a sampling period, and mark it as a set RWx, RWx = {data transmission time cumulative amount RW1, data signal-to-noise ratio cumulative amount RW2}, and at the same time determine the problem proportion set FWx according to the fusion anomaly proportion RH and the compression misjudgment proportion YS, FWx = {fusion anomaly proportion cumulative amount WRH, compression misjudgment proportion cumulative amount WYS}, and correspondingly set the problem category generated by the influencing factor to set Fx, Fx = {fusion anomaly problem F1, compression misjudgment problem F2}; Step C2, analyzing the influence relationship between the influencing factor Rx and each element in the problem category Fx in the same sampling period, and determining the single influence mode of the single influencing factor at the same time, and then assigning corresponding weight coefficients to each element in the proportion set FWx of each problem category according to the different influence degrees of the single influencing factor on each problem category, and then obtaining the comprehensive influence degree value Mki, and constructing a dynamic relationship diagram S0 of the single influencing factor cumulative amount RWx-comprehensive influence degree value Mki with the single influencing factor cumulative amount RWx as the X-axis and the comprehensive influence degree value Mki as the Y-axis; Step C3, first assign corresponding weight correction coefficients to the cumulative amount set RWx of each influencing factor according to the different influence degrees of each element in the set RWx on each problem category, and then obtain the comprehensive influence value Gzx through the formula; then, with the comprehensive influence value Gzx as the X-axis and the comprehensive influence degree value Mki as the Y-axis, establish a dynamic relationship diagram S1 of the comprehensive influence value Gzx-comprehensive influence degree value Mki, and at the same time determine the comprehensive influence mode of the comprehensive influencing factors.

9. The compression processing method based on multi-source heterogeneous data fusion according to claim 1 is characterized in that: The specific work process of designing a solution to the problem is as follows: Step D1, first according to the dynamic relationship diagram S1 of the comprehensive influence value Gzx-comprehensive influence degree value Mki, at the same time preset a threshold relationship diagram S2 in the coordinate system where the dynamic relationship diagram S1 is located, the dynamic relationship diagram S1 and the threshold relationship diagram are overlapped with each other, and then the line segment corresponding to the dynamic relationship diagram S1 is obtained above the line segment corresponding to the dynamic relationship diagram S2, and it is marked as Xd, and then the line segment Xd is cut and divided into multiple small segments, and then the average value and fluctuation value corresponding to each small segment are obtained to determine the curve evaluation value Zm corresponding to the small segment, and then the horizontal coordinate Xzm of the critical point where the corresponding Zm on the line segment Xd is from 0 to a positive value is marked as the comprehensive factor threshold, and then the preset interval of the comprehensive influence value Gzx is obtained as [0, Xzm), and the comprehensive influence value Gzx is calculated by real-time monitoring of the data transmission time and the data signal-to-noise ratio, and it is limited to the corresponding preset interval; Step D2, when the comprehensive impact value Gzx exceeds the preset interval, combined with the dynamic relationship diagram S0 of the single impact factor accumulation amount RWx-comprehensive impact degree value Mki, determine the single impact factor that does not meet the preset interval to optimize and adjust the corresponding algorithm.

10. The compression processing method based on multi-source heterogeneous data fusion according to claim 9, characterized in that: The process of determining a single influencing factor that is not within the preset range is as follows: Obtain all points on the dynamic relationship diagram S0 of the cumulative amount of a single influencing factor RWx-the comprehensive influence degree value Mki, and then obtain the critical point where the influence degree value Mki is located from a value of 0 to a positive value and the critical point where the influence degree value Mki is located from a negative value to a value of 0, and set the horizontal coordinate of the position of the critical point as the threshold of the single influencing factor, and then determine the preset interval of the single influencing factor, where the preset interval of the data transmission time is [0, qj1], and the preset interval of the data signal-to-noise ratio is [qj2, 0].

Images