A port intelligent construction site multi-source data fusion method and system based on an internet of things

Abstract

This invention provides a method and system for multi-source data fusion at a smart port construction site based on the Internet of Things (IoT). The method includes: acquiring initial sensor data and video image data from the port construction site, where the initial sensor data includes initial equipment status data, initial personnel safety data, and initial environmental monitoring data; acquiring fused equipment status data, fused personnel safety data, and fused environmental monitoring data through data cleaning and fusion; converting the video image data into scene description metadata; performing time alignment on the fused equipment status data, fused personnel safety data, fused environmental monitoring data, and scene description metadata; and acquiring multi-source fused data based on a multi-head attention mechanism. By performing time alignment, asynchronous data streams that are out of sync in time are synchronized, and the multi-head attention mechanism enables cross-modal data fusion, strengthening the correlation between modalities and improving the efficiency of data fusion.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for multi-source data fusion in smart port construction sites based on the Internet of Things. Background Technology

[0002] Port construction sites are complex environments that integrate cargo loading and unloading, warehousing and transshipment, and engineering construction. Their working environment is characterized by dense equipment (cranes, transport vehicles, loading and unloading machinery, etc.), frequent personnel movement, dispersed work areas, and many environmental interference factors (wind, rain, tides, dust, etc.). Traditional management models face many pain points: severe data fragmentation, single perception dimension, low data processing efficiency, and insufficient level of intelligence.

[0003] Existing IoT data fusion technologies are mostly applied to scenarios such as urban security and industrial production, lacking adaptability to the special characteristics of port construction sites (such as complex outdoor environments, multi-device collaborative operations, and cross-management of multiple businesses). In particular, existing deep learning methods (such as CNN-LSTM) struggle to effectively handle the challenges of multi-stream asynchronous time-series data (device data once / second, personnel data once / 3 seconds, environment data once / minute) and cross-modal feature alignment (numerical sensor data and image-based video data) when processing multi-source heterogeneous data at port construction sites, resulting in insufficient feature extraction, missing intermodal correlations, and low fusion efficiency. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a multi-source data fusion method and system for smart port construction sites based on the Internet of Things. This invention addresses the technical problems of existing technologies, such as the difficulty in adapting multi-source data fusion to the special characteristics of port construction sites, the inability to handle multi-stream asynchronous time-series data and cross-modal feature alignment, resulting in insufficient feature extraction, missing inter-modal correlations, and low fusion efficiency.

[0005] To achieve the above objectives, in a first aspect, embodiments of this application provide a multi-source data fusion method for smart port construction sites based on the Internet of Things, comprising the following steps:

[0006] Acquire initial sensor data and video image data from the port construction site. The initial sensor data includes initial equipment status data, initial personnel safety data, and initial environmental monitoring data with different sampling frequencies. Clean the initial sensor data to obtain updated sensor data, which includes updated equipment status data, updated personnel safety data, and updated environmental monitoring data.

[0007] The updated device status data is fused to obtain fused device status data, and the updated personnel safety data and the updated environment monitoring data are fused to obtain fused personnel safety data and fused environment monitoring data, respectively.

[0008] The video image data is converted into scene description metadata, and the sampling frequency of the scene description metadata is the same as that of the initial device status data. The fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata are time-aligned.

[0009] The multi-head attention mechanism is used to fuse the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata into multi-source fusion data.

[0010] Furthermore, the initial equipment status data, initial personnel safety data, and initial environmental monitoring data each include several initial data sets. Each initial data set corresponds to a sensor and includes several initial time-series data points. The step of cleaning the initial sensor data includes:

[0011] The initial time series data in the initial acquisition data group are subjected to moving average filtering based on a sliding time window to convert the initial time series data into corrected time series data. Several corrected time series data are combined into a corrected acquisition data group.

[0012] Obtain the sample mean and sample standard deviation corresponding to the calibration data set; select the maximum deviation data from several calibration time series data based on the sample mean and the sample standard deviation; and compare the maximum deviation data with the deviation threshold.

[0013] If the maximum deviation data is greater than the deviation threshold, the maximum deviation data is replaced by padding data to complete a single round of screening. This single round of screening is repeated several times until the maximum deviation data is less than the deviation threshold to obtain an updated collection data set.

[0014] Furthermore, the formula for obtaining the correction time series data is:

[0015] ,

[0016] in, This represents the correction timing data in the i-th time frame. This represents the initial time series data in the j-th time frame. This represents a sliding time window, and =5, Indicates rounding down;

[0017] The formula for obtaining the maximum deviation data is:

[0018] ,

[0019] in, This represents the maximum deviation data in the m-th calibration data set. This represents the sample mean of the m-th calibration data set. This represents the sample standard deviation of the m-th calibration data set. This indicates taking the maximum value.

[0020] Furthermore, the updated device status data includes several updated data collection groups, each corresponding to a sensor. The step of fusing the updated device status data to obtain fused device status data includes:

[0021] Based on the sensor type, several updated data acquisition groups are divided into several sensor acquisition data sets, and the updated data acquisition groups in the sensor acquisition data sets are selected as the data groups to be fused.

[0022] Obtain the credibility factor and environmental impact factor of each data group to be fused in the sensor acquisition data set, and obtain the weight coefficient based on the credibility factor and the environmental impact factor;

[0023] The fusion weights are obtained through the weight coefficients, and all the fusion weights are normalized to obtain normalized weights. Based on the normalized weights, several data groups to be fused are fused into sub-fusion data groups, and several sub-fusion data groups are combined into fusion device status data.

[0024] Furthermore, the formula for obtaining the credibility factor is:

[0025] ,

[0026] in, This represents the confidence factor of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame. This represents the number of abnormal data points in the b-th data group to be fused within 600 time frames within the sensor acquisition data set corresponding to the a-th sensor type.

[0027] The formula for obtaining the environmental impact factors is as follows:

[0028] ,

[0029] in, This represents the environmental impact factor of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame. This represents the correction time-series data in the i-th time frame of the b-th data group to be fused within the sensor acquisition data set corresponding to the a-th sensor type. This represents the correction time-series data in the (i-1)th time frame of the b-th data group to be fused within the sensor acquisition data set corresponding to the a-th sensor type. Represents the time constant. Represents an exponential function;

[0030] The formula for obtaining the weighting coefficient is:

[0031] ,

[0032] in, This represents the weight coefficient of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame;

[0033] The formula for obtaining the fusion weight is:

[0034] ,

[0035] in, The fusion weight of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type is shown in the i-th time frame.

[0036] Furthermore, the step of time-aligning the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata includes:

[0037] Within the fusion time window, based on the sampling frequency of the fusion device status data and the fusion personnel safety data, several first interpolation points corresponding to the fusion personnel safety data are obtained. A first fill value for the first interpolation point is obtained through a linear interpolation algorithm, and the first fill value is filled into the first interpolation point.

[0038] Within the fusion time window, based on the sampling frequency of the fusion device status data and the fusion environment monitoring data, a number of second interpolation points corresponding to the fusion environment monitoring data are obtained, and preset fill values ​​are filled into the second interpolation points;

[0039] Within the fusion time window, the scene description metadata is dimensionality reduced based on a preset sampling factor to obtain dimensionality-reduced metadata. Several third interpolation points corresponding to the dimensionality-reduced metadata are obtained through the sampling frequency between the dimensionality-reduced metadata and the fusion device status data. The second fill value of the third interpolation point is obtained through a linear interpolation algorithm, and the second fill value is filled into the third interpolation point.

[0040] Furthermore, the formula for obtaining the first fill value is:

[0041] ,

[0042] in, This represents the first fill value of the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the qth known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the (q+1)th known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the time frame corresponding to the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the q-th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the (q+)th time frame in the sub-fusion data group corresponding to the (a)th sensor type in the fused personnel safety data. , All with Adjacent.

[0043] Furthermore, the step of fusing the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata into multi-source fused data based on a multi-head attention mechanism includes:

[0044] Based on the multi-head attention mechanism, the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata are updated into multi-dimensional device status data, multi-dimensional personnel safety data, multi-dimensional environment monitoring data, and multi-dimensional scene metadata, respectively.

[0045] Based on gating weights, the multi-dimensional device status data, the multi-dimensional personnel safety data, the multi-dimensional environmental monitoring data, and the multi-dimensional scene metadata are weighted and fused into multi-source fused data.

[0046] Furthermore, after the step of fusing the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata into multi-source fused data based on the multi-head attention mechanism, the method further includes:

[0047] Based on the multi-source fusion data, equipment fault diagnosis, safety risk warning, and construction progress optimization are performed.

[0048] Secondly, embodiments of this application provide an IoT-based multi-source data fusion system for smart port construction sites, applied to the IoT-based multi-source data fusion method for smart port construction sites described in the first aspect above. The system includes:

[0049] The acquisition module is used to acquire initial sensor data and video image data of the port construction site. The initial sensor data includes initial equipment status data, initial personnel safety data and initial environmental monitoring data with different sampling frequencies. The initial sensor data is cleaned to obtain updated sensor data, which includes updated equipment status data, updated personnel safety data and updated environmental monitoring data.

[0050] The fusion module is used to perform data fusion on the updated device status data to obtain fused device status data, and to perform data fusion on the updated personnel safety data and the updated environment monitoring data to obtain fused personnel safety data and fused environment monitoring data respectively.

[0051] The correction module is used to convert the video image data into scene description metadata. The sampling frequency of the scene description metadata is the same as that of the initial device status data. The module performs time alignment on the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata.

[0052] The execution module is used to fuse the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata into multi-source fusion data based on a multi-head attention mechanism.

[0053] Thirdly, embodiments of this application provide a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the IoT-based multi-source data fusion method for smart construction sites in ports as described in the first aspect above.

[0054] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the Internet of Things-based multi-source data fusion method for smart port construction sites as described in the first aspect above.

[0055] Compared with existing technologies, the beneficial effects of this invention are as follows: For port construction site applications, the initial sensor data and video image data are selected to construct a multi-dimensional data foundation for port construction sites. Data cleaning ensures the accuracy and reliability of subsequent multi-source fusion data. Through data fusion, monitoring data from multiple sensors are considered for the same monitoring data. In the presence of local interference, data complementarity from multiple sensors can be achieved to further improve data accuracy and fully extract features. By aligning the four types of data based on sampling frequency, data from different modalities are unified onto the same high-frequency, equally spaced virtual timestamp sequence, enabling time synchronization of asynchronous data streams that are misaligned in time. Furthermore, a multi-head attention mechanism is used to complete cross-modal data fusion, strengthening the correlation between modalities and improving the efficiency of data fusion. Attached Figure Description

[0056] Figure 1 This is a flowchart of the multi-source data fusion method for smart port construction sites based on the Internet of Things in the first embodiment of the present invention;

[0057] Figure 2 This is a structural block diagram of the multi-source data fusion system for smart port construction sites based on the Internet of Things in the second embodiment of the present invention;

[0058] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0059] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0060] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0062] Please see Figure 1 The first embodiment of the present invention provides a multi-source data fusion method for smart port construction sites based on the Internet of Things, comprising the following steps:

[0063] S10: Acquire initial sensor data and video image data of the port construction site. The initial sensor data includes initial equipment status data, initial personnel safety data and initial environmental monitoring data with different sampling frequencies. Perform data cleaning on the initial sensor data to obtain updated sensor data. The updated sensor data includes updated equipment status data, updated personnel safety data and updated environmental monitoring data.

[0064] The initial equipment status data, initial personnel safety data, and initial environmental monitoring data all include several initial data acquisition groups. Each initial data acquisition group corresponds to a sensor and includes initial time-series data over several time periods. Taking the initial equipment status data as an example, it includes vibration sensors and current sensors; there are four vibration sensors and four current sensors. Therefore, the initial equipment status data includes eight initial data acquisition groups. For a specific vibration sensor, the initial data acquisition group includes vibration amplitude values ​​over several time periods. In this embodiment, the sampling frequency of the initial equipment status data is 60 times / min (i.e., once per second), the initial personnel safety monitoring data is 20 times / min (i.e., once every 3 seconds), and the initial environmental monitoring data is 1 time / min. It can be understood that the video image data is a video stream, with one frame per second. The initial personnel safety data includes heart rate data and location data, and the initial environmental monitoring data includes dust concentration, wind speed, and temperature.

[0065] Step S10 includes:

[0066] S110: Perform a sliding average filter on the initial time series data in the initial acquisition data group based on a sliding time window to convert the initial time series data into corrected time series data, and combine several of the corrected time series data into a corrected acquisition data group.

[0067] The formula for obtaining the correction time series data is:

[0068] ,

[0069] in, This represents the correction timing data in the i-th time frame. This represents the initial time series data in the j-th time frame. This represents a sliding time window, and =5, This indicates rounding down. It should be noted that when j ≤ 0, the lower limit is set to 1, and the upper limit is set to M.

[0070] S120: Obtain the sample mean and sample standard deviation corresponding to the calibration data set; select the maximum deviation data from several calibration time series data based on the sample mean and the sample standard deviation; and compare the maximum deviation data with the deviation threshold.

[0071] The sample mean is obtained by weighted averaging of the time-series data in the calibration collection data set. The sample standard deviation is obtained based on the sample mean, and its application is widespread, so it will not be elaborated here. The formula for obtaining the maximum deviation data is as follows:

[0072] ,

[0073] in, This represents the maximum deviation data in the m-th calibration data set. This represents the sample mean of the m-th calibration data set. This represents the sample standard deviation of the m-th calibration data set. This indicates taking the maximum value.

[0074] S130: If the maximum deviation data is greater than the deviation threshold, the maximum deviation data is replaced by padding data to complete the single round of screening. The single round of screening is repeated several times until the maximum deviation data is less than the deviation threshold to obtain the updated collection data group.

[0075] The filling data is the mean of the two corrected time series data adjacent to the maximum deviation data.

[0076] S20: Perform data fusion on the updated device status data to obtain fused device status data, and perform data fusion on the updated personnel safety data and the updated environment monitoring data respectively to obtain fused personnel safety data and fused environment monitoring data;

[0077] Understandably, the updated device status data, the updated personnel safety data, and the updated environmental monitoring data all include several updated data collection groups, which are the same as the initial data collection groups and correspond to the sensors. Specifically, step S20 includes:

[0078] S210: Based on the sensor type, the several updated data acquisition groups are divided into several sensor acquisition data sets, and the updated data acquisition groups in the sensor acquisition data sets are selected as the data groups to be fused.

[0079] In the updated device status data, taking the number of vibration sensors as 4 and the number of current sensors as 4 as an example, the 8 updated data acquisition groups form two sensor acquisition data sets, which correspond to the vibration sensors and the current sensors respectively. In this step, the purpose is to internally fuse the data from the 4 vibration sensors and the 4 current sensors respectively.

[0080] S220: Obtain the credibility factor and environmental impact factor of each data group to be fused in the sensor acquisition data set, and obtain the weight coefficient based on the credibility factor and the environmental impact factor;

[0081] The formula for obtaining the credibility factor is:

[0082] ,

[0083] in, This represents the confidence factor of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame. This indicates the number of abnormal data in the b-th data group to be fused within 600 time frames in the sensor acquisition data set corresponding to the a-th sensor type; the number of abnormal data in the b-th data group to be fused within 600 time frames in the sensor acquisition data set corresponding to the a-th sensor type can be obtained through the data cleaning process in step S10.

[0084] The formula for obtaining the environmental impact factors is as follows:

[0085] ,

[0086] in, This represents the environmental impact factor of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame. This represents the correction time-series data in the i-th time frame of the b-th data group to be fused within the sensor acquisition data set corresponding to the a-th sensor type. This represents the correction time-series data in the (i-1)th time frame of the b-th data group to be fused within the sensor acquisition data set corresponding to the a-th sensor type. Represents the time constant. Represents an exponential function;

[0087] The formula for obtaining the weighting coefficient is:

[0088] ,

[0089] in, This represents the weight coefficient of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame.

[0090] S230: Obtain the fusion weight through the weight coefficient, normalize all the fusion weights to obtain normalized weight, and merge several data groups to be fused into sub-fusion data groups based on the normalized weight, and combine several sub-fusion data groups into fusion device status data.

[0091] The formula for obtaining the fusion weight is:

[0092] ,

[0093] in, The fusion weight of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type is shown in the i-th time frame. After obtaining the fusion weight, the fusion weights of all the data groups to be fused in the i-th time frame are added together to obtain the total fusion weight. Each fusion weight is divided by the total fusion weight to obtain the normalized weight. Then, the corrected time-series data in the i-th time frame of each data group to be fused is multiplied by the corresponding normalized weight, and then summarized into fused time-series data. Several fused time-series data are combined into the sub-fused data group, and the sub-fused data group corresponds to the sensor type.

[0094] S30: Convert the video image data into scene description metadata. The sampling frequency of the scene description metadata is the same as the sampling frequency of the initial device status data. Time alignment is performed on the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata.

[0095] In this embodiment, the scene description metadata can be obtained through target detection and feature extraction (such as YOLO and Faster R-CNN recognition models to identify the video image data). The scene description metadata includes the number of targets, target location, target type, and target speed.

[0096] Step S30 includes:

[0097] S310: Within the fusion time window, based on the sampling frequency of the fusion device status data and the fusion personnel safety data, obtain a number of first interpolation points corresponding to the fusion personnel safety data, obtain the first fill value of the first interpolation point through a linear interpolation algorithm, and fill the first fill value into the first interpolation point;

[0098] In this embodiment, the fusion time window is 60 seconds. Based on the foregoing, the sampling frequency of the fusion device status data is 60 times / min, and the sampling frequency of the fusion personnel safety data is 20 times / min. Therefore, 40 first interpolation points need to be generated. After determining the first interpolation points, the formula for obtaining the first fill value is:

[0099] ,

[0100] in, This represents the first fill value of the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the qth known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the (q+1)th known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the time frame corresponding to the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the q-th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the (q+1)th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. , All with Adjacent.

[0101] S320: Within the fusion time window, based on the sampling frequency of the fusion device status data and the fusion environment monitoring data, obtain a number of second interpolation points corresponding to the fusion environment monitoring data, and fill the second interpolation points with preset fill values;

[0102] The sampling frequency of the fused environmental monitoring data is 1 time / min, which corresponds to the generation of 59 second interpolation points. It should be noted that since the fused environmental monitoring data monitors data such as dust concentration and noise, the changes in the short time are small. Therefore, the sampling frequency is low. The existing data in the fused time window is selected as the preset fill value and repeatedly filled into each second interpolation point.

[0103] S330: Within the fusion time window, the scene description metadata is dimensionality reduced based on a preset sampling factor to obtain dimensionality-reduced metadata. Several third interpolation points corresponding to the dimensionality-reduced metadata are obtained through the sampling frequency between the dimensionality-reduced metadata and the fusion device status data. The second fill value of the third interpolation point is obtained through a linear interpolation algorithm, and the second fill value is filled into the third interpolation point.

[0104] Taking the preset sampling factor of 3 as an example, the 60-dimensional scene description metadata is divided into 20 datasets with 3 frames as the dividing point. The datasets are averaged to generate 20-dimensional scene description metadata. Then, the dimensions are increased to 60 dimensions by padding. Through the process of first reducing the dimensions and then increasing the dimensions, not only can the time alignment of the overall data be completed, but also the random errors that may exist in single-frame detection can be filtered out, and the overall state of the scene can be preserved.

[0105] S40: Based on a multi-head attention mechanism, the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata are fused into multi-source fusion data;

[0106] Specifically, step S40 includes:

[0107] S410: Based on the multi-head attention mechanism, update the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata to multi-dimensional device status data, multi-dimensional personnel safety data, multi-dimensional environment monitoring data, and multi-dimensional scene metadata respectively;

[0108] Taking the acquisition process of the multidimensional device status data as an example, firstly, the query vector of the fused device status data is obtained through linear transformation, and then the key vectors and value vectors of the fused personnel safety data, the fused environment monitoring data, and the scene description multi-dimensional data are obtained respectively. Based on the query vector of the fused device status data, the key vector and value vector of the fused personnel safety data, the attention vector between the fused device status data and the fused personnel safety data can be obtained. The acquisition method of the attention vector between the fused device status data, the fused environment monitoring data, and the scene description multi-dimensional data is similar. These three are combined to form the multidimensional device status data. The acquisition methods of the multidimensional personnel safety data, the multidimensional environment monitoring data, and the multidimensional scene metadata are the same as those of the multidimensional device status data, and will not be elaborated here.

[0109] S420: Based on gating weights, the multi-dimensional device status data, the multi-dimensional personnel safety data, the multi-dimensional environmental monitoring data, and the multi-dimensional scene metadata are weighted and fused into multi-source fused data;

[0110] Specifically, the multi-dimensional device status data, multi-dimensional personnel safety data, multi-dimensional environmental monitoring data, and multi-dimensional scene metadata are input into the gating network. Based on the current comprehensive information, the gating network determines the proportion of information occupied by each modality in the final decision, and after normalization, forms the gating weight of each modality. The sum of the four gating weights is 1. Then, the multi-source fusion data is formed by weighted summation of the data of the four modalities with the four gating weights respectively.

[0111] For port construction site applications, the initial sensor data and video image data were selected to construct a multi-dimensional data foundation for port construction sites. Data cleaning ensured the accuracy and reliability of subsequent multi-source fusion data. Through data fusion, monitoring data from multiple sensors were considered for the same monitoring data. In the presence of local interference, data from multiple sensors could be used for data complementarity to further improve data accuracy and achieve full feature extraction. By aligning the four types of data based on sampling frequency, data from different modalities were unified onto the same high-frequency, equally spaced virtual timestamp sequence, enabling time synchronization of asynchronous data streams that were misaligned in time. Furthermore, a multi-head attention mechanism was used to complete cross-modal data fusion, strengthening the correlation between modalities and improving the efficiency of data fusion.

[0112] Preferably, the method further includes:

[0113] S50: Based on the multi-source fusion data, perform equipment fault diagnosis, safety risk warning, and construction progress optimization;

[0114] Specifically, the equipment fault diagnosis, the safety risk warning, and the construction progress optimization are all implemented through a pre-set neural network model. The neural network model for equipment fault diagnosis includes two fully connected layers. The neural network model for equipment fault diagnosis is as follows:

[0115] ,

[0116] in, This represents the probability value of a failure occurring within the next time window. The weight matrix of the first fully connected layer in the second layer. This represents the weight matrix of the first fully connected layer. This indicates multi-source fused data. This represents the bias amount of the first fully connected layer. This represents the bias amount of the first fully connected layer in the second layer. Represents the linear rectification activation function. Represents the Sigmoid function;

[0117] The neural network model for security risk early warning includes two second fully connected layers. The neural network model for security risk early warning is as follows:

[0118] ,

[0119] in, This represents the probability distribution vector of the risk level within the next time window. The weight matrix of the second fully connected layer. This represents the weight matrix of the first and second fully connected layers. This represents the bias amount of the first and second fully connected layers. This indicates the bias amount of the second fully connected layer;

[0120] The neural network model for optimizing construction progress includes two fully connected third layers. The neural network model for optimizing construction progress is as follows:

[0121] ,

[0122] in, This indicates the predicted schedule deviation time. The weight matrices of the second and third fully connected layers, This represents the weight matrix of the first and third fully connected layers. This represents the bias of the first and third fully connected layers. This represents the bias value of the second and third fully connected layers. After obtaining the schedule deviation time, the A* algorithm is invoked to dynamically plan the scheduling paths of on-site transport vehicles and cranes to minimize the total operation delay. By introducing three types of neural models, the multi-source fused data is combined with the actual application at the port construction site, improving the overall intelligence level of operation and maintenance.

[0123] Please see Figure 2 The second embodiment of the present invention provides a multi-source data fusion system for smart port construction sites based on the Internet of Things (IoT). This system is applied to the multi-source data fusion method for smart port construction sites based on the IoT described in the above embodiments, and will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0124] The system includes:

[0125] The acquisition module 10 is used to acquire initial sensor data and video image data of the port construction site. The initial sensor data includes initial equipment status data, initial personnel safety data and initial environmental monitoring data with different sampling frequencies. The initial sensor data is cleaned to obtain updated sensor data. The updated sensor data includes updated equipment status data, updated personnel safety data and updated environmental monitoring data.

[0126] The acquisition module 10 includes:

[0127] The first unit is used to perform a sliding average filter on the initial time series data in the initial acquisition data group based on a sliding time window, so as to convert the initial time series data into corrected time series data, and a plurality of the corrected time series data are combined into a corrected acquisition data group.

[0128] The second unit is used to obtain the sample mean and sample standard deviation corresponding to the calibration data set, select the maximum deviation data from a number of calibration time series data based on the sample mean and the sample standard deviation, and compare the maximum deviation data with the deviation threshold.

[0129] The third unit is used to replace the maximum deviation data with padding data if the maximum deviation data is greater than the deviation threshold, so as to complete a single round of screening. The single round of screening is repeated several times until the maximum deviation data is less than the deviation threshold, so as to obtain an updated collection data group.

[0130] The fusion module 20 is used to perform data fusion on the updated device status data to obtain fused device status data, and to perform data fusion on the updated personnel safety data and the updated environment monitoring data to obtain fused personnel safety data and fused environment monitoring data respectively.

[0131] The fusion module 20 includes:

[0132] The fourth unit is used to divide several updated data acquisition groups into several sensor acquisition data sets based on sensor type, and to select the updated data acquisition groups in the sensor acquisition data sets as the data groups to be fused.

[0133] The fifth unit is used to obtain the credibility factor and environmental impact factor of each data group to be fused in the sensor acquisition data set, and to obtain the weight coefficient based on the credibility factor and the environmental impact factor.

[0134] The sixth unit is used to obtain the fusion weight through the weight coefficient, normalize all the fusion weights to obtain normalized weights, and merge several data groups to be fused into sub-fusion data groups based on the normalized weights, and combine several sub-fusion data groups into fusion device status data.

[0135] The correction module 30 is used to convert the video image data into scene description metadata, wherein the sampling frequency of the scene description metadata is the same as the sampling frequency of the initial device status data, and to perform time alignment on the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata;

[0136] The correction module 30 includes:

[0137] The seventh unit is used to obtain a number of first interpolation points corresponding to the safety data of the personnel being integrated based on the sampling frequency of the integrated device status data and the integrated personnel safety data within the integration time window, obtain the first fill value of the first interpolation point through a linear interpolation algorithm, and fill the first fill value into the first interpolation point;

[0138] The eighth unit is used to obtain a number of second interpolation points corresponding to the fusion environment monitoring data based on the sampling frequency of the fusion device status data and the fusion environment monitoring data within the fusion time window, and fill the second interpolation points with preset fill values;

[0139] The ninth unit is used to perform dimensionality reduction processing on the scene description metadata based on a preset sampling factor within the fusion time window to obtain dimensionality-reduced metadata, obtain a number of third interpolation points corresponding to the dimensionality-reduced metadata through the sampling frequency between the dimensionality-reduced metadata and the fusion device status data, obtain the second fill value of the third interpolation point through a linear interpolation algorithm, and fill the third interpolation point with the second fill value.

[0140] Execution module 40 is used to fuse the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata into multi-source fusion data based on a multi-head attention mechanism;

[0141] The execution module 40 includes:

[0142] The tenth unit is used to update the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata into multi-dimensional device status data, multi-dimensional personnel safety data, multi-dimensional environment monitoring data, and multi-dimensional scene metadata based on a multi-head attention mechanism.

[0143] The eleventh unit is used to weightedly fuse the multi-dimensional device status data, the multi-dimensional personnel safety data, the multi-dimensional environmental monitoring data, and the multi-dimensional scene metadata into multi-source fusion data based on gating weights.

[0144] Preferably, the system further includes:

[0145] The prediction module 50 is used to perform equipment fault diagnosis, safety risk warning and construction progress optimization based on the multi-source fusion data.

[0146] The present invention also provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-source data fusion method for smart port construction sites based on the Internet of Things as described in the above technical solutions.

[0147] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-source data fusion method for smart port construction sites based on the Internet of Things as described in the above technical solution.

[0148] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0149] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for multi-source data fusion in a smart port construction site based on the Internet of Things, characterized in that, Includes the following steps: Acquire initial sensor data and video image data from the port construction site. The initial sensor data includes initial equipment status data, initial personnel safety data, and initial environmental monitoring data with different sampling frequencies. Clean the initial sensor data to obtain updated sensor data, which includes updated equipment status data, updated personnel safety data, and updated environmental monitoring data. The updated device status data is fused to obtain fused device status data, and the updated personnel safety data and the updated environment monitoring data are fused to obtain fused personnel safety data and fused environment monitoring data, respectively. The video image data is converted into scene description metadata, and the sampling frequency of the scene description metadata is the same as that of the initial device status data. The fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata are time-aligned. The step of time-aligning the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata includes: Within the fusion time window, based on the sampling frequency of the fusion device status data and the fusion personnel safety data, several first interpolation points corresponding to the fusion personnel safety data are obtained. A first fill value for the first interpolation point is obtained through a linear interpolation algorithm, and the first fill value is filled into the first interpolation point. The formula for obtaining the first fill value is: ， in, This represents the first fill value of the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the qth known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the (q+1)th known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the time frame corresponding to the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the q-th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the (q+1)th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. , All with Adjacent; Within the fusion time window, based on the sampling frequency of the fusion device status data and the fusion environment monitoring data, a number of second interpolation points corresponding to the fusion environment monitoring data are obtained, and preset fill values ​​are filled into the second interpolation points; Within the fusion time window, the scene description metadata is dimensionality reduced based on a preset sampling factor to obtain dimensionality-reduced metadata. Several third interpolation points corresponding to the dimensionality-reduced metadata are obtained through the sampling frequency between the dimensionality-reduced metadata and the fusion device status data. The second fill value of the third interpolation point is obtained through a linear interpolation algorithm, and the second fill value is filled into the third interpolation point. The multi-head attention mechanism is used to fuse the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata into multi-source fusion data.

2. The method for multi-source data fusion of smart port construction sites based on the Internet of Things as described in claim 1, characterized in that, The initial equipment status data, initial personnel safety data, and initial environmental monitoring data each include several initial data sets. Each initial data set corresponds to a sensor and includes several initial time-series data sets. The step of cleaning the initial sensor data includes: The initial time series data in the initial acquisition data group are subjected to moving average filtering based on a sliding time window to convert the initial time series data into corrected time series data. Several corrected time series data are combined into a corrected acquisition data group. Obtain the sample mean and sample standard deviation corresponding to the calibration data set; select the maximum deviation data from several calibration time series data based on the sample mean and the sample standard deviation; and compare the maximum deviation data with the deviation threshold. If the maximum deviation data is greater than the deviation threshold, the maximum deviation data is replaced by padding data to complete a single round of screening. This single round of screening is repeated several times until the maximum deviation data is less than the deviation threshold to obtain an updated collection data set.

3. The method for multi-source data fusion of smart port construction sites based on the Internet of Things as described in claim 2, characterized in that, The formula for obtaining the correction time series data is: ， in, This represents the correction timing data in the i-th time frame. This represents the initial time series data in the j-th time frame. This represents a sliding time window, and =5, Indicates rounding down; The formula for obtaining the maximum deviation data is: ， in, This represents the maximum deviation data in the m-th calibration data set. This represents the sample mean of the m-th calibration data set. This represents the sample standard deviation of the m-th calibration data set. This indicates taking the maximum value.

4. The method for multi-source data fusion of smart port construction sites based on the Internet of Things as described in claim 2, characterized in that, The updated device status data includes several updated data collection groups, each corresponding to a sensor. The step of fusing the updated device status data to obtain fused device status data includes: Based on the sensor type, several updated data acquisition groups are divided into several sensor acquisition data sets, and the updated data acquisition groups in the sensor acquisition data sets are selected as the data groups to be fused. Obtain the credibility factor and environmental impact factor of each data group to be fused in the sensor acquisition data set, and obtain the weight coefficient based on the credibility factor and the environmental impact factor; The fusion weights are obtained through the weight coefficients, and all the fusion weights are normalized to obtain normalized weights. Based on the normalized weights, several data groups to be fused are fused into sub-fusion data groups, and several sub-fusion data groups are combined into fusion device status data.

5. The method for multi-source data fusion of smart port construction sites based on the Internet of Things as described in claim 4, characterized in that, The formula for obtaining the credibility factor is: ， in, This represents the confidence factor of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame. This represents the number of abnormal data points in the b-th data group to be fused within 600 time frames within the sensor acquisition data set corresponding to the a-th sensor type. The formula for obtaining the environmental impact factors is as follows: ， in, This represents the environmental impact factor of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame. This represents the correction time-series data in the i-th time frame of the b-th data group to be fused within the sensor acquisition data set corresponding to the a-th sensor type. This represents the correction time-series data in the (i-1)th time frame of the b-th data group to be fused within the sensor acquisition data set corresponding to the a-th sensor type. Represents the time constant. Represents an exponential function; The formula for obtaining the weighting coefficient is: ， in, This represents the weight coefficient of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type at the i-th time frame; The formula for obtaining the fusion weight is: ， in, The fusion weight of the b-th data group to be fused in the sensor acquisition data set corresponding to the a-th sensor type is shown in the i-th time frame.

6. The method for multi-source data fusion of smart port construction sites based on the Internet of Things as described in claim 1, characterized in that, The step of fusing the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata into multi-source fused data based on a multi-head attention mechanism includes: Based on the multi-head attention mechanism, the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata are updated into multi-dimensional device status data, multi-dimensional personnel safety data, multi-dimensional environment monitoring data, and multi-dimensional scene metadata, respectively. Based on gating weights, the multi-dimensional device status data, the multi-dimensional personnel safety data, the multi-dimensional environmental monitoring data, and the multi-dimensional scene metadata are weighted and fused into multi-source fused data.

7. The method for multi-source data fusion of smart port construction sites based on the Internet of Things as described in claim 1, characterized in that, After the step of fusing the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata into multi-source fused data based on the multi-head attention mechanism, the method further includes: Based on the multi-source fusion data, equipment fault diagnosis, safety risk warning, and construction progress optimization are performed.

8. A multi-source data fusion system for smart port construction sites based on the Internet of Things (IoT), applied to the multi-source data fusion method for smart port construction sites based on the IoT as described in any one of claims 1 to 7, characterized in that, The system includes: The acquisition module is used to acquire initial sensor data and video image data of the port construction site. The initial sensor data includes initial equipment status data, initial personnel safety data and initial environmental monitoring data with different sampling frequencies. The initial sensor data is cleaned to obtain updated sensor data, which includes updated equipment status data, updated personnel safety data and updated environmental monitoring data. The fusion module is used to perform data fusion on the updated device status data to obtain fused device status data, and to perform data fusion on the updated personnel safety data and the updated environment monitoring data to obtain fused personnel safety data and fused environment monitoring data respectively. The correction module is used to convert the video image data into scene description metadata. The sampling frequency of the scene description metadata is the same as that of the initial device status data. The module performs time alignment on the fused device status data, the fused personnel safety data, the fused environment monitoring data, and the scene description metadata. The correction module includes: The seventh unit is used to obtain a number of first interpolation points corresponding to the safety data of the personnel being integrated based on the sampling frequency of the integrated device status data and the integrated personnel safety data within the integration time window, obtain the first fill value of the first interpolation point through a linear interpolation algorithm, and fill the first fill value into the first interpolation point; The formula for obtaining the first fill value is: ， in, This represents the first fill value of the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the qth known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the data of the (q+1)th known time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the time frame corresponding to the p-th first interpolation point in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the q-th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. This represents the (q+1)th time frame in the sub-fusion data group corresponding to the a-th sensor type in the fused personnel safety data. , All with Adjacent; The eighth unit is used to obtain a number of second interpolation points corresponding to the fusion environment monitoring data based on the sampling frequency of the fusion device status data and the fusion environment monitoring data within the fusion time window, and fill the second interpolation points with preset fill values; The ninth unit is used to perform dimensionality reduction processing on the scene description metadata based on a preset sampling factor within the fusion time window to obtain dimensionality-reduced metadata, obtain a number of third interpolation points corresponding to the dimensionality-reduced metadata through the sampling frequency between the dimensionality-reduced metadata and the fusion device status data, obtain the second fill value of the third interpolation point through a linear interpolation algorithm, and fill the third interpolation point with the second fill value. The execution module is used to fuse the fusion device status data, the fusion personnel safety data, the fusion environment monitoring data, and the scene description metadata into multi-source fusion data based on a multi-head attention mechanism.

Images