Multi-source heterogeneous data automatic processing fusion method and system based on convolutional neural network

Abstract

The invention relates to the field of data processing, and provides a multi-source heterogeneous data automatic processing fusion method and system based on a convolutional neural network, and the method comprises the steps: obtaining sensor data at different positions, and obtaining image data including the installation position of a sensor; establishing a spatial mapping relation according to a spatial corresponding relation between the sensor mounting position and the image feature area; organizing the sensor data into sensor two-dimensional data according to the mapping relation; determining a resonance region of the sensor two-dimensional data and the image data using an LSTM model; extracting a first high-order feature of the image data by using a first two-dimensional convolutional neural network, and extracting a second high-order feature of the sensor two-dimensional data by using a second two-dimensional convolutional neural network; fusing the first high-order feature and the second high-order feature by using convolution operation to obtain a first fused feature; and based on the resonance region and the first fusion feature, determining a fusion feature.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a fusion method and system for automatic processing of multi-source heterogeneous data based on convolutional neural networks. Background Art

[0002] In complex environmental monitoring, intelligent transportation, industrial equipment fault diagnosis and other fields, it is usually necessary to process multi-source heterogeneous data from a variety of sensors and visual devices. These data come from various sources, including temperature, humidity, vibration and other sensors installed in different locations, as well as image or video data collected by cameras. With the increase and diversity of data sources, how to efficiently integrate data from different modalities and make full use of their respective advantages has become the key to improving prediction accuracy and system robustness.

[0003] Traditional multimodal data fusion methods usually rely on manual feature extraction and rule formulation, which has the following problems: Traditional manual feature extraction methods rely on the knowledge of domain experts, which is not only time-consuming and labor-intensive, but also prone to missing potential key features, resulting in low data utilization and affecting model performance.

[0004] Sensor data and image data are usually spatially inconsistent. How to accurately align data from different modalities to achieve effective fusion is a major challenge faced by traditional methods.

[0005] Due to the significant differences in structure, scale and dimension of sensor and image data, traditional fusion methods have difficulty in capturing the deep correlation of multimodal data, resulting in poor fusion effect.

[0006] In the process of feature fusion of multimodal data, traditional methods often ignore the dynamic interaction between multimodal features, resulting in the model being unable to fully explore the potential patterns and associations in the data. Summary of the invention

[0007] In order to solve the problems in the prior art, the present invention provides a fusion method for automatic processing of multi-source heterogeneous data based on a convolutional neural network, the method comprising the following steps: Acquire sensor data at different positions, and acquire image data including the sensor installation position; establish a spatial mapping relationship according to the spatial correspondence between the sensor installation position and the image feature area; organize the sensor data into sensor two-dimensional data according to the mapping relationship; use an LSTM model to determine the resonance area between the sensor two-dimensional data and the image data; use a first two-dimensional convolutional neural network to extract a first high-order feature of the image data, and use a second two-dimensional convolutional neural network to extract a second high-order feature of the sensor two-dimensional data; use a convolution operation to fuse the first high-order feature and the second high-order feature to obtain a first fused feature; and determine a fused feature based on the resonance area and the first fused feature.

[0008] Furthermore, the establishing of the spatial mapping relationship includes: Obtain the physical installation location of each sensor, including geographic coordinates, relative height, and direction angle information to form three-dimensional coordinates; Collect image data and obtain pixel coordinates of image feature areas; The following formula is used for projection transformation:

[0009] in are the horizontal and vertical pixel coordinates of the image, are the coordinates of the sensor in three directions of the three-dimensional space; is the intrinsic parameter matrix of the camera, is the rotation matrix, which represents the rotation posture of the camera in three-dimensional space; The corresponding pixel area of ​​each sensor in the image is determined through projection transformation, the spatial alignment of sensor data and image feature area is achieved, and a spatial mapping relationship table is generated.

[0010] Further, organizing the sensor data into sensor two-dimensional data according to the mapping relationship includes: According to the spatial mapping relationship, the time series data of each sensor is mapped to the corresponding image plane coordinates; Construct a two-dimensional matrix whose size is consistent with the spatial distribution of the image feature area, and each element corresponds to the sensor data at a location; Fill the measurement data of each sensor position into the corresponding pixel position to form a two-dimensional representation; The interpolation method is used to fill in the pixel positions without filling data.

[0011] Further, using the LSTM model to determine the resonance region between the two-dimensional sensor data and the image data includes: The resonance region is specifically a region where sensor data and image data change significantly in both time and space; The two-dimensional data of the sensor is regarded as a time series feature, where the two-dimensional matrix at each time point represents the spatial distribution of the sensor at the corresponding time; Extract image data into time series form; Build a dual-input LSTM model to process the sensor two-dimensional data and image time series features in parallel. Use LSTM to extract time-related spatial features, and use another LSTM to extract time-related image features. Temporally align sensor features and image features and conduct similarity analysis; The similarity between the sensor features and the image features is calculated in the time series. When the similarity exceeds a preset threshold, the corresponding time point is determined to be a time resonance point. For the time resonance point, the common activation area of ​​sensor data and image features in two-dimensional space is further identified; The two-dimensional features of the sensor and the image features are spatially matched, and the resonance intensity is calculated using the Hadamard product.

[0012] among them Indicates the horizontal position of the pixel, Indicates the vertical position of the pixel. Represents the time parameter, represents the Hadamard product, represents the resonance intensity, Represents the sensor characteristics, Represents image features; by setting a resonance threshold, the position where the resonance intensity is greater than the resonance threshold is marked as a spatial resonance point, thereby determining the resonance area in the two-dimensional space.

[0013] Further, based on the resonance region and the first fusion feature, determining the fusion feature includes: The high-order features of the image data are extracted by the first two-dimensional convolutional neural network to obtain the high-order features of the image; The high-order features of the two-dimensional data of the sensor are extracted by the second two-dimensional convolutional neural network to obtain the high-order features of the sensor; Aligning the image high-order features and the sensor high-order features in a spatial dimension; Splicing the image high-order features and the sensor high-order features in the channel dimension to form a new joint feature matrix splicing feature; A new 2D convolutional layer is used to convolve the concatenated features to extract the deep interaction patterns between multimodal features. After the convolution operation, a nonlinear activation function is applied to enhance the expressiveness of the features and retain the fused feature information of the forward activation; The result obtained after the convolution operation is the first fusion feature; Generate a binary mask matrix mask of the same size as the first fusion feature to mark the position of the resonance area; Multiply the mask matrix pixel by pixel with the first fusion feature to obtain a weighted feature matrix weighted fusion feature; The weighted fusion features are subjected to nonlinear activation processing, the weighted features are normalized, and finally the fusion features are output.

[0014] On the other hand, the present invention also provides a fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural networks, the system comprising the following modules: A data acquisition module, used to acquire sensor data at different positions, and acquire image data including the sensor installation position; A mapping module, used to establish a spatial mapping relationship according to the spatial correspondence between the sensor installation position and the image feature area; A processing module, used for organizing the sensor data into sensor two-dimensional data according to the mapping relationship; A detection module, used for determining a resonance area between the two-dimensional sensor data and the image data using an LSTM model; An extraction module, configured to extract a first high-order feature of the image data using a first two-dimensional convolutional neural network, and to extract a second high-order feature of the two-dimensional sensor data using a second two-dimensional convolutional neural network; A fusion module is used to fuse the first high-order feature and the second high-order feature using a convolution operation to obtain a first fused feature; and determine a fused feature based on the resonance area and the first fused feature.

[0015] Furthermore, the establishing of the spatial mapping relationship includes: Obtain the physical installation location of each sensor, including geographic coordinates, relative height, and direction angle information to form three-dimensional coordinates; Collect image data and obtain pixel coordinates of image feature areas; The following formula is used for projection transformation: , in are the horizontal and vertical pixel coordinates of the image, are the coordinates of the sensor in three directions of the three-dimensional space; is the intrinsic parameter matrix of the camera, is the rotation matrix, which represents the rotation posture of the camera in three-dimensional space; The corresponding pixel area of ​​each sensor in the image is determined through projection transformation, the spatial alignment of sensor data and image feature area is achieved, and a spatial mapping relationship table is generated.

[0016] Further, organizing the sensor data into sensor two-dimensional data according to the mapping relationship includes: According to the spatial mapping relationship, the time series data of each sensor is mapped to the corresponding image plane coordinates; Construct a two-dimensional matrix whose size is consistent with the spatial distribution of the image feature area, and each element corresponds to the sensor data at a location; Fill the measurement data of each sensor position into the corresponding pixel position to form a two-dimensional representation; The interpolation method is used to fill in the pixel positions without filling data.

[0017] Further, using the LSTM model to determine the resonance region between the two-dimensional sensor data and the image data includes: The resonance region is specifically a region where sensor data and image data change significantly in both time and space; The two-dimensional data of the sensor is regarded as a time series feature, where the two-dimensional matrix at each time point represents the spatial distribution of the sensor at the corresponding time; Extract image data into time series form; Build a dual-input LSTM model to process the sensor two-dimensional data and image time series features in parallel. Use LSTM to extract time-related spatial features, and use another LSTM to extract time-related image features. Temporally align sensor features and image features and conduct similarity analysis; The similarity between the sensor features and the image features is calculated in the time series. When the similarity exceeds a preset threshold, the corresponding time point is determined to be a time resonance point. For the time resonance point, the common activation area of ​​sensor data and image features in two-dimensional space is further identified; The two-dimensional features of the sensor and the image features are spatially matched, and the resonance intensity is calculated using the Hadamard product. , among them Indicates the horizontal position of the pixel, Indicates the vertical position of the pixel. Represents the time parameter, represents the Hadamard product, represents the resonance intensity, Represents the sensor characteristics, Represents image features; by setting a resonance threshold, the position where the resonance intensity is greater than the resonance threshold is marked as a spatial resonance point, thereby determining the resonance area in the two-dimensional space.

[0018] Further, based on the resonance region and the first fusion feature, determining the fusion feature includes: The high-order features of the image data are extracted by the first two-dimensional convolutional neural network to obtain the high-order features of the image; The high-order features of the two-dimensional data of the sensor are extracted by the second two-dimensional convolutional neural network to obtain the high-order features of the sensor; Aligning the image high-order features and the sensor high-order features in a spatial dimension; Splicing the image high-order features and the sensor high-order features in the channel dimension to form a new joint feature matrix splicing feature; A new 2D convolutional layer is used to convolve the concatenated features to extract the deep interaction patterns between multimodal features. After the convolution operation, a nonlinear activation function is applied to enhance the expressiveness of the features and retain the fused feature information of the forward activation; The result obtained after the convolution operation is the first fusion feature; Generate a binary mask matrix mask of the same size as the first fusion feature to mark the position of the resonance area; Multiply the mask matrix pixel by pixel with the first fusion feature to obtain a weighted feature matrix weighted fusion feature; The weighted fusion features are subjected to nonlinear activation processing, the weighted features are normalized, and finally the fusion features are output.

[0019] The fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural network proposed in the present invention provides the following beneficial effects by making full use of the characteristics of multimodal data: By utilizing the automatic feature extraction capabilities of convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), the present invention can efficiently extract and integrate high-order features in sensor data and image data, avoiding the inefficiency of manual feature extraction in traditional methods and greatly improving the fusion efficiency.

[0020] By establishing a spatial correspondence between the sensor installation position and the image feature area, the present invention can accurately align data of different modalities in space, so that multimodal data can be feature fused in a unified spatial coordinate system, effectively solving the problem of spatial inconsistency of multimodal data in traditional methods.

[0021] The LSTM model is used to identify the "resonance areas" of sensor data and image features, that is, the common activation areas of multimodal data in time and space, so as to better capture the key patterns in the data and improve the correlation mining effect of multimodal data.

[0022] By deeply fusing high-order features of sensors and images through convolution operations and combining them with weighted processing of resonance areas, the model can more accurately identify and predict changing trends and abnormal situations in complex environments, thereby improving the prediction accuracy and robustness of the system.

[0023] The method of the present invention has a high degree of automation and can adaptively adjust the fusion strategy according to the characteristics of different modal data without human intervention. It is suitable for a variety of complex practical application scenarios, such as disaster monitoring, intelligent transportation, industrial equipment fault diagnosis, etc.

[0024] In summary, the present invention not only solves the shortcomings of traditional methods in feature extraction, spatial alignment, heterogeneous data integration, etc. through deep fusion of multimodal data, but also significantly improves the accuracy and real-time performance of data processing, and has broad application value and innovative significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0027] Below, the invention is preferably described in conjunction with the accompanying drawings and specific implementation methods.

[0028] This embodiment solves the above problem through the following steps: In one embodiment, reference Figure 1 The present invention provides a fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural networks. The method uses convolutional neural networks to perform feature extraction and fusion processing on sensor data and image data from different locations to achieve automatic alignment and deep interaction of multi-source heterogeneous data. The method specifically includes the following steps: Sensor data at different positions are acquired, and image data including the sensor installation position is acquired.

[0029] Acquire sensor data collected by sensors installed at multiple different locations, where the sensors include temperature sensors, humidity sensors, vibration sensors or gas sensors, and the sensors are arranged at different physical locations, such as different parts of industrial equipment, different floors or areas of building structures, different geographical locations of outdoor environments, etc.; the sensor data include numerical information of physical quantities such as temperature values, humidity values, vibration intensity or gas concentration at corresponding locations, and these data are collected in the form of time series or timed sampling.

[0030] The image data corresponding to the installation position of the sensor is obtained, and the image data is collected by a camera or monitoring equipment installed near the location, and the image data includes the coverage area of ​​the sensor installation position. For example, when the sensor is installed at the bearing of a certain industrial equipment, the image data is an equipment monitoring image containing the bearing and its surrounding environment; when the sensor is installed on a certain floor of a building, the image data is a video frame or image containing the internal environment of the floor; when the sensor is arranged at a certain geographical location, the image data is a ground monitoring image or a drone aerial image covering the geographical location and its surrounding area.

[0031] The image data not only includes the spatial position information corresponding to the sensor installation location, but also includes the environmental feature information of the location, such as the operating status of the equipment, the appearance of the building structure, the activities of people in the environment, the presence of flames or smoke and other visual features; by acquiring the above-mentioned sensor data and image data, the collection of multimodal data at different locations is realized, providing data support for subsequent multimodal data alignment, feature extraction and fusion.

[0032] According to the spatial correspondence between the sensor installation position and the image feature area, a spatial mapping relationship is established.

[0033] In this step, a spatial mapping relationship of multimodal data is constructed based on the spatial correspondence between the installation position of the sensor and the image feature area, and the spatial mapping relationship is used to achieve alignment and matching of sensor data and image data in the same coordinate system. Specifically, the absolute position of the sensor in three-dimensional space is determined by measuring the sensor's geographical location, installation height, orientation and other physical parameters, and converted into pixel coordinates in the image data to generate a spatial mapping model between the sensor data and the image feature area.

[0034] The mapping relationship is established as follows: The physical installation location of each sensor is obtained, including geographic coordinates, relative height, direction angle and other information, to form three-dimensional coordinates.

[0035] Image data is collected through a camera or other imaging device to obtain the pixel coordinates of the image feature area.

[0036] The following formula is used for projection transformation: , in are the horizontal and vertical pixel coordinates of the image, are the coordinates of the sensor in three directions of the three-dimensional space; is the intrinsic parameter matrix of the camera, is the rotation matrix, which represents the rotation posture of the camera in three-dimensional space.

[0037] The corresponding pixel area of ​​each sensor in the image is determined through projection transformation, the spatial alignment of sensor data and image feature area is achieved, and a spatial mapping relationship table is generated.

[0038] For example, when a temperature sensor is installed at the bearing of industrial equipment, the corresponding pixel area of ​​the location in the equipment monitoring image is determined by measuring its specific physical coordinates, thereby establishing a spatial mapping relationship between the sensor data and the image features of the area; when a humidity sensor is installed on the ceiling of a building floor, the characteristic area of ​​the location in the indoor monitoring image is determined by measuring its floor position and relative height, and the humidity sensor value is aligned with the image features through spatial mapping; when a vibration sensor is arranged at a pier of an outdoor bridge, its position in the bridge structure monitoring image is obtained through GPS positioning information, and a spatial mapping relationship between the sensor data and the pier image features is established.

[0039] Through the above-mentioned spatial mapping relationship, the physical coordinates of the sensor data are aligned with the pixel coordinates of the image feature area, so as to maintain spatial consistency in the process of multimodal data fusion and ensure the feature interaction and integration of different modal data in the same spatial range.

[0040] The sensor data is organized into sensor two-dimensional data according to the mapping relationship.

[0041] In this step, the spatial mapping relationship of multimodal data is constructed based on the spatial correspondence between the installation position of the sensor and the image feature area, and the spatial mapping relationship is used to achieve alignment and matching of sensor data and image data in the same coordinate system. By measuring the physical parameters such as the geographical location, installation height, and orientation of the sensor, the absolute position of the sensor in three-dimensional space is determined and converted into pixel coordinates in the image data, and a spatial mapping model between the sensor data and the image feature area is generated.

[0042] Physical quantity data, including temperature, humidity, vibration or other measurement data, are collected from sensors installed at multiple different locations. The data of each sensor is represented in the form of a time series.

[0043] According to the previously established spatial mapping relationship, the time series data of each sensor is mapped to the corresponding image plane coordinates.

[0044] Construct a two-dimensional matrix whose size is consistent with the spatial distribution of the image feature area, and each element corresponds to the sensor data at a specific location.

[0045] The measurement data of each sensor position is filled into the corresponding pixel position to form a two-dimensional representation.

[0046] If there is no corresponding sensor data for some image pixel positions, an interpolation method (such as bilinear interpolation or Gaussian filling) can be used to complete the two-dimensional matrix.

[0047] The LSTM model is used to determine the resonance area between the sensor two-dimensional data and the image data.

[0048] Resonance regions refer to areas where sensor data and image data change significantly in time and space at the same time, that is, multimodal data have similar responses to an event or change in a specific time period. This region often reflects the high correlation between data and is the most critical part of multimodal fusion, which can reveal the synchronous response of different modalities to the same event.

[0049] The present invention determines resonance from two dimensions, time and space.

[0050] Resonance in the time dimension: Sensor and image data have similar fluctuation patterns in time series, such as when the sensor measurement changes significantly within a certain period of time, the change is also detected in the corresponding image frame.

[0051] Resonance in the spatial dimension: The 2D sensor data and image features show overlapping activation regions in space, indicating that the two modalities have similar spatial responses in this region.

[0052] The two-dimensional data of the sensor is regarded as a time series feature, where the two-dimensional matrix at each time point represents the spatial distribution of the sensor at the corresponding time.

[0053] The image data is extracted into a time series form, which can be a time series of image frames or specific image features (such as high-order features extracted by convolutional neural networks), to form a corresponding feature matrix.

[0054] A dual-input LSTM model is constructed to process sensor two-dimensional data and image time series features in parallel. Time-related spatial features are extracted through LSTM, and time-related image features are extracted through another LSTM.

[0055] The sensor features and image features are temporally aligned and similarity analyzed to determine the temporal co-activation periods of the two.

[0056] The similarity between sensor features and image features can be calculated in time series through measurement methods such as cosine similarity or correlation coefficient. When the similarity exceeds a preset threshold, the time point is determined to be a time resonance point.

[0057] For the temporal resonance points, the common activation areas of sensor data and image features in two-dimensional space are further identified.

[0058] The two-dimensional features of the sensor and the image features are spatially matched, and the resonance intensity is calculated using the Hadamard product. , among them Indicates the horizontal position of the pixel, Indicates the vertical position of the pixel. Represents the time parameter, represents the Hadamard product, represents the resonance intensity, Represents the sensor characteristics, Represents image features. By setting a resonance threshold, the position where the resonance intensity is greater than the resonance threshold is marked as a spatial resonance point, thereby determining the resonance area in the two-dimensional space.

[0059] The resonance results in time and space are combined to form the final resonance area. This resonance area can reflect the common response mode of the sensor and image in time and space, and is an important basis for subsequent multimodal feature fusion and decision-making.

[0060] A first two-dimensional convolutional neural network is used to extract first high-order features of the image data, and a second two-dimensional convolutional neural network is used to extract second high-order features of the sensor two-dimensional data.

[0061] Image data is an input two-dimensional matrix that represents the visual information around the sensor installation location. It may be a static image, a single frame of a video frame, or a pre-processed feature matrix. The input image data contains spatial distribution information, has a certain width and height, and usually has multiple color channels, such as red, green, and blue (RGB).

[0062] The first two-dimensional convolutional neural network (CNN1) is used to extract high-order features of image data. The network consists of multiple convolutional layers, pooling layers, and activation functions to extract spatial features of the image layer by layer.

[0063] Convolutional layer: Each convolutional layer scans the local area of ​​the image through several convolution kernels and extracts the feature patterns in the area. These features may be simple edges, textures, or higher-level complex image features.

[0064] Pooling layer: After the convolution layer, a pooling layer is added to downsample the extracted features. The pooling layer reduces the spatial resolution of the features by selecting the maximum or average value of the local area while retaining important information, improving computational efficiency and feature robustness.

[0065] Multi-layer convolution and pooling: Through multiple convolution and pooling operations, CNN1 gradually extracts higher-level image features so that the output features can represent the deep information of the image, such as shape, contour, or texture features of a specific area.

[0066] Finally, after multiple layers of convolution and pooling, CNN1 outputs a high-order feature matrix of the image. This feature matrix retains the deep spatial features of the image and compresses the size of the original image, making it easier to process without losing important information.

[0067] The two-dimensional sensor data represents the spatial measurement results of the sensor, such as the spatial distribution of values ​​such as temperature, humidity, vibration, etc. The data is organized into a two-dimensional matrix consistent with the image space to ensure spatial alignment.

[0068] The second 2D convolutional neural network (CNN2) is used to extract high-order features of the sensor 2D data. The structure of CNN2 is similar to CNN1, consisting of multiple layers of convolutional layers, pooling layers, and activation functions, which extract spatial patterns in the sensor data layer by layer.

[0069] Convolutional layer: Each convolutional layer scans local areas of sensor data and identifies characteristic patterns in different spatial regions. For example, in temperature sensor data, the convolutional layer may identify patterns such as high temperature areas and temperature gradients.

[0070] Pooling layer: After the convolution layer, a pooling layer is added to downsample the extracted features. The pooling layer reduces the spatial dimension by selecting the local maximum or average value and retains the main features in the sensor data.

[0071] Multi-layer convolution and pooling: Through multiple convolution and pooling operations, CNN2 gradually extracts higher-level sensor features so that the output features can reflect the deep patterns in the sensor data.

[0072] Finally, after multiple layers of convolution and pooling, CNN2 outputs a high-order feature matrix of the sensor's two-dimensional data. This feature matrix reflects the deep spatial distribution of the sensor data and provides a basis for subsequent fusion with the image's high-order features.

[0073] The first high-order feature and the second high-order feature are fused using a convolution operation to obtain a first fused feature; and a fused feature is determined based on the resonance region and the first fused feature. Specifically, the following steps are included: Preliminary convolutional fusion of high-level features Input high-level features First high-order features: The high-order features of the image data extracted by the first two-dimensional convolutional neural network, denoted as "image high-order features", contain the deep spatial patterns and information of the image.

[0074] Second high-order features: The high-order features of the sensor two-dimensional data extracted by the second two-dimensional convolutional neural network, denoted as “sensor high-order features”, reflect the spatial distribution and deep patterns of the sensor data.

[0075] Feature Alignment Before fusion, make sure that the "image high-order features" and "sensor high-order features" are aligned in the spatial dimension, that is, they have the same height and width. This ensures that the two can be fused pixel by pixel in space.

[0076] Feature concatenation and convolution fusion The "image high-order features" and "sensor high-order features" are spliced ​​in the channel dimension to form a new joint feature matrix "spliced ​​features", which contains multi-level feature information of the two modalities.

[0077] A new 2D convolutional layer is used to convolve the “concatenated features” to extract the deep interaction patterns between multimodal features.

[0078] This convolutional layer captures the correlation patterns of image features and sensor features in space and channels by scanning the local area of ​​the "stitching features", and extracts the common information and mutual influence patterns between them.

[0079] After the convolution operation, a nonlinear activation function (such as ReLU or Sigmoid) is applied to enhance the expressiveness of the features and retain the fused feature information of the forward activation.

[0080] Output of the first fusion feature The result obtained after the convolution operation is called the "first fusion feature", which contains the preliminary fusion information of the image data and sensor data in space and channel.

[0081] This feature matrix represents the interaction pattern of multimodal data in the same space and is the basis for further fusion and analysis.

[0082] Fusion feature determination based on resonance region The resonance region is the synchronous activation region of multimodal data in time and space identified by the LSTM model, indicating the common response region of image data and sensor data in a specific time period.

[0083] This region reflects similar activations in space between the image and sensor data, and is usually represented by a mask matrix of the same size as the “first fused feature”.

[0084] Mask generation of the resonance region Generate a binary mask matrix "mask" of the same size as the "first fusion feature" to mark the location of the resonance area.

[0085] The pixel position with a value of 1 in the mask matrix indicates that it belongs to the resonance region, and the pixel position with a value of 0 indicates that it does not belong to the resonance region.

[0086] The mask is generated based on the detection results of the resonant regions, ensuring that it is spatially aligned with the “first fused feature”.

[0087] Weighting and masking Multiply the mask matrix by the “first fusion feature” pixel by pixel to obtain the weighted feature matrix “weighted fusion feature”.

[0088] The features in the resonant region are retained and amplified, while the features in the non-resonant region are weakened or ignored. This can highlight the strong correlation features of multimodal data in the common activation area.

[0089] This weighted processing ensures that the characteristic response of the resonance area is spatially enhanced, so that the final fusion feature can more concentratedly reflect the common characteristics of multimodal data.

[0090] Nonlinear activation and normalization Perform nonlinear activation processing on the "weighted fusion features", such as ReLU or Sigmoid, to further enhance the expressive power of the features.

[0091] Afterwards, the weighted features are normalized (such as batch normalization or layer normalization) to ensure the balance and stability of the fused features in different channels and spatial positions.

[0092] Output of fused features The final output feature matrix is ​​called “fused features”, which not only contains the deep interaction information of image and sensor features, but also highlights the influence of resonance areas.

[0093] This fused feature represents the joint response of image data and sensor data in time and space. It is the final result of multimodal fusion and can be used for subsequent classification, detection or prediction tasks.

[0094] On the other hand, the present invention also provides a fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural networks, comprising: A data acquisition module, used to acquire sensor data at different positions, and acquire image data including the sensor installation position; A mapping module, used to establish a spatial mapping relationship according to the spatial correspondence between the sensor installation position and the image feature area; A processing module, used for organizing the sensor data into sensor two-dimensional data according to the mapping relationship; A detection module, used for determining a resonance area between the two-dimensional sensor data and the image data using an LSTM model; An extraction module, configured to extract a first high-order feature of the image data using a first two-dimensional convolutional neural network, and to extract a second high-order feature of the two-dimensional sensor data using a second two-dimensional convolutional neural network; A fusion module is used to fuse the first high-order feature and the second high-order feature using a convolution operation to obtain a first fused feature; and determine a fused feature based on the resonance area and the first fused feature.

[0095] The prior art mentioned in the above background technology section and specific embodiment section of the present invention can be used as part of the present invention to understand the meaning of some technical features or parameters.

Claims

1. A fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural network, characterized in that: The method comprises the following steps: Acquire sensor data at different positions, and acquire image data including the sensor installation position; According to the spatial correspondence between the sensor installation position and the image feature area, a spatial mapping relationship is established; Organizing the sensor data into sensor two-dimensional data according to the mapping relationship; Determine the resonance area between the two-dimensional sensor data and the image data using an LSTM model; Extracting first high-order features of the image data using a first two-dimensional convolutional neural network, and extracting second high-order features of the two-dimensional sensor data using a second two-dimensional convolutional neural network; The first high-order feature and the second high-order feature are fused using a convolution operation to obtain a first fused feature; and a fused feature is determined based on the resonance area and the first fused feature.

2. The fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 1 is characterized in that: The establishing of the spatial mapping relationship comprises: Obtain the physical installation location of each sensor, including geographic coordinates, relative height, and direction angle information to form three-dimensional coordinates; Collect image data and obtain pixel coordinates of image feature areas; The following formula is used for projection transformation: ， in are the horizontal and vertical pixel coordinates of the image, are the coordinates of the sensor in three directions of the three-dimensional space; is the intrinsic parameter matrix of the camera, is the rotation matrix, which represents the rotation posture of the camera in three-dimensional space; The corresponding pixel area of ​​each sensor in the image is determined through projection transformation, the spatial alignment of sensor data and image feature area is achieved, and a spatial mapping relationship table is generated.

3. The fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 1 is characterized in that Organizing the sensor data into sensor two-dimensional data according to the mapping relationship comprises: According to the spatial mapping relationship, the time series data of each sensor is mapped to the corresponding image plane coordinates; Construct a two-dimensional matrix whose size is consistent with the spatial distribution of the image feature area, and each element corresponds to the sensor data at a location; Fill the measurement data of each sensor position into the corresponding pixel position to form a two-dimensional representation; The pixel positions without fill data are filled using interpolation method.

4. The fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 1 is characterized in that Determining the resonance region between the two-dimensional sensor data and the image data using the LSTM model includes: The resonance region is specifically a region where sensor data and image data change significantly in both time and space; The two-dimensional data of the sensor is regarded as a time series feature, where the two-dimensional matrix at each time point represents the spatial distribution of the sensor at the corresponding time; Extract image data into time series form; Build a dual-input LSTM model to process the sensor two-dimensional data and image time series features in parallel. Use LSTM to extract time-related spatial features, and use another LSTM to extract time-related image features. Temporally align sensor features and image features and conduct similarity analysis; The similarity between the sensor features and the image features is calculated in the time series. When the similarity exceeds a preset threshold, the corresponding time point is determined to be a time resonance point. For the time resonance point, the common activation area of ​​sensor data and image features in two-dimensional space is further identified; The two-dimensional features of the sensor and the image features are spatially matched, and the resonance intensity is calculated using the Hadamard product. ， in Indicates the horizontal position of the pixel, Indicates the vertical position of the pixel. Represents the time parameter, represents the Hadamard product, represents the resonance intensity, Represents the sensor characteristics, Represents image features; by setting a resonance threshold, the position where the resonance intensity is greater than the resonance threshold is marked as a spatial resonance point, thereby determining the resonance area in the two-dimensional space.

5. The fusion method for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 1 is characterized in that Determining the fusion feature based on the resonance region and the first fusion feature includes: The high-order features of the image data are extracted by the first two-dimensional convolutional neural network to obtain the high-order features of the image; The high-order features of the two-dimensional data of the sensor are extracted by the second two-dimensional convolutional neural network to obtain the high-order features of the sensor; Aligning the image high-order features and the sensor high-order features in a spatial dimension; Splicing the image high-order features and the sensor high-order features in the channel dimension to form a new joint feature matrix splicing feature; A new 2D convolutional layer is used to convolve the concatenated features to extract the deep interaction patterns between multimodal features. After the convolution operation, a nonlinear activation function is applied to enhance the expressiveness of the features and retain the fused feature information of the forward activation; The result obtained after the convolution operation is the first fusion feature; Generate a binary mask matrix mask of the same size as the first fusion feature to mark the position of the resonance area; Multiply the mask matrix pixel by pixel with the first fusion feature to obtain a weighted feature matrix weighted fusion feature; The weighted fusion features are subjected to nonlinear activation processing, the weighted features are normalized, and finally the fusion features are output.

6. A fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural networks, characterized in that: The system includes the following modules: A data acquisition module, used to acquire sensor data at different positions, and acquire image data including the sensor installation position; A mapping module, used to establish a spatial mapping relationship according to the spatial correspondence between the sensor installation position and the image feature area; A processing module, used for organizing the sensor data into sensor two-dimensional data according to the mapping relationship; A detection module, used for determining a resonance area between the two-dimensional sensor data and the image data using an LSTM model; An extraction module, configured to extract a first high-order feature of the image data using a first two-dimensional convolutional neural network, and to extract a second high-order feature of the two-dimensional sensor data using a second two-dimensional convolutional neural network; A fusion module is used to fuse the first high-order feature and the second high-order feature using a convolution operation to obtain a first fused feature; and determine a fused feature based on the resonance area and the first fused feature.

7. The fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 6 is characterized in that: The establishing of the spatial mapping relationship comprises: Obtain the physical installation location of each sensor, including geographic coordinates, relative height, and direction angle information to form three-dimensional coordinates; Collect image data and obtain pixel coordinates of image feature areas; The following formula is used for projection transformation: ， in are the horizontal and vertical pixel coordinates of the image, are the coordinates of the sensor in three directions of the three-dimensional space; is the intrinsic parameter matrix of the camera, is the rotation matrix, which represents the rotation posture of the camera in three-dimensional space; The corresponding pixel area of ​​each sensor in the image is determined through projection transformation, the spatial alignment of sensor data and image feature area is achieved, and a spatial mapping relationship table is generated.

8. The fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 6 is characterized in that Organizing the sensor data into sensor two-dimensional data according to the mapping relationship includes: According to the spatial mapping relationship, the time series data of each sensor is mapped to the corresponding image plane coordinates; Construct a two-dimensional matrix whose size is consistent with the spatial distribution of the image feature area, and each element corresponds to the sensor data at a location; Fill the measurement data of each sensor position into the corresponding pixel position to form a two-dimensional representation; The pixel positions without fill data are filled using interpolation method.

9. The fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural networks according to claim 6 is characterized in that Determining the resonance region between the two-dimensional sensor data and the image data using the LSTM model includes: The resonance region is specifically a region where sensor data and image data change significantly in both time and space; The two-dimensional data of the sensor is regarded as a time series feature, where the two-dimensional matrix at each time point represents the spatial distribution of the sensor at the corresponding time; Extract image data into time series form; Build a dual-input LSTM model to process the sensor two-dimensional data and image time series features in parallel. Use LSTM to extract time-related spatial features, and use another LSTM to extract time-related image features. Temporally align sensor features and image features and conduct similarity analysis; The similarity between the sensor features and the image features is calculated in the time series. When the similarity exceeds a preset threshold, the corresponding time point is determined to be a time resonance point. For the time resonance point, the common activation area of ​​sensor data and image features in two-dimensional space is further identified; The two-dimensional features of the sensor and the image features are spatially matched, and the resonance intensity is calculated using the Hadamard product. ， among them Indicates the horizontal position of the pixel, Indicates the vertical position of the pixel. Represents the time parameter, represents the Hadamard product, represents the resonance intensity, Represents the sensor characteristics, Represents image features; by setting a resonance threshold, the position where the resonance intensity is greater than the resonance threshold is marked as a spatial resonance point, thereby determining the resonance area in the two-dimensional space.

10. The fusion system for automatic processing of multi-source heterogeneous data based on convolutional neural network according to claim 6 is characterized in that Determining the fusion feature based on the resonance region and the first fusion feature includes: The high-order features of the image data are extracted by the first two-dimensional convolutional neural network to obtain the high-order features of the image; The high-order features of the two-dimensional data of the sensor are extracted by the second two-dimensional convolutional neural network to obtain the high-order features of the sensor; Aligning the image high-order features and the sensor high-order features in a spatial dimension; Splicing the image high-order features and the sensor high-order features in the channel dimension to form a new joint feature matrix splicing feature; A new 2D convolutional layer is used to convolve the concatenated features to extract the deep interaction patterns between multimodal features. After the convolution operation, a nonlinear activation function is applied to enhance the expressiveness of the features and retain the fused feature information of the forward activation; The result obtained after the convolution operation is the first fusion feature; Generate a binary mask matrix mask of the same size as the first fusion feature to mark the position of the resonance area; Multiply the mask matrix pixel by pixel with the first fusion feature to obtain a weighted feature matrix weighted fusion feature; The weighted fusion features are subjected to nonlinear activation processing, the weighted features are normalized, and finally the fusion features are output.

Images