An affinity analysis method for multimodal data streams based on intelligent Internet of Things

Abstract

This paper discloses an affinity analysis method for multimodal data streams based on the intelligent Internet of Things. The method comprises two parts: a method for measuring the affinity of multimodal data and constructing an affinity matrix based on the analytic hierarchy process. This method effectively calculates the hidden affinities between multimodal data, ensuring accuracy and adaptability to a variety of heterogeneous sensors. The method is versatile and robust, capable of processing different types of sensor data and reliably operating in a variety of practical application scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things, and in particular relates to an affinity analysis method for multimodal data streams based on intelligent Internet of Things. Background Art

[0002] The intelligent Internet of Things (IoT) has become an indispensable tool across various fields, with widespread application in environmental monitoring, healthcare, intelligent transportation, industrial control, and more. The data sensed by sensors in these networks is often heterogeneous, both in terms of data size and transmission speed. This heterogeneity enables sensor systems to simultaneously acquire information from multiple perspectives and dimensions, providing more comprehensive and accurate data support.

[0003] This feature has greatly improved the reasoning accuracy of multimodal learning. In previous work, researchers have improved the model effect by fusing data from different modalities. This is because they utilized the implicit correlation information between the modalities, such as audio stream and video stream data. Video stream data reveals the visual information of the scene, and audio stream provides auditory information of the same scene. There is a certain correlation information between them. We call this implicit information "affinity of multimodal data."

[0004] However, this data heterogeneity also poses a challenge to multimodal learning. Because data from different modalities, due to structural differences and other factors, cannot arrive at the cloud simultaneously for fusion as ideally desired, some data flows are slow or easily corrupted. These slow flows require waiting for the fast flows to arrive at the cloud, increasing latency. If these wait times are not met, accuracy is reduced. Most current solutions block the fast flows and wait for the slow flows to arrive at the cloud together. This reduces latency to maintain accuracy, but does not address the fundamental issues inherent in the different modal data flows. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the present invention provides an affinity analysis method for multimodal data streams based on the intelligent Internet of Things. The method comprises two parts: a method for measuring the affinity of multimodal data and constructing an affinity matrix based on the analytic hierarchy process. This method effectively calculates the hidden affinities between multimodal data, ensuring accuracy and adaptability to a variety of heterogeneous sensors. The method is versatile and robust, capable of processing different types of sensor data and reliably operating in a variety of practical application scenarios.

[0006] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0007] Step 1: Affinity measurement of multimodal data;

[0008] Step 1-1: For data streams of different modalities, select different feature extraction networks to obtain the features of the modal data generated by each different sensor;

[0009] Step 1-2: Perform t-SNE dimensionality reduction on the acquired features to obtain low-dimensional feature representation, and then calculate the average cosine similarity between the low-dimensional features as a preliminary affinity measure;

[0010] Step 2: Affinity matrix construction based on analytic hierarchy process;

[0011] Step 2-1: All sensors in the entire sensor network are divided into sensors of interest and sensors of need; the sensors of interest include slow flow sensors and fragile sensors, and the sensors of need include fast flow sensors;

[0012] Step 2-2: Divide the sensors of interest into the focus layer and the required sensors into the candidate layer;

[0013] Step 2-3: Construct the criteria layer;

[0014] The criteria for the quasi-measurement layer are selected based on the initial affinity;

[0015] Step 2-4: Construct a judgment matrix between the criterion layer and the focus layer, and between the criterion layer and the alternative layer;

[0016] Step 2-5: Calculate the weight matrix of each judgment matrix;

[0017] Step 2-6: Multiply the weight matrix of the criterion layer and the focus layer with the weight matrix of the criterion layer and the alternative layer to obtain the final affinity matrix.

[0018] Preferably, in step 1-1, U-net is used for image data and PointNet network is used for 3D point cloud data.

[0019] Preferably, the average cosine similarity calculation formula is:

[0020]

[0021] Among them, A and B are two eigenvectors.

[0022] Preferably, the judgment matrix construction algorithm in steps 2-4 is specifically as follows:

[0023] Step 2-4-1: Construct a square matrix according to the number of elements in the hierarchy;

[0024] Step 2-4-2: Fill in the matrix content according to the relative importance of each element;

[0025] The value of a square matrix element is 1: it means that the two elements have the same importance;

[0026] The value of the square matrix element is 3: it means that the former is slightly more important than the latter.

[0027] The value of the matrix element is 5: it means that the former is more important than the latter.

[0028] The value of the square matrix element is 7: it means that the former is more important than the latter.

[0029] The value of the square matrix element is 9: it means that the former is more important than the latter.

[0030] The values ​​of the square matrix elements are 2, 4, 6, and 8: representing the middle values ​​of the above adjacent judgments;

[0031] The values ​​of the matrix elements are the reciprocals of 1-9: indicating the importance of comparing the order of the corresponding two elements;

[0032] Step 2-4-3: The judgment matrix is ​​constructed.

[0033] Preferably, the weight matrix calculation formula in steps 2-5 is as follows:

[0034]

[0035] The beneficial effects of the present invention are as follows:

[0036] 1. This method effectively calculates hidden affinities between multimodal data, ensuring accuracy and adaptability to a wide variety of heterogeneous sensors. The method is versatile and robust, capable of processing different types of sensor data, such as text, images, and audio, and operates reliably in a variety of practical application scenarios.

[0037] 2. The matrix structure of this invention is used to represent the architecture and state parameters of the entire system. This data structure allows for further optimization and refinement of affinity. This involves analyzing the relationships between individual sensor data and the system's performance under different states, thereby constructing an efficient data structure capable of representing the affinity relationships between all heterogeneous sensors in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a structural diagram of the method of the present invention.

[0039] Figure 2 Schematic diagram of the hierarchical structure of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and examples.

[0041] The first objective of the present invention is to develop a highly accurate and efficient computational method to quantify the affinity between multimodal data (heterogeneous sensors), and it is hoped that this method can be adapted to different heterogeneous sensors.

[0042] The technical solution for achieving the first object of the present invention is: a method for measuring affinity of multimodal data, comprising the following steps:

[0043] ① For data streams of different modalities, different feature extraction networks are selected, such as U-net for image data and PointNet for 3D point cloud data. First, the features of the modal data generated by each different sensor are obtained;

[0044] ② Perform t-SNE dimensionality reduction on the acquired features to obtain low-dimensional feature representation, and then calculate the average cosine similarity between the low-dimensional features as a preliminary affinity measure.

[0045] The specific average cosine similarity calculation formula in step ② is (A, B are two feature vectors):

[0046]

[0047] The second objective of this invention is to develop a matrix structure that can be used to further optimize and refine affinity. This involves analyzing the relationships between individual sensor data and the system's performance under different states, thereby constructing an efficient data structure that can represent the affinity relationships between all heterogeneous sensors in the system.

[0048] The technical solution for achieving the second object of the present invention is: a method for constructing an affinity matrix based on the hierarchical analysis method, comprising the following steps:

[0049] ① For all sensors in the entire sensor network, distinguish between the sensors we are interested in (slow-flow sensors or fragile sensors) and the sensors we need (fast-flow sensors);

[0050] ②Divide the sensors of interest into the focus layer and the required sensors into the alternative layer;

[0051] ③Build the criterion layer. The criterion of the measurement layer selects the preliminary affinity obtained in the previous solution. Other criteria are added according to the actual situation, such as the sensor's cross FOV (Field of View);

[0052] ④ Construct a judgment matrix between the criterion layer and the focus layer, as well as between the criterion layer and the alternative layer;

[0053] ⑤Calculate the weight matrix of each judgment matrix;

[0054] ⑥ Multiply the weight matrix of the criterion layer and the focus layer with the weight matrix of the criterion layer and the alternative layer to obtain the final affinity matrix;

[0055] The weight matrix construction algorithm in step ④ is specifically as follows:

[0056] I. Construct a square matrix according to the number of elements in the hierarchy;

[0057] II. Fill in the matrix content according to the relative importance of each element. For example, when constructing the matrix of the criterion layer and the focus layer, if the criterion layer has two criteria, then construct a two-dimensional matrix and fill in the matrix content according to the following importance table:

[0058]

[0059] III. The judgment matrix is ​​constructed;

[0060] The weight matrix calculation formula in step ⑤ is as follows:

[0061]

[0062] Example:

[0063] Step 1: Sensor affinity measurement;

[0064] Taking the onboard sensor network of an autonomous driving vehicle as an example, the entire network consists of cameras and lidars. This method first uses different feature extraction networks to extract features for different sensors, ensuring that each feature extraction network can adapt to the corresponding sensor modality data. For example, for 3D radar point cloud data and image data, classic PointNet and U-net can be used. For sensors {S1, S2, ..., Sn}, features {x1, x2, ..., xn} are obtained, and then the t-SNE method is used to reduce their dimensionality to a low-dimensional space. t-SNE converts high-dimensional features xi(i) into low-dimensional features yi, ensuring that the features after dimensionality reduction retain the information of the original features. After obtaining the low-dimensional features, the average cosine similarity CosSimilarity between the low-dimensional features of each sensor is calculated, and the obtained cosine similarity is used as the preliminary affinity measure;

[0065] Step 2: A method for constructing sensor network affinity matrix based on analytic hierarchy process;

[0066] In the sensor affinity measurement method, we can finally get an average cosine similarity CosSimilarity as a preliminary measure of affinity, which is used to represent the affinity relationship between each sensor. It is abbreviated as CSY in the following text. The similarity between the i-th sensor and the j-th sensor is expressed as CSY ij , it is obvious that CSYij=CSY jiThe reason why it is called preliminary affinity is that it is only a characteristic distance calculation at the mathematical level and cannot fully interpret the affinity relationship of the entire sensor network. Therefore, in this step, the affinity matrix of the entire sensor network is constructed, which takes into account the influencing factors of the sensor network and reflects the affinity relationship of the entire sensor network.

[0067] Step 2-1: Constructing the hierarchical analysis structure;

[0068] According to the hierarchical analysis method, the focus layer, alternative layer, and criterion layer are set. The slow flow and vulnerable sensors {f1, f2, ..., fn} of interest are selected as the focus layer Focus, and the slow flow sensors {a1, a2, ..., an} are selected as the alternative layer Altenatives. The criterion layer Criteria selects factors {C1, C2, ..., Cn} that may affect affinity in the sensor network, such as the cross FOV (Field Of View) of the sensor, etc., and the CSY obtained in the previous step is also added to the criterion layer. The entire criterion layer is constructed as shown in the figure below. Figure 2 As shown in .

[0069] Step 2-2: Use the analytic hierarchy process to construct an affinity matrix;

[0070] There are two types of matrices for the criterion layer and the alternative layer, and the quasi-measurement layer and the focus layer. The first is the criterion layer and the focus layer matrix A. Assuming there are two criteria C1 and C2, then for each focus layer sensor, construct a 2*2 matrix A, and the element a in A is ij Represents the importance of the i-th criterion and the j-th criterion relative to the focus sensor, then the matrix A can be obtained in the end:

[0071]

[0072] If a 12 =7, which means that the C1 criterion is more important than C2. So, taking the previous example, cosine similarity is more important than cross FOV.

[0073] The next step is to solve the matrix Bn, b between the criterion layer and the alternative layer ij The value of also represents the importance of candidate sensor i relative to j on criterion n. Assuming that there are only three sensors in the device selection layer, we can get B1 and B2 (the values ​​are filled in as examples and have no practical significance):

[0074]

[0075] The next step is to solve the weight matrices of matrices A and B, and then obtain the weight matrix WA of A, which should be a 2*1 matrix. Then solve the weight matrices WB1 and WB2 of B1 and B2, and merge them into WB=[WB1, WB2].

[0076] Finally, multiply WA and WB and normalize them to get the final weight matrix W (a 3*1 matrix in this example). In this example, W represents the expected affinity relationship between the three alternative sensors for a focus sensor. The element value of the matrix w i It represents the affinity of the i-th sensor in the entire sensor network relative to the focal sensor, that is, the size of the association relationship between the two sensors.

Claims

1. A method for affinity analysis of multimodal data streams based on intelligent Internet of Things, characterized in that: The steps include: Step 1: Affinity measurement of multimodal data; Step 1-1: For data streams of different modalities, select different feature extraction networks to obtain the features of the modal data generated by each different sensor; Step 1-2: Perform t-SNE dimensionality reduction on the acquired features to obtain low-dimensional feature representation, and then calculate the average cosine similarity between the low-dimensional features as a preliminary affinity measure; Step 2: Affinity matrix construction based on analytic hierarchy process; Step 2-1: All sensors in the entire sensor network are divided into sensors of interest and sensors of need; the sensors of interest include slow flow sensors and fragile sensors, and the sensors of need include fast flow sensors; Step 2-2: Divide the sensors of interest into the focus layer and the required sensors into the candidate layer; Step 2-3: Construct the criteria layer; The criteria for the quasi-measurement layer are selected based on the initial affinity; Step 2-4: Construct a judgment matrix between the criterion layer and the focus layer, and between the criterion layer and the alternative layer; Step 2-5: Calculate the weight matrix of each judgment matrix; Step 2-6: Multiply the weight matrix of the criterion layer and the focus layer with the weight matrix of the criterion layer and the alternative layer to obtain the final affinity matrix.

2. The affinity analysis method for multimodal data stream based on intelligent Internet of Things according to claim 1 is characterized in that: In step 1-1, U-net is used for image data and PointNet network is used for 3D point cloud data.

3. The affinity analysis method for multimodal data stream based on intelligent Internet of Things according to claim 1 is characterized in that: The average cosine similarity calculation formula is: Among them, A and B are two eigenvectors.

4. The affinity analysis method for multimodal data stream based on intelligent Internet of Things according to claim 1 is characterized in that: The judgment matrix construction algorithm in steps 2-4 is specifically as follows: Step 2-4-1: Construct a square matrix according to the number of elements in the hierarchy; Step 2-4-2: Fill in the matrix content according to the relative importance of each element; The value of a square matrix element is 1: it means that the two elements have the same importance; The value of the square matrix element is 3: it means that the former is slightly more important than the latter. The value of the matrix element is 5: it means that the former is more important than the latter. The value of the square matrix element is 7: it means that the former is more important than the latter. The value of the square matrix element is 9: it means that the former is more important than the latter. The values ​​of the square matrix elements are 2, 4, 6, and 8: representing the middle values ​​of the above adjacent judgments; The values ​​of the matrix elements are the reciprocals of 1-9: indicating the importance of comparing the order of the corresponding two elements; Step 2-4-3: The judgment matrix is ​​constructed.

Images