Affinity analysis method of multi-modal data stream based on intelligent Internet of Things

The affinity matrix is ​​constructed through the affinity measurement and hierarchical analysis method of multimodal data, which solves the problem of transmission inconsistency caused by the heterogeneity of multimodal data in the Internet of Things system, and realizes efficient processing and accurate affinity of heterogeneous sensors and sexual analysis improves the universality and robustness of the system.

CN119071327AActive Publication Date: 2024-12-03NORTHWESTERN POLYTECHNICAL UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411327743.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-24
Publication Date
2024-12-03
Estimated Expiration
2044-09-24

AI Technical Summary

Technical Problem

The heterogeneity of multimodal data in IoT systems leads to inconsistent data transmission speed, resulting in increased latency and reduced accuracy. The existing technology mainly solves the problem by blocking fast flow and waiting for slow flow, but it fails to solve the essential problem.

Method used

Using the affinity measurement method of multimodal data and the affinity matrix construction based on hierarchical analysis method, the average cosine similarity is calculated through feature extraction network and t-SNE dimensionality reduction, combined with hierarchical analysis method, the sensor hierarchical analysis method is used to construct the weight matrix to The final affinity matrix is ​​calculated to represent the affinity relationship between sensors.

Benefits of technology

It realizes accurate calculation of hidden affinity between multimodal data, adapts to heterogeneous sensors, improves the versatility and robustness of data processing, optimizes the system architecture and state parameters, and ensures the reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071327A_ABST
    Figure CN119071327A_ABST
Patent Text Reader

Abstract

The invention discloses an affinity analysis method of a multi-modal data stream based on an intelligent Internet of Things. The affinity analysis method comprises an affinity measurement method of multi-modal data and affinity matrix construction based on an analytic hierarchy process. According to the method, the hidden affinity among the multi-modal data can be well calculated, the accuracy is guaranteed, and the method can adapt to various different heterogeneous sensors. The method has universality and robustness, can process different types of sensor data, and can work reliably in various practical application scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things, and in particular relates to an affinity analysis method for multimodal data streams based on intelligent Internet of Things. Background Art

[0002] At present, the intelligent Internet of Things has become an indispensable tool in various fields, and is widely used in environmental monitoring, health care, intelligent transportation, industrial control and other fields. The data perceived by sensors in these networks are often heterogeneous, whether in terms of data size or data transmission speed. This heterogeneity enables the sensor system to obtain information from multiple angles and dimensions at the same time, thereby providing more comprehensive and accurate data support.

[0003] This feature has greatly improved the reasoning accuracy of multimodal learning. In previous work, researchers have improved the model effect by fusing data from different modalities. This is because they utilized the implicit correlation information between the modalities, such as audio stream and video stream data. Video stream data reveals the visual information of the scene, and audio stream provides auditory information of the same scene. There is some kind of correlation information between them. We call this implicit information "affinity of multimodal data."

[0004] However, the heterogeneity of this data also poses a challenge to multimodal learning, because data of different modalities cannot reach the cloud at the same time for fusion as we would like due to differences in structure. There are always some data with slow transmission speeds or data that is easily damaged. These slow streams will cause waiting when the fast stream has already reached the cloud, resulting in increased latency. If there is no waiting, the accuracy will be reduced. At present, most solutions are to block the fast stream and wait for the slow stream to arrive at the cloud together. In order to ensure accuracy, the latency is reduced, but the essential problem of data streams of different modalities is not solved. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides an affinity analysis method for multimodal data streams based on the intelligent Internet of Things, including an affinity measurement method for multimodal data and an affinity matrix construction based on the hierarchical analysis method. The present invention can well calculate the hidden affinity between multimodal data, ensure accuracy and adapt to a variety of different heterogeneous sensors. The method of the present invention is universal and robust, can process different types of sensor data, and can work reliably in various practical application scenarios.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step 1: Affinity measurement of multimodal data;

[0008] Step 1-1: For data streams of different modes, different feature extraction networks are selected to obtain the features of the modal data generated by different sensors;

[0009] Step 1-2: Perform t-SNE dimensionality reduction on the acquired features to obtain low-dimensional feature representation, and then calculate the average cosine similarity between the low-dimensional features as a preliminary affinity measure;

[0010] Step 2: Affinity matrix construction based on analytic hierarchy process;

[0011] Step 2-1: for all sensors in the entire sensor network, divide them into sensors of interest and sensors of need; the sensors of interest include slow flow sensors and vulnerable sensors, and the sensors of need include fast flow sensors;

[0012] Step 2-2: Divide the sensors of interest into the focus layer and the required sensors into the candidate layer;

[0013] Step 2-3: Construct the criteria layer;

[0014] The criteria for the quasi-test layer select the initial affinity;

[0015] Step 2-4: Construct a judgment matrix between the criterion layer and the focus layer, and between the criterion layer and the alternative layer;

[0016] Step 2-5: Calculate the weight matrix of each judgment matrix;

[0017] Step 2-6: Multiply the weight matrix of the criterion layer and the focal layer with the weight matrix of the criterion layer and the candidate layer to obtain the final affinity matrix.

[0018] Preferably, in step 1-1, U-net is used for image data, and PointNet network is used for 3D point cloud data.

[0019] Preferably, the average cosine similarity calculation formula is:

[0020]

[0021] Among them, A and B are two eigenvectors.

[0022] Preferably, the judgment matrix construction algorithm in step 2-4 is specifically:

[0023] Step 2-4-1: Construct a square matrix according to the number of elements in the hierarchy;

[0024] Step 2-4-2: Fill in the matrix content according to the relative importance of each element;

[0025] The value of the square matrix element is 1: it means that the two elements have the same importance;

[0026] The value of the square matrix element is 3: it means that compared with the two elements, the former is slightly more important than the latter;

[0027] The value of the square matrix element is 5: it means that compared with the two elements, the former is obviously more important than the latter;

[0028] The value of the square matrix element is 7: it means that compared with the two elements, the former is extremely important than the latter;

[0029] The value of the square matrix element is 9: it means that compared with the two elements, the former is more important than the latter;

[0030] The values ​​of the square matrix elements are 2, 4, 6, and 8: representing the middle values ​​of the above adjacent judgments;

[0031] The values ​​of the square matrix elements are the reciprocals of 1-9: indicating the importance of comparing the order of the corresponding two elements;

[0032] Step 2-4-3: The judgment matrix is ​​constructed.

[0033] Preferably, the weight matrix calculation formula in steps 2-5 is as follows:

[0034]

[0035] The beneficial effects of the present invention are as follows:

[0036] 1. The present invention can well calculate the hidden affinity between multimodal data. It ensures accuracy and can adapt to a variety of different heterogeneous sensors. The method of the present invention is versatile and robust, can process different types of sensor data, such as text, images, audio, etc., and can work reliably in various practical application scenarios.

[0037] 2. The matrix structure of the present invention is used to represent the architecture and state parameters of the entire system. Through this data structure, affinity can be further optimized and refined. This includes analyzing the relationship between the data of each sensor and the performance of the system in different states, thereby constructing an efficient data structure that can represent the affinity relationship between all heterogeneous sensors in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a structural diagram of the method of the present invention.

[0039] Figure 2 It is a schematic diagram of the hierarchical structure of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0041] The first objective of the present invention is to develop a highly accurate and efficient computational method to quantify the affinity between multimodal data (heterogeneous sensors), and it is hoped that this method can be adapted to different heterogeneous sensors.

[0042] The technical solution for achieving the first object of the present invention is: a method for measuring affinity of multimodal data, comprising the following steps:

[0043] ① For data streams of different modes, different feature extraction networks are selected, such as U-net for image data and PointNet network for 3D point cloud data. First, the features of the modal data generated by different sensors are obtained;

[0044] ② Perform t-SNE dimensionality reduction on the acquired features to obtain low-dimensional feature representation, and then calculate the average cosine similarity between the low-dimensional features as a preliminary affinity measure.

[0045] The specific average cosine similarity calculation formula in step ② is (A, B are two feature vectors):

[0046]

[0047] The second objective of the present invention is to study a matrix structure through which affinity can be further optimized and refined. This involves analyzing the relationship between the data of each sensor and the performance of the system in different states, thereby constructing an efficient data structure that can represent the affinity relationship between all heterogeneous sensors in the system.

[0048] The technical solution for achieving the second object of the present invention is: a method for constructing an affinity matrix based on a hierarchical analysis method, comprising the following steps:

[0049] ① For all sensors in the entire sensor network, distinguish the sensors we are concerned about (slow-flow sensors or vulnerable sensors) from the sensors we need (fast-flow sensors);

[0050] ② Divide the sensors of interest into the focus layer and the required sensors into the alternative layer;

[0051] ③Build the criterion layer. The criterion of the criterion layer selects the preliminary affinity obtained in the previous solution, and other criteria are added according to the actual situation, such as the cross FOV (Field Of View) of the sensor;

[0052] ④ Construct a judgment matrix between the criterion layer and the focus layer, and between the criterion layer and the alternative layer;

[0053] ⑤Calculate the weight matrix of each judgment matrix;

[0054] ⑥ Multiply the weight matrix of the criterion layer and the focus layer with the weight matrix of the criterion layer and the candidate layer to obtain the final affinity matrix;

[0055] The weight matrix construction algorithm in step ④ is specifically as follows:

[0056] I. Construct a square matrix according to the number of elements in the hierarchy;

[0057] II. Fill in the matrix content according to the relative importance of each element. For example, when constructing the matrix of the criterion layer and the focus layer, if the criterion layer has two criteria, then construct a two-dimensional matrix and fill in the matrix content according to the following importance table:

[0058]

[0059] III. The judgment matrix is ​​constructed;

[0060] The weight matrix calculation formula in step ⑤ is as follows:

[0061]

[0062] Example:

[0063] Step 1: Sensor affinity measurement;

[0064] Taking the onboard sensor network of autonomous driving as an example, the entire network consists of cameras and lidars. This method first uses different feature extraction networks to extract features for different sensors to ensure that each feature extraction network can adapt to the corresponding sensor modality data. For example, for 3D radar point cloud data and image data, classic PointNet and U-net can be used. For sensors {S1, S2, ..., Sn}, features {x1, x2, ..., xn} are obtained, and then the t-SNE method is used to reduce their dimensions to low-dimensional space. t-SNE converts high-dimensional features xi(i) into low-dimensional features yi to ensure that the features after dimensionality reduction retain the information of the original features. After obtaining the low-dimensional features, the average cosine similarity CosSimilarity between the low-dimensional features of each sensor is calculated, and the obtained cosine similarity is used as the preliminary affinity measurement;

[0065] Step 2: A method for constructing sensor network affinity matrix based on analytic hierarchy process;

[0066] In the sensor affinity measurement method, an average cosine similarity CosSimilarity can be finally obtained as a preliminary measure of affinity, which is used to represent the affinity relationship between each sensor, abbreviated as CSY in the following text. The similarity between the i-th sensor and the j-th sensor is expressed as CSY ij , it is obvious that CSYij=CSY jiThe reason why it is called preliminary affinity is that it is only a characteristic distance calculation at the mathematical level and cannot fully interpret the affinity relationship of the entire sensor network. Therefore, in this step, the affinity matrix of the entire sensor network is constructed, which takes into account the influencing factors of the sensor network and reflects the affinity relationship of the entire sensor network.

[0067] Step 2-1: Construct the hierarchical analysis structure;

[0068] According to the hierarchical analysis method, the focus layer, alternative layer, and criterion layer are set. The slow flow and vulnerable sensors {f1, f2, ..., fn} are selected as the focus layer Focus, and the slow flow sensors {a1, a2, ..., an} are selected as the alternative layer Altenatives. The criterion layer Criteria selects factors that may affect affinity in the sensor network {C1, C2, ..., Cn}, such as the cross FOV (Field Of View) of the sensor, etc., and the CSY obtained in the previous step is also added to the criterion layer. The entire criterion layer is constructed as shown in the figure below. Figure 2 as shown in .

[0069] Step 2-2: Use the analytic hierarchy process to construct an affinity matrix;

[0070] There are two types of matrices for the criterion layer and the candidate layer, and the quasi-measurement layer and the focus layer. The first is the criterion layer and the focus layer matrix A. Assuming there are two criteria C1 and C2, then for each sensor in the focus layer, a 2*2 matrix A is constructed, and the element a in A is ij represents the importance of the i-th criterion and the j-th criterion relative to the focus sensor, then the matrix A can be finally obtained:

[0071]

[0072] If a 12 =7, which means that the C1 criterion is more important than C2. So, taking the previous example, cosine similarity is more important than cross FOV.

[0073] The next step is to solve the matrix Bn, b between the criterion layer and the candidate layer ij The value of also represents the importance of candidate sensor i relative to j on criterion n. Assuming that there are only three sensors in the device selection layer, we can get B 1 ,B 2 (The values ​​are filled in as examples and have no practical significance):

[0074]

[0075] The next step is to solve the weight matrix of matrix A and B, and then we can get the weight matrix WA of A, which should be a 2*1 matrix, and then solve B 1 and B2 The weight matrices WB1, WB2 are combined into WB = [WB1, WB2].

[0076] Finally, multiply WA and WB and normalize them to get the final weight matrix W (a 3*1 matrix in this example). In this example, W represents the expected affinity relationship between the three candidate sensors for a focus sensor. The element value of the matrix w i It represents the affinity of the i-th sensor in the entire sensor network relative to the focal sensor, that is, the size of the association relationship between the two sensors.

Claims

1. A method for affinity analysis of multimodal data streams based on intelligent Internet of Things. It is characterized in that The steps include: Step 1: Affinity measurement of multimodal data; Step 1-1: For data streams of different modes, different feature extraction networks are selected to obtain the features of the modal data generated by different sensors; Step 1-2: Perform t-SNE dimensionality reduction on the acquired features to obtain low-dimensional feature representation, and then calculate the average cosine similarity between the low-dimensional features as a preliminary affinity measure; Step 2: Affinity matrix construction based on analytic hierarchy process; Step 2-1: for all sensors in the entire sensor network, divide them into sensors of interest and sensors of need; the sensors of interest include slow flow sensors and vulnerable sensors, and the sensors of need include fast flow sensors; Step 2-2: Divide the sensors of interest into the focus layer and the required sensors into the candidate layer; Step 2-3: Construct the criteria layer; The criteria for the quasi-test layer select the initial affinity; Step 2-4: Construct a judgment matrix between the criterion layer and the focus layer, and between the criterion layer and the alternative layer; Step 2-5: Calculate the weight matrix of each judgment matrix; Step 2-6: Multiply the weight matrix of the criterion layer and the focal layer with the weight matrix of the criterion layer and the candidate layer to obtain the final affinity matrix.

2. According to claim 1, a method for affinity analysis of multimodal data streams based on intelligent Internet of Things, It is characterized in that In step 1-1, U-net is used for image data, and PointNet network is used for 3D point cloud data.

3. According to claim 1, a method for affinity analysis of multimodal data streams based on intelligent Internet of Things, It is characterized in that The average cosine similarity calculation formula is: Among them, A and B are two eigenvectors.

4. According to claim 1, a method for affinity analysis of multimodal data streams based on intelligent Internet of Things, It is characterized in that The judgment matrix construction algorithm in step 2-4 is specifically as follows: Step 2-4-1: Construct a square matrix according to the number of elements in the hierarchy; Step 2-4-2: Fill in the matrix content according to the relative importance of each element; The value of the square matrix element is 1: it means that the two elements have the same importance; The value of the square matrix element is 3: it means that compared with the two elements, the former is slightly more important than the latter; The value of the square matrix element is 5: it means that compared with the two elements, the former is obviously more important than the latter; The value of the square matrix element is 7: it means that compared with the two elements, the former is extremely important than the latter; The value of the square matrix element is 9: it means that compared with the two elements, the former is more important than the latter; The values ​​of the square matrix elements are 2, 4, 6, and 8: representing the middle values ​​of the above adjacent judgments; The values ​​of the square matrix elements are the reciprocals of 1-9: indicating the importance of comparing the order of the corresponding two elements; Step 2-4-3: The judgment matrix is ​​constructed.

5. According to claim 1, a method for affinity analysis of multimodal data streams based on intelligent Internet of Things, It is characterized in that The weight matrix calculation formula in steps 2-5 is as follows:

Citation Information

Patent Citations

  • Collection method and collection device

    CN103729244A

  • Computing network selection method and platform based on analytic hierarchy process

    CN109067581A

  • Zero sample cross-modal retrieval method and system based on tensor product graph fusion diffusion

    CN117992805A