A cross-temporal and spatial multi-level perception data association method based on target attribute semantic consistency

Abstract

The present invention discloses a method for associating multi-level perception data across time and space based on semantic consistency of target attributes. First, an attribute trajectory is constructed to integrate multi-dimensional data from different sensors and form a time series to reflect the changes in the comprehensive state of the target. Then, data entity alignment is performed, and the correlation between target motion and stillness is calculated to associate multi-dimensional information of the same target. Then, nodes containing the original attribute information of the target are created and weighted edges are constructed according to the spatiotemporal correlation, and multi-class representations are obtained through a graph attention network (GAT). After that, the target semantic consistency constraint is implemented and the corresponding loss function is calculated. In addition, self-supervised learning and network training are adopted, the extracted features are first clustered and the results are used as pseudo-labels, and then a network collaborative self-supervised learning containing clustering and classification branches is constructed, and the network parameters are updated and optimized by the back propagation algorithm according to the target semantic consistency loss and clustering loss, and it is iterated repeatedly until the network converges, thereby realizing the effective association and processing of multi-level perception data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data association, and more specifically, relates to a cross-temporal and multi-level perception data association method based on semantic consistency of target attributes. Background Art

[0002] With the rapid development of sensor technology, various types of sensors are widely used in many fields. Different sensors differ in accuracy, resolution, perception range, perception dimension, etc. Taking cameras and radars in traffic scenes as an example, cameras can provide high-resolution visual images and can clearly identify the appearance characteristics of vehicles, such as vehicle model, color, license plate, etc., but may be limited in bad weather or long distances. Radar can accurately measure the target's distance, speed and other motion parameters, and is not affected by lighting and weather conditions, but has a weaker perception of the target's appearance characteristics. In practical applications, these multi-level perception data across time and space need to be associated to obtain a more comprehensive and accurate description of the target.

[0003] Traditional data association methods face many challenges when dealing with multi-level perception data across time and space. Simple rule-based association methods, such as matching based on timestamps and spatial positions, often have difficulty accurately associating data when faced with complex target behaviors and changing environments. For example, when a target moves quickly in a short period of time or when data is lost or noise interferes, relying solely on temporal and spatial rules may lead to association errors. Therefore, a new association method is needed to solve problems such as target state loss and mixed pseudo-target information. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for associating cross-temporal and multi-level perception data based on semantic consistency of target attributes, so as to achieve effective association of cross-temporal and multi-level perception data in real scenarios.

[0005] To achieve the above-mentioned object of the invention, the present invention provides a method for associating multi-level perception data across time and space based on semantic consistency of target attributes, which is characterized by comprising the following steps:

[0006] (1) Constructing attribute trajectories: Integrate multi-dimensional data from different sensors, arrange them in chronological order, form a continuous time series, and construct attribute trajectories that can reflect the changes in the comprehensive state of the target in different time and space;

[0007] (2) Data entity alignment: When multiple targets appear in the perception scene, data entity alignment is required to associate the multi-dimensional information of the same target. When calculating target correlation, consider the correlation between motion and stillness. Since stillness correlation is easier to judge and more reliable, it can be used for grouping. When the target moves continuously without obvious pauses, use motion correlation. Motion correlation calculates the correlation of time series in different dimensions to find the best matching group so that the average correlation within the group is maximized.

[0008] (3) Extract features using graph attention network: Create nodes corresponding to the perceived data points, including the original attribute information of the target, and construct weighted edges based on the spatiotemporal correlation of the target (weights are calculated based on spatiotemporal distance, attribute similarity, etc.). Use graph attention network (GAT) to obtain representations such as spatiotemporal position, motion attributes, and target attributes from different angles. Taking motion representation as an example, first calculate the attention score to measure the importance of neighboring nodes, then normalize it to obtain the corresponding coefficient, and then derive the new feature vector of the node. The spatiotemporal position representation is calculated through a multi-layer GAT network, and when learning motion attribute representation and target attribute representation, the GAT network focuses on motion feature similarity and target attribute similarity respectively.

[0009] (4) Target semantic consistency constraints: First, we need to determine which specific constraints to use (e.g., based on the same target constraint, ensure that targets belonging to the same target are consistent at the semantic level; based on the spatiotemporal position representation consistency constraint, expect nodes belonging to the same target to have consistency in their spatiotemporal position representation in terms of spatial position and time sequence; based on the target state consistency constraint, nodes of the same target should have continuity and consistency in their motion states). Then, when actually associating multi-level perception data, we use these selected constraints to determine whether targets in different perception data meet the semantic consistency requirements;

[0010] (5) Self-supervised learning and network training: Since it is difficult to obtain target label information in real complex scenes and there may be unknown types of targets, self-supervised learning methods are proposed for training. The extracted features are classified by similarity using clustering algorithms such as K-Means and Sync, and the clustering results are assigned to the feature data as pseudo-labels. In subsequent network training, they play a role similar to real labels and guide the network to learn patterns that conform to the data feature structure. Then, a network with clustering branches and classification branches is constructed, and the two branches work together to achieve self-supervised learning. Based on the target semantic consistency loss plus the clustering loss, the back-propagation algorithm is used to update and optimize the network parameters, and the two-branch training based on pseudo-labels, total loss calculation and parameter update process are repeated, and iterated multiple times until the network converges.

[0011] The object of the present invention is achieved in this way.

[0012] The present invention is based on a method for associating multi-level perception data across time and space based on semantic consistency of target attributes. First, attribute trajectories are constructed, multi-sensor multi-dimensional data are integrated and arranged into a continuous sequence in time sequence, so as to reflect the comprehensive state change of the target in time and space. Then, a graph attention network (GAT) is used to extract features, nodes containing original attribute information of the target are constructed, weighted edges are constructed according to spatiotemporal correlation, and multi-class representations are obtained through GAT. Then, the target semantic consistency constraint is implemented, and when associating multi-level perception data, it is judged whether the target meets the semantic consistency requirements based on these conditions. Finally, since it is difficult to obtain target labels in real scenes and there may be unknown types of targets, self-supervised learning and network training are adopted. First, the extracted features are classified by a clustering algorithm, and the clustering results are used as pseudo labels to guide network learning. Then, a network collaborative self-supervised learning containing clustering and classification branches is constructed, and according to the target semantic consistency loss and clustering loss, the network parameters are updated and optimized by the back propagation algorithm, and branch training based on pseudo labels, total loss calculation and parameter update are repeatedly performed until the network converges, thereby realizing the effective association and processing of multi-level perception data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of a specific implementation method of the cross-temporal and spatial multi-level perception data association method based on the semantic consistency of target attributes of the present invention;

[0014] Figure 2 It is a schematic diagram of constructing multiple dimensional feature trajectories in the present invention, including time features, space features, motion features, etc.;

[0015] Figure 3 It is a schematic diagram of aligning data entities of different dimensions in the present invention, demonstrating how to align data collected by an image sensor and a radar sensor;

[0016] Figure 4 It is a schematic diagram of the overall network of the cross-temporal and spatial multi-level perception data association method based on the semantic consistency of target attributes in the present invention. DETAILED DESCRIPTION

[0017] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0018] Figure 1 It is a flow chart of a specific implementation method of the cross-temporal and spatial multi-level perception data association method based on the semantic consistency of target attributes of the present invention.

[0019] In this embodiment, if Figure 1As shown, the cross-temporal and spatial multi-level perception data association method based on the semantic consistency of target attributes of the present invention includes the following steps:

[0020] S1: Constructing attribute tracks

[0021] First, we need to collect multi-dimensional data from different sensors, including the target’s location information and implicit feature information such as target attributes (such as vehicle color, model, etc.), motion information (speed, acceleration, etc.), and parameter information (size, weight, etc.).

[0022] The integrated information is arranged in chronological order to form a continuous time series, thus constructing an attribute trajectory that can reflect the comprehensive state changes of the target in different time and space. For example, the vehicle's position, appearance, speed and other aspects are recorded once every certain time interval (such as 5 seconds). The dynamic changes of the vehicle in the entire traffic environment over time are intuitively displayed in a string. This trajectory is the basic data carrier for subsequent analysis and characterization.

[0023] like Figure 2 As shown, the trajectory features mentioned in the present invention come from three main sets: time feature set (T), space feature set (S), and motion feature set (M). The elements in any set contain features related to themselves, which can be specifically expressed as a series of attribute values. The formation of the trajectory is not instantaneous, but takes a certain amount of time, so time and space are inseparable components of the trajectory formation process. With the change of time, other features will also change accordingly. For example, with the passage of time intervals, the spatial position will change, that is, T→S; with the change of time and space, a series of motion features or motion parameter values ​​will also continue to change, that is, T→M. At the same time, these mapping relationships should not be simply treated separately, but should be considered simultaneously.

[0024] S2: Data Entity Alignment

[0025] When the number of targets N>1 in the perception scene, it is necessary to align the data entities, that is, to associate the multi-dimensional information of the same target.

[0026] Figure 3 The specific process of data entity alignment is shown. Assuming there is a scene with a person and a car, the image sensor and radar sensor receive target information respectively. When calculating the target correlation, the present invention considers both motion correlation and static correlation. The following is an explanation of the two:

[0027] Motion correlation: On the one hand, data belonging to the same target should have more consistent semantic information than data from different targets. The best matching group is found by calculating the correlation between time series of different dimensions and maximizing the average correlation within the group.

[0028] Stillness correlation: Stillness can be regarded as a special state of motion. Unlike distinguishing the state of motion, stillness can be easily judged. Compared with motion correlation, stillness correlation is generally much more reliable. Therefore, the data series can be grouped according to stillness correlation first, and motion correlation can be used when the target is in continuous motion without obvious pauses.

[0029] The specific correlation formula can be expressed as follows:

[0030]

[0031] Here | S i |Indicates the number of data segments split within a fixed time period. is the indicator function. i1 and i2 refer to two different modes. L is the data segment S i [n] labels. Both motion correlation and stationary correlation are used to calculate L. For motion correlation, L is given by the corresponding model M i is given, while for stationary correlation, a threshold is used for detection and then calculation.

[0032] S3: Graph Attention Network Feature Extraction

[0033] For each perception data point (corresponding to the target observation at a specific time and space), a node is created. The node contains the original attribute information of the target, such as the target appearance feature vector obtained from the image sensor, the coordinate information obtained from the position sensor, the motion parameter vector such as speed and acceleration obtained from the motion sensor, etc. The edges are constructed according to the spatiotemporal correlation of the target. For example, if two perception data points are adjacent in time and close in spatial position (which can be determined by setting the time threshold and the spatial distance threshold), an edge is established between their corresponding nodes. The edge can be weighted, and the weight can be calculated based on factors such as the spatiotemporal distance between nodes and the attribute similarity. For example, the closer the spatiotemporal distance and the higher the attribute similarity, the greater the weight of the edge.

[0034] From different perspectives, the graph attention network (GAT) is used to obtain representations such as spatiotemporal position representation, motion attribute representation, and target attribute representation. Taking motion representation as an example, let the feature vector of node i be x i , whose neighborhood node set is N(i), calculate the attention score e ij To measure the importance of node j to node j:

[0035]

[0036] where q and W are learnable parameters, represents the vector concatenation operation. Then the attention score is normalized by the softmax function to obtain α ij :

[0037]

[0038] The new feature vector x of the last node i i ′ for:

[0039]

[0040] After calculation of the multi-layer GAT network, the spatiotemporal position representation of the nodes is obtained.

[0041] Similarly, when learning motion attribute representation, the GAT network focuses on neighborhood nodes with similar motion features; for target attribute representation, the GAT network considers the similarity of target attributes between nodes.

[0042] S4: Target semantic consistency constraints

[0043] In the multi-level perceptual data association method, the target semantic consistency constraint plays a key role, aiming to ensure that the network learns accurate and logical target association feature representation. The core points are as follows:

[0044] 4.1) Same target constraint

[0045] Define the discriminant function of the same target. For two nodes i and j, calculate the probability p that they belong to the same target ij For example, we can input their feature vectors (including comprehensive representations such as spatiotemporal position, motion attributes, and target attributes) into a fully connected layer and then pass it through the softmax function to obtain:

[0046] p ij =softmax(f(x i ′ ,x j ′ ))

[0047] Where f is the fully connected layer function.

[0048] Calculate the cross entropy loss L for the same objective constraint cross For a node pair (i, j) that is known to belong to the same target, its label y ij = 1; for node pairs that are known not to belong to the same target, their labels y ij =0. Then the cross entropy loss is:

[0049]

[0050] This loss function forces the network to learn feature representations that can accurately distinguish between the same object and different objects.

[0051] 4.2) Consistency constraints on spatiotemporal position representation

[0052] Compute the difference measure of spatiotemporal position representation between nodes, such as Euclidean distance or cosine similarity. For pairs of nodes belonging to the same target, their spatiotemporal position representation is expected to be consistent in spatial position and time order. For example, if a target has position nodes at consecutive time points, its spatial position should change continuously and conform to the law of target movement.

[0053] Define the consistency loss L of spatiotemporal position representation pos The position representation difference metric can be compared with the set threshold. If the difference exceeds the threshold, a certain loss penalty is given. For example, let d ij The spatial and temporal position difference metric of nodes i and j is represented by a simple Euclidean distance, i.e.

[0054]

[0055] τ pos is the threshold, then L pos It can be expressed as:

[0056]

[0057] This loss function helps ensure that the spatiotemporal position representation learned by the network can accurately reflect the actual spatiotemporal position changes of the target.

[0058] 4.3) Target state consistency constraints

[0059] Similarly, the difference measures of the motion attribute representations between nodes are calculated, such as the rate of change of velocity, the rate of change of acceleration, etc. For the node pairs of the same target, their motion states should be continuous and consistent. For example, during normal driving of a vehicle, its speed and acceleration will not suddenly jump (unless a special event occurs, such as a collision, but this situation can be handled separately through anomaly detection).

[0060] Define the target motion state consistency loss L motion . The difference between the motion attribute representation and the set threshold is compared. If the difference is too large, a loss penalty is generated:

[0061]

[0062] Where m ijis the difference measure of the motion attribute representation between nodes i and j, τ motion is the motion state consistency threshold.

[0063] As mentioned above, the total loss function is as follows:

[0064] L same =L cross +L pos +L motion

[0065] Through the back-propagation algorithm, the total loss function is used to optimize the parameters of the graph attention network (including the attribute trajectory representation network part and the association network part). During the training process, the network parameters are continuously adjusted so that the network can automatically extract the spatiotemporal features of target associations and meet various target semantic consistency constraints. After training with a large amount of data, the network can accurately perform data association and feature extraction when facing new multi-level perception data, providing strong support for subsequent analysis and application.

[0066] S5: Self-supervised learning and network training

[0067] In the target semantic consistency association technology, the basic assumption is that multiple sensor data of some targets are known. Considering that it is difficult to obtain target label information in real complex scenes and unknown types of targets may also appear, it is proposed to use self-supervised learning methods for training.

[0068] 5.1) Feature Clustering

[0069] First, clustering operations are performed on the extracted features, aiming to divide them into different categories based on the similarity of the features. Different clustering algorithms can be used, such as the common K-Means and Sync, which can mine the inherent distribution structure of the data, group similar features together, and form clustering results.

[0070] The results obtained through feature clustering are used as pseudo labels and assigned to the corresponding feature data. These pseudo labels are equivalent to the role of real labels in supervised learning in the subsequent network training process, guiding the network to learn a model that conforms to the data feature structure.

[0071] 5.2) Network construction

[0072] Construct a network for training. The network has two branches at the output, namely the clustering branch and the classification branch. The two branches work together to achieve the self-supervised learning process:

[0073] The clustering branch is mainly responsible for inputting the features of the network output into the clustering model again for clustering operations. The clustering model here is associated with the clustering algorithm used for feature clustering in the first step. Its purpose is to continuously perform clustering analysis on the features during the network training process so as to compare with the pseudo-labels and adjust the network parameters later. After the clustering branch inputs the features of the network output into the clustering model for clustering, it will obtain the clustering result of the current network output features. This result will be compared with the pseudo-labels generated previously, and the corresponding loss will be calculated (for example, the difference between the two can be reflected by measuring the accuracy indicators of clustering, such as the purity of clustering, the Rand index, etc., and converted into a loss function) to measure the degree of deviation between the output of the clustering branch and the expected pseudo-label, thereby providing feedback for the adjustment of network parameters.

[0074] The classification branch follows the method used in previous supervised learning, aiming to learn the similarities between different types of data samples. By grasping the similarity of samples, it provides a basis for subsequent training based on pseudo labels, thereby optimizing the network's ability to distinguish different target features. The classification branch uses the similarity learning method in previous supervised learning to analyze different types of data samples, train based on pseudo labels, and also calculate a classification loss (such as the common cross entropy loss, etc.), which reflects the accuracy of the classification branch in classifying samples based on pseudo labels.

[0075] 5.3) Network training iteration

[0076] The loss function of the network is the target semantic consistency loss plus the clustering loss:

[0077]

[0078] Among them is Loss function, measuring the predicted value With the true value The difference between is the input feature of the nth sample at time t, is the corresponding true label or target value; Represents the weight matrix associated with cluster l Perform vectorized operations; represents the inverse of the covariance matrix; The entropy associated with cluster l; Calculate the squared error between the prediction and the reconstruction based on the cluster assignment; It is a constraint that means each sample can only belong to one cluster.

[0079] Then, based on this total loss function, the network parameters are updated and optimized through the back propagation algorithm. The back propagation process will perform gradient calculations on the weights and biases of each layer of the network according to the total loss function, and adjust the parameters according to the set learning rate, so that the network gradually learns in the direction that can reduce the total loss, that is, to make the feature clustering results output by the network more consistent with the pseudo-label, and the classification branch can better classify the samples based on the pseudo-label. The process of training the clustering branch and the classification branch based on pseudo-labels, calculating the total loss, and updating the parameters through back propagation will be repeated continuously. In multiple iterations, the network continues to learn and optimize, gradually mastering the feature association patterns of the targets in the data and the basis for distinguishing different targets, until the network converges (for example, the loss function no longer decreases significantly or reaches the preset number of training rounds and other stopping conditions).

[0080] Figure 4 It is a schematic diagram of the overall network of the cross-temporal and multi-level perception data association method based on the semantic consistency of target attributes in the present invention. The attribute representation network is used to obtain the multi-dimensional spatiotemporal features under different sensors, and then the association is automatically performed through the target semantic consistency constraint. The extracted features are the motion trajectories of different targets. Due to the semantic consistency of the targets, the outputs of the two neural networks should be similar trajectories, and their features are similar. The subspace clustering method can be used to perform clustering to assist the automatic extraction of the associated features of the target.

[0081] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for associating multi-level perception data across time and space based on semantic consistency of target attributes, characterized in that: The following steps are involved: (1) Constructing attribute trajectories: Integrate multi-dimensional data from different sensors, arrange them in chronological order, form a continuous time series, and construct attribute trajectories that can reflect the changes in the comprehensive state of the target in different time and space; (2) Data entity alignment: When multiple targets appear in the perception scene, data entity alignment is required to associate the multi-dimensional information of the same target. When calculating target correlation, consider the correlation between motion and stillness. Since stillness correlation is easier to judge and more reliable, it can be used for grouping. When the target moves continuously without obvious pauses, use motion correlation. Motion correlation calculates the correlation of time series in different dimensions to find the best matching group so that the average correlation within the group is maximized. (3) Extract features using graph attention network: Create nodes corresponding to the perceived data points, including the original attribute information of the target, and construct weighted edges based on the spatiotemporal correlation of the target (weights are calculated based on spatiotemporal distance, attribute similarity, etc.). Use graph attention network (GAT) to obtain representations such as spatiotemporal position, motion attributes, and target attributes from different angles. Taking motion representation as an example, first calculate the attention score to measure the importance of neighboring nodes, then normalize it to obtain the corresponding coefficient, and then derive the new feature vector of the node. The spatiotemporal position representation is calculated through a multi-layer GAT network, and when learning motion attribute representation and target attribute representation, the GAT network focuses on motion feature similarity and target attribute similarity respectively. (4) Target semantic consistency constraints: First, we need to determine which specific constraints to use (e.g., based on the same target constraint, ensure that targets belonging to the same target are consistent at the semantic level; based on the spatiotemporal position representation consistency constraint, expect nodes belonging to the same target to have consistency in their spatiotemporal position representation in terms of spatial position and time sequence; based on the target state consistency constraint, nodes of the same target should have continuity and consistency in their motion states). Then, when actually associating multi-level perception data, we use these selected constraints to determine whether targets in different perception data meet the semantic consistency requirements; (5) Self-supervised learning and network training: Since it is difficult to obtain target label information in real complex scenes and there may be unknown types of targets, self-supervised learning methods are proposed for training. The extracted features are classified by similarity using clustering algorithms such as K-Means and Sync, and the clustering results are assigned to the feature data as pseudo-labels. In subsequent network training, they play a role similar to real labels and guide the network to learn patterns that conform to the data feature structure. Then, a network with clustering branches and classification branches is constructed, and the two branches work together to achieve self-supervised learning. Based on the target semantic consistency loss plus the clustering loss, the back-propagation algorithm is used to update and optimize the network parameters, and the two-branch training based on pseudo-labels, total loss calculation and parameter update process are repeated, and iterated multiple times until the network converges.

2. The method for associating multi-level perception data across time and space based on semantic consistency of target attributes according to claim 1 is characterized in that: In step (4), the target consistency is constrained, and in step (5), the network is trained using self-supervised learning: 4.1) Same target constraint Define the discriminant function of the same target. For two nodes i and j, calculate the probability p that they belong to the same target ij For example, we can input their feature vectors (including comprehensive representations such as spatiotemporal position, motion attributes, and target attributes) into a fully connected layer and then pass it through the softmax function to obtain: p ij =softmax(f(x′ i ,x′ j )) Where f is the fully connected layer function. Calculate the cross entropy loss L for the same objective constraint cross For a node pair (i, j) that is known to belong to the same target, its label y ij = 1; for node pairs that are known not to belong to the same target, their labels y ij =0. Then the cross entropy loss is: This loss function forces the network to learn feature representations that can accurately distinguish between the same object and different objects. 4.2) Consistency constraints on spatiotemporal position representation Compute the difference measure of spatiotemporal position representation between nodes, such as Euclidean distance or cosine similarity. For pairs of nodes belonging to the same target, their spatiotemporal position representation is expected to be consistent in spatial position and time order. For example, if a target has position nodes at consecutive time points, its spatial position should change continuously and conform to the law of target movement. Define the consistency loss L of spatiotemporal position representation pos The position representation difference metric can be compared with the set threshold. If the difference exceeds the threshold, a certain loss penalty is given. For example, let d ij The spatial and temporal position difference metric of nodes i and j is represented by a simple Euclidean distance, i.e. τ pos is the threshold, then L pos It can be expressed as: This loss function helps ensure that the spatiotemporal position representation learned by the network can accurately reflect the actual spatiotemporal position changes of the target. 4.3) Target state consistency constraints Similarly, the difference measures of the motion attribute representations between nodes are calculated, such as the rate of change of velocity, the rate of change of acceleration, etc. For the node pairs of the same target, their motion states should be continuous and consistent. For example, during normal driving of a vehicle, its speed and acceleration will not suddenly jump (unless a special event occurs, such as a collision, but this situation can be handled separately through anomaly detection). Define the target motion state consistency loss L motion . The difference between the motion attribute representation and the set threshold is compared. If the difference is too large, a loss penalty is generated: Where m ij is the difference measure of the motion attribute representation between nodes i and j, τ motion is the motion state consistency threshold. As mentioned above, the total loss function is as follows: L same =L cross +L pos +L motion 5.1) Feature Clustering First, clustering operations are performed on the extracted features, aiming to divide them into different categories based on the similarity of the features. Different clustering algorithms can be used, such as the common K-Means and Sync, which can mine the inherent distribution structure of the data, group similar features together, and form clustering results. The results obtained through feature clustering are used as pseudo labels and assigned to the corresponding feature data. These pseudo labels are equivalent to the role of real labels in supervised learning in the subsequent network training process, guiding the network to learn a model that conforms to the data feature structure. 5.2) Network construction Construct a network for training. The network has two branches at the output, namely the clustering branch and the classification branch. The two branches work together to achieve the self-supervised learning process: The clustering branch is mainly responsible for inputting the features of the network output into the clustering model again for clustering operations. The clustering model here is associated with the clustering algorithm used for feature clustering in the first step. Its purpose is to continuously perform clustering analysis on the features during the network training process so as to compare with the pseudo-labels and adjust the network parameters later. After the clustering branch inputs the features of the network output into the clustering model for clustering, it will obtain the clustering result of the current network output features. This result will be compared with the pseudo-labels generated previously, and the corresponding loss will be calculated (for example, the difference between the two can be reflected by measuring the accuracy indicators of clustering, such as the purity of clustering, the Rand index, etc., and converted into a loss function) to measure the degree of deviation between the output of the clustering branch and the expected pseudo-label, thereby providing feedback for the adjustment of network parameters. The classification branch follows the method used in previous supervised learning, aiming to learn the similarities between different types of data samples. By grasping the similarity of samples, it provides a basis for subsequent training based on pseudo labels, thereby optimizing the network's ability to distinguish different target features. The classification branch uses the similarity learning method in previous supervised learning to analyze different types of data samples, train based on pseudo labels, and also calculate a classification loss (such as the common cross entropy loss, etc.), which reflects the accuracy of the classification branch in classifying samples based on pseudo labels. 5.3) Network training iteration The loss function of the network is the target semantic consistency loss plus the clustering loss: Among them is Loss function, measuring the predicted value With the true value The difference between is the input feature of the nth sample at time t, is the corresponding true label or target value; Represents clustering The associated weight matrix Perform vectorized operations; represents the inverse of the covariance matrix; With clustering The associated entropy; Calculate the squared error between the prediction and the reconstruction based on the cluster assignment; It is a constraint that means each sample can only belong to one cluster.

Images