Sample marking method and device for multi-view data, equipment, medium and product

Abstract

The invention discloses a sample marking method and device for multi-view data, equipment, a medium and a product, and relates to the field of machine learning and artificial intelligence, and the method comprises the steps: constructing a training data set; training a classifier by using the training data set to obtain the weight and bias of each full connection layer in the classifier; marginal contribution values of different view features are obtained based on the weights and offsets of all the full-connection layers; updating parameters of the classifier and the training data set based on the marginal contribution values of different view features to obtain a current classifier and a current training data set; training a current classifier by using the current training data set, and updating the current classifier and the current training data set again by using the current marginal contribution value after the training is completed until the number of training iterations of the classifier reaches a preset round, thereby obtaining a trained classifier; and marking the unmarked samples by using the trained classifier to obtain marked samples. According to the method, each view marginal contribution can be dynamically quantified.

Description

Technical Field

[0001] This application relates to the fields of machine learning and artificial intelligence, and in particular to a method, apparatus, device, medium and product for sample labeling of multi-view data. Background Technology

[0002] In the implementation of machine learning and artificial intelligence technologies, multi-view data has become a key support for improving the generalization ability of classifiers. This type of data describes the same object through multi-dimensional features (such as images, which can extract three heterogeneous features: color, texture, and shape, forming multi-view data), and is widely used in fields such as computer vision and intelligent analysis. However, the cost of obtaining high-quality labeled data is extremely high. Manual labeling is not only time-consuming and labor-intensive, but also limited by professional barriers, becoming a core bottleneck restricting the large-scale application of the technology.

[0003] To overcome the above problems, existing technologies often use multi-view active learning (MVAL) methods to annotate multi-view data in order to reduce the reliance on manual annotation. However, existing technologies cannot dynamically quantify the actual value of features in each dimension, which can easily lead to a single view dominating decision-making and marginalizing weak and key views. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the related technologies, the purpose of this application is to provide a sample labeling method, apparatus, device, medium and product for multi-view data, which can dynamically quantify the marginal contribution of each view and effectively solve the problem of high signal-to-noise ratio views dominating and weak key views being marginalized.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a sample labeling method for multi-view data, comprising: constructing a training dataset based on an unlabeled image data pool; the training dataset includes unlabeled samples and labeled samples; training a classifier using the training dataset, and during the classifier training process, calculating a loss function based on the output of the classifier and the true labels, and performing backpropagation on the classifier based on the loss function to obtain the weights and biases of each fully connected layer in the classifier; calculating the marginal contribution of different view features of each sample in the training dataset based on the weights and biases of each fully connected layer in the classifier to obtain the marginal contribution value of different view features; updating the parameters of the classifier and the training dataset based on the marginal contribution values ​​of different view features to obtain the current classifier and the current training dataset; training the current classifier using the current training dataset, and updating the current classifier and the current training dataset again using the current marginal contribution value after training is completed, until the number of training iterations of the classifier reaches a preset number of rounds to obtain a trained classifier; and labeling the unlabeled samples using the trained classifier to obtain labeled samples.

[0006] Optionally, the step of calculating the marginal contribution of different view features of each sample in the training dataset based on the weights and biases of each fully connected layer in the classifier to obtain the marginal contribution value of different view features includes: taking the partial derivative with respect to the weights and biases of each fully connected layer in the classifier to obtain the gradient of each fully connected layer; updating the weights and biases of each fully connected layer based on the gradients of each fully connected layer to obtain the current weights and current biases of each connected layer; and calculating the marginal contribution of different view features of each sample in the training dataset based on the current weights and current biases of each connected layer to obtain the marginal contribution value of different view features.

[0007] Optionally, updating the classifier parameters and training dataset based on the marginal contribution values ​​of different view features to obtain the current classifier and the current training dataset includes: normalizing the marginal contribution values ​​of different view features to obtain contribution weights for different view features; weighting and aggregating the loss gradients of different view features based on their contribution weights to obtain an aggregated gradient; backpropagating the classifier parameters using the aggregated gradient to obtain the current classifier; quantifying the sample uncertainty of the unlabeled image data pool based on the L2 norm of the aggregated gradient to obtain a preset number of unlabeled samples; the unlabeled samples becoming the current diversified labeled samples after manual labeling; and removing the current diversified labeled samples from the unlabeled image data pool and adding them to the training dataset to obtain the current unlabeled image data pool and the current training dataset.

[0008] Optionally, constructing the training dataset based on the unlabeled image data pool includes: extracting different view features of each unlabeled sample in the unlabeled image data pool, and concatenating the different view features of each unlabeled sample to obtain a multi-view feature space corresponding to each unlabeled sample; selecting multiple initial centroids based on distance-weighted probability in the multi-view feature space for clustering iteration to obtain multiple cluster centroids; using the unlabeled samples corresponding to the multiple cluster centroids as diversified initial samples; the diversified initial samples becoming diversified labeled samples after manual labeling; and constructing the training dataset based on the diversified labeled samples.

[0009] Optionally, the step of selecting multiple initial center points in the multi-view feature space based on distance-weighted probability for clustering iteration to obtain multiple cluster center points includes: performing initial clustering using the K-means++ algorithm in the multi-view features, randomly selecting a center point as the first cluster center, iteratively selecting subsequent center points according to the probability distribution of the squared shortest distance between the current sample and the first cluster center, until N initial cluster center points are determined; assigning each sample in the unlabeled image data pool to the nearest initial cluster center point, recalculating the mean of each cluster as the new center point, updating and iterating until the center points no longer change, thus obtaining multiple cluster center points.

[0010] Optionally, the method further includes: counting the amount of sample data in the unlabeled image data pool; when the amount of sample data in the unlabeled image data pool is greater than a preset threshold, dividing the unlabeled image data pool to obtain multiple sub-unlabeled image data pools. The training dataset constructed based on the unlabeled image data pool includes: A training dataset is constructed based on the sub-unlabeled image data pools; multiple sub-unlabeled image data pools are processed sequentially by a classifier.

[0011] Secondly, this application provides a sample labeling device for multi-view data, comprising: A construction module is used to build a training dataset based on an unlabeled image data pool; the training dataset includes unlabeled samples and labeled samples. The first training module is used to train the classifier using the training dataset, and during the classifier training process, calculates the loss function based on the output of the classifier and the true label, and performs backpropagation on the classifier based on the loss function to obtain the weights and biases of each fully connected layer in the classifier. The calculation module is used to calculate the marginal contribution of different view features of each sample in the training dataset based on the weights and biases of each fully connected layer in the classifier, and obtain the marginal contribution value of different view features. The update module is used to update the parameters of the classifier and the training dataset based on the marginal contribution values ​​of different view features, so as to obtain the current classifier and the current training dataset. The second training module is used to train the current classifier using the current training dataset, and after the training is completed, to update the current classifier and the current training dataset again using the current marginal contribution value, until the number of updates reaches a preset number of rounds, and then the trained classifier is obtained. The labeling module is used to label unlabeled samples using the trained classifier to obtain labeled samples.

[0012] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the sample labeling method for multi-view data as described above.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the sample labeling method for multi-view data described above.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the sample labeling method for multi-view data described above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, medium, and product for sample labeling of multi-view data. A training dataset is constructed using an unlabeled image data pool. A classifier is trained using this training dataset. During classifier training, a loss function is calculated based on the classifier's output and the true labels. Backpropagation is then performed on the classifier based on the loss function to obtain the weights and biases of each fully connected layer. The marginal contribution of different view features for each sample in the training dataset is calculated using the weights and biases of each fully connected layer. The marginal contribution value of each view feature is dynamically quantified using this value. View weights can be adaptively allocated based on the marginal contribution of each view, effectively avoiding the problem of high signal-to-noise ratio views dominating and weak, key views being marginalized. This allows the classifier to fully utilize complementary information from multiple view features for parameter updates, ensuring stable performance even in complex scenarios such as missing views and noise interference. This effectively improves the accuracy of image sample labeling and reduces reliance on manual labeling. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a sample labeling method for multi-view data provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the extraction of a color histogram according to an embodiment of this application; Figure 3 A schematic diagram illustrating the extraction of LBP values ​​from a local binary pattern histogram according to an embodiment of this application; Figure 4 This is a schematic diagram of the gradient direction in the extracted directional gradient histogram provided in an embodiment of this application; Figure 5 This is a schematic diagram of an embodiment of the present application for extracting the directional gradient histogram; Figure 6 Clustering illustration provided for an embodiment of this application Figure 1 ; Figure 7 Clustering illustration provided for an embodiment of this application Figure 2 ; Figure 8 Clustering illustration provided for an embodiment of this application Figure 3 ; Figure 9 A schematic diagram of the functional modules of a sample marking device for multi-view data provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, such as Figure 1As shown, a sample labeling method for multi-view data is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method includes the following steps S101 to S106. Wherein: Step S101: Construct a training dataset based on the unlabeled image data pool.

[0021] In the example embodiment, the unlabeled image data pool is a collection of image data that has not been manually labeled in a multi-view active learning scenario. In this embodiment, it also serves as the data source for selecting training samples. The image data in the unlabeled image data pool contains multiple view features (such as color, texture, and shape features), but lacks manually labeled categories, attributes, etc. During subsequent classifier training, the number of images in the training dataset needs to be increased sequentially with each iteration of the classifier to improve its prediction accuracy. Correspondingly, as the number of images in the training dataset is increased, the number of images in the unlabeled image data pool is removed (reduced) sequentially until the classifier training is complete.

[0022] It should be noted that the training dataset includes both unlabeled and labeled samples.

[0023] Step S102: The classifier is trained using the training dataset. During the classifier training process, the loss function is calculated based on the output of the classifier and the true label. The classifier is backpropagated based on the loss function to obtain the weights and biases of each fully connected layer in the classifier.

[0024] In the example embodiment, the view features of the samples in the training dataset include the color features, texture features, and shape features of the image. In this embodiment, the classifier is constructed using the color features, texture features, and shape features, and the classifier structure is as follows: The input layer is divided into three channels, corresponding to color features, texture features, and shape features, respectively: Channel 1 (RGB view): Input 512-dimensional color histogram features, divided into 2 fully connected layers, activated by the ReLU function. Fully connected layer 1 (Equation 1 below) inputs a 512-dimensional vector and outputs a 256-dimensional vector. Fully connected layer 2 (Equation 2 below) inputs a 256-dimensional vector and outputs a 128-dimensional vector.

[0025] (1) (2) Where W is the weight matrix and b is the bias vector.

[0026] The ReLU function is as follows (3): (3) Channel 2 (LBP view): Input 4096-dimensional texture features, divided into 3 fully connected layers, activated by the ReLU function. Fully connected layer 1 (Equation 4 below) inputs a 4096-dimensional vector and outputs a 512-dimensional vector. Fully connected layer 2 (Equation 5 below) inputs a 512-dimensional vector and outputs a 256-dimensional vector. Fully connected layer 3 (Equation 5 below) inputs a 256-dimensional vector and outputs a 128-dimensional vector.

[0027] (4) (5) (6) Channel 3 (HOG view): Input 900-dimensional shape features, divided into 2 fully connected layers, activated by the ReLU function. Fully connected layer 1 (Equation 7 below) inputs a 900-dimensional vector and outputs a 256-dimensional vector. Fully connected layer 2 (Equation 8 below) inputs a 256-dimensional vector and outputs a 128-dimensional vector.

[0028] (7) (8) It should be noted that the weights of channels 1, 2 and 3 are initialized according to the He normal distribution, and the bias is initialized to a constant of 0.01; the L2 weights of channels 1 and 3 are decayed to 1e-4, and the L2 weights of channel 2 are decayed to 1e-3.

[0029] Hidden layer: Fully connected layer (Equation 9 below), activated by the ReLU function, outputting a 64-dimensional vector from the three 128-dimensional vectors obtained from the input layer.

[0030] (9) It should be noted that the concatenation order of the fully connected layers in the hidden layers is as follows: the 128-dimensional channel 1 is concatenated with the 128-dimensional channel 2, and then concatenated with the 128-dimensional channel 3, resulting in a total input dimension of 384. The weights of the fully connected layers in the hidden layers are randomly selected from a normal distribution with a mean of 0 and a standard deviation of ≈0.0722, and the bias is initialized with a constant of 0.01. The L2 weight decay is 1e-4.

[0031] Output layer: The 64-dimensional vector obtained from the hidden layer is activated by the softmax function, and a vector is output to represent the prediction probability of each category. The calculation process is as follows (10): (10) The classifier uses backpropagation during training, as shown in equation (11): (11) in, and , .

[0032] The cross-entropy loss function is as follows (12): (12) Where N is the number of samples in the training dataset, and K is the number of classes. It is the one-hot encoding of the true label of the sample in the nth training dataset, that is, it is 1 if the sample belongs to the kth class, and 0 otherwise. It is the probability that the classifier predicts that the sample belongs to the k-th class.

[0033] It should be noted that the weights of the fully connected layers in the output layer are initialized according to the Xavier normal distribution, and the biases are initialized to a constant of 0; the number of classes K can be set to 5 by default.

[0034] By adapting the graph data to multiple view features through independent branches—that is, processing features of different dimensions (such as 512-dimensional color features and 4096-dimensional texture features)—a single input layer can avoid the one-size-fits-all approach to heterogeneous features, providing a structural foundation for quantifying the marginal contribution of subsequent views. The fusion mechanism of the hidden layers allows the features of each view to complement each other. Combined with subsequent Shapley-weighted gradient updates, the classifier can fully utilize the advantages of multiple views, avoiding the dominance of a single view. The probabilistic certainty score output by the output layer can be directly used to calculate the marginal contribution and sample uncertainty of each view, effectively connecting the entire process of view evaluation, parameter update, and sample selection.

[0035] Step S103: Based on the weights and biases of each fully connected layer in the classifier, calculate the marginal contribution of different view features for each sample in the training dataset to obtain the marginal contribution value of different view features.

[0036] In the example embodiment, the marginal contribution value in this application refers to the difference between the predictive performance (such as accuracy and determination) of the classifier when the view is retained and the performance of the classifier after the view is hidden. In this application embodiment, the Shapley value is used to calculate the marginal contribution of each view, that is, to traverse all samples in the training dataset and find the view in all samples. v and no view v The average of the prediction certainty differences is used to obtain the marginal contribution score for each view, where a higher score indicates a higher marginal contribution of the view. v The greater the training aid for the classifier, the lower the score. v Redundancy or limited information value.

[0037] It should be noted that marginal contribution is the core basis for subsequent weight normalization and gradient modulation. That is, views with high marginal contribution will be assigned higher weights so that they can play a greater role in training; views with low marginal contribution will have their weights automatically reduced to avoid consuming too many computing resources, thereby solving the problem of single view dominating and key weak views being marginalized.

[0038] Step S104: Update the parameters of the classifier and the training dataset based on the marginal contribution values ​​of different view features to obtain the current classifier and the current training dataset.

[0039] Step S105: Train the current classifier using the current training dataset, and after training, update the current classifier and the current training dataset again using the current marginal contribution value until the classifier training iterations reach the preset number of rounds, and obtain the trained classifier.

[0040] Step S106: Use the trained classifier to label the unlabeled samples to obtain labeled samples.

[0041] By implementing steps S101 to S106 above, a training dataset is constructed using an unlabeled image data pool. The classifier is then trained using this training dataset. During training, the classifier is backpropagated based on its output to obtain the weights and biases of each fully connected layer. The marginal contribution of different view features for each sample in the training dataset is calculated using the weights and biases of each fully connected layer. The marginal contribution value of each view feature is then dynamically quantified using these values. Based on the marginal contribution of each view, view weights can be adaptively allocated, effectively avoiding the problem of high signal-to-noise ratio views dominating and weak key views being marginalized. This allows the classifier to fully utilize complementary information from multiple view features for parameter updates, ensuring that the trained classifier maintains stable performance even in complex scenarios such as missing views and noise interference. This effectively improves the accuracy of image sample labeling and reduces reliance on manual labeling.

[0042] In another exemplary embodiment of this application, in step S101 above, the step of constructing a training dataset based on an unlabeled image data pool can be achieved through steps S1011 to S1014.

[0043] Step S1011: Extract different view features of each unlabeled sample in the unlabeled image data pool, and concatenate the different view features of each unlabeled sample to obtain a multi-view feature space corresponding to each unlabeled sample; Step S1012: Select multiple initial center points in the multi-view feature space based on distance-weighted probability and perform clustering iteration to obtain multiple cluster center points; Step S1013: The unlabeled samples corresponding to multiple cluster centers are used as initial samples for diversification; the initial samples for diversification are then manually labeled as labeled samples for diversification. Step S1014: Construct a training dataset based on diverse labeled samples.

[0044] As can be understood from the above embodiments, taking any image data from the unlabeled image data pool as an example, the color histogram of the image data is calculated to obtain color features, the local binary pattern histogram of the image data is calculated to obtain texture features, and the orientation gradient histogram of the image data is calculated to obtain shape features. The color features, texture features, and shape features are each treated as a view. Specifically: like Figure 2 As shown, the color histogram counts the number of pixels in different intervals for each color component R, G, and B in the image. RGB is divided into 8 intervals each, so the color histogram has 512 dimensions. The color histogram values ​​for R in the i-th interval, G in the j-th interval, and B in the k-th interval are given by the following formula (13): (13) in, is the number of pixels falling within the interval, and N is the total number of pixels.

[0045] It should be noted that, Figure 2 The horizontal axis represents 512 different color ranges, and the vertical axis represents the frequency (normalized value) of the corresponding color range in the image.

[0046] The Local Binary Pattern Histogram (LBP) is calculated as follows: The image is divided into several blocks. For each pixel in a block, it is compared with its neighboring pixels. Based on the comparison results, an LBP value is generated. The LBP value is calculated as follows: Figure 3 As shown, the frequency distribution of LBP values ​​was then statistically analyzed for each block. The image has 400×400 pixels and is divided into 4×4 blocks. The LBP values ​​are generally... Then the dimension of the local binary pattern histogram is 4096, and the LBP value of each pixel is calculated using the following formula (14): (14) Where P is the number of neighboring pixels, typically 8. It is the gray value of the p-th pixel in the neighborhood. It is the grayscale value of the center pixel. For symbolic functions, It is 1 if it is greater than or equal to 0, otherwise it is 0.

[0047] The formula for calculating the grayscale value of a color image is as follows (15): (15).

[0048] The directional gradient histogram is calculated as follows: Figure 4 As shown, first, gradient calculation is performed to obtain the gradient direction of each pixel, then... Figure 5 The image is divided into several blocks, and the gradient direction is divided into several intervals. A gradient direction histogram is generated for each block. The image is divided into 10×10 blocks, and the direction is divided into 9 intervals. Then the gradient direction histogram has 900 dimensions. Each dimension counts the sum of the gradient magnitudes. The gradient direction calculation formula is as follows (16): (16) In the above formula It is the gradient component in the horizontal direction (x-axis). Represents the pixel grayscale value, that is: , In the above formula It is the gradient component in the horizontal direction (x-axis), that is: , In the above formula It is the gradient component in the horizontal direction (x-axis), that is: For the features of the above three views, feature normalization is first performed to eliminate the difference in dimensions. Then, Z-score normalization is used to compress the features of each view into the interval [-1,1]. The calculation formula is as follows (17): (17) in, It is the average value of the view features. This represents the variance of the view features.

[0049] Then, the features of the three views are concatenated to obtain a unified feature vector (i.e., multi-view features) for each unlabeled sample, as shown in equation (18): (18) in, This is a color view feature of the unlabeled sample. It is the texture view feature of the unlabeled sample. It is the shape view feature of the unlabeled sample. It is the uniform feature vector of the unlabeled sample, with a total of 5508 dimensions.

[0050] It should be noted that, Figure 5The horizontal axis divides the gradient direction (0° to 180°) into nine equal-width intervals (each interval 20°), representing different edge directions. However, in computer vision gradient features (such as HOG), the gradient direction is undirected, meaning 0° and 180° represent the same edge direction (e.g., the left and right directions of a horizontal edge are equivalent). Therefore, the last interval [160, 180) is merged with the first interval [0, 20), forming eight valid intervals. The vertical axis represents the sum of the gradient magnitudes of all pixels within the corresponding directional interval; a higher value indicates a more pronounced edge or texture in that direction.

[0051] For ease of understanding, this application embodiment takes the initial construction of the training dataset as an example. First, all unlabeled samples are added to the unlabeled image data pool U, and the training dataset L is set to an empty set.

[0052] like Figure 6 As shown, the first center point is randomly selected in the spliced ​​feature space: from An unlabeled sample is randomly selected as the first cluster center. .

[0053] like Figure 7 As shown, the subsequent center point is selected iteratively: for each unlabeled sample, the shortest distance between it and the currently selected center point in the concatenated feature space is calculated. The probability that each unlabeled sample is selected as the next center point is proportional to the square of its distance, as shown in the following formula (19): (19) in, Indicates unlabeled samples The shortest distance to the currently selected center point. It represents the probability distribution of points in space.

[0054] According to probability distribution Randomly select an unlabeled sample As the new center, join Select q centroids. Run the standard K-means algorithm: for the q initial centroids, assign each unlabeled sample to the nearest centroid, and recalculate the mean of each cluster as the new centroid. Repeat until the center point no longer changes (or the maximum number of iterations is reached). The calculation formula is as follows (20), that is: (20) like Figure 8 As shown, all of the final ones The diverse initial sample I is manually labeled and then added to the training dataset L. It is also removed from the unlabeled image data pool U, resulting in L = I and U = U \ I.

[0055] In another exemplary embodiment of this application, step S103 may include steps S1031 to S1033, specifically: Step S1031: Take the partial derivative of the weights and biases of each fully connected layer of the classifier to obtain the gradient of each fully connected layer. Step S1032: Update the weights and biases of each fully connected layer based on the gradients of each fully connected layer to obtain the current weights and biases of each connected layer; Step S1033: Calculate the marginal contribution of different view features for each sample in the training dataset based on the current weights and biases of each connection layer, and obtain the marginal contribution value of different view features.

[0056] As can be understood from the above embodiments, the gradient of each fully connected layer is obtained by taking the partial derivatives with respect to the weights and biases of each fully connected layer. and Then update all weights and biases, as shown in equation (21): (twenty one) in, The learning rate is a hyperparameter that controls the step size of parameter updates, determining the magnitude of parameter adjustments in each iteration. In the embodiments of this application, For example, a fixed value is not dynamically adjusted. =0.01.

[0057] For each view v Through this view v The marginal contribution of a sample is calculated using its Shapley value, as shown in equation (22): (twenty two) in, This represents the complete input of the i-th sample. Indicates that the i-th sample is masked by the i-th sample. v The input after each view is used to set the corresponding view data to zero, where N is the total number of samples. This represents the prediction function of the classifier, returning the classifier's certainty score for the samples. These are the trainable parameters of the classifier, namely the weights and biases of each fully connected layer.

[0058] The contribution weights of each view are normalized using the mathematical method Softmax, as shown in equation (23): (twenty three) This makes all and .

[0059] For each unlabeled sample By calculating the loss function of each view As shown in equation (24): (twenty four) Assuming the classifier predicts the sample has the highest probability of belonging to class k, in the formula... This represents the probability that the classifier predicts the sample belongs to the k-th class.

[0060] And classifier parameters Calculate the partial derivatives to obtain the gradients for each view. And aggregated according to contribution weight, as shown in the following formula (25): (25) The classifier parameters are updated using the backpropagation algorithm by aggregating gradients, thereby reducing the loss function, as shown in equation (26): (26) in This is the learning rate.

[0061] Calculate the L2 norm (Euclidean norm) of the aggregated gradient, which is the magnitude of the aggregated gradient vector, to obtain the uncertainty of the sample, as shown in equation (27): (27) A higher L2 norm of the gradient indicates a lower confidence level in the classifier's predictions and a higher degree of uncertainty.

[0062] The top 2k samples with the highest uncertainty ranking from the unlabeled image data pool U are selected to form a candidate set C, which is as follows (28): (28) In the feature space of candidate set C, the K-means++ algorithm is applied to select k cluster centers as the sample set to be labeled B; after manually labeling B, the training dataset is updated to obtain the current training dataset, which is: The current unlabeled image data pool is .

[0063] Retrain the classifier on the updated training dataset to obtain an optimized classifier. If the number of active learning rounds reaches the maximum number of rounds T, stop training and use the current classifier as the final classifier. Use the classifier to label unlabeled samples and display them to humans on the visualization platform. Otherwise, repeat the active learning loop.

[0064] In another exemplary embodiment of this application, the method further includes: counting the amount of sample data in the unlabeled image data pool; when the amount of sample data in the unlabeled image data pool is greater than a preset threshold, dividing the unlabeled image data pool to obtain multiple sub-unlabeled image data pools; then constructing a training dataset based on the unlabeled image data pools may include: constructing a training dataset based on the sub-unlabeled image data pools; the multiple sub-unlabeled image data pools are processed sequentially by a classifier.

[0065] Understandably, in order to avoid memory overflow due to an excessively large number of unlabeled image data pools and to better adapt to large-scale data scenarios, the size of the unlabeled image data pool U is first calculated. When the size of the unlabeled image data pool exceeds the threshold τ, the unlabeled image data pool U is divided into multiple sub-blocks. When the size of the unlabeled image data pool does not exceed the threshold τ, the unlabeled image data pool U is processed as a single sub-block. The above embodiment has already explained the process of processing the unlabeled image data pool U as a single sub-block, and this embodiment will not repeat it.

[0066] It should be noted that the threshold τ is dynamically adjusted according to the computing device. For example, when using a single GPU (16GB of video memory), τ=10000, and when using multiple GPUs (≥32GB), τ=50000.

[0067] The process involves multiple sub-blocks, each processed independently. A training dataset is created for each sub-block. Multiple view features are concatenated for each unlabeled sample in the training dataset to obtain multi-view features. Initial samples are selected based on these multi-view features by randomly selecting a center point for iterative clustering, resulting in multiple cluster centers. The unlabeled samples corresponding to these cluster centers are used as initial diversification samples. These initial diversification samples are then manually labeled to obtain diversified labeled samples. A training dataset is constructed based on these diversified labeled samples. Samples from the training dataset are removed from the unlabeled image data pool, updating the unlabeled image data pool. Throughout this process, multiple sub-unlabeled image data pools are processed sequentially by a single classifier.

[0068] The above embodiments have the following effective effects: First, by dynamically quantifying the marginal contribution of each view through Shapley value, and combining Softmax normalization and gradient modulation, the view weights are adaptively allocated, which effectively avoids the problem of high signal-to-noise ratio views dominating and weak key views being marginalized. This allows the classifier to make full use of the complementary information of multiple views and maintain stable performance even in complex scenarios such as missing views and noise interference.

[0069] Second, a two-stage strategy of high-uncertainty initial screening + K-means++ diversity selection is adopted to ensure that the selected samples can effectively reduce the uncertainty of the classifier and cover the global distribution of data, avoiding sample redundancy or information sparsity, so as to maximize the classifier performance with fewer labeled samples, which is especially suitable for high-cost labeling scenarios in professional fields.

[0070] Third, by flexibly splitting the ultra-large-scale unlabeled image data pool, the risk of memory overflow can be avoided, and the computational complexity can be reduced from exponential to linear, meeting the low-latency processing requirements of massive multi-view data in scenarios such as autonomous driving and industrial inspection.

[0071] Fourth, Shapley values ​​provide clear quantitative criteria for evaluating view contributions, making classifier decisions traceable and auditable; the end-to-end framework can adapt to multiple types of multi-view data such as images and sensors, and has strong adaptability to view quality fluctuations and distribution shifts, and can be widely applied in multiple fields such as medical diagnosis and intelligent recommendation.

[0072] Based on the same inventive concept, this application also provides a sample marking device for multi-view data to implement the sample marking method for multi-view data described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more sample marking device embodiments for multi-view data provided below can be found in the limitations of the sample marking method for multi-view data above, and will not be repeated here.

[0073] In one exemplary embodiment, such as Figure 9 As shown, a sample labeling device for multi-view data is provided. The sample labeling device 900 for multi-view data includes: a construction module 901, a first training module 902, a calculation module 903, an update module 904, a second training module 905, and a labeling module 906, wherein... Module 901 is used to construct a training dataset based on an unlabeled image data pool; the training dataset includes both unlabeled and labeled samples. The first training module 902 is used to train the classifier using the training dataset. During the classifier training process, the loss function is calculated based on the output of the classifier and the true label. The classifier is backpropagated based on the loss function to obtain the weights and biases of each fully connected layer in the classifier. The calculation module 903 is used to calculate the marginal contribution of different view features of each sample in the training dataset based on the weights and biases of each fully connected layer in the classifier, and obtain the marginal contribution value of different view features. The update module 904 is used to update the parameters of the classifier and the training dataset based on the marginal contribution values ​​of different view features, so as to obtain the current classifier and the current training dataset; The second training module 905 is used to train the current classifier using the current training dataset, and after training, it updates the current classifier and the current training dataset again using the current marginal contribution value until the number of updates reaches a preset number of rounds, and then obtains the trained classifier. The labeling module 906 is used to label unlabeled samples using a trained classifier to obtain labeled samples.

[0074] As an optional implementation, the above-mentioned calculation module 903 is specifically used to: calculate the partial derivative of the weights and biases of each fully connected layer of the classifier to obtain the gradient of each fully connected layer; update the weights and biases of each fully connected layer based on the gradients of each fully connected layer to obtain the current weights and current biases of each connected layer; and calculate the marginal contribution of different view features of each sample in the training dataset based on the current weights and current biases of each connected layer to obtain the marginal contribution value of different view features.

[0075] As an optional implementation, the update module 904 is specifically used to: normalize the marginal contribution values ​​of different view features to obtain the contribution weights of different view features; weight and aggregate the loss gradients of different view features based on the contribution weights of different view features to obtain the aggregated gradient; backpropagate the parameters of the classifier using the aggregated gradient to obtain the current classifier; quantify the sample uncertainty of the unlabeled image data pool based on the L2 norm of the aggregated gradient to obtain a preset number of unlabeled samples; after manual labeling, the unlabeled samples become the current diversified labeled samples; remove the current diversified labeled samples from the unlabeled image data pool and add them to the training dataset to obtain the current unlabeled image data pool and the current training dataset.

[0076] As an optional implementation, the aforementioned classifier 901 is specifically used to: extract different view features of each unlabeled sample in the unlabeled image data pool, and concatenate the different view features of each unlabeled sample to obtain a multi-view feature space corresponding to each unlabeled sample; select multiple initial centroids based on distance-weighted probability in the multi-view feature space for clustering iteration to obtain multiple cluster centroids; use the unlabeled samples corresponding to the multiple cluster centroids as diversified initial samples; after manual labeling, the diversified initial samples become diversified labeled samples; and construct a training dataset based on the diversified labeled samples.

[0077] As an optional implementation, the aforementioned classifier 901 is further specifically used to perform initial clustering in the multi-view features using the K-means++ algorithm, randomly select a center point as the first cluster center, iteratively select subsequent center points according to the probability distribution of the square of the shortest distance between the current sample and the first cluster center, until N initial cluster center points are determined; assign each sample in the unlabeled image data pool to the nearest initial cluster center point, recalculate the mean of each cluster as the new center point, update and iterate until the center points no longer change, and obtain multiple cluster center points.

[0078] As an optional implementation, the above-mentioned sample labeling device for multi-view data further includes a splitting module, which is used to count the amount of sample data in the unlabeled image data pool. When the amount of sample data in the unlabeled image data pool is greater than a preset threshold, the unlabeled image data pool is divided to obtain multiple sub-unlabeled image data pools. The above-mentioned classifier construction 901 is also specifically used to construct a training dataset based on the sub-unlabeled image data pools. The multiple sub-unlabeled image data pools are processed sequentially by a classifier.

[0079] This implementation method involves constructing a training dataset using an unlabeled image data pool. The classifier is then trained on this dataset. During training, backpropagation is performed on the classifier based on its output to obtain the weights and biases of each fully connected layer. The marginal contribution of different view features for each sample in the training dataset is calculated using these weights and biases. The marginal contribution value of each view feature is then dynamically quantified based on these values. View weights can be adaptively allocated according to these marginal contributions, effectively avoiding the problem of high signal-to-noise ratio views dominating and weak, critical views being marginalized. This allows the classifier to fully utilize complementary information from multiple view features for parameter updates, ensuring stable performance even in complex scenarios such as missing views and noise interference. This significantly improves the accuracy of image sample labeling and reduces reliance on manual labeling.

[0080] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores sample labeling data for multi-view data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a sample labeling method for multi-view data.

[0081] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0082] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0083] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0084] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0085] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0086] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A sample labeling method for multi-view data, characterized in that, The sample labeling method for multi-view data includes: A training dataset is constructed based on an unlabeled image data pool; the constructed training dataset includes unlabeled samples and labeled samples; The classifier is trained using the training dataset. During the classifier training process, a loss function is calculated based on the output of the classifier and the true label. Backpropagation is then performed on the classifier based on the loss function to obtain the weights and biases of each fully connected layer in the classifier. Based on the weights and biases of each fully connected layer in the classifier, the marginal contribution of different view features for each sample in the training dataset is calculated to obtain the marginal contribution value of different view features. The parameters of the classifier and the training dataset are updated based on the marginal contribution values ​​of different view features to obtain the current classifier and the current training dataset. The current classifier is trained using the current training dataset, and after training is completed, the current classifier and the current training dataset are updated again using the current marginal contribution value until the number of training iterations of the classifier reaches a preset number of rounds, and then a trained classifier is obtained. The trained classifier is used to label the unlabeled samples to obtain labeled samples.

2. The sample labeling method for multi-view data according to claim 1, characterized in that, The method of calculating the marginal contribution of different view features for each sample in the training dataset based on the weights and biases of each fully connected layer in the classifier, and obtaining the marginal contribution value of different view features, includes: The gradient of each fully connected layer is obtained by taking the partial derivative of the weights and biases of each fully connected layer of the classifier. The weights and biases of each fully connected layer are updated based on the gradients of each fully connected layer to obtain the current weights and biases of each connected layer. The marginal contribution of different view features for each sample in the training dataset is calculated based on the current weights and biases of each connection layer, thus obtaining the marginal contribution value of different view features.

3. The sample labeling method for multi-view data according to claim 2, characterized in that, The marginal contribution values ​​based on different view features are used to update the parameters of the classifier and the training dataset to obtain the current classifier and the current training dataset, including: The marginal contribution values ​​of different view features are normalized to obtain the contribution weights of different view features. The loss gradients of different view features are weighted and aggregated based on their contribution weights to obtain the aggregated gradient. The parameters of the classifier are backpropagated using the aggregated gradient to obtain the current classifier. The L2 norm of the aggregated gradient is used to quantify the sample uncertainty of the unlabeled image data pool to obtain a preset number of unlabeled samples; the unlabeled samples are then manually labeled as the current diversified labeled samples. The current diverse labeled samples are removed from the unlabeled image data pool and added to the training dataset to obtain the current unlabeled image data pool and the current training dataset.

4. The sample labeling method for multi-view data according to claim 1, characterized in that, The training dataset constructed based on the unlabeled image data pool includes: Extract the different view features of each unlabeled sample in the unlabeled image data pool, and stitch the different view features of each unlabeled sample together to obtain the multi-view feature space corresponding to each unlabeled sample; In the multi-view feature space, multiple initial center points are selected based on distance-weighted probability for clustering iteration to obtain multiple cluster center points; The unlabeled samples corresponding to the multiple cluster centers are used as initial diversification samples; the initial diversification samples are then manually labeled as diversified labeled samples. The training dataset is constructed based on the diverse labeled samples.

5. The sample labeling method for multi-view data according to claim 4, characterized in that, The step involves selecting multiple initial center points in the multi-view feature space based on distance-weighted probability for clustering iterations, resulting in multiple cluster center points, including: In the multi-view features, the K-means++ algorithm is used to perform initial clustering. A center point is randomly selected as the first cluster center. Subsequent center points are selected iteratively according to the probability distribution of the square of the shortest distance between the current sample and the first cluster center, until N initial cluster center points are determined. Each sample in the unlabeled image data pool is assigned to the nearest initial cluster center point, the mean of each cluster is recalculated as the new center point, and the process is repeated until the center point no longer changes, resulting in multiple cluster center points.

6. The sample labeling method for multi-view data according to claim 1, characterized in that, The method further includes: The amount of sample data in the unlabeled image data pool is counted. When the amount of sample data in the unlabeled image data pool is greater than a preset threshold, the unlabeled image data pool is divided to obtain multiple sub-unlabeled image data pools. The training dataset constructed based on the unlabeled image data pool includes: A training dataset is constructed based on the sub-unlabeled image data pool; multiple sub-unlabeled image data pools are processed sequentially by a classifier.

7. A sample labeling device for multi-view data, characterized in that, The sample labeling device for multi-view data includes: A construction module is used to build a training dataset based on an unlabeled image data pool; the training dataset includes unlabeled samples and labeled samples. The first training module is used to train the classifier using the training dataset, and during the classifier training process, calculates the loss function based on the output of the classifier and the true label, and performs backpropagation on the classifier based on the loss function to obtain the weights and biases of each fully connected layer in the classifier. The calculation module is used to calculate the marginal contribution of different view features of each sample in the training dataset based on the weights and biases of each fully connected layer in the classifier, and obtain the marginal contribution value of different view features. The update module is used to update the parameters of the classifier and the training dataset based on the marginal contribution values ​​of different view features, so as to obtain the current classifier and the current training dataset. The second training module is used to train the current classifier using the current training dataset, and after the training is completed, to update the current classifier and the current training dataset again using the current marginal contribution value, until the number of updates reaches a preset number of rounds, and then the trained classifier is obtained. The labeling module is used to label unlabeled samples using the trained classifier to obtain labeled samples.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the sample labeling method for multi-view data as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the sample labeling method for multi-view data as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the sample labeling method for multi-view data as described in any one of claims 1-6.

Images