A visual analysis system and method for weighting samples

Abstract

The present invention proposes a visual analysis system and method for weighting samples, wherein the method includes: receiving multiple verification samples and multiple training samples, wherein the first part of the samples in the multiple training samples are known to have correct category labels, the second part of the samples are known to have incorrect category labels, and the remaining third part of the samples are unknown whether their category labels are correct; generating a weighted relationship between each verification sample and each training sample based on the multiple verification samples and the multiple training samples; generating a final weight of each verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding multiple verification samples based on the first part of the samples and the second part of the samples, and the weighted relationship between each verification sample and each training sample; and displaying the multiple verification samples, the multiple training samples, the weighted relationship between each verification sample and each training sample, and the final weight of each verification sample.

Description

Technical Field

[0001] The present invention belongs to the technical field of sample visual analysis, and in particular proposes a visual analysis system and method for weighting samples. Background Art

[0002] With the advent of the big data era, data-driven machine learning methods have achieved success in many fields. A key factor in this success is the use of high-quality training samples. However, in real-world applications, training samples may contain quality issues such as incorrect class labeling and an imbalance in the number of samples across different categories, which can affect the quality of the trained machine learning models. Existing training sample weighting methods use a small number of validation samples to estimate model performance. By calculating the impact of training samples on model performance, they identify samples that degrade model performance and assign lower weights to them. In this process, the quality of validation samples directly affects both the estimated model performance and the training sample weighting results. Therefore, ensuring the quality of validation samples during the training sample weighting process is crucial to the success of training sample weighting methods. Using visualization techniques can better demonstrate the impact of validation samples on training sample weighting, helping users understand this process and identify and improve low-quality validation samples, ultimately enhancing the effectiveness of training sample weighting. Summary of the Invention

[0003] The following description includes exemplary methods, systems, techniques, and instruction sequences that embody the present invention. However, it will be appreciated that, in one or more aspects, the described invention may be practiced without these specific details. In other instances, well-known protocols, structures, and techniques are not shown in detail in order to avoid obscuring the present invention. One of ordinary skill in the art will appreciate that the described techniques and mechanisms can be applied to a variety of architectures for visualizing samples.

[0004] According to an example of one aspect of the present invention, a visual analysis system for weighting samples is proposed, comprising: a receiving module configured to receive a plurality of verification samples and a plurality of training samples, wherein a first portion of samples in the plurality of training samples are known to have correct category labels, a second portion of samples are known to have incorrect category labels, and the remaining third portion of samples are unknown to have correct category labels; a weighted relationship generation module configured to generate a weighted relationship of each verification sample to each training sample based on the plurality of verification samples and the plurality of training samples; a verification sample weight generation module configured to generate a final weight of each verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding plurality of verification samples based on the first portion of samples and the second portion of samples, and the weighted relationship of each verification sample to each training sample; and a visualization module configured to display the plurality of verification samples, the plurality of training samples, the weighted relationship of each verification sample to each training sample, and the final weight of each verification sample.

[0005] According to another aspect of the present invention, an example is proposed, comprising: receiving a plurality of verification samples and a plurality of training samples, wherein a first portion of samples in the plurality of training samples are known to have correct category labels, a second portion of samples are known to have incorrect category labels, and the remaining third portion of samples is unknown whether their category labels are correct; generating a weighted relationship between each verification sample and each training sample based on the plurality of verification samples and the plurality of training samples; generating a final weight for each verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding plurality of verification samples based on the first portion of samples and the second portion of samples, and the weighted relationship between each verification sample and each training sample; and displaying the plurality of verification samples, the plurality of training samples, the weighted relationship between each verification sample and each training sample, and the final weight of each verification sample.

[0006] According to yet another aspect of the present invention, an example provides a computer-readable storage medium for visualizing a sample. The computer-readable storage medium has program instructions stored therein. The program instructions can be executed by a computing device to enable the computing device to perform the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The invention itself, as well as the mode of use, objects, features and advantages of its preferred embodiments, may be better understood by reading the following detailed description of illustrative embodiments with reference to the accompanying drawings in which:

[0008] Figure 1 A shows an example of weighting training images in a classification task;

[0009] Figure 1 B shows an example of obtaining better training image weighting results by reducing the weight of the validation image;

[0010] Figure 2 FIG2 shows a structural block diagram of a visual analysis system for weighting samples according to an embodiment of the present invention;

[0011] Figure 3 A flowchart of a method for generating a weighted relationship between a validation sample and a training sample according to one or more embodiments of the present invention is shown;

[0012] Figure 4 A flowchart of a method for generating a final weight for each validation sample according to one or more embodiments of the present invention is shown;

[0013] Figure 5 An interface of a visualization module of a visual analysis system for weighting samples according to an embodiment of the present invention is shown;

[0014] Figure 6 A flow chart of a visual analysis method for weighting samples is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present invention are described below with reference to the accompanying drawings. In the following description, many specific details are set forth in order to provide a more comprehensive understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be implemented without some of these specific details. In addition, it should be understood that the present invention is not limited to the specific embodiments described. On the contrary, any combination of the following features and elements may be considered to implement the present invention, regardless of whether they relate to different embodiments. Furthermore, the specific steps of the methods of the different embodiments are not in a strict order, that is, when a method includes a first step and a second step, the first step may be performed before the second step, or the second step may be performed before the first step. Therefore, the following aspects, features, embodiments and advantages are for illustrative purposes only and should not be regarded as elements or limitations of the appended claims unless expressly set forth in the claims.

[0016] As mentioned earlier, with the advent of the big data era, data-driven machine learning methods have achieved success in many fields. A key factor in this success is the use of high-quality training samples. However, in real-world applications, training samples may contain quality issues such as incorrect category labeling and an imbalance in the number of training samples across different categories, which can affect the quality of the trained machine learning models. Therefore, existing techniques use training sample weighting to address these issues. Existing training sample weighting methods use a small number of validation samples to estimate model performance. By calculating the impact of training samples on model performance, they identify training samples that degrade model performance and assign lower weights. This allows for the use of training samples with different weights to produce a higher-quality model. In this process, the quality of the validation samples directly influences the results of the training sample weighting. However, existing sample weighting methods treat all validation samples as a whole to consider their impact on the training sample weighting results. This method fails to identify and mitigate the impact of low-quality validation samples, resulting in poor training sample weighting and, consequently, a low-quality model.

[0017] In the present invention, the influence of each verification sample on the weighted result of each training sample can be considered in the process of training sample weighting, and the weight of the verification sample can be automatically adjusted to obtain better training sample weighted results, and a higher quality model can be obtained. The weighted situation of the sample can be well displayed through visualization technology, and interaction with the user can be performed to further improve the sample weighted results and improve the model quality.

[0018] For example, Figure 1Figure A shows an example of weighting training images in a classification task, where there are three verification images 101, 102, and 103 and four training images 104, 105, 106, and 107. The numbers on the lines connecting the verification and training images represent the weighted relationship between the corresponding verification image and the training image. Subsequently, for each training image, the sum of the weighted relationships of all verification images with respect to it is calculated to obtain the weighted result for that training image. For example, the sub-weight of verification image 101 with respect to training image 105 is 1, the sub-weight of verification image 102 with respect to training image 105 is -2, and the sub-weight of verification image 103 with respect to training image 105 is 0. If the weights of verification images 101, 102, and 103 are all 1, then the weight of training image 105 is calculated as 1*1+1*(-2)+1*0=-1. The present invention finds that when the category labels of some verification images are incorrect, an incorrect weighted relationship between the verification images and the training images may be obtained, which in turn causes the weighting method to give incorrect training image weights, that is, a lower weight is given to the training images with correct known category labels, such as the training image 105 with correct category labeling, which obtains a weight of -1; a higher weight is given to the images with incorrect known category labels, such as the training image 106 with incorrect category labeling, which obtains a weight of 1; ultimately resulting in a decrease in the performance of the model trained using incorrect weights.

[0019] Figure 1 B shows an example of a better training image weighting result obtained by reducing the weight of the verification image 102. Figure 1 In Figure 2, training images 105 and 107, which are correctly labeled with known categories, receive a higher weight of 1, while training image 106, which is incorrectly labeled with known categories, receives a lower weight of -1. This ultimately leads to improved performance of the model trained with the correct weights. Those skilled in the art will appreciate that the three validation images and four training images described above are merely exemplary, and that any number of validation images and training images can be used in practical applications.

[0020] The above describes the sample weighting method using images as an example. In fact, those skilled in the art will appreciate that the sample weighting method is not only applicable to image samples, but also to video samples, text samples, table samples, or user-specified samples, etc.

[0021] Figure 2 FIG. 2 shows a block diagram of a visual analysis system 200 for weighting samples according to an embodiment of the present invention. Figure 2 As shown, the system 200 includes a receiving module 210 , a weighted relationship generating module 220 , a verification sample weight generating module 230 , and a visualization module 250 .

[0022] exist Figure 2In the embodiment, the receiving module 210 is configured to receive a plurality of verification samples 201 and a plurality of training samples 202, wherein a first portion of samples in the plurality of training samples 202 are known to have correct category labels, a second portion of samples in the plurality of training samples 202 are known to have incorrect category labels, and a remaining third portion of samples in the plurality of training samples 202 are unknown to have correct category labels.

[0023] The weighted relationship generating module 220 is configured to generate a weighted relationship 203 between the validation samples and the training samples based on the multiple validation samples 201 and the multiple training samples 202 .

[0024] The verification sample weight generation module 230 is configured to generate a final weight 204 of the verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding multiple verification samples based on the first part of samples and the second part of samples in the above-mentioned multiple training samples 202 and the weighted relationship 203 of each verification sample to each training sample.

[0025] The visualization module 250 is configured to display a plurality of validation samples 201 , a plurality of training samples 202 , a weighted relationship 203 between each validation sample and each training sample, and a final weight 204 of each validation sample 201 .

[0026] In one embodiment, the visual analysis system 200 further includes a training sample weight generation module 240. The training sample weight generation module 240 is configured to generate a weight 205 for each training sample based on the weighted relationship 203 between each validation sample and each training sample, and the final weight 204 of the validation sample. The visualization module 250 is further configured to display the weight 205 of each training sample.

[0027] The visual analysis system 200 for weighting samples can be implemented as an application on a general computer system, or as an application on a server system, or as a network application, or as an application on a cloud platform.

[0028] In one embodiment, the receiving module 210 may be configured to receive multiple validation samples 201 and multiple training samples 202 uploaded by a user through a graphical user interface or user commands. The receiving module 210 may also be configured to store the multiple validation samples 201 and multiple training samples 202 somewhere for reading by the receiving module 210. The receiving module 210 may also be configured to obtain the multiple validation samples 201 and multiple training samples 202 via network transmission, etc.

[0029] In one embodiment, the weighted relationship generating module 220 may be configured to calculate a weighted relationship between each validation sample and each training sample. Figure 3 FIG. 3 is a flow chart showing a method 300 for generating a weighted relationship between each validation sample and each training sample according to one or more embodiments of the present invention. Figure 3 In step 310, the initial model is trained using multiple training samples to obtain a trained model. The initial model can be uploaded by the user, built into the system, or obtained from another system via the network, among other methods. This training can utilize any existing artificial intelligence training method, which will not be described in detail here. In step 320, for each of the multiple validation samples, the performance metrics of the trained model for that validation sample are calculated, such as cross entropy, mean squared error, mean absolute error, etc. The calculation of these performance metrics is well known to those skilled in the art. In step 330, the derivative of the performance metrics of the trained model for that validation sample with respect to each training sample is calculated, thereby obtaining a weighted relationship between the validation sample and each training sample. Calculating the derivative of the performance metrics of the trained model for that validation sample with respect to each training sample can be performed using existing techniques. The inverse of the obtained derivative for each training sample is the sub-weight for each training sample, which indicates the weighted relationship between the validation sample and each training sample. In step 340, a determination is made as to whether any validation samples remain to be calculated. If yes, return to step 320 and continue to calculate the weighted relationship between other verification samples and each training sample. Otherwise, return to the weighted relationship 203 of each verification sample to the training sample and end this process.

[0030] Those skilled in the art will appreciate that the aforementioned use of the inverse of the derivative of each training sample as a sub-weight for each training sample, thereby obtaining a weighted relationship between the validation sample and each training sample, is merely an exemplary embodiment. Other transformations of the derivative of each training sample may also be used, such as normalizing after taking the inverse, performing a tangent transformation after taking the inverse, etc. Furthermore, the stationary point of each training sample may also be used, and other transformations of the stationary point may be used, such as adding random perturbations, adding scaling factors, etc.

[0031] In one embodiment, the verification sample weight generation module 230 can be configured to generate a final weight 204 of the verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding multiple verification samples based on the first part of samples and the second part of samples in the above-mentioned multiple training samples 202 and the weighted relationship 203 of each verification sample to each training sample. Figure 4 FIG. 4 is a flow chart showing a method 400 for generating the final weight 204 for each validation sample according to one or more embodiments of the present invention. Figure 4In step 410, the current weight of each validation sample is set. The weights are typically initialized to 1. Users can also set different weights. In step 420, the correctness of the multiple validation samples is calculated based on the first and second sample portions, the weighted relationship 203 between each validation sample and each training sample, and the current weight of each validation sample. Correctness indicates the accuracy of the training sample weights generated using the multiple validation samples; a higher accuracy is better. In step 430, the balance of the multiple validation samples is calculated based on the first sample portion, the weighted relationship between each validation sample and each training sample, and the current weight of each validation sample. Balance indicates the degree of balance between the weights of the training samples generated using the multiple validation samples across different categories; a higher accuracy is better. In step 440, it is determined whether the calculated correctness and balance meet predetermined conditions. These predetermined conditions can include ensuring that the calculated correctness and balance are as high as possible, exceeding a first threshold, or that the difference between the correctness and balance calculated in the current iteration and the correctness and balance calculated in the previous iteration is within a second threshold, etc. If the predetermined condition is met, in step 450, the current weight of each validation sample is returned as the final weight 204, and the process ends. Otherwise, in step 460, the current weight of each validation sample is updated, and then the process returns to step 420 to continue the iterative process.

[0032] In one embodiment, the correctness of the verification sample can be indicated by the weight of the first part of samples (training samples with correct category labels) and the second part of samples (training samples with incorrect category labels). The correctness of the verification sample can be calculated using the following formula (1):

[0033]

[0034] where s j represents the training sample, w j Indicates the use of the weight corresponding to the current verification sample, S + Represents the first part of the sample, S - represents the second part of the sample. φ(x)=1 / (1+e -x ) is the sigmoid function, and its value falls within (0,1). For the first part of the samples, since their categories are correctly labeled, their weights are expected to be higher. At this time, φ(w j ) is close to 1, the accuracy of the verification sample is higher; for the second part of the samples, since their category labels are incorrect, their weights are expected to be lower. At this time, φ(w j ) is close to 0, and the accuracy of the verification sample is higher.

[0035] In another embodiment, the correctness of the verification sample can be calculated using the following formula (2):

[0036]

[0037] The meaning of the parameters in formula (2) is the same as that in formula (1). For the first part of samples, since their categories are correctly labeled, their weights are expected to be higher. At this time, φ(w j ) is close to 1, and the accuracy of the verification sample is higher; for the second part of the samples, since their category labels are incorrect, their weights are expected to be lower. At this time, 1-φ(w j ) is close to 1, the accuracy of the verification sample is higher.

[0038] In another embodiment, the correctness of the verification sample can be calculated using the following formula (3):

[0039]

[0040] The meaning of the parameters in formula (3) is the same as that in formula (1). In addition, max(a,b) represents the larger value of a and b. For the first part of samples, since their categories are correctly labeled, their weight is expected to be higher. In this case, w j The larger the value, the higher the accuracy of the verification sample. For the second part of the samples, since their category labels are incorrect, their weights are expected to be lower. In this case, -w j The larger the size, the higher the accuracy of the validation sample.

[0041] The larger the correctness values ​​of the multiple verification samples calculated by the above formulas (1), (2), and (3), the better the correctness. Inspired by the above methods, those skilled in the art can also develop other methods for calculating the correctness of verification samples.

[0042] In one embodiment, the balance of the validation sample can be indicated by the proportion of the weights of the training samples of different categories in the first part of the sample. The balance of the validation sample can be calculated using the following formula (4):

[0043]

[0044] Where c is a specific category among all the samples, C is the total number of categories of all the samples, and p c is the proportion of the sample weight of category c in the first part of the sample, that is, in is the sample set of category c in the first part of the samples.

[0045] In another embodiment, the balance of the verification sample can be calculated using the following formula (5):

[0046]

[0047] The meaning of the parameters in formula (5) is the same as that in formula (4). It measures the proportion of sample weights of each category and the expected proportion The absolute deviation.

[0048] In another embodiment, the balance of the verification sample can be calculated using the following formula (6):

[0049]

[0050] The meaning of the parameters in formula (6) is the same as that in formula (4). It measures the proportion of sample weights of each category and the expected proportion The squared deviation.

[0051] The larger the balance value of the multiple verification samples calculated by the above formulas (4), (5), and (6), the better the balance. Inspired by the above methods, those skilled in the art can also develop other methods for calculating the balance of verification samples.

[0052] The verification sample weight generation module 230 expects to generate verification sample weights that satisfy the following requirements: the correctness and balance of its verification samples are both high. Therefore, the optimization objective can be defined as the sum of two items, where the first item is the correctness of the verification sample and the second item is the balance of the verification sample. Those skilled in the art will know that summation is only a specific implementation method, and other functional relationships can also be used, such as weighted summation, performing a logarithmic transformation before summing, etc. The verification sample weight is then obtained by maximizing the optimization objective. Those skilled in the art will know that the method for maximizing the optimization objective used in the verification sample weight generation module 230 can be any existing optimization solution method (such as a search method, a gradient descent method, an interior point method, etc.) or any optimization solution method developed in the future. Therefore, the predetermined condition can be set to the correctness and balance of the calculation being as large as possible; or the correctness and balance of the calculation being greater than a first threshold; or the difference between the correctness and balance of the current iterative calculation and the correctness and balance of the previous iterative calculation is within a second threshold range. Here, the first threshold and the second threshold can be a vector respectively. For example, the first threshold vector contains two elements, namely the first threshold for correctness and the first threshold for balance; the second threshold vector contains two elements, namely the second threshold for correctness and the second threshold for balance. The first threshold and the second threshold can also be a numerical value respectively, indicating the threshold of the calculated sum of correctness and balance (including weighted sum, sum of various transformations, etc.).

[0053] In one embodiment, the visualization module 250 may be configured to present multiple validation samples 201 , multiple training samples 202 , the relationship 203 between validation samples and training samples, and the weight 204 of the validation samples in a visual form. Figure 5The interface of a visualization module 250 of a visual analysis system for weighting samples according to an embodiment of the present invention is shown. In one embodiment, a dimensionality reduction algorithm (e.g., a tSNE algorithm) can be used to perform one-dimensional dimensionality reduction on multiple validation samples 201, various graphics (e.g., circles, rectangles, etc.) can be used to represent samples, and the properties of the graphics (e.g., the x-coordinate and y-coordinate of the graphics) can be used to represent the results of the validation sample projection and the weights 204 of the validation samples, thereby converting the multiple validation samples 201 into two-dimensional scatter plots. A dimensionality reduction algorithm (e.g., a tSNE algorithm) can be used to perform one-dimensional dimensionality reduction on multiple training samples 202, various graphics (e.g., circles, rectangles, etc.) can be used to represent samples, and the properties of the graphics (e.g., the x-coordinate and y-coordinate of the graphics) can be used to represent the results of the training sample projection, thereby converting the multiple training samples 202 into two-dimensional scatter plots. Edges can be used to show the weighted relationship between multiple validation samples and multiple training samples. Figure 5 Box 510 in the figure shows an example of using the tSNE algorithm to reduce the dimensionality of multiple validation samples 201, using squares to represent validation samples, using the x-coordinates of the squares to represent the dimensionality reduction results, and using the y-coordinates of the squares to represent the validation sample weights 204, thereby converting the multiple validation samples 201 into a two-dimensional scatter plot. Figure 5 Box 530 in FIG. 5 shows an example of using the tSNE algorithm to reduce the dimensionality of the multiple training samples 202 , using circles to represent the training samples, and using the x-coordinates and y-coordinates of the circles to represent the dimensionality reduction results, thereby converting the multiple training samples 202 into a two-dimensional scatter plot. Figure 5 Box 520 in the figure shows an example of using edges to show the weighted relationship of multiple verification samples to multiple training samples, where the color of the edge indicates the positive or negative of the weighted relationship, and the thickness of the edge indicates the absolute value of the weighted relationship. Those skilled in the art will know that when performing dimensionality reduction on multiple verification samples 201 or training samples 202, MDS algorithm, UMAP algorithm, Isomap algorithm, etc. can also be used. These dimensionality reduction algorithms are well known to those skilled in the art and will not be described in detail here. Those skilled in the art will know that when converting the projection results of multiple verification samples 201 and multiple training samples 202 into a two-dimensional scatter plot, other graphics such as triangles and diamonds can also be used to represent the samples. These visualization methods are well known to those skilled in the art and will not be described in detail here.

[0054] The user can adjust the weight of a portion of the multiple validation samples 201 through the visualization module 250, thereby modifying the validation sample weight 204 generated by the validation sample weight generation module 230. The user can annotate a portion of the multiple training samples 202, thereby adjusting the samples included in the first and second portions of samples. The validation sample weight generation module 230 generates an adjusted validation sample final weight based on the adjusted validation sample weight and the adjusted first and second portions of samples, and generates an adjusted training sample weight based on the adjusted validation sample final weight. Through these visualization operations, a more effective training sample weighting result can be achieved.

[0055] In one embodiment, the system 200 further includes a training sample weight generation module 240, which can generate the training sample weight 205 by performing a weighted average on the weighted relationship 203 between the verification sample and the training sample using the verification sample weight 204 to obtain the training sample weight 205. For example, Figure 1 In A, the sub-weight of verification image 101 to training image 105 is 1, the sub-weight of verification image 102 to training image 105 is -2, and the sub-weight of verification image 103 to training image 105 is 0; the weights of verification images 101, 102, and 103 are all 1; then the weight of training image 105 is calculated as 1*1+1*(-2)+1*0=-1. After adjustment, as Figure 1 As shown in B, the weight of the verification image 102 is updated to 0; then the weight of the training image 105 is updated to 1*1+0*(-2)+1*0=1.

[0056] Through system 200, the weighted relationship between each validation sample and each training sample can be obtained, thereby identifying poor-quality validation samples at a finer granularity and making adjustments to improve the training sample weighting results. By calculating the correctness and balance of the validation samples, an optimization algorithm can be used to automatically adjust the validation sample weights, improving the training sample weighting results. Through the visualization module, users can gain a deeper understanding of the training sample weighting process and more easily improve the training sample weighting results by providing feedback, such as adjusting the validation sample weights or adjusting the samples included in the first or second sample parts.

[0057] The present invention also proposes a visual analysis method for weighting samples. Figure 6 FIG. 6 is a flow chart showing a visual analysis method 600 for weighting samples. Figure 6As shown, in step 610, a plurality of verification samples and a plurality of training samples are received, wherein the first part of the samples in the plurality of training samples are known to have correct category labels, the second part of the samples are known to have incorrect category labels, and the remaining third part of the samples are unknown whether their category labels are correct. In step 620, a weighted relationship of each verification sample to each training sample is generated based on the plurality of verification samples and the plurality of training samples. In step 630, based on the first part of the samples and the second part of the samples, and the weighted relationship of each verification sample to each training sample, the final weight of each verification sample is generated by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding plurality of verification samples. In step 650, the plurality of verification samples, the plurality of training samples, the weighted relationship of each verification sample to each training sample, and the final weight of each verification sample are displayed.

[0058] In one embodiment, the method 600 further includes: generating the weight of each training sample according to the weighted relationship between each validation sample and each training sample and the final weight of each validation sample in step 640; and further including displaying the weight of each training sample in step 650. In one embodiment, the method 600 may be implemented in step 630. Figure 4 The method steps shown, in which the calculation method of correctness and balance has been introduced above, will not be repeated here. The predetermined condition is one of the following: (1) the correctness and balance of the calculation are as large as possible; (2) the correctness and balance of the calculation are greater than the first threshold; and (3) the difference between the correctness and balance of the current iterative calculation and the correctness and balance of the previous iterative calculation is within the second threshold range. Those skilled in the art will know that other predetermined conditions can also be set based on the above predetermined conditions. As long as the basic principles described above are met, they are within the scope of protection of the present invention. In addition, those skilled in the art will know that step 630 can also be adopted Figure 4 Variations of the steps shown, such as steps 420 and 430 may be swapped in order, using different correctness and balance calculation methods, different predetermined conditions, using different looping methods, etc., are all within the scope of protection of the present invention.

[0059] In one embodiment, step 620 of method 600 may be performed using Figure 3 Those skilled in the art will appreciate that step 620 may also be performed using Figure 3 Variations of the steps shown, such as using different models, different performance indicators, using mathematical forms other than derivatives, using different looping methods, etc., are all within the scope of protection of the present invention.

[0060] In one embodiment, step 650 of method 600 also includes reducing the dimensions of the multiple validation samples and the multiple training samples respectively by using a dimensionality reduction algorithm, converting the dimensionality reduction results of the multiple samples into a two-dimensional scatter plot, wherein the samples are represented graphically; receiving input to adjust the samples included in the first part of samples and the second part of samples to form an adjusted final weight of the validation samples; and generating further adjusted training sample weights based on the adjusted final weight of the validation samples.

[0061] The present invention may be implemented as a system, method, and / or computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for causing a processor to implement various aspects of the present invention. The method of the present invention may be executed on a standalone computer system, a distributed computing system, or even a cloud platform.

[0062] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer-readable storage media according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0063] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer-readable storage medium according to multiple embodiments of the present invention.In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and the part for the module, program segment or instruction comprises one or more executable instructions for realizing the logical function of regulation.In some alternative implementations, the function marked in the box can also occur in a sequence different from that marked in the accompanying drawings.For example, two continuous boxes can actually be performed substantially in parallel, and they can also be performed in reverse order sometimes, depending on the function involved.

[0064] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual analysis system for weighting samples, the system comprising: A receiving module is configured to receive a plurality of validation samples and a plurality of training samples, wherein a first portion of the samples in the plurality of training samples are known to have correct category labels, a second portion of the samples are known to have incorrect category labels, and a remaining third portion of the samples are unknown to have correct category labels; a weighted relationship generating module, configured to generate a weighted relationship between each verification sample and each training sample based on the multiple verification samples and the multiple training samples; a verification sample weight generation module, configured to generate a final weight for each verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding multiple verification samples based on the first part of samples and the second part of samples, and the weighted relationship between each verification sample and each training sample; as well as a visualization module configured to display the plurality of validation samples, the plurality of training samples, a weighted relationship between each validation sample and each training sample, and a final weight of each validation sample; The system further comprises: a training sample weight generation module, configured to generate a weight of each training sample based on a weighted relationship between each validation sample and each training sample, and a final weight of each validation sample, wherein the visualization module is further configured to display the weight of each training sample; The verification sample weight generation module generates the final weight of each verification sample by iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding multiple verification samples, including: Set the current weight of each validation sample; Iterate and perform the following steps: Calculating the correctness of the multiple verification samples based on the first part of samples and the second part of samples, a weighted relationship between each verification sample and each training sample, and a current weight of each verification sample; Calculating the balance of the multiple validation samples based on the first part of samples, the weighted relationship between each validation sample and each training sample, and the current weight of each validation sample; Determine whether the correctness and balance of the calculations meet the predetermined conditions; In response to the correctness and balance of the calculation not satisfying the predetermined conditions, updating the current weight of each verification sample and restarting the next iterative execution; and In response to the correctness and balance of the calculation satisfying the predetermined conditions, the current weight of each verification sample at this time is the final weight of each verification sample, and the iterative execution is stopped; The step of generating a weighted relationship between each validation sample and each training sample includes: Using the multiple training samples to train the initial model to obtain a trained model; For each validation sample in the plurality of validation samples: Calculate the performance index of the trained model under the validation sample; Calculate the derivative of the performance index under the verification sample with respect to each training sample to obtain the weighted relationship between the verification sample and each training sample; wherein the visualization module is further configured to use a dimensionality reduction algorithm to perform dimensionality reduction on the multiple validation samples and the multiple training samples, respectively, and convert the dimensionality reduction results of the samples into a two-dimensional scatter plot, wherein the samples are represented graphically; wherein the visualization module is further configured to receive input to adjust the samples included in the first part of samples and the second part of samples to form adjusted final weights of validation samples, and wherein the training sample weight generation module is further configured to generate adjusted training sample weights according to the adjusted final weights of validation samples; The sample type includes one of the following: Image data; Video data; Text data.

2. The system according to claim 1, wherein the predetermined condition is one of the following: The correctness and balance of calculations are as great as possible; The correctness and balance of the calculation are greater than a first threshold; and A difference between the correctness and balance of the current iterative calculation and the correctness and balance of the previous iterative calculation is within a second threshold range.

3. The system of claim 1 , wherein the correctness of the plurality of validation samples is calculated using one of the following: Correctness of multiple validation samples ; Correctness of multiple validation samples ; Correctness of multiple validation samples ; in represents the training sample, Indicates that the validation sample weights in this iteration are used to generate The weight of represents the first part of the sample, represents the second part of the sample, is monotonically increasing and has a range of The sigmoid function, express The larger value in .

4. The system of claim 1 , wherein the balance of the plurality of validation samples is calculated using one of the following: Balance of multiple validation samples ; Balance of multiple validation samples ; Balance of multiple validation samples ; in is a specific category among all samples, is the total number of categories of all samples, For the first part of the sample The proportion of sample weights of the category, that is ,in For the first part of the sample The sample set of the category.

5. A visual analysis method for weighting a sample, comprising: Receiving a plurality of validation samples and a plurality of training samples, wherein a first portion of the samples in the plurality of training samples are known to have correct category labels, a second portion of the samples are known to have incorrect category labels, and a remaining third portion of the samples are unknown to have correct category labels; Generating a weighted relationship between each verification sample and each training sample according to the multiple verification samples and the multiple training samples; According to the first part of samples and the second part of samples, and the weighted relationship between each verification sample and each training sample, iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding multiple verification samples, thereby generating a final weight for each verification sample; as well as Displaying the multiple validation samples, the multiple training samples, the weighted relationship between each validation sample and each training sample, and the final weight of each validation sample; Also includes: Generate the weight of each training sample according to the weighted relationship between each verification sample and each training sample, and the final weight of each verification sample, wherein displaying the multiple verification samples, the multiple training samples, the weighted relationship between each verification sample and each training sample, and the final weight of each verification sample further includes displaying the weight of each training sample; The step of iteratively updating the weight of each verification sample and calculating the correctness and balance of the corresponding verification samples to generate the final weight of each verification sample includes: Set the current weight of each validation sample; Iterate and perform the following steps: Calculating the correctness of the multiple verification samples based on the first part of samples and the second part of samples, a weighted relationship between each verification sample and each training sample, and a current weight of each verification sample; Calculating the balance of the multiple validation samples based on the first part of samples, the weighted relationship between each validation sample and each training sample, and the current weight of each validation sample; Determine whether the correctness and balance of the calculations meet the predetermined conditions; In response to the correctness and balance of the calculation not satisfying the predetermined conditions, updating the current weight of each verification sample and restarting the next iterative execution; and In response to the correctness and balance of the calculation meeting the predetermined conditions, the current weight of each verification sample is the final weight of each verification sample, and the iterative execution is stopped; The step of generating a weighted relationship between each validation sample and each training sample includes: Using the multiple training samples to train the initial model to obtain a trained model; For each validation sample in the plurality of validation samples: Calculate the performance index of the trained model under the validation sample; Calculate the derivative of the performance index under the verification sample with respect to each training sample to obtain the weighted relationship between the verification sample and each training sample; The displaying of the plurality of validation samples, the plurality of training samples, the weighted relationship of each validation sample to each training sample, and the final weight of each validation sample further comprises: Performing dimensionality reduction on the plurality of validation samples and the plurality of training samples respectively by using a dimensionality reduction algorithm, and converting the dimensionality reduction results of the samples into a two-dimensional scatter plot, wherein the samples are represented graphically; receiving input to adjust samples included in the first portion of samples and the second portion of samples to form adjusted validation sample final weights; and generating an adjusted training sample weight according to the adjusted validation sample final weight; The sample type includes one of the following: Image data; Video data; Text data.

6. The method according to claim 5, wherein the predetermined condition is one of the following: The correctness and balance of calculations are as great as possible; The correctness and balance of the calculation are greater than a first threshold; and A difference between the correctness and balance of the current iterative calculation and the correctness and balance of the previous iterative calculation is within a second threshold range.

7. The method according to claim 5, wherein the correctness of the plurality of verification samples is calculated using one of the following: Correctness of multiple validation samples ; Correctness of multiple validation samples ; Correctness of multiple validation samples ; in represents the training sample, Indicates that the validation sample weights in this iteration are used to generate The weight of represents the first part of the sample, represents the second part of the sample, is monotonically increasing and has a range of The sigmoid function, express The larger value in .

8. The method of claim 5, wherein the balance of the plurality of validation samples is calculated using one of the following: Balance of multiple validation samples ; Balance of multiple validation samples ; Balance of multiple validation samples ; in is a specific category among all samples, is the total number of categories of all samples, For the first part of the sample The proportion of sample weights of the category, that is ,in For the first part of the sample The sample set of the category.

9. A computer-readable storage medium having program instructions stored therein, wherein the program instructions can be executed by a computing device to cause the computing device to perform the method according to any one of claims 5 to 8.

10. A computer system comprising: Memory; as well as At least one processor is operatively coupled to the memory and configured to execute the method according to any one of claims 5-8.

Images