Methods, systems, and readable storage media for visualizing a sample

Abstract

The application discloses a method, a system and a readable storage medium for visualizing samples. The system comprises: a receiving module for receiving a plurality of samples, wherein the information of each sample comprises a feature vector corresponding to the sample, a clustering category of the sample and a labeled category; a projection module for obtaining the position of the projection point of each sample in a two-dimensional plane according to the feature of each sample; a grid generation module for obtaining the corresponding grid of each sample in the two-dimensional plane according to the position of the projection point of each sample; a grid adjustment module for iteratively adjusting the position of the grid corresponding to the sample, thereby obtaining the updated position of the grid corresponding to the sample; and a visualization module configured to visualize and display the plurality of samples in the two-dimensional plane according to the grid corresponding to each sample and the updated position of the grid.

Description

Technical Field

[0001] This invention relates to the field of data visualization technology, and in particular to methods, systems and readable storage media for visualizing samples. Background Technology

[0002] Data visualization can present abstract data in an intuitive way, helping users understand patterns in the data and discover potential problems, thereby improving the efficiency of data analysis. For example, by mapping high-dimensional data to a low-dimensional space and displaying it as a scatter plot, users can intuitively observe the distribution of the data, understand the classification of the data, and more easily find misclassified points in the data, which are called outliers. However, scatter plots suffer from problems such as over-plotting and overlapping visual labels, which are not conducive to users' observation and analysis of samples of interest. Grid layout is a commonly used display form in data visualization, with advantages such as simple and easy-to-understand visual encoding, beautiful layout, and high space utilization. In a grid layout, each sample is represented by a grid, and the color of the grid is usually used to indicate the category of its corresponding sample (cluster category or label category, etc.). These grids are arranged closely together. Since there is no overlap between different grids in a grid layout, it can be used to conveniently perform tasks that require exploring samples, such as finding samples that have been incorrectly predicted by artificial intelligence models. Summary of the Invention

[0003] The following description includes exemplary methods, systems, techniques, and sequences of instructions embodying the techniques of this invention. However, it should be understood that the described invention can be practiced in one or more aspects without these specific details. In other instances, well-known protocols, structures, and techniques have not been shown in detail so as not to obscure the invention. Those skilled in the art will understand that the described techniques and mechanisms can be applied to various architectures for visualizing samples.

[0004] According to one aspect of the present invention, a system for visualizing samples is proposed, comprising: a receiving module configured to receive a plurality of samples, wherein information of each sample includes a feature vector corresponding to the sample, a clustering category of the sample, and a labeling category; a projection module configured to obtain the position of a projection point of the plurality of samples on a two-dimensional plane based on the feature vector of each sample; a mesh generation module configured to obtain a mesh corresponding to the plurality of samples on the two-dimensional plane based on the position of each sample and its projection point, wherein the position of the mesh corresponding to a specific sample is related to the position of the projection point of that specific sample; a mesh adjustment module configured to iteratively adjust the position of the mesh corresponding to each sample based on the clustering category, the position of the corresponding projection point, and the position of the corresponding mesh of each sample, such that predetermined conditions are met, thereby obtaining the updated position of the mesh corresponding to the plurality of samples; and a visualization module configured to visualize the plurality of samples on the two-dimensional plane based on the mesh corresponding to each sample and the updated position of the mesh.

[0005] According to another aspect of the present invention, a method for visualizing samples is proposed, comprising: receiving a plurality of samples, wherein information of each sample includes a feature vector corresponding to the sample, a clustering category of the sample, and a labeling category; obtaining the position of a projection point of the plurality of samples on a two-dimensional plane based on the feature vector of each sample; obtaining a grid corresponding to the plurality of samples on the two-dimensional plane based on the position of each sample and its projection point, wherein the position of the grid corresponding to a specific sample is related to the position of the projection point of that specific sample; iteratively adjusting the position of the grid corresponding to each sample based on the clustering category, the corresponding projection point position, and the corresponding grid position of each sample to satisfy predetermined conditions, thereby obtaining the updated position of the grid corresponding to the plurality of samples; and visualizing the plurality of samples on the two-dimensional plane based on the updated grid corresponding to each sample.

[0006] According to another aspect of the invention, a computer-readable storage medium for visualizing samples is provided, the computer-readable storage medium having program instructions stored therein, the program instructions being executable by a computing device to cause the computing device to perform the method described above. Attached Figure Description

[0007] The invention itself, its preferred modes of use, objectives, features, and advantages can be better understood by referring to the following detailed description of illustrative embodiments, in which:

[0008] Figure 1A Nine samples from a digital image dataset are shown.

[0009] Figure 1B It shows Figure 1A The results were obtained by visualizing the nine samples in the image using a grid layout.

[0010] Figure 2 The visualization results of grid layout for several image samples using existing methods are shown;

[0011] Figure 3 A structural block diagram of a system for visualizing samples according to one or more embodiments of the present invention is shown;

[0012] Figures 4A-4C The projection points corresponding to the nine samples projected onto the two-dimensional plane and the generated mesh corresponding to the samples are shown.

[0013] Figure 5 A flowchart is shown for a method for adjusting the position of a grid corresponding to each of a plurality of samples according to one or more embodiments of the present invention;

[0014] Figures 6A-6B It shows the use of Figure 5 The method shown is the process of adjusting the grid position of the sample shown in Figure 4;

[0015] Figure 7 A flowchart is shown illustrating another method for adjusting the position of a grid corresponding to each of a plurality of samples according to one or more embodiments of the present invention;

[0016] Figures 8A-8E It shows the use of Figure 7 The method shown is the process of adjusting the grid position of the sample shown in Figure 4;

[0017] Figure 9 A method for visualizing samples according to one or more embodiments of the present invention is shown;

[0018] Figure 10A The diagram shows an interface for visualizing an image dataset with clustering (AI model prediction) category errors using an existing method (i.e., skipping the grid adjustment module 340) in a visualization system developed based on the present invention.

[0019] Figure 10B This illustrates the execution of the grid adjustment module 340 in a visualization system developed based on the present invention. Figure 5 The method shown provides an interface for visualizing an image dataset with clustering (predicted by an artificial intelligence model) category errors;

[0020] Figure 10C This illustrates the execution of the grid adjustment module 340 in a visualization system developed based on the present invention. Figure 7 The method shown provides an interface for visualizing an image dataset with clustering (predicted by an artificial intelligence model) category errors;

[0021] Figure 10D This illustrates the sequential execution of the grid adjustment module 340 in the visualization system developed based on this invention. Figure 5 The method shown is the same as Figure 7 The method shown provides an interface for visualizing an image dataset with clustering (predicted by an AI model) category errors; and

[0022] Figure 10E This illustrates a visualization system developed based on the method of the present invention, in which the grid adjustment module 340 is executed sequentially. Figure 7 The method shown is the same as Figure 5 The method shown provides an interface for visualizing an image dataset with clustering (predicted by an artificial intelligence model) category errors. Detailed Implementation

[0023] Embodiments of the present invention will now be described with reference to the accompanying drawings. Numerous specific details are set forth in the following description to provide a more complete understanding of the invention. However, it will be apparent to those skilled in the art that implementations of the invention may not include some of these specific details. Furthermore, it should be understood that the invention is not limited to the specific embodiments described. Rather, the invention can be practiced with any combination of the following features and elements, regardless of whether they relate to different embodiments. Therefore, the following aspects, features, embodiments, and advantages are illustrative only and should not be construed as elements or limitations of the appended claims unless expressly set forth in the claims.

[0024] With the development of network technology, big data research has become a hot topic, and the demand for big data classification and in-depth analysis is increasing. Data visualization can present abstract data in an intuitive way, helping users understand patterns in the data and discover potential problems, thereby improving the efficiency of data analysis. Grid layout is a commonly used display form in data visualization, with advantages such as simple and easy-to-understand visual encoding, beautiful layout, and high space utilization. In grid layout, each sample is represented by a grid, and the color of the grid is usually used to indicate the category of its corresponding sample (cluster category or label category, etc.). These grids are arranged closely together. Because there is no overlap between different grids in grid layout, it can be used to conveniently perform tasks that require exploring samples, such as finding samples that have been incorrectly predicted by artificial intelligence models.

[0025] For example, Figure 1A The image shows nine samples from a digital image dataset, where samples 101-109 represent digital images. Each sample corresponds to a feature set (also called a feature vector), a cluster category (predicted by the model), and a label. Sample 102 has a label category of "8", while samples 101 and 103-109 have a label category of "3". However, samples 101-109 all have a cluster category of "3". Figure 1B It shows Figure 1A The results of visualizing 9 samples using the existing grid layout, among which Figure 1A The grids corresponding to samples 101-109 in the data are respectively Figure 1B In the grid, grids 110-190 are not colored the same as grids 110 and 130-190 (color indicates labeled category). The sample corresponding to grid 120 is an outlier, indicating an error in the clustering (model prediction) category.

[0026] Because the number of samples in Figure 1 is relatively small, the grid-based regions have little impact on user perception; users can easily identify outliers through color differences. However, when a large number of image samples need to be visualized, the grid-based regions significantly affect user perception. Figure 2 This demonstrates a visualization of the grid layout results for several image samples using existing methods. Figure 2 In this context, the grayscale of the grid (i.e., the color of the grid) indicates the labeled category of the image.

[0027] As can be seen, the images of "3" and "5" are mixed up in the visualization, making it difficult to determine the boundaries. Similar problems exist between other cluster categories, making it difficult for users to perceive the cluster structure of the samples, determine the cluster category, and thus identify outliers. A detailed description of these situations is provided below. Figure 10A The description.

[0028] The inventors of this invention believe that the main reason why existing grid layout visualization methods make it difficult for users to perceive the cluster structure of samples and to determine the cluster category of samples is that the grid layout does not consider preserving the cluster structure of samples and performing targeted optimization. The boundaries between the corresponding regions of samples of different cluster categories in the grid layout visualization results are very complex, and samples of the same cluster category present complex disconnected graphics in the visualization. Therefore, it is difficult to perceive the cluster structure of samples and it is also difficult to find outliers in samples.

[0029] This invention proposes a method for visualizing the grid layout of multiple samples. The method projects the samples onto a planar graph and divides the planar graph area into grids. Similar data samples are placed in adjacent grid positions, while minimizing the distance between grid positions corresponding to samples of the same cluster category, and / or maximizing the average convexity of multiple graphs composed of all grids corresponding to all samples in each of the multiple cluster categories. This visualization method helps users perceive the similarity and clustering structure of samples. The method of this invention also helps users visually analyze outliers in the samples, which is beneficial for analyzing clustering errors and difficult-to-distinguish samples in the sample data.

[0030] The samples described in this invention may include one of the following: images, videos, tables, documents, etc.

[0031] Figure 3 A structural block diagram of a system 300 for visualizing samples according to one or more embodiments of the present invention is shown.

[0032] like Figure 3 As shown, system 300 includes a receiving module 310, a projection module 320, a grid generation module 330, a grid adjustment module 340, and a visualization module 350.

[0033] exist Figure 3 In this embodiment, the receiving module 310 is configured to receive multiple samples 301. The information for each sample 301 includes its corresponding feature vector, cluster category, and label category. The cluster category, depending on the application scenario, could be a predicted category based on the model's corresponding feature vector, a manually labeled category, or a classification category obtained using any clustering method. The label category, if manually labeled, could also be changed to other categories depending on the application scenario. In one implementation, the receiving module 310 can directly read the multiple samples 301 stored somewhere (local or remote disk). In another implementation, the receiving module 310 can obtain the multiple samples 301 transmitted to it via a network.

[0034] The projection module 320 is configured to obtain the position 302 of the projection point of the plurality of samples 301 on a two-dimensional plane based on the features of each sample 301. In one embodiment, projecting each sample of the plurality of samples onto the two-dimensional plane can be achieved using the existing dimensionality reduction algorithm t-SNE. Those skilled in the art will know that projecting each sample of the plurality of samples onto the two-dimensional plane can also be achieved using other existing dimensionality reduction algorithms, such as MDS, PCA, etc. To facilitate subsequent processing, all projection points can be uniformly transformed using linear coordinates so that their position variation range (two-dimensional bounding box) is a specified region (e.g., a square). Figure 4A The diagram shows the locations 401-409 of the projection points obtained after projecting a set of samples onto a two-dimensional plane using existing projection techniques. The solid and hollow projection points represent different clustering categories of the corresponding samples.

[0035] The mesh generation module 330 is configured to obtain a mesh 303 corresponding to multiple samples on a two-dimensional plane based on the positions 302 of multiple samples 301 and their projection points, wherein the position of the mesh corresponding to a specific sample is related to the position of the projection point of that specific sample.

[0036] In one implementation, by calculating and dividing the bounding box of multiple projection points using existing techniques, a square grid of at least the number of samples can be obtained. Those skilled in the art will recognize that other existing techniques can also be used to obtain a square grid of at least the number of samples. Figure 4B It shows the result of Figure 4A The nine grids 411-419 obtained from the sample projection points are at this point, and the correspondence between the nine samples and the nine grids has not yet been established.

[0037] In existing technologies, there are various methods for establishing the correspondence between samples and grids. One method involves first treating the sample set and the grid set as two point sets in a bipartite graph, calculating the distance between the projection point of the sample and the grid position to obtain the cost of matching the edge between the sample and the grid, and then using any bipartite graph matching algorithm (such as the JV algorithm) to solve the bipartite graph matching problem, thereby obtaining the correspondence between samples and grids. Those skilled in the art will understand that other existing techniques can also be used to establish the correspondence between samples and grids. After establishing the correspondence, each sample corresponds to one and only one grid, and each grid corresponds to at most one sample.

[0038] Then, when the grid generation module 330 generates the grid corresponding to each sample according to the established correspondence, each sample corresponds to one and only one grid, each grid corresponds to at most one sample, the grid position corresponding to each sample is as close as possible to the projection point position of the sample, and the color of the grid corresponding to each sample is related to the labeling category of the sample.

[0039] like Figures 4A-4C As shown, where Figure 4C Medium grid 421-429 and Figure 4B Mesh sets 411-419 are the same, and Figure 4C Grids 421-429 are numbered in ascending order and respectively connected to... Figure 4A The projection points 401-409 correspond to the same samples. Figure 4C The solid and hollow squares in the grid represent the different cluster categories of the corresponding samples (unlike the grid color, which is related to the labeled category of the corresponding sample).

[0040] In the process of establishing the correspondence between samples and grids using existing technology, the grid generation module 330 can make the positions of different grids corresponding to similar samples close. However, this process does not directly take into account the clustering structure contained in the samples, which may result in the graphs formed by the grids corresponding to samples of the same cluster category being disconnected and irregular, which is not conducive to the user perceiving it as a whole structure.

[0041] The grid adjustment module 340 is configured to iteratively adjust the position of the grid corresponding to each sample based on the clustering category, the corresponding projection point position, and the corresponding grid position of each sample in the multiple samples 301, so as to meet the predetermined conditions, thereby obtaining the updated grid corresponding to the multiple samples.

[0042] In one embodiment of the present invention, the position of the grid corresponding to the sample is iteratively adjusted to satisfy a predetermined condition, which may be: minimizing the distance between the positions of the grids corresponding to samples of the same cluster category, referred to as predetermined condition A. Figure 5 A flowchart of a method 500 for adjusting the position of a grid corresponding to a sample to satisfy a predetermined condition A is shown according to one or more embodiments of the present invention.

[0043] like Figure 5 As shown, in step 510, based on the projection point position of each sample in the multiple samples on the two-dimensional plane and the position of each grid in the entire grid, the distance between the projection point position of each sample and each grid position is calculated, obtaining the proximity cost matrix Prox that matches the multiple samples with all grid positions. This proximity cost matrix Prox is an n x m matrix, where n is the number of samples and m is the number of grids (n ≤ m). Each item in the matrix is ​​Prox. i，j This represents the distance in a two-dimensional plane between the projection point of the i-th sample and the j-th grid position (e.g., the grid center, the position of a specific point on the grid, etc.). Taking the squared Euclidean distance as an example, Prox... i，j It can be obtained from the following formula:

[0044] Prox i，j =||i-th sample projection point position - j-th grid position|| 2 (1)

[0045] In step 520, based on the grid position corresponding to each sample in the two-dimensional plane, for each cluster category, the centroid of the grid region corresponding to all samples included in that cluster category is calculated. The centroid of the region corresponding to all samples included in the k-th cluster category can be obtained by the following formula:

[0046]

[0047] Then calculate the centroid μ respectively. k The distance between a location and every other grid location in the entire grid is used to obtain the compact cost vector CompK, which matches each sample in cluster category k with all grid locations. k The compact cost vector CompK k Let CompK be a vector with m terms, where m is the number of grid cells and the vector's terms are CompK. k，j The center of gravity μ k The distance between the current location and the j-th grid location in the two-dimensional plane. Taking the squared Euclidean distance as an example, CompK... k，j It can be obtained from the following formula:

[0048] CompK k，j =||μ k Position - j-th grid position || 2 (3)

[0049] Assuming the k-th cluster category includes the i-th sample, then the compactness cost vector CompI corresponding to the i-th sample is... i The j-th term is:

[0050] CompI i，j =CompK k，j (4)

[0051] In other words, formula (4) indicates the compactness cost vector CompK corresponding to the k-th cluster category. k It can be copied as a compact cost vector CompI corresponding to each sample i included in the cluster category. i .

[0052] In step 530, the compact cost vectors corresponding to each sample from the obtained samples are concatenated into a compact cost matrix Comp. This compact cost matrix Comp is an n x m matrix, where n is the number of samples and m is the number of grid cells (n ≤ m). The matrix items Comp... i，jThat is, the j-th term CompI of the compactness cost vector corresponding to the i-th sample. i，j .

[0053] In step 540, the proximity cost matrix Prox and the compactness cost matrix Comp obtained in steps 510 and 530 are weighted and averaged to obtain the weighted average cost matrix Cost. This weighted average cost matrix Cost is an n x m matrix, where n is the number of samples and m is the number of grid cells (n ≤ m). The matrix's terms are: Cost... i，j It can be obtained from the following formula:

[0054] Cost i，j =λProx i，j +(1-λ)Comp i，j (5)

[0055] Where λ is the weighted average coefficient, which can be set to a fixed value or dynamically set using any existing multi-objective optimization method.

[0056] In step 550, based on the weighted average cost matrix Cost, any existing bipartite graph matching algorithm is used to perform bipartite graph matching to obtain the correspondence between the sample and the grid position, so as to satisfy the predetermined condition A.

[0057] After obtaining the correspondence between the sample and the grid position by performing bipartite graph matching, if the predetermined condition A is met, proceed to step 560; otherwise, return to step 510.

[0058] Those skilled in the art should know that different distance calculation methods can be used in the above process, different weighted average coefficients λ can be set in different ways, and different bipartite graph matching algorithms can be used. As long as the correspondence can satisfy the predetermined condition A, they are all within the protection scope of this invention.

[0059] In step 560, based on the correspondence between the samples and grid positions obtained in step 550, the updated positions of the grids corresponding to multiple samples are obtained, and the process ends.

[0060] Figure 6A and Figure 6B Together they showed the Figure 4C The diagram shows a grid undergoing position updates. Specifically, for... Figure 4C The samples shown and their corresponding grids, the proximity cost matrix Prox uses Figure 4A The positions of the projection points 401-409 corresponding to the samples in the image are... Figure 4B The positions of grids 411-419 in the calculation were obtained. Figure 6A It shows according to Figure 4CThe sample and its corresponding grid, along with the region centers μ1 and μ2 for the two clustering categories shown, are 601 and 602 respectively. The solid and hollow centers correspond to different clustering categories. For... Figure 4C The samples shown and their corresponding grids, the proximity cost vectors CompK1 and CompK2 are used Figure 6A The position of the cluster centroids 601-602 in the cluster is related to Figure 4B The positions of grids 411-419 in the calculation were obtained. Figure 6B It shows that for Figure 4C The samples and their corresponding grids are used, and method 500 is used to adjust the positions of the grids corresponding to the samples. Each sample corresponds to grids 611-619, arranged in ascending order of their numbers, and is respectively matched with... Figure 4C The same samples are represented in grids 421-429.

[0061] Therefore, Figure 5 The method shown for adjusting the position of the grid corresponding to the sample can make the grid positions of samples of the same cluster category closer together, which helps users perceive them as a whole.

[0062] In another embodiment of the present invention, the predetermined condition may also be: maximizing the average convexity of multiple graphs composed of all grids corresponding to all samples included in each of the multiple cluster categories, referred to as predetermined condition B. Figure 7 A flowchart of another method 700 for adjusting the position of a grid corresponding to a sample to satisfy predetermined condition B according to one or more embodiments of the present invention is shown.

[0063] like Figure 7 As shown, in step 710, based on the clustering category and corresponding grid of each sample in the multiple samples, a first graph is obtained consisting of all grids corresponding to all samples included in each of the multiple clustering categories, wherein multiple clustering categories correspond to multiple first graphs.

[0064] In step 720, the positions of two or two sets of grids located at the edges of multiple first graphics are swapped to obtain multiple second graphics after the multiple first graphics are updated.

[0065] In step 730, it is determined whether the average convexity of the multiple second shapes after the swap is better than the average convexity of the multiple first shapes before the swap. If yes, proceed to step 740 and accept the swap of the grid position here; if no, proceed to step 750, reject the swap of the grid position here, and count the number of times the swap is rejected.

[0066] After step 740, return to step 710.

[0067] After step 750, if the number of times the grid position exchange is rejected is greater than or equal to the threshold, it is determined that the predetermined condition B has been met and proceeds to step 760; otherwise, it returns to step 710.

[0068] In step 760, based on the position exchange results of the previous steps, the updated positions of the grids corresponding to multiple samples are obtained, and the process ends.

[0069] Those skilled in the art should know that in the above process, the convexity calculation of the graphic can be performed using any existing convexity measurement method, such as calculating the ratio of the area of ​​the graphic to the area of ​​the convex hull of the graphic, or replacing it with a custom measurement method. As long as the updated positions of the grids corresponding to the multiple samples obtained satisfy the predetermined condition B, they are all within the protection scope of this invention.

[0070] Figure 8A It shows according to Figure 4C The clustering category and corresponding grid for each sample are obtained through step 710, and multiple first graphs 801-802 corresponding to each clustering category are obtained.

[0071] Figure 8B The diagram shows grids 811-819 corresponding to each sample after a position swap performed in step 720. The swapped grids... Figure 4C Medium grid 424 and 428 ( Figure 8B The positions are 818 and 814. Here, grids 811-819 are arranged in ascending order of their numbers, corresponding to... Figure 4C The same samples are represented in grids 421-429. Figure 8A The multiple first graphics 801-802 shown, after being swapped, become respectively... Figure 8C The multiple second figures 821-822 shown.

[0072] According to step 730, for Figure 8A Given two figures, 801-802, calculate the area of ​​each figure and its convex hull. Figure 8D The ratio of the areas of 801'-802' in the first figure is used to obtain the average convexity of multiple first figures. for Figure 8C Given two second figures, 821-822, calculate the area of ​​each figure and its convex hull. Figure 8E The ratio of the areas of 821'-822' in the figure is used to obtain the average convexity of multiple second figures, which is...

[0073] Based on the branch decision in step 730, swap. Figure 4C Medium grid 424 and 428 ( Figure 8BAfter the positions of 818 and 814 are given, the average convexity of the multiple second figures is better than the average convexity of the multiple first figures before the exchange, therefore the exchange is accepted.

[0074] Assuming the average convexity of the multiple first figures is 0.8 and the average convexity of the multiple second figures is 0.7, then the average convexity of the multiple second figures is worse than the average convexity of the multiple first figures before the swap. Therefore, the swap is rejected, the grid position before the swap is maintained, and the number of rejected swaps is incremented by 1.

[0075] Therefore, Figure 7 The method shown for adjusting the position of the grid corresponding to the sample makes the graph of the grid corresponding to the sample of the same cluster category simpler and more regular, which helps the user perceive it as a whole.

[0076] In module 340, methods 500 and 700 can be executed individually or in combination. Combined execution can involve executing method 500 first, followed by method 700, or vice versa. This combined execution results in closer grid positions for samples within the same cluster category, and also makes the resulting graph simpler, more regular, and easier for the user to perceive as a unified whole.

[0077] Back Figure 3 The visualization module 350 is configured to visualize multiple samples on a two-dimensional plane based on the grid corresponding to each sample in the multiple samples 301 and the updated position of the grid. Because the generated grid layout enhances the compactness of the grids corresponding to samples of the same cluster category and the convexity of the resulting graphic, the grids corresponding to samples of the same cluster category present a compact and regular graphic in the visualization.

[0078] In one embodiment, system 300 further includes an outlier identification module 360, configured to identify a specific sample as an outlier in response to the fact that the color of the grid corresponding to a specific sample on a two-dimensional plane is different from the color of the surrounding grid. In visualization, the color of the grid is used to represent the label category of the corresponding sample. Grids located at the edge or inside of the graph with a color different from the surrounding grids correspond to samples whose label category is different from other samples in the same cluster category, and therefore can be identified as outliers. This invention maintains the proximity of different grid positions corresponding to similar samples, while improving the user's perception of the cluster structure (the graph composed of grids) in visualization, making it easier for the user to discover outliers.

[0079] The visualization system 300 can be implemented as an application on a general computer system, or as an application on a server system, or as a network application, or as an application on a cloud platform.

[0080] Based on the same inventive concept, this invention also discloses a method for visualizing samples. Figure 9 A flowchart of a method 900 for visualizing a sample according to one or more embodiments of the present invention is shown.

[0081] according to Figure 9 In step 910, multiple samples are received, and the information of each sample includes the feature vector corresponding to the sample, the clustering category and the labeling category of the sample.

[0082] In step 920, the position of the projection point of the plurality of samples on the two-dimensional plane is obtained based on the feature vector of each of the plurality of samples.

[0083] In step 930, based on the position of each of the plurality of samples and its projection point, a grid corresponding to the plurality of samples on the two-dimensional plane is obtained, wherein the position of the grid corresponding to a specific sample is related to the position of the projection point of that specific sample.

[0084] In step 940, based on the clustering category, corresponding projection point position, and corresponding grid position of each sample in the plurality of samples, the position of the grid corresponding to the sample is iteratively adjusted to meet a predetermined condition, thereby obtaining the updated position of the grid corresponding to the plurality of samples.

[0085] In step 950, the multiple samples are visualized on the two-dimensional plane according to the grid corresponding to each sample and the updated position of the grid.

[0086] In one implementation, the predetermined condition in step 940 can be: minimizing the distance between the positions of the grids corresponding to samples of the same cluster category, which is called predetermined condition A.

[0087] In one implementation, based on the clustering category, corresponding projection point position, and corresponding grid position of each sample in the plurality of samples, the position of the grid corresponding to the sample is iteratively adjusted to satisfy a predetermined condition A, so that it can be used. Figure 5 The steps of method 500 are shown.

[0088] In one implementation, the predetermined condition in step 940 can be: maximizing the average convexity of multiple graphs composed of all grids corresponding to all samples included in each of the multiple cluster categories, referred to as predetermined condition B.

[0089] In one implementation, based on the clustering category, corresponding projection point position, and corresponding grid position of each sample in the plurality of samples, the position of the grid corresponding to the sample is iteratively adjusted to satisfy a predetermined condition B, so that it can be used. Figure 7 The steps of method 700 are shown.

[0090] In one implementation, based on the clustering category, corresponding projection point position, and corresponding grid position of each sample among the plurality of samples, the position of the grid corresponding to the sample is iteratively adjusted to meet predetermined conditions, allowing for initial use. Figure 5 The steps of method 500 shown are then used. Figure 7 The steps of method 700 are shown; alternatively, you can first use... Figure 7 The steps of method 700 shown are then used. Figure 5 The steps of method 500 are shown.

[0091] In one implementation, when generating the grid corresponding to each sample in step 930, each sample corresponds to one and only one grid, each grid corresponds to at most one sample, the grid position corresponding to each sample is as close as possible to the projection point position of the sample, and the color of the grid corresponding to each sample is related to the labeling category of the sample.

[0092] In one embodiment, method 900 further includes step 960, determining the specific sample as an outlier in response to the fact that the color of the grid corresponding to a specific sample on the two-dimensional plane is different from the color of the surrounding grid.

[0093] Based on the method of this invention, a visualization system was developed to analyze image data. This system can visually display datasets on an interface. Through this interface, users can explore the similarity relationships, clustering structures, and outliers of samples in the dataset, and discover potentially erroneous clustering (model prediction) categories. For the same digital image dataset containing clustering (artificial intelligence model prediction) category errors, Figure 10A To apply existing methods in the system Figure 2 The visualization of the corresponding sample's generated grid layout (i.e., skipping the grid adjustment module 340) shows a grid layout that is consistent with... Figure 2 Consistent; Figure 10B To Figure 2 The corresponding sample is a visualization of the grid layout generated by method 500 in the grid adjustment module 340 of the system; Figure 10C To Figure 2 The corresponding sample is visualized using the grid layout generated by method 700 in the grid adjustment module 340 of the system; Figure 10D To Figure 2The corresponding samples are visualized in the grid adjustment module 340 of the system using the grid layout generated by method 500 and method 700 respectively. Figure 10E To Figure 2 The corresponding samples are visualized using the grid layouts generated by methods 700 and 500 sequentially in the grid adjustment module 340 of the system. Figure 10A In the image, four cluster structures are represented by approximate region graphics 1001-1004 (solid borders are for illustration only, not the actual visualization result), corresponding to images of the numbers "2", "3", "5", and "8" in the cluster categories. Samples within the same cluster category appear as complex graphics in the visualization; the boundaries between the four graphics are highly curved and difficult to discern, especially at the boundary between "3" and "5," where the two types of samples are mixed together, making the cluster boundaries even more difficult to determine (solid borders in the image represent approximate boundaries). Therefore, users find it difficult to perceive the cluster structure of the samples and to identify outliers within the samples. Figures 10B-10E In the data, the four cluster structures present region graphs of 1011–1014, 1021–1024, 1031–1034, and 1041–1044, respectively. The boundaries between the four graphs are simpler, more regular, and less easily identifiable. Among them, the region graphs of the four cluster structures are... Figure 10D Taking 1032 as an example, several grids with colors different from the surrounding grids can be observed within this image, such as 1061-1062. The color (grayscale) of grid 1061 represents its labeled category "5," and the color of grid 1062 represents its labeled category "8." Image samples on these grids can be identified as outliers incorrectly clustered (predicted) as "2." In 1001-1004, the two outliers mentioned above correspond to 1051-1052 respectively. These outliers are easily confused with surrounding non-outliers (for example, the other "8"s around 1052 are actually correctly clustered and not outliers), making them more difficult to identify. Figures 10B-10E The specific advantages and disadvantages of the corresponding methods depend on the dataset, but they are all superior to... Figure 10A The corresponding method.

[0094] This invention can be a system, a method, and / or a computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for causing a processor to implement various aspects of the invention. The methods of this invention can be executed on a standalone computer system, on a distributed computing system, or even on a cloud platform.

[0095] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable storage media according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable storage media according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0097] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A system for visualizing samples, the system comprising: The receiving module is configured to receive multiple samples, wherein the information of each sample includes the feature vector corresponding to the sample, the clustering category and the labeling category of the sample; The projection module is configured to obtain the position of the projection point of the plurality of samples on a two-dimensional plane based on the feature vector of each of the plurality of samples; The mesh generation module is configured to obtain a mesh corresponding to each of the plurality of samples and its projection point on the two-dimensional plane based on the position of each sample and its projection point. The position of the mesh corresponding to a specific sample is related to the position of the projection point of that specific sample. Furthermore, each sample corresponds to one and only one mesh, each mesh corresponds to at most one sample, the position of the mesh corresponding to each sample is as close as possible to the position of the projection point of that sample, and the color of the mesh corresponding to each sample is related to the labeling category of that sample. A grid adjustment module is configured to iteratively adjust the position of the grid corresponding to each sample based on the clustering category, corresponding projection point position, and corresponding grid position of each sample in the plurality of samples, such that predetermined conditions are met, thereby obtaining the updated position of the grid corresponding to the plurality of samples; wherein, the predetermined conditions include: minimizing the distance between the positions of the grids corresponding to samples of the same clustering category, and maximizing the average convexity of the plurality of graphs composed of all grids corresponding to all samples included in each of the plurality of clustering categories; and The visualization module is configured to visualize the plurality of samples on the two-dimensional plane based on the grid corresponding to each sample and the updated position of the grid.

2. The system according to claim 1, wherein the position of the grid corresponding to each sample is iteratively adjusted based on the clustering category, the corresponding projection point position, and the corresponding grid position of each sample in the plurality of samples, so as to satisfy a predetermined condition, including: Calculate the distance between the projection point position of each of the multiple samples and each grid position in the entire grid to obtain the proximity cost matrix between the multiple samples and the entire grid position; For each cluster category: Calculate the centroid of the grid region corresponding to all samples included in this cluster category; Calculate the distance between the centroid location and each grid location in the entire grid to obtain the compactness cost vector that matches each sample in the cluster category with all grid locations; Each compact cost vector corresponding to each of the multiple samples obtained is concatenated into a compact cost matrix; The proximity cost matrix and the compact cost matrix are weighted and averaged to obtain a weighted average cost matrix. Based on the obtained weighted average cost matrix, the correspondence between the sample and the grid position is obtained by performing bipartite graph matching to satisfy the predetermined conditions; as well as Based on the obtained correspondence between the samples and the grid positions, the updated positions of the grids corresponding to the multiple samples are obtained.

3. The system according to claim 1, wherein the position of the grid corresponding to each sample is iteratively adjusted based on the clustering category, the corresponding projection point position, and the corresponding grid position of each sample in the plurality of samples, so as to satisfy a predetermined condition, including: Based on the clustering category and corresponding grid of each sample in the plurality of samples, a first graphic is obtained which is composed of all grids corresponding to all samples included in each of the plurality of clustering categories, and the plurality of clustering categories correspond to a plurality of first graphics; By swapping the positions of two or two sets of grids located at the edges of the plurality of first graphics, a plurality of second graphics are obtained after the plurality of first graphics are updated; In response to the fact that the average convexity of the plurality of second graphics is better than the average convexity of the plurality of first graphics, the grid positions are swapped, and the process returns to the step of obtaining the first graphic composed of all grids corresponding to all samples in each of the plurality of cluster categories based on the cluster category of each sample in the plurality of samples and the corresponding grid. In response to the fact that the average convexity of the plurality of second graphics is inferior to the average convexity of the plurality of first graphics, the exchange of the grid positions is rejected, and the number of times the exchange is rejected is counted; If the number of rejections for a grid position is less than a threshold, the process returns to the step of obtaining a first graph consisting of all grids corresponding to all samples in each of the multiple cluster categories, based on the cluster category of each sample in the multiple samples and the corresponding grid. as well as In response to the number of times the grid position is rejected for exchange being greater than or equal to a threshold, the acquisition of the plurality of first graphics and the exchange of positions are stopped, thereby obtaining the updated positions of the grids corresponding to the plurality of samples.

4. The system according to claim 1, further comprising: An outlier identification module is configured to identify a particular sample as an outlier in response to the fact that the color of the grid corresponding to a particular sample on the two-dimensional plane is different from the color of the surrounding grid.

5. The system according to any one of claims 1-4, wherein the sample is one of the following: image; video; sheet; document.

6. A method for visualizing samples, comprising: Receive multiple samples, and the information of each sample includes the feature vector corresponding to the sample, the cluster category and the label category of the sample; Based on the feature vector of each of the plurality of samples, the position of the projection point of the plurality of samples on the two-dimensional plane is obtained; Based on the position of each sample and its projection point in the plurality of samples, a grid corresponding to the plurality of samples on the two-dimensional plane is obtained, wherein the position of the grid corresponding to a specific sample is related to the position of the projection point of that specific sample; and each sample corresponds to one and only one grid, each grid corresponds to at most one sample, the position of the grid corresponding to each sample is as close as possible to the position of the projection point of that sample, and the color of the grid corresponding to each sample is related to the labeling category of that sample; Based on the cluster category, corresponding projection point position, and corresponding grid position of each sample in the plurality of samples, the grid position corresponding to the sample is iteratively adjusted to satisfy predetermined conditions, thereby obtaining the updated position of the grid corresponding to the plurality of samples; wherein, the predetermined conditions include: minimizing the distance between the positions of the grids corresponding to samples of the same cluster category, and maximizing the average convexity of the plurality of graphs composed of all grids corresponding to all samples included in each of the plurality of cluster categories; and Based on the grid corresponding to each of the multiple samples and the updated position of the grid, the multiple samples are visualized on the two-dimensional plane.

7. The method according to claim 6, wherein iteratively adjusting the position of the grid corresponding to each sample based on the clustering category, the corresponding projection point position, and the corresponding grid position of each sample in the plurality of samples, to satisfy a predetermined condition, includes: Calculate the distance between the projection point position of each of the multiple samples and each grid position in the entire grid to obtain the proximity cost matrix between the multiple samples and the entire grid position; For each cluster category: Calculate the centroid of the grid region corresponding to all samples included in this cluster category; Calculate the distance between the centroid location and each grid location in the entire grid to obtain the compactness cost vector that matches each sample in the cluster category with all grid locations; Each compact cost vector corresponding to each of the multiple samples obtained is concatenated into a compact cost matrix; The proximity cost matrix and the compact cost matrix are weighted and averaged to obtain a weighted average cost matrix. Based on the obtained weighted average cost matrix, the correspondence between the sample and the grid position is obtained by performing bipartite graph matching to satisfy the predetermined conditions; as well as Based on the obtained correspondence between the samples and the grid positions, the updated positions of the grids corresponding to the multiple samples are obtained.

8. The method according to claim 6, wherein iteratively adjusting the position of the grid corresponding to each sample based on the clustering category, the corresponding projection point position, and the corresponding grid position of each sample in the plurality of samples, to satisfy a predetermined condition, includes: Based on the clustering category and corresponding grid of each sample in the plurality of samples, a first graphic is obtained which is composed of all grids corresponding to all samples included in each of the plurality of clustering categories, and the plurality of clustering categories correspond to a plurality of first graphics; By swapping the positions of two or two sets of grids located at the edges of the plurality of first graphics, a plurality of second graphics are obtained after the plurality of first graphics are updated; In response to the fact that the average convexity of the plurality of second graphics is better than the average convexity of the plurality of first graphics, the grid positions are swapped, and the process returns to the step of obtaining the first graphic composed of all grids corresponding to all samples in each of the plurality of cluster categories based on the cluster category of each sample in the plurality of samples and the corresponding grid. In response to the fact that the average convexity of the plurality of second graphics is inferior to the average convexity of the plurality of first graphics, the exchange of the grid positions is rejected, and the number of times the exchange is rejected is counted; If the number of rejections for a grid position is less than a threshold, the process returns to the step of obtaining a first graph consisting of all grids corresponding to all samples in each of the multiple cluster categories, based on the cluster category of each sample in the multiple samples and the corresponding grid. as well as In response to the number of times the grid position is rejected for exchange being greater than or equal to a threshold, the acquisition of the plurality of first graphics and the exchange of positions are stopped, thereby obtaining the updated positions of the grids corresponding to the plurality of samples.

9. The method according to claim 6, further comprising: In response to the fact that the color of the grid corresponding to a specific sample on the two-dimensional plane is different from the color of the surrounding grid, the specific sample is determined to be an outlier.

10. The method according to any one of claims 6-9, wherein the sample is one of the following: image; video; sheet; document.

11. A computer-readable storage medium for visualizing a grid layout of multiple samples, the computer-readable storage medium having program instructions stored therein, the program instructions being executable by a computing device to cause the computing device to perform the method as described in any one of claims 6-10.

12. A system for visualizing the grid layout of multiple samples, comprising: Memory; as well as At least one processor is operatively coupled to a memory and configured to perform the method as described in any one of claims 6-10.

Images