Visual analytics system and method for unified analysis of multiple computer vision tasks

Abstract

The application provides a visual analysis system and method for unified analysis of multiple computer vision tasks, which comprises the following steps: receiving multiple pictures containing at least one object, annotation results of the object in each picture and model prediction results of the object level in each picture, wherein the annotation results and the model prediction results both comprise detection classification results of the object level and detection position results of the object level or segmentation classification results of the object level and segmentation position results of the object level in each picture, the detection classification results and the segmentation classification results comprise discrete values of discrete attributes, and the detection position results and the segmentation position results comprise continuous values of continuous attributes; according to the annotation results of the object in each picture and the model prediction results of the object level in each picture, unified modeling of discrete attributes and continuous attributes in the results is performed by using a joint probability distribution, so that a multivariate probability distribution is obtained; and the multivariate probability distribution is displayed in multiple levels.

Description

TECHNICAL FIELD

[0001] The present application relates to visual analysis of pictures, and in particular, the present application relates to a visual analysis system, method, and computer readable storage medium for unified analysis of multiple computer vision tasks. BACKGROUND

[0002] With the development of machine learning technology, more and more image processing scenarios can be assisted by machine learning to liberate manpower and improve work efficiency. In the actual use of processing pictures by using machine learning, computer vision model evaluation is a very important step. At present, the most widely used method for evaluating computer vision models is to use performance indicators such as accuracy and average precision. However, using these indicators alone can sometimes be misleading. For example, in the picture classification task, a classification model that always predicts the majority class in an unbalanced picture dataset can achieve high accuracy, but has no predictive ability at all. In order to better understand the performance of the computer vision model, more fine-grained evaluation is needed. Existing methods have tried to do this, for example, the existing confusion matrix method provides picture class level evaluation by presenting the confusion between picture classes, the grid layout method provides picture sample level evaluation by placing similar pictures in close proximity, and the ConfusionWheel method combines a chord diagram and a histogram to provide confidence distribution information of the model's classification pictures while showing the confusion of the picture classes. Many existing methods can make the performance of the computer vision model presented to the user in an intuitive way by using visualization technology, helping the user better understand the behavior of the computer vision model and providing help for the user to select and use the computer vision model. SUMMARY

[0003] The description that follows includes exemplary methods, systems, techniques, and instruction sequences that embody the present technology. However, it is understood that in one or more aspects, the described technology can be practiced without these specific details. In other instances, well-known protocols, structures and techniques have not been shown in detail in order not to obscure the application. Those of ordinary skill in the art will understand that the described technology and mechanisms can be applied to a variety of architectures for visualizing samples.

[0004] According to an aspect of the present disclosure, there is provided a visual analysis system for unified analysis of multiple computer vision tasks, the system comprising: a receiving module configured to receive a plurality of images each containing at least one object, a labeled result of the object in each image, and a model prediction result of the object in each image at an object level, the labeled result of the object in each image and the model prediction result of the object in each image at the object level each comprising one of: (1) a detection classification result of the object in each image at the object level and a detection location result of the object in each image at the object level; (2) a segmentation classification result of the object in each image at the object level and a segmentation location result of the object in each image at the object level; the detection classification result and the segmentation classification result comprising discrete values of discrete attributes, and the detection location result and the segmentation location result comprising continuous values of continuous attributes; a data modeling module configured to model the discrete attributes and the continuous attributes in the results using a joint probability distribution according to the labeled result of the object in each image and the model prediction result of the object in each image at the object level, thereby obtaining a multivariate probability distribution; and a visualization module configured to display the multivariate probability distribution at multiple levels. The system can perform unified evaluation and analysis on multiple results obtained by performing detection tasks and segmentation tasks on a large number of images. According to another aspect of the present disclosure, there is provided a visual analysis method for unified analysis of multiple computer vision tasks, the method comprising: receiving a plurality of images each containing at least one object, a labeled result of the object in each image, and a model prediction result of the object in each image at an object level, the labeled result of the object in each image and the model prediction result of the object in each image at the object level each comprising one of: (1) a detection classification result of the object in each image at the object level and a detection location result of the object in each image at the object level; (2) a segmentation classification result of the object in each image at the object level and a segmentation location result of the object in each image at the object level; the detection classification result and the segmentation classification result comprising discrete values of discrete attributes, and the detection location result and the segmentation location result comprising continuous values of continuous attributes; modeling the discrete attributes and the continuous attributes in the results using a joint probability distribution according to the labeled result of the object in each image and the model prediction result of the object in each image at the object level, thereby obtaining a multivariate probability distribution; and displaying the multivariate probability distribution at multiple levels. The method can perform unified evaluation and analysis on multiple results obtained by performing detection tasks and segmentation tasks on a large number of images.

[0005] According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium having program instructions stored therein, the program instructions executable by a computing device to cause the computing device to perform the method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present invention itself, as well as the preferred mode of use, further objectives, features, and advantages of the present invention, will best be understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, in which:

[0007] Figure 1 Object detection in pictures and object segmentation in pictures in the field of computer vision are shown;

[0008] Figure 2 A structural block diagram of a visual analysis system for computer vision models according to an embodiment of the present invention is shown;

[0009] Figure 3 A flow chart of a method for obtaining a matching result between a labeling result of an object and a model prediction result at an object level according to one or more embodiments of the present invention is shown;

[0010] Figure 4 The steps of Figure 3 are shown in a specific computing process for a picture and the corresponding results obtained;

[0011] Figure 5 An interface of a visualization module of a visual analysis system for unified analysis of multiple computer vision tasks according to an embodiment of the present invention is shown;

[0012] Figure 6 Interfaces of three modes of a matrix visualization module in the visualization module according to an embodiment of the present invention are shown;

[0013] Figure 7 An interface of a table visualization module in the visualization module according to an embodiment of the present invention is shown;

[0014] Figure 8 A flow chart of a visual analysis method for unified analysis of multiple computer vision tasks is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present application will be described below with reference to the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to unnecessarily obscure the present application. In addition, it is to be understood that the present application is not limited in its application to the particular implementation, implementations, conditions, or environments of operation illustrated herein, which are provided for description sake. Rather, the application is capable of other implementations and of being practiced, or being carried out in various ways. Also, the specific steps of the methods of the different embodiments are not strictly ordered, that is, the first step of a method can be performed before or after the second step, and so on. Thus, the following aspects, features, embodiments, and advantages are merely illustrative and are not to be construed as limiting the scope of the application, unless the claims state otherwise.

[0016] As mentioned before, with the wide use of computer vision models, computer vision model fine-grained evaluation with visualization techniques helps users to select and use computer vision models. However, these methods only focus on the picture classification task, i.e. the accuracy of picture classification. However, there are other tasks widely used in real-world applications, such as object detection in pictures, object segmentation in pictures, etc. These computer vision model evaluations not only involve the accuracy of picture classification, but also involve, for example, the precision of object detection in the object detection in pictures task, the precision of object segmentation in the object segmentation in pictures task, etc. For these tasks, existing methods such as confusion matrix method are not applicable because they only focus on the picture classification task and do not consider other tasks such as object detection, object segmentation, etc. In addition, in real picture processing scenarios, multiple tasks often need to be considered simultaneously, such as the multiple tasks in the picture processing described above. If different fine-grained computer vision model evaluation tools are designed for different tasks, it will increase the learning cost and cognitive burden of users when switching between multiple different evaluation tools. Therefore, a visual analytics system and method that can analyze multiple computer vision tasks uniformly is needed.

[0017] For example, Figure 1 Object detection in pictures and object segmentation in pictures tasks in the field of computer vision are shown. In Figure 1In the prior art, the object detection task is performed on the picture 101, i.e., the picture 101 is input into the detection model 102 to obtain the detection classification result 103 at the object level and the detection position result 104 at the object level; the object segmentation task is performed on the picture 101, i.e., the picture 101 is input into the segmentation model 105 to obtain the segmentation classification result 106 at the object level and the segmentation position result 107 at the object level. The existing fine-grained model evaluation method can only analyze and evaluate the classification result in the picture classification task, but cannot analyze and evaluate the detection result in the detection task, the segmentation result in the segmentation task, and the combination of various results. The analysis and evaluation of the detection result in the detection task and the segmentation result in the segmentation task need to use the annotation result of the object in the picture and the model prediction result at the object level in the picture. Figure 1 Only one picture is shown, and in fact, an evaluation and analysis system needs to process a large number of pictures, and it is necessary for the evaluation and analysis system to uniformly evaluate and analyze various results obtained by performing various tasks on the large number of pictures.

[0018] Therefore, in the present application, a visual analysis system and method for unified analysis of various computer vision tasks are proposed, so as to analyze and evaluate the detection classification result and the detection position result in the picture detection task or the segmentation classification result and the segmentation position result in the picture segmentation task. The system and method receive a plurality of pictures containing at least one object, the annotation result of the object in each picture, and the model prediction result at the object level in each picture, the annotation result of the object in each picture and the model prediction result at the object level in each picture each include one of the following: (1) the detection classification result at the object level and the detection position result at the object level in each picture; (2) the segmentation classification result at the object level and the segmentation position result at the object level in each picture, the detection classification result and the segmentation classification result include discrete values of discrete attributes, and the detection position result and the segmentation position result include continuous values of continuous attributes; then according to the annotation result of the object in each picture and the model prediction result at the object level in each picture, the discrete attributes and the continuous attributes in the result are uniformly modeled by using a joint probability distribution, so as to obtain a multivariate probability distribution; finally, the multivariate probability distribution is displayed in multiple levels. By using the above visual analysis system and method for unified analysis of various computer vision tasks, the results of various computer vision tasks can be modeled, and then fine-grained model evaluation is supported by visualization technology, not only supporting multi-level evaluation at the class level, but also supporting multi-level evaluation at the subset level and the sample level, so that users can analyze and evaluate different task results in the system by using the method.

[0019] Figure 2A structural block diagram of a visual analysis system 200 for computer vision models according to embodiments of the present application is shown. As shown Figure 2 The system 200 includes a receiving module 210, a data modeling module 220, and a visualization module 230.

[0020] In Figure 2 The receiving module 210 is configured to receive a plurality of pictures 201 containing at least one object, annotation results 202 of the object in each picture, and model prediction results 203 of the object level in each picture, wherein the annotation results 202 of the object in each picture and the model prediction results 203 of the object level in each picture each include one of the following: (1) detection classification results 204 of the object level in each picture and detection location results 205 of the object level; (2) segmentation classification results 206 of the object level in each picture and segmentation location results 207 of the object level Figure 2 The detection classification results 204 of the object level and the segmentation classification results 206 of the object level include discrete values of discrete attributes, and the detection location results 205 of the object level and the segmentation location results 207 of the object level include continuous values of continuous attributes.

[0021] The data modeling module 220 is configured to generate a multivariate probability distribution 209 according to the annotation results 202 of the object in each picture and the model prediction results 203 of the object level in each picture.

[0022] The visualization module 230 is configured to display the multivariate probability distribution 209 in multiple levels. Specifically, the visualization module 230 can display the plurality of pictures 201 containing at least one object, the annotation results 202 of the object in the plurality of pictures, the model prediction results 203 of the object level in the plurality of pictures, and the multivariate probability distribution 209.

[0023] The visual analysis system 200 for computer vision tasks can be implemented as an application program on a general computer system, or as an application program on a server system, or as a web application program, or as an application program on a cloud platform.

[0024] In one implementation, the receiving module 210 can be configured to receive the uploaded plurality of pictures 201 containing at least one object, the annotation results 202 of the object in each picture, and the model prediction results 203 of the object level in each picture from the user through a graphical user interface or user commands, etc. The receiving module 210 can also be configured to store the plurality of pictures 201 containing at least one object, the annotation results 202 of the object in each picture, and the model prediction results 203 of the object level in each picture somewhere to be read by the receiving module 210. The receiving module 210 can also be configured to obtain the plurality of pictures 201 containing at least one object, the annotation results 202 of the object in each picture, and the model prediction results 203 of the object level in each picture through network transmission, etc.

[0025] The pictures 201 contain at least one object, and the following assumes that the picture set contains three pictures, picture A, picture B, and picture C. Picture A contains a cat (object) and a dog (object), picture B contains a tree (object), and picture C contains several houses (objects), etc. The size specifications of picture A, picture B, and picture C are not limited and can be the same or different.

[0026] The annotation result 202 of the object in the picture is a result of human processing of the picture. For example, in a human detection task, a human identifies the object in the picture, obtains an annotation detection classification result of the object (corresponding to the detection classification result 204 at the object level in the picture), draws a minimum rectangular frame for the object in the picture, and obtains parameters of the rectangular frame as an annotation detection position result of the object (corresponding to the detection position result 205 at the object level in the picture). In a human segmentation task, a human identifies the object in the picture, obtains an annotation segmentation classification result of the object (corresponding to the segmentation classification result 206 at the object level in the picture), and draws a minimum polygon frame for the object in the picture and obtains parameters of the polygon frame as an annotation segmentation position result (corresponding to the segmentation position result 207 at the object level in the picture). On the other hand, the model prediction result 203 at the object level in the picture is a result of processing of the picture by using a computer vision model. For example, in a computer detection task, a detection classification model identifies the object in the picture, obtains a prediction detection classification result of the object (corresponding to the detection classification result 204 at the object level in the picture), a detection position model predicts a minimum rectangular frame of each object existing in the picture and outputs parameters of the rectangular frame as a prediction detection position result of the object (corresponding to the detection position result 205 at the object level in the picture). In a computer segmentation task, a segmentation classification model identifies the object in the picture, obtains a prediction segmentation classification result of the object (corresponding to the segmentation classification result 206 at the object level in the picture), and a segmentation position model predicts a minimum polygon frame of each object existing in the picture and obtains parameters of the polygon frame as a prediction segmentation position result (corresponding to the segmentation position result 207 at the object level in the picture).

[0027] According to human operation, for example, the object-level detection classification result 204 in the object annotation result 202 of the object of picture A is cat and dog; the object-level detection classification result 204 in the object annotation result 202 of the object of picture B is tree; and the object-level detection classification result 204 in the object annotation result 202 of the object of picture C is house. The above detection classification results 204 are different categories, which can be represented by different integers, so the detection classification result 204 is a discrete attribute with discrete values. On the other hand, the object-level segmentation classification result 206 in the object annotation result 202 of the object of picture A is cat and dog; the object-level segmentation classification result 206 in the object annotation result 202 of the object of picture B is tree; and the object-level segmentation classification result 206 in the object annotation result 202 of the object of picture C is house. The above segmentation classification results 206 are also different categories, which can be represented by different integers, so the segmentation classification result 206 is also a discrete attribute with discrete values. In addition, the object-level detection location result 205 in the object annotation result 202 of the object of picture A is the minimum rectangular box containing the objects corresponding to cat and dog; the object-level detection location result 205 in the object annotation result 202 of the object of picture B is the minimum rectangular box containing the object corresponding to tree; and the object-level detection location result 205 in the object annotation result 202 of the object of picture C is the minimum rectangular box containing the object corresponding to each house. The above detection location results 205 actually include the center point position coordinates of the rectangular box, the width and height of the rectangular box, which are real values, and the derived attributes such as the area of the rectangular box and the aspect ratio of the rectangular box are also real values, so the detection location result 205 is a continuous attribute with continuous values. On the other hand, the object-level segmentation location result 207 in the object annotation result 202 of the object of picture A is the minimum polygonal box containing the objects corresponding to cat and dog; the object-level segmentation location result 207 in the object annotation result 202 of the object of picture B is the minimum polygonal box containing the object corresponding to tree; and the object-level segmentation location result 207 in the object annotation result 202 of the object of picture C is the minimum polygonal box containing the object corresponding to each house. The above segmentation location results 207 actually include the vertex position coordinates of the polygonal box and other parameters, which are real values, and the derived attributes such as the area of the polygonal box and the aspect ratio of the minimum circumscribed rectangle of the polygonal box are also real values, so the segmentation location result 207 is a continuous attribute with continuous values.

[0028] The model prediction result 203 is different from the annotation result 202 in that the model prediction result 203 is a result of computer operation, but the model prediction result 203 also includes the object-level detection classification result 204 and the object-level detection position result 205, or the object-level segmentation classification result 206 and the object-level segmentation position result 207, wherein the object-level detection classification result 204 and the object-level segmentation classification result 206 are discrete attributes with discrete values, and the object-level detection position result 205 and the object-level segmentation position result 207 are continuous attributes with continuous values.

[0029] In an embodiment, the data modeling module 220 uses a joint probability distribution to uniformly model the discrete attributes and the continuous attributes in the results according to the annotation result of the object in each picture and the object-level model prediction result in the object in each picture, so that in the implementation of the multivariate probability distribution, the matching result 208 between the annotation result of the object in each picture and the object-level model prediction result in the object in the picture can be obtained first, and then the matching result 208 in each picture is taken as a sample, and the multivariate probability distribution 209 with the discrete attributes and the continuous attributes possessed by multiple samples in multiple pictures as variables is calculated.

[0030] In an embodiment, the at least one matching result of each picture is based on the reward score between the annotation result of the at least one object in the picture and the object-level model prediction result of the at least one object in the picture.

[0031] Figure 3 A flowchart of a method 300 of obtaining the matching result 208 between the annotation result of the object and the object-level model prediction result is shown according to one or more embodiments of the present application. According to the method 300, the matching result 208 between the annotation result of the object and the object-level model prediction result can be obtained by the following steps. Figure 3 In step 310, for each picture in the received multiple pictures 201 containing at least one object, the multiple object-level model prediction results in the picture are sorted according to the classification confidence in the detection classification result 204 or the segmentation classification result 206 included therein. Those skilled in the art can know that the confidence here is a probability value output by the model at the same time as the classification prediction result or the segmentation classification result. In an embodiment, the above sorting can be from high to low; in another embodiment, the above sorting can also be from low to high.

[0032] At step 320, for each object-level model prediction result in the picture, a reward score of matching between the model prediction result and the annotation result of each object in the picture is calculated according to a ranking result, wherein the ranking result affects the reward score. At step 330, the annotation result with the largest reward score is selected as the matching result of the model prediction result. In this way, each model prediction result is matched with only one annotation result, but one annotation result can be matched with multiple model prediction results.

[0033] At step 340, it is determined whether there are still model prediction results to be matched. If yes, return to step 320 to continue calculating the reward score of matching between the model prediction result and all annotation results. Otherwise, go to step 350 to determine whether there are still pictures without matching results. If yes, return to step 310 to continue calculating the reward score of matching between the model prediction result and the annotation result in the picture. Otherwise, go to step 360 to return the matching result of the annotation result of the object and the object-level model prediction result of each object in the picture, and end the process.

[0034] In an embodiment, the reward score between the annotation result of the object in the picture and the object-level model prediction result can be calculated by the following formula (1):

[0035] Reward score = λ1c ij + λ2p ij + (1 - λ1- λ2)u i (1)

[0036] wherein c ij is a category consistency score indicating whether the paired object-level model prediction result and the annotation result of the object are consistent in category, taking 1 when consistent in category and 0 when inconsistent in category, p ij is a position consistency score indicating how close the paired object-level model prediction result and the annotation result of the object are in position, and u i is a uniqueness score indicating a specific score of pairing the annotation result of each object with several object-level model prediction results, and λ1and λ2are parameters for balancing the three addends of the reward score.

[0037] In an embodiment, the position consistency score p ij can be calculated by the following formula (2):

[0038]

[0039] wherein G i represents the position of the detection position result or the position of the segmentation position result in the annotation result of the i-th object, and P jrepresents the position of the detection position result or the position of the segmentation position result in the model prediction result of the jth object level, |·| operation represents the area of the position inside |·|, G i ∩P j represents the position of the detection position result or the position of the segmentation position result in the model prediction result of the jth object level, |·| operation represents the area of the position inside |·|, G i ∩P j represents the position of the detection position result or the position of the segmentation position result in the model prediction result of the jth object level, |·| operation represents the area of the position inside |·|, G i ∪P j represents the position of the detection position result or the position of the segmentation position result in the model prediction result of the jth object level, |·| operation represents the area of the position inside |·|, G i ∪P j represents the position of the detection position result or the position of the segmentation position result in the model prediction result of the jth object level, |·| operation represents the area of the position inside |·|, G

[0040] In another embodiment, the position consistency score p ij can be calculated by the following formula (3):

[0041]

[0042] The meanings of the parameters in formula (3) are the same as those in formula (2). In addition, C ij represents the minimum rectangle covering G i and P j . Compared with formula (2), this way has different penalties for the label result and the model prediction result without overlapping.

[0043] In yet another embodiment, the position consistency score p ij can be calculated by the following formula (4):

[0044]

[0045] The meanings of the parameters in formula (4) are the same as those in formula (2) and (3). In addition, ρ ij represents the Euclidean distance between the center point of G i and the center point of P j , and l(C ij ) represents the diagonal length of C ij .

[0046] The higher the position consistency score calculated by the above formula (2) (3) (4), the better the position overlapping relationship between the label result and the model prediction result. Those skilled in the art can also develop other methods for calculating the position overlapping relationship between the label result and the model prediction result under the inspiration of the above method.

[0047] In one embodiment, the uniqueness score u i in formula (2) can be calculated by the following formula (5):

[0048]

[0049] wherein z ijz ij z ij z i The uniqueness score u

[0050] In another embodiment, the uniqueness score u i The uniqueness score u

[0051] The uniqueness score u i = 1 -∑ k z ik (6)

[0052] The meanings of the parameters in formula (6) are the same as in formula (5).

[0053] In yet another embodiment, the uniqueness score u i The uniqueness score u

[0054]

[0055] The meanings of the parameters in formula (7) are also the same as in formula (5).

[0056] The greater the uniqueness score calculated by the above formulas (5), (6), and (7), the more suitable the annotation result is to represent the matching with the current model prediction result. Those skilled in the art can also develop other methods for calculating the uniqueness score of the annotation result under the inspiration of the above method.

[0057] For example, Figure 4 The steps of Figure 3 are shown in a specific calculation process for one picture and the corresponding results obtained. There are two annotation results 401 and 402 of objects and two model prediction results 403 and 404 of object levels, and their object class results are both zebras. Assuming that the classification confidence contained in the model prediction result 403 is 0.4, and the classification confidence contained in the model prediction result 404 is 0.6, then according to the steps of Figure 3 , the sorting of the classification confidence from large to small is taken here, and the matching result between the model prediction result 404 and the annotation result will be calculated first, and then the matching result between the model prediction result 403 and the annotation result will be calculated. Assuming that the matching of the model prediction result 404 has been completed, and the matching result is the annotation result 402, now the matching of the model prediction result 403 needs to be performed. According to the reward score = λ1c ij + λ2P ij + (1 - λ1- λ2)ui Here, take λ1=0.5, λ2=0.25 to illustrate the process of calculation, and then calculate the reward score of the matching between the model prediction result 403 and the labeled results 401 and 402 respectively. For the labeled result 401, the coordinates of the upper left corner point are (50, 0), and the coordinates of the lower right corner point are (500, 300); for the labeled result 402, the coordinates of the upper left corner point are (150, 150), and the coordinates of the lower right corner point are (500, 500); for the model prediction result 403, the coordinates of the upper left corner point are (100, 50), and the coordinates of the lower right corner point are (480, 480). Then, for the matching between the model prediction result 403 and the labeled result 401, the reward score is For the matching between the model prediction result 403 and the labeled result 402, the reward score is Therefore, the labeled result that produces the largest reward score for matching with the model prediction result 403 is 401, and the two are matched together to complete the matching of the model prediction result 403. Assuming that the reward score for the matching between the model prediction result 403 and the labeled result 401 is 0.7, and the reward score for the matching between the model prediction result 403 and the labeled result 402 is 0.8, then the labeled result that produces the largest reward score for matching with the model prediction result 403 is 402, and the two are matched together, so that the labeled result 402 is matched with the model prediction result 403 and the model prediction result 404 at the same time.

[0058] In an embodiment, Figure 2 The data modeling module 220 takes the matching result 208 in each picture as a sample to calculate the multivariate probability distribution 209 with discrete attributes and continuous attributes in multiple samples in multiple pictures as variables. In the implementation of the multivariate probability distribution 209, the probability distribution of each discrete attribute and continuous attribute can be calculated respectively, and the multivariate probability distribution 209 containing both discrete attributes and continuous attributes can be calculated by using conditional probability. For a discrete attribute, the probability value of the discrete attribute falling within a specific value is calculated by using a probability mass function, and the probability mass function is established based on the frequency of the discrete attribute taking each value. For a continuous attribute, the probability value of the continuous attribute falling within a specific value range is calculated by using a cumulative distribution function, and the cumulative distribution function is estimated by using an empirical cumulative distribution function, which calculates the frequency of the attribute being less than a specific value after discretizing the continuous attribute. Based on the probability value of each attribute falling within a specific value or within a specific value range, the multivariate probability distribution 209 with the discrete attribute and the continuous attribute as variables is calculated by using conditional probability.

[0059] For example, this example illustrates how to calculate the value of a multivariate probability distribution function by using a multivariate function P containing two variables. This multivariate function includes two variables: the classification result Label from the object's annotation results. X , is a discrete variable; the confidence result in the model prediction results of the object. Y is a continuous variable. Specifically, we want to calculate the multivariate probability distribution function P when the labeled category is cat (represented by the integer 0) and the model's prediction confidence score is greater than 0.5, i.e., P(Label... X =0, Confidence Y >0.5). According to the conditional probability formula, the above multivariate probability distribution function value can be calculated by decomposing it using the following formula (8):

[0060] P(Label X =0, Confidence Y >0.5)

[0061] =P(Label) X =0)P(Confidence Y >0.5|Label X =0) (8)

[0062] Wherein, P(Label) X =0) can be determined based on the discrete attribute Label X The probability mass function, calculate Label X The probability value is obtained from P = 0, and P(Confidence) Y >0.5|Label X =0) can be based on the continuous attribute Confidence Y In Label X The empirical cumulative distribution function under the condition that = 0 is calculated in Label. X Confidence = 0 Y It is obtained from a probability value greater than 0.5.

[0063] In practical analysis, multivariate probability distributions 209 often contain a large number of variables, such as category results, position results, and size results in the labeling results. Such a large number of variables can pose significant challenges to the user's analysis, making direct analysis of the original multivariate probability distribution impractical. However, this paper utilizes transformations of the multivariate probability distribution to help users complete the analysis of the distribution function formed by specified variables. For example, by finding the marginal distributions of the multivariate probability distribution on a few specified variables, some variables irrelevant to the current analysis task can be discarded, focusing on a small number of specified variables; by using some variables as conditional variables, the conditional distributions of the multivariate probability distribution on other variables can be found, allowing analysis to be performed on the data subset represented by the specified conditional variables. This analysis can be performed using the visualization module 230.

[0064] In one implementation, the visualization module 230 can be further configured to display multiple images 201 containing at least one object, the annotation results of the objects in the multiple images 202, the model prediction results 203 at the object level in the multiple images, and the multivariate probability distribution 209 in a visual form. Figure 5 The diagram illustrates the interface of a visualization module 230 of a visual analysis system for unified analysis of multiple computer vision tasks, according to an embodiment of the present invention. When displaying the multivariate probability distribution 209, the marginal distribution is calculated so that each analysis involves only a few variables of interest from the original multivariate probability distribution 209, thus reducing the difficulty of analysis for the user.

[0065] In one implementation, the visualization module 230 of the visual analysis system for unified analysis of multiple computer vision tasks may further include a matrix visualization module. Figure 2 (Not shown), which can output interface 510. This matrix visualization module is configured to display at least one of three modes—obfuscation mode, size mode, or orientation mode—in matrix form. Figure 6 The diagram shows the interface of the matrix visualization module in three modes according to an embodiment of the present invention.

[0066] In one implementation, the confusion mode can be configured to display the confusion between different categories in the matching results of multiple images and the model prediction results at the object level, based on the confusion matrix. Figure 6Fig. 6 shows an example of the confusion pattern. In the confusion pattern, each row represents a labeled class result, each column represents a model-predicted class result, and the intersection of a row and a column represents the confusion between the labeled class result and the model-predicted class result. The confusion is represented by the color depth. From the confusion pattern, the user can see the confusion between the model-predicted class result and the labeled class result, and locate the more serious confusion by the color depth. For example, in Fig. 6, the confusion between the large vehicle and the small vehicle can be seen, and the confusion of the helicopter being identified as an airplane is more serious, so the related problems can be analyzed in depth. This pattern retains the two variables of the labeled class result and the model-predicted class result in the multivariate probability distribution 209 by solving the marginal distribution, and discards other variables, so that the user can focus on analyzing the confusion between the classes.

[0067] In an embodiment, the size pattern can be configured to use a graph to display the accuracy of the object size in the labeled result and the model-predicted result of the object in the matching result of the plurality of pictures. The graph used to display the size accuracy can use a pie chart, a concentric circle, and a histogram, etc. The rows and columns of the matrix have the same meaning as in the confusion pattern, and will not be described again here. Figure 6 Fig. 6 shows an example of the confusion pattern. In the confusion pattern, each row represents a labeled class result, each column represents a model-predicted class result, and the intersection of a row and a column represents the confusion between the labeled class result and the model-predicted class result. The confusion is represented by the color depth. From the confusion pattern, the user can see the confusion between the model-predicted class result and the labeled class result, and locate the more serious confusion by the color depth. For example, in Fig. 6, the confusion between the large vehicle and the small vehicle can be seen, and the confusion of the helicopter being identified as an airplane is more serious, so the related problems can be analyzed in depth. This pattern retains the two variables of the labeled class result and the model-predicted class result in the multivariate probability distribution 209 by solving the marginal distribution, and discards other variables, so that the user can focus on analyzing the confusion between the classes.

[0068] In one implementation, the directional mode can be configured to graphically display the object labeling results in the matching results of multiple images and the offset of object positions in the model prediction results at the object level. The graphics used to display the positional offset can include arrow charts, radar charts, and histograms, etc. The rows and columns of the matrix have the same meaning as in the obfuscation mode, and will not be repeated here. Figure 6 Example 630 shows a directional pattern display, using arrows to illustrate positional offsets. The circles in the center of the arrows indicate that the object position predicted by the model is close to the labeled object position. The eight arrows correspond to the percentage of matches where the predicted object position deviates from the labeled object position in that direction. Other graphs can be designed similarly, reflecting both proximity and the percentage of offsets in each of the eight directions. By examining the object position offsets in the graphs, users can identify cells in the matrix where the predicted object position has a significant percentage of offsets, allowing for further analysis. For example, in example 630, in matches where the labeled category is "human" but the model predicts "skis," the predicted object position is significantly offset downwards compared to the labeled object position, requiring further analysis. This mode also retains three variables in the multivariate probability distribution 209—the labeled category result, the model-predicted category result, and the offset relationship between the labeled result and the model-predicted object position—by solving the marginal distribution, while discarding other variables. This allows users to focus on analyzing the accuracy of the model-predicted object position and its relationship with the category.

[0069] In one implementation, the visualization module 230 of a visual analysis system that performs unified analysis of multiple computer vision tasks may further include a table visualization module. Figure 2 (Not shown), which can output Figure 5 The interface 520. This table visualization module can be configured to use a frequent pattern mining algorithm to find subsets of a certain size among the matching results 208 of labeled results and model prediction results in multiple images, while satisfying a certain attribute range, so that users can filter and sort the subsets according to different attributes, thereby finding the subsets of interest. Figure 7 The diagram illustrates the interface of a table visualization module within a visualization module proposed according to an embodiment of the present invention. Each row of the table represents a subset of 208 matching results found by the frequent pattern mining algorithm that has a certain size and satisfies a certain attribute range. Each column represents an attribute. For discrete attributes, such as predicted category or labeled category, the value of the attribute for each row's corresponding subset is represented by text. For continuous attributes, such as accuracy, regression value, and confidence, a rectangular bar is used to display the attribute value. This can be used...Figure 7 The buttons above the table header allow you to sort, filter, and find the complement of subsets, exploring data subsets of interest. For example... Figure 7 The example shows the first two subsets after filtering by predicted category and labeled category as traffic light, sorted by regression value from smallest to largest. The subset with lower regression value indicates that the model performs poorly on it, suggesting that users can further explore the problems in this subset.

[0070] In one implementation, the visualization module 230 of a visual analysis system that performs unified analysis of multiple computer vision tasks may further include a grid visualization module. Figure 2 (Not shown), which can output Figure 5 Interface 530. This grid visualization module can be configured to display multiple images using a grid layout, along with the model predictions and their matching annotations for each image.

[0071] The preceding description indicates that System 200 can perform unified evaluation and analysis of detection and segmentation tasks. Based on the above description, those skilled in the art will understand that System 200 can also perform unified evaluation and analysis of existing image classification tasks.

[0072] The present invention also proposes a visual analysis method for weighting samples. Figure 8 A flowchart of a visual analytics method 800 for unified analysis of multiple computer vision tasks is shown. Figure 8 As shown, in step 810, multiple images containing at least one object are received, along with the annotation results of the objects in each image and the model prediction results at the object level in each image. The annotation results of the objects in each image and the model prediction results at the object level in each image each include one of the following: (1) the detection and classification results and the detection location results at the object level in each image; (2) the segmentation and classification results and the segmentation location results at the object level in each image. The detection and classification results and the segmentation and classification results include discrete values ​​of discrete attributes, and the detection location results and the segmentation location results include continuous values ​​of continuous attributes. In step 820, based on the annotation results of the objects in each image and the model prediction results at the object level in each image, a joint probability distribution is used to uniformly model the discrete and continuous attributes in the results, thereby obtaining a multivariate probability distribution. In step 830, the multivariate probability distribution is displayed in multiple layers.

[0073] In one implementation, step 820 of method 800 further includes ( Figure 8The matching result between the annotation result of the object in each picture and the model prediction result of the object level in the picture is obtained, and the matching result in each picture is taken as a sample. A multivariate probability distribution is calculated with the discrete attributes and continuous attributes possessed by multiple samples in multiple pictures as variables.

[0074] In an embodiment, the at least one matching result of each picture is based on a reward score between the annotation result of the at least one object of the picture and the model prediction result of the at least one object level of the picture.

[0075] In an embodiment, the step of obtaining the matching result between the annotation result of the object in each picture and the model prediction result of the object level in the picture of the method 800 further comprises Figure 3 The steps shown are not repeated here. The calculation of the reward score and the calculation of the parameters therein are described in detail in the foregoing, and are not repeated here.

[0076] In an embodiment, the step 830 of the method 800 further comprises displaying at least one of the confusion pattern, the size pattern and the direction pattern in the form of a matrix. The confusion pattern, the size pattern and the direction pattern have been described in the foregoing, and are not repeated here.

[0077] In an embodiment, the step 830 of the method 800 further comprises using a frequent pattern mining algorithm to find a subset in the matching results of the multiple pictures that has a certain size and satisfies a certain attribute range, so that the user can filter and sort the subset according to different attributes to find a subset of interest.

[0078] The method 800 can perform unified evaluation and analysis on detection tasks and segmentation tasks, as described in the foregoing. Through the foregoing description, those skilled in the art can see that the method can also perform unified evaluation and analysis on existing picture classification tasks. The present application can be a system, a method and / or a computer readable storage medium. The computer readable storage medium has computer readable program instructions for causing a processor to implement various aspects of the present application. The method of the present application can be executed on an independent computer system, a distributed computing system or even a cloud platform.

[0079] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer readable storage media according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0080] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0081] Embodiments of the application have been described above, with the understanding that these embodiments are exemplary only, and not exhaustive, and are not limited to the embodiments disclosed. Many modifications and variations of the described embodiments are possible, without departing from the scope and spirit of the described embodiments. The selection of terms to be used in the description herein is intended to best explain the principles of the embodiments, practical application, or technical improvement over the prior art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual analytics system for unified analysis of multiple computer vision tasks, the system comprising: The receiving module is configured to receive multiple images containing at least one object, the annotation results of the objects in each image, and the model prediction results at the object level in each image, wherein the annotation results of the objects in each image and the model prediction results at the object level in each image each include one of the following: (1) Detection and classification results at the object level and detection location results at the object level in each image; (2) Segmentation and classification results at the object level and segmentation location results at the object level in each image; The detection classification result and the segmentation classification result include discrete values ​​of discrete attributes, and the detection location result and the segmentation location result include continuous values ​​of continuous attributes; The data modeling module is configured to model the discrete and continuous attributes in the results using a joint probability distribution based on the annotation results of objects in each image and the model prediction results at the object level in each image, thereby obtaining a multivariate probability distribution. as well as The visualization module is configured to display the multivariate probability distribution in multiple levels; The data modeling module includes: Obtain the matching result between the annotation result of the object in each image and the model prediction result at the object level in the image; use the matching result in each image as a sample to calculate the multivariate probability distribution of the discrete attribute and the continuous attribute as variables of multiple samples in multiple images; The process of calculating a multivariate probability distribution with discrete and continuous attributes as variables for multiple samples in multiple images, using the matching results in each image as samples, includes: For the discrete attribute, a probability mass function is used to calculate the probability value of the discrete attribute falling on a specific value. The probability mass function is established based on the frequency of the discrete attribute taking each value. For the continuous attribute, a cumulative distribution function is used to calculate the probability value of the continuous attribute falling within a specific value range. The cumulative distribution function is estimated using an empirical cumulative distribution function, which discretizes the continuous attribute and calculates the frequency of the attribute being less than a specific value. Based on the probability values ​​of each attribute falling on a specific value or within a specific value range, a multivariate probability distribution with the discrete and continuous attributes as variables is calculated using conditional probability.

2. The system of claim 1, wherein at least one matching result for each image is based on a reward score for matching between the annotation result of at least one object in the image and the model prediction result at the level of at least one object in the image.

3. The system according to claim 2, wherein obtaining the matching result between the annotation result of the object in each image and the model prediction result at the object level in the image includes: Perform the following steps for each image: The model prediction results for multiple objects in the image are sorted according to the classification confidence in the detection or segmentation classification results they include. For the model prediction results at the level of each object in the image, perform the following steps according to the sorting results: Calculate the reward score for matching the model's prediction results with the annotation results of each object in the image, wherein the ranking results affect the reward score; The labeled result with the highest reward score is selected as the matching result of the model's prediction.

4. The system according to claim 3, wherein the reward score , wherein The category consistency score indicates whether the model prediction results and the labeled results of the paired objects are consistent in category; a score of 1 indicates consistency, and a score of 0 indicates inconsistency. The location consistency score indicates the degree to which the paired object-level model predictions and the object's labeled locations are similar. The uniqueness score, which is influenced by the ranking results, indicates a specific score that pairs the annotation results of each object with model prediction results at several object levels. and These are the parameters used to balance the three addends of the reward score.

5. The system of claim 4, wherein the position consistency score Calculate using one of the following methods: Positional Consistency Score ; Positional Consistency Score ; Positional Consistency Score ; in Indicates the first The position of the detection position result or the position of the segmentation position result in the annotation results of the object. Indicates the first The location of the detection location result or the location of the segmentation location result in the model prediction results at the object level. Operation representation Area of ​​the interior location, express and The position where they intersect, express and The combined position, Indicates coverage and The smallest rectangle, express The length of the diagonal, express The center point and The Euclidean distance between the center points.

6. The system of claim 5, wherein the uniqueness score Calculate using one of the following methods: Uniqueness score ; Uniqueness score ; Uniqueness score ; Indicates the first The annotation results of the first object and the first Whether the model prediction results at the object level match, if they match Take 1, otherwise Take 0.

7. The system of claim 6, wherein the visualization module further comprises: The matrix visualization module is configured to display at least one of the following three modes in matrix form: The obfuscation mode is configured to display the obfuscation between different categories in the matching results of the multiple images and the model prediction results at the object level, based on the obfuscation matrix. Size mode is configured to use graphics to display the accuracy of object size in the matching results of the multiple images and the object size in the model prediction results at the object level; as well as The orientation mode is configured to graphically display the annotation results of objects in the matching results of the multiple images and the offset of object positions in the model prediction results at the object level.

8. The system of claim 7, wherein the visualization module further comprises: The table visualization module is configured to use a frequent pattern mining algorithm to find subsets of the matching results of the multiple images that have a certain size and satisfy a certain attribute range, so that users can filter and sort the subsets according to different attributes to find the subsets of interest.

9. A visual analytics method for unified analysis of multiple computer vision tasks, the method comprising: The system receives multiple images containing at least one object, annotation results for the objects in each image, and model prediction results at the object level for each image. The annotation results for the objects in each image and the model prediction results at the object level for each image each include one of the following: (1) Detection and classification results at the object level and detection location results at the object level in each image; (2) Segmentation and classification results at the object level and segmentation location results at the object level in each image; The detection classification result and the segmentation classification result include discrete values ​​of discrete attributes, and the detection location result and the segmentation location result include continuous values ​​of continuous attributes; Based on the annotation results of objects in each image and the model prediction results at the object level in each image, the discrete and continuous attributes in the results are modeled in a unified manner using the joint probability distribution, thereby obtaining a multivariate probability distribution; as well as The multivariate probability distribution is displayed in multiple levels; Based on the annotation results of objects in each image and the model prediction results at the object level in each image, a joint probability distribution is used to uniformly model the discrete and continuous attributes in the results, thereby obtaining a multivariate probability distribution, including: Obtain the matching result between the annotation result of the object in each image and the model prediction result at the object level in the image; use the matching result in each image as a sample to calculate the multivariate probability distribution of the discrete attribute and the continuous attribute as variables of multiple samples in multiple images; The process of calculating a multivariate probability distribution with discrete and continuous attributes as variables for multiple samples in multiple images, using the matching results in each image as samples, includes: For the discrete attribute, a probability mass function is used to calculate the probability value of the discrete attribute falling on a specific value. The probability mass function is established based on the frequency of the discrete attribute taking each value. For the continuous attribute, a cumulative distribution function is used to calculate the probability value of the continuous attribute falling within a specific value range. The cumulative distribution function is estimated using an empirical cumulative distribution function, which discretizes the continuous attribute and calculates the frequency of the attribute being less than a specific value. Based on the probability values ​​of each attribute falling on a specific value or within a specific value range, a multivariate probability distribution with the discrete and continuous attributes as variables is calculated using conditional probability.

10. The method of claim 9, wherein at least one matching result for each image is based on a reward score between a labeling result of at least one object in the image and a model prediction result at the level of at least one object in the image.

11. The method of claim 10, wherein obtaining the matching result between the annotation result of the object in each image and the model prediction result at the object level in the image comprises: Perform the following steps for each image: The model prediction results for multiple objects in the image are sorted from high to low according to the classification confidence of the included detection classification results or segmentation classification results. For the model prediction results at the level of each object in the image, perform the following steps according to the sorting results: Calculate the reward score for matching the model's prediction results with the annotation results of each object in the image, wherein the ranking results affect the reward score; The labeled result with the highest reward score is selected as the matching result of the model's prediction.

12. The method of claim 11, wherein the reward score , wherein The category consistency score indicates whether the model prediction results and the labeled results of the paired objects are consistent in category; a score of 1 indicates consistency, and a score of 0 indicates inconsistency. The location consistency score indicates the degree to which the paired object-level model predictions and the object's labeled locations are similar. The uniqueness score is a specific score that pairs the labeled results for each object with model predictions at several object levels. and These are the parameters used to balance the three addends of the reward score.

13. The method of claim 12, wherein the positional consistency score Calculate using one of the following methods: Positional Consistency Score ; Positional Consistency Score ; Positional Consistency Score ; in Indicates the first The position of the detection position result or the position of the segmentation position result in the annotation results of the object. Indicates the first The location of the detection location result or the location of the segmentation location result in the model prediction results at the object level. Operation representation Area of ​​the interior location, express and The position where they intersect, express and The combined position, Indicates coverage and The smallest rectangle, express The length of the diagonal, express The center point and The Euclidean distance between the center points.

14. The method of claim 13, wherein the uniqueness score Calculate using one of the following methods: Uniqueness score ; Uniqueness score ; Uniqueness score ; Indicates the first The annotation results of the first object and the first Whether the model prediction results at the object level match, if they match Take 1, otherwise Take 0.

15. The method of claim 14, wherein displaying the multivariate probability distribution at multiple levels further comprises: Display at least one of the following three patterns using matrix form: The obfuscation mode is configured to display the obfuscation between different categories in the matching results of the multiple images and the model prediction results at the object level, based on the obfuscation matrix. Size mode is configured to use graphics to display the accuracy of object size in the matching results of the multiple images and the object size in the model prediction results at the object level; as well as The orientation mode is configured to graphically display the annotation results of objects in the matching results of the multiple images and the offset of object positions in the model prediction results at the object level.

16. The method of claim 15, wherein displaying the multivariate probability distribution at multiple levels further comprises: The frequent pattern mining algorithm is used to find subsets of the matching results of the multiple images that have a certain size and satisfy a certain attribute range, so that users can filter and sort the subsets according to different attributes, thereby finding the subsets of interest.

17. A computer-readable storage medium having program instructions stored therein, the program instructions being executable by a computing device to cause the computing device to perform the method as described in any one of claims 9-16.

18. A computer system, comprising: Memory; as well as At least one processor is operatively coupled to the memory and configured to perform the method as described in any one of claims 9-16.

Images