Method and system for visualizing output of image object detection model

Abstract

The present invention proposes a system and method for visualizing the output of an image target detection model trained with text corresponding to an image, wherein the system comprises: a receiving module for receiving images and texts of a plurality of training samples; a category extraction module for extracting the categories of targets contained in the images; a target detection module for training an image target detection model, and using the trained model to extract the positions and categories of targets contained in the images of unlabeled training samples; an image and category double clustering module for clustering the images and categories of a plurality of training samples to obtain clustering results of the images of the training samples, clustering results of the categories, and image and category co-occurrence clustering results; and a visualization module for displaying a representative image of each cluster in the image clustering results, a name of each cluster in the category clustering results, and each co-occurrence cluster in the image and category co-occurrence clustering results.

Description

Technical Field

[0001] The present invention relates to object detection in images, and in particular, to a method, a computer system and a computer-readable storage medium for visualizing the output of an image object detection model trained using text corresponding to an image. Technical Background

[0002] Object detection in images has achieved great success in many fields, such as autonomous driving and augmented reality. However, training a high-performance image object detection model requires a large amount of labeled data, which is very expensive. To this end, the prior art uses images and their corresponding texts to train image object detection models. These texts are often obtained without annotation, so the cost of data annotation can be saved. For example, medical images are often accompanied by a diagnosis report (text) given by a doctor. However, since the text contains noise and the text cannot provide accurate location information of the target, the performance of the image object detection model obtained in this way is often poor, and the user needs to provide certain confirmation information to remove the noise in the text and correct the erroneous detection results, thereby improving the performance of the image object detection model. Summary of the invention

[0003] According to one aspect of the present invention, a system for visualizing the output of an image target detection model trained using text corresponding to an image is proposed, comprising: a receiving module, configured to receive a plurality of training samples, the plurality of training samples including a limited number of labeled training samples and a plurality of unlabeled training samples, an unlabeled training sample including an image and its corresponding text, and a labeled training sample including an image and its corresponding text, as well as targets contained in the image and their positions; a category extraction module, configured to extract the categories of targets contained in the images corresponding to the plurality of unlabeled training samples from a section of text contained in each of the plurality of unlabeled training samples using the plurality of training samples received by the receiving module; and a target detection module, configured to extract the categories of targets contained in the images corresponding to the plurality of unlabeled training samples extracted from the category extraction module, as well as the limited number of labeled training samples received by the receiving module. An image target detection model is trained together with the labeled training samples and the images of the multiple unlabeled training samples, and the positions and categories of the targets contained in the images of the multiple unlabeled training samples are extracted using the trained image target detection model; an image and category dual clustering module is configured to cluster the images and categories of the multiple training samples using the positions and categories of the targets contained in the images of the multiple unlabeled training samples obtained by the image target detection module and the positions and categories of the targets contained in the images of the limited number of labeled training samples received by the receiving module, so as to obtain clustering results of the images of the multiple training samples, clustering results of the categories, and image and category co-occurrence clustering results; and a visualization module is configured to display a representative image of each cluster in the clustering results of the images of the multiple training samples, the name of each cluster in the clustering results of the categories, and each co-occurrence cluster in the image and category co-occurrence clustering results.

[0004] According to another aspect of the present invention, a method for visualizing the output of an image target detection model trained using text corresponding to an image is proposed, comprising: receiving a plurality of training samples, wherein the plurality of training samples include a limited number of labeled training samples and a plurality of unlabeled training samples, wherein an unlabeled training sample includes an image and its corresponding text, and a labeled training sample includes an image and its corresponding text, as well as targets contained in the image and their positions; extracting the categories of targets contained in the images corresponding to the plurality of unlabeled training samples from a section of text contained in each training sample of the plurality of unlabeled training samples using the received plurality of training samples; and visualizing the categories of targets contained in the images corresponding to the plurality of unlabeled training samples using the extracted categories of targets contained in the images corresponding to the plurality of unlabeled training samples and the received limited number of labeled training samples. This method trains an image target detection model together with the images of the multiple unlabeled training samples; uses the trained image target detection model to extract the positions and categories of the targets contained in the images of the multiple unlabeled training samples; uses the obtained positions and categories of the targets contained in the images of the multiple unlabeled training samples and the positions and categories of the targets contained in the received images of the limited number of labeled training samples to cluster the images and categories of the multiple training samples to obtain clustering results of the images of the multiple training samples, clustering results of the categories, and image and category co-occurrence clustering results; and displays a representative image of each cluster in the clustering results of the images of the multiple training samples, the name of each cluster in the clustering results of the categories, and each co-occurrence cluster in the image and category co-occurrence clustering results.

[0005] According to another aspect of the present invention, a computer-readable storage medium is proposed, on which computer-readable code is stored. When the computer-readable code is executed by a processor, the method of visualizing the output of an image object detection model trained using text corresponding to an image as proposed in another aspect of the present invention is implemented.

[0006] According to another aspect of the present invention, a computer program product is proposed. When an instruction processor in the computer program product is executed, it performs the method of visualizing the output of an image object detection model trained using text corresponding to an image as proposed in another aspect of the present invention.

[0007] Additional aspects and advantages of the present invention will be given in the following detailed description, and the advantages of the present invention will become apparent from the following description or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The invention itself, as well as its preferred mode of use, objects, features and advantages may be better understood by reading the following detailed description of illustrative embodiments with reference to the accompanying drawings, in which:

[0009] Figure 1 An example of an unlabeled training sample is shown;

[0010] Figure 2 An example of a labeled training sample is shown;

[0011] Figure 3 A structural block diagram of a system for visualizing the output of an image object detection model trained using text corresponding to an image according to an embodiment of the present invention is shown;

[0012] Figure 4 The process of calculating the co-occurrence frequencies between four images and four categories according to an embodiment of the present invention is schematically shown;

[0013] Figure 5 Schematically illustrates a process of performing biclustering of images and categories according to an embodiment of the present invention;

[0014] Figure 6 An example of an interface displayed by a visualization module in a system for visualizing the output of an image object detection model trained using text corresponding to an image according to an embodiment of the present invention is schematically shown;

[0015] Figure 7 A flow chart showing a method for visualizing the output of an object detection model trained using text corresponding to an image. DETAILED DESCRIPTION

[0016] The following description includes exemplary methods, systems, and storage media that embody the present invention. However, it should be understood that in one or more aspects, the described invention can be practiced without these specific details. In other cases, well-known protocols, structures, and techniques are not shown in detail to avoid obscuring the present invention. One of ordinary skill in the art will appreciate that the described techniques and mechanisms can be applied to systems, methods, and computer-readable storage media for training object detection models using image descriptions.

[0017] Embodiments of the present invention are described below with reference to the accompanying drawings. In the following description, many specific details are set forth in order to more fully understand the present invention. However, it is obvious to those skilled in the art that the present invention may be implemented without some of these specific details. In addition, it should be understood that the present invention is not limited to the specific embodiments described. On the contrary, any combination of the following features and elements may be considered to implement the present invention, regardless of whether they relate to different embodiments. Therefore, the following aspects, features, embodiments and advantages are intended for illustration only and should not be considered as elements or limitations of the appended claims unless clearly stated in the claims.

[0018] Image object detection models have achieved great success in many fields, such as autonomous driving and augmented reality. However, training high-performance image object detection models requires a large amount of labeled data, which is very expensive. To this end, existing technologies use images and their corresponding texts to train image object detection models. These texts are often obtained without annotation. For example, medical images are often accompanied by a diagnosis report (text) given by a doctor. Or when users upload images to social media, they often add a text description. Figure 1 An example of an unlabeled training sample is shown, which includes an image 110 and its corresponding text 120 . Figure 2 An example of a labeled training sample is shown, which includes not only an image 200 and its corresponding text 220, but also an object (whose category is dog 230) contained in the image 200 and its position. The position of the object is represented by a rectangular box 210 in the image.

[0019] Although the existing technology can use a small number of labeled training samples and a large number of unlabeled training samples with images and their corresponding texts to train image object detection models, its performance is often poor. There are two reasons for this. First, the text of the unlabeled training samples contains noise, and its description of the image is often incomplete. For example, Figure 1 In the example, text 120 only describes the horse taking a shower in image 110, but ignores the person showering it and the van behind it. Such unlabeled training sample text will introduce noise to the training of image object detection models. Secondly, image object detection models need to accurately detect the location of the target in the image. However, during the training phase, unlabeled training sample text can only provide the category information of the target (coarse-grained annotation), but cannot provide its accurate location in the image (fine-grained annotation). For example Figure 1 In the example, the text 120 only describes that there is a horse in the image 110, but does not explain where the horse is in the image 110. These two reasons lead to poor performance of the image object detection model.

[0020] To this end, the user needs to provide certain confirmation information to remove the noise in the text and correct the erroneous detection results, thereby improving the performance of the image target detection model. Therefore, the present invention proposes a method and system for visualizing the output of the image target detection model trained using the text corresponding to the image, effectively displaying the text, image, and the detection results of the target detection model, and helping the user find noisy text and erroneous detection results. By reducing the noise of the text and correcting the erroneous detection results, the image target detection model is updated, thereby improving the detection accuracy of the image target detection model.

[0021] Figure 3FIG. 3 is a block diagram of a system 300 for visualizing the output of an image object detection model trained using text corresponding to an image according to an embodiment of the present invention. Figure 3As shown, the system 300 according to the embodiment of the present invention includes a receiving module 320, a category extraction module 330, an image target detection module 340, an image and category biclustering module 350, and a visualization module 360. The receiving module 320 is configured to receive multiple training samples 310. The multiple training samples 310 include a limited number of labeled training samples 311 and multiple unlabeled training samples 312. An unlabeled training sample includes an image and its corresponding text; a labeled training sample includes not only an image and its corresponding text, but also an object contained in the image and its position. In one embodiment, the position of the object is represented as a rectangular box in an image. In another embodiment, the position of the object can be represented as the position of a pixel in an image expressed by text. The category extraction module 330 is configured to use the multiple training samples 310 received by the receiving module 320 to extract the category of the object contained in the image corresponding to the unlabeled training sample from a section of text contained in each training sample in the multiple unlabeled training samples 312. The image object detection module 340 is configured to train an image object detection model 341 using the categories of objects contained in the images corresponding to the multiple unlabeled training samples 312 extracted from the category extraction module 330, and the limited number of labeled training samples 311 and the images of the multiple unlabeled training samples 312 received by the receiving module 320, and extract the positions and categories of objects contained in the images of the multiple unlabeled training samples 312 using the trained image object detection model 341. The image and category dual clustering module 350 is configured to cluster the images and categories of all training samples 310 using the positions and categories of objects contained in the images of the multiple unlabeled training samples 312 obtained from the image object detection module 340, and the positions and categories of objects contained in the images of the limited number of labeled training samples 311 received by the receiving module 320, so as to obtain clustering results of images of all training samples, clustering results of categories, and image and category co-occurrence clustering results. The visualization module 360 ​​is configured to display the representative image of each cluster in the clustering results of the images of all training samples 310, the name of each cluster in the clustering results of the categories, and each co-occurrence cluster in the co-occurrence clustering results of the images and categories. In addition, the visualization module 360 ​​is also configured to receive user input through the receiving module 320 to interact with the visualization module 360, so as to iteratively update the image target detection model obtained by the image target detection module 340 and improve the detection accuracy of the image target detection model. The system 300 for visualizing the output of the target detection model trained with the text corresponding to the image can be implemented as an application on a general computer system, or as an application on a server system, or as a network application, or as an application on a cloud platform.

[0022] In one embodiment, the receiving module 320 may be configured to receive multiple training samples 310 uploaded from a user through a graphical user interface or a user command, etc. The receiving module 320 may also be configured to store the multiple training samples 310 in an internal or external storage medium, and read by the receiving module 320. The receiving module 320 may also be configured to obtain the multiple training samples 310 through network transmission, etc.

[0023] In one embodiment, the category extraction module 330 can be configured to train a category extraction model 331 using a limited number of labeled training samples 311 included in multiple training samples, and use the category extraction model 331 to extract the category of the target contained in the image of the unlabeled training sample 312.

[0024] In one embodiment, the category extraction model 331 may be a keyword matching model. The keyword matching model determines whether to extract a category from the text by detecting whether the text contains a category name. For example, Figure 1 The Chinese text 120 contains the category name "horse", and the keyword matching model extracts the category "horse" from the text 120. Those skilled in the art should know that the category extraction model can also be a neural network model, a support vector machine, etc. for natural language processing. These models are well known to those skilled in the art and will not be described in detail here.

[0025] In one embodiment, the image target detection model 341 trained by the image target detection module 340 can be a CSD-SSD model. The CSD-SSD model is an image target detection model based on a neural network. By using images, the categories of targets contained in the images, and a limited number of labeled training samples, an effective image target detection model can be trained. The trained image target detection model 341 can detect the location and category of the targets contained in the image. Those skilled in the art should know that the image target detection model 341 can also be a Cap2Det model, a Cap2Seg model, and the like. These models and how to train these models are well known to those skilled in the art and will not be described in detail here.

[0026] In one embodiment, the image and category dual clustering module 350 is further configured to calculate the co-occurrence relationship between each image and each category using the location and category of the target contained in the image of each training sample in the obtained multiple training samples 311, and to obtain clustering results of the images of all training samples, clustering results of the categories, and image and category co-occurrence clustering results through clustering.

[0027] In one embodiment, the co-occurrence (co-occurrence) relationship between an image and a category can be related to the co-occurrence frequency between the image and the category. The co-occurrence relationship between multiple images and multiple categories can be represented by a co-occurrence frequency matrix. For example, if an image contains a category, the image and the category are considered to co-occur, and their co-occurrence frequency is 1. Figure 4 The process of calculating the co-occurrence frequency matrix between four images (including images 401-404) and four categories (including categories 411-414) is shown. Figure 4 , category 411 (cat) and category 412 (dog) appear in image 401, then the co-occurrence frequencies of image 401 with category 411 (cat) and category 412 (dog) are both 1. Category 413 (sheep) and category 414 (cow) do not appear in image 401, then the co-occurrence frequencies of image 401 with category 413 (sheep) and category 414 (cow) are both 0. Similarly, the co-occurrence frequency of image 402 with category 411 (cat) and category 412 (dog) is 1, and the co-occurrence frequency with category 413 (sheep) and category 414 (cow) is 0; the co-occurrence frequency of image 403 with category 411 (cat) and category 412 (dog) is 0, and the co-occurrence frequency with category 413 (sheep) and category 414 (cow) is 1; the co-occurrence frequency of image 404 with category 411 (cat) and category 412 (dog) is 0, and the co-occurrence frequency with category 413 (sheep) and category 414 (cow) is 1. Those skilled in the art should know that the 4 images and 4 categories described above are exemplary, and in actual applications, there can be any number of images and any number of categories. Those skilled in the art should also know that the co-occurrence relationship between images and categories can also be the square of the co-occurrence frequency between images and categories, the inverse of the co-occurrence frequency between images and categories, etc., all of which are related to the co-occurrence frequency.

[0028] In one embodiment, the co-occurrence relationship between images and categories is used to obtain an image and category co-occurrence frequency matrix, and the co-occurrence clustering result of the parameters of the matrix can be achieved by minimizing the sum of the distances (e.g., squared differences) between each parameter of the co-occurrence frequency matrix (the value of the elements of the matrix) and the center of the co-occurrence cluster to which it belongs. A co-occurrence cluster refers to a set of several co-occurrence frequencies between each image in an image cluster and each category in a category cluster. Those skilled in the art will know that the squared difference is only a specific implementation method, and other functional relationships may also be used, such as the absolute value of the difference, weighted difference, etc., which can all be considered as a manifestation of distance.

[0029] In one embodiment, the center of a co-occurrence cluster may be the average value of the parameters belonging to the co-occurrence cluster in the co-occurrence frequency matrix. Those skilled in the art may know that the average value is only a specific embodiment, and other functional relationships may also be used, such as a weighted average value, a harmonic mean value, and the like.

[0030] Figure 5 The process of co-occurrence clustering of images and categories according to an embodiment of the present invention is schematically shown. In this process, for the co-occurrence frequency matrix between multiple images and multiple categories, the sum of the distances (e.g., squared differences) between the parameters of each co-occurrence frequency matrix (i.e., the values ​​of the elements of the matrix) and the center of the co-occurrence cluster to which it belongs is minimized. In this way, two image clusters (including image clusters 501 and 502), two category clusters (including category clusters 511 and 512), and clusters of parameters of image and category co-occurrence frequency matrices (including clusters 521-524 of parameters of co-occurrence frequency matrices) can be obtained. Cluster 521 of co-occurrence frequency matrix parameters is a set of parameters of the co-occurrence frequency matrix between each image in image cluster 501 and each category in category cluster 511; cluster 522 of co-occurrence frequency matrix parameters is a set of parameters of the co-occurrence frequency matrix between each image in image cluster 502 and each category in category cluster 511; cluster 523 of co-occurrence frequency matrix parameters is a set of parameters of the co-occurrence frequency matrix between each image in image cluster 501 and each category in category cluster 512; cluster 524 of co-occurrence frequency matrix parameters is a set of parameters of the co-occurrence frequency matrix between each image in image cluster 502 and each category in category cluster 512. The center of cluster 521 of the parameters of the co-occurrence frequency matrix is ​​1, and in cluster 521 of the parameters of the co-occurrence frequency matrix, the sum of the square differences between the parameters of the co-occurrence frequency matrix and the center of cluster 521 is 0; the center of cluster 522 of the parameters of the co-occurrence frequency matrix is ​​0, and in cluster 522 of the parameters of the co-occurrence frequency matrix, the sum of the square differences between the parameters of the co-occurrence frequency matrix and the center of cluster 522 is 0; the center of cluster 523 of the parameters of the co-occurrence frequency matrix is ​​1, and in cluster 423 of the parameters of the co-occurrence frequency matrix, the sum of the square differences between the parameters of the co-occurrence frequency matrix and the center of cluster 423 is 0; the center of cluster 524 of the parameters of the co-occurrence frequency matrix is ​​0, and in cluster 524 of the parameters of the co-occurrence frequency matrix, the sum of the square differences between the parameters of the co-occurrence frequency matrix and the center of cluster 524 is 0. The sum of the square differences between the parameters of each co-occurrence frequency matrix and the center of the cluster to which it belongs is 0, which is a clustering result in which the sum of the square differences between the parameters of each co-occurrence frequency matrix and the center of the cluster to which it belongs is the smallest.

[0031] In one embodiment, the visualization module 360 ​​may be configured to display the image clusters and representative images of the image clusters for all training samples 310 , the name of each cluster in the category clusters, and the co-occurrence clusters between images and categories. Figure 6 An example of an interface displayed by a visualization module 360 ​​of a system for visualizing the output of an image object detection model trained using text corresponding to an image according to an embodiment of the present invention is shown. In one implementation, the visualization module 360 ​​may display image clusters and representative images of image clusters, the name of each cluster in the category clusters, and co-occurrence clusters between images and categories on the interface. Figure 6 Box 610 in shows examples of image clusters and representative images of image clusters, wherein each row is an image cluster and each row shows a partial representative image of the image cluster. Figure 6 Box 620 in shows the names of multiple clusters of all categories. Figure 6 In the box 630 in FIG. 1 , an example of co-occurrence clustering between images and categories is shown using connecting lines. In one embodiment, the representative image of the image cluster can be obtained by using random sampling of all images in the image cluster. Those skilled in the art will know that the representative image of the image cluster can also be obtained by using blue noise sampling or density-based sampling of all images in the image cluster. These sampling methods are well known to those skilled in the art and will not be described in detail here. Those skilled in the art will know that Figure 6 Rectangles are used to represent image clusters and representative images of image clusters, rectangles and text are used to represent the names of multiple clusters of all categories, and lines are used to represent co-occurrence clusters between images and categories. Those skilled in the art will know that other graphics, such as circles, triangles, diamonds, etc., can also be used to represent relevant information. These visualization methods are well known to those skilled in the art and will not be described in detail here.

[0032] In one embodiment, the receiving module 320 can be configured to receive a certain category cluster name displayed by the visualization module 360 ​​through a mouse, keyboard or other peripherals (here, a certain category cluster name input or selected by the user is received), and the visualization module 360 ​​displays the text of the unlabeled training sample corresponding to the category cluster name accordingly; the receiving module 320 is further configured to receive the modification of the text of the part of the unlabeled training sample displayed by the visualization module 360 ​​through a mouse, keyboard or other peripherals, and then the category extraction module 330 uses the modification of the text of the part of the unlabeled training sample displayed by the visualization module 360 ​​received by the receiving module 320 to update the unlabeled training sample 312, and obtain the updated unlabeled training sample 312', thereby extracting the category of the target contained in the image of the updated unlabeled training sample 312' using the category extraction model. The image target detection module 340 can use the category of the target contained in the image of the updated unlabeled training sample 312' to obtain an updated image target detection model 341'. Since the text is modified more accurately, the noise in the text is reduced, so that the updated image object detection model 341 ′ is more accurate than the original object detection model 341 .

[0033] In one embodiment, the receiving module 320 can be configured to receive a certain image cluster displayed by the visualization module 360 ​​through peripherals such as a mouse and a keyboard (here, a certain image cluster input or selected by the user is received), and the visualization module 360 ​​correspondingly displays the position and category of the target contained in the image of the unlabeled training sample corresponding to the image cluster; the receiving module 320 is further configured to receive modifications to the target position or category contained in the image of some unlabeled training samples displayed by the visualization module 360 ​​through peripherals such as a mouse and a keyboard, and then the image target detection module 340 uses the modifications to the target position or category contained in the image of the unlabeled training sample received by the receiving module 320 to obtain an updated image target detection model 341". Since the target position or category contained in the image of some unlabeled training samples is modified more accurately, the updated image target detection model 341" detects more accurately than the original target detection model 341.

[0034] In fact, there are multiple category cluster names and image clusters. The user can quickly select which category cluster name and which image cluster contains representative images by viewing the co-occurrence clusters between images and categories represented by connecting lines. The details will be described in detail in the following examples.

[0035] like Figure 6As shown, the first row in 610 is an image cluster. It can be seen that these images are mainly scenes on the dining table. It is reasonable for this image cluster to be connected to the category cluster names "dining room" and "fruit". However, it is unreasonable to connect it to the category cluster name "vehicle". The user first selected the category cluster name "vehicle" through this line. The text of the unlabeled training sample corresponding to the category cluster name "vehicle" is displayed in box 640. The user modifies the text of some unlabeled training samples in box 640. After the receiving module 320 receives the user's modification, the category extraction module 330 uses the user's modification received by the receiving module 320 to obtain an updated unlabeled training sample 312', thereby using the category extraction model to extract the category of the target contained in the image of the updated unlabeled training sample 312'. Then the image target detection module 340 can use the category of the target contained in the image of the updated unlabeled training sample 312' to obtain an updated image target detection model 341'. Since the text is modified more accurately, the noise in the text is reduced, so that the updated image object detection model 341 ′ is more accurate than the original object detection model 341 .

[0036] On the other hand, the user also selects the image cluster corresponding to the first row in box 610 through this line. Box 650 displays the location and category of the target contained in the image of the unlabeled training sample corresponding to the image cluster. The user modifies the location or category of the target contained in the image of some unlabeled training samples in box 650. After the receiving module 320 receives the user's modification, the image target detection module 340 uses the user's modification received by the receiving module to obtain an updated image target detection model 341". Since the location or category of the target contained in the image of some unlabeled training samples is modified more accurately, the updated image target detection model 341" is more accurate than the original target detection model 341.

[0037] When the user inputs, he can click an image cluster or a category cluster name with the mouse, or input an image cluster or a category cluster name with the keyboard, or use other peripherals to input an image cluster or a category cluster name. Figure 6 The text in box 640 can be modified by deleting, adding, or modifying keywords in the text. Figure 6 The modification of the target position or category included in the image in the frame 650 can be achieved by mouse dragging, text editing, etc. Those skilled in the art know that user input and modification can also be achieved by other methods, which will not be described here.

[0038] In the system 300, through repeated interaction between the user and the visualization module 360 ​​and the receiving module 320, the category extraction module 330, the image target detection module 340 and the image and category biclustering module 350 are iteratively run, and the image target detection model 341 can be iteratively updated to obtain a target detection model with higher accuracy.

[0039] The present invention also proposes a method for visualizing the output of an object detection model trained using text corresponding to an image. Figure 7 A flowchart 700 of a method for visualizing the output of an object detection model trained using text corresponding to an image is shown. Figure 7 As shown, in step 710, multiple training samples are received, wherein the multiple training samples include a limited number of labeled training samples and multiple unlabeled training samples, wherein one unlabeled training sample includes an image and its corresponding text, and one labeled training sample includes an image and its corresponding text, as well as objects contained in the image and their positions. In step 720, the categories of objects contained in the images corresponding to the multiple unlabeled training samples are extracted from a section of text contained in each of the multiple unlabeled training samples using the received multiple training samples. In step 730, an image object detection model is trained using the categories of objects contained in the images corresponding to the extracted multiple unlabeled training samples, as well as the images of the limited number of labeled training samples and the multiple unlabeled training samples received. In step 740, the positions and categories of objects contained in the images of the multiple unlabeled training samples are extracted using the trained image object detection model. In step 750, the positions and categories of the objects contained in the images of the obtained multiple unlabeled training samples and the positions and categories of the objects contained in the images of the received limited number of labeled training samples are used to cluster the images and categories of the multiple training samples to obtain clustering results of the images of the multiple training samples, clustering results of the categories, and image and category co-occurrence clustering results. In step 760, the representative image of each cluster in the clustering results of the images of the multiple training samples, the name of each cluster in the clustering results of the categories, and each co-occurrence cluster in the image and category co-occurrence clustering results are displayed.

[0040] In one embodiment, method 700 further includes: in step 770, in response to receiving a displayed name of a certain category cluster, displaying the text of the unlabeled training sample corresponding to the category cluster name. In one embodiment, method 700 further includes: in step 780, in response to receiving a modification of the text of a portion of the unlabeled training samples displayed, using the modification of the text of the portion of the unlabeled training samples displayed to update the unlabeled training samples, obtain an updated category extraction model, and thereby obtain an updated image object detection model. Method 700 further includes: in step 790, in response to receiving a displayed image cluster, displaying the position and category of the target contained in the image of the unlabeled training sample corresponding to the image cluster. Method 700 further includes: in step 795, in response to receiving a modification of the position or category of the target contained in the image of the portion of the unlabeled training samples, using the modification of the position or category of the target contained in the image of the unlabeled training samples, to obtain an updated image object detection model.

[0041] In one embodiment, step 750 further includes: calculating the co-occurrence relationship between each image and each category using the target position and target category contained in the image of each training sample among the obtained multiple training samples; and obtaining the clustering results of the images of the multiple training samples, the clustering results of the categories, and the co-occurrence clustering results of the images and the categories.

[0042] In one embodiment, the co-occurrence relationship between each image and each category is related to the co-occurrence frequency between each image and each category, the co-occurrence relationship between multiple images and multiple categories is represented by a co-occurrence frequency matrix, and the image and category co-occurrence clustering result is obtained by minimizing the sum of the distances between each parameter in the co-occurrence frequency matrix and the parameter corresponding to the co-occurrence cluster center to which it belongs.

[0043] In one embodiment, step 720 further includes: training a category extraction model using the limited number of labeled training samples; and extracting the category of the target contained in the image of the unlabeled training sample using the category extraction model.

[0044] In method 700 , through repeated interaction between the user and method 700 , category extraction, image object detection, and image and category biclustering are iteratively performed, and the image object detection model can be iteratively updated to obtain an object detection model with higher accuracy.

[0045] The present invention may be a system, a method and / or a computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for causing a processor to implement various aspects of the present invention. The method of the present invention may be executed on an independent computer system, on a distributed computing system, or even on a cloud platform.

[0046] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer-readable storage media according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of the boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0047] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer-readable storage medium according to multiple embodiments of the present invention. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be performed substantially in parallel, and they can also be performed in reverse order sometimes, depending on the function involved.

[0048] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system for visualizing the output of an image object detection model trained using text corresponding to an image, comprising: The receiving module is configured to receive a plurality of training samples, wherein the plurality of training samples include a limited number of labeled training samples and a plurality of unlabeled training samples, wherein an unlabeled training sample includes an image and its corresponding text, and a labeled training sample includes an image and its corresponding text, as well as an object contained in the image and its position; A category extraction module is configured to extract the category of the target contained in the image corresponding to the plurality of unlabeled training samples from a section of text contained in each training sample of the plurality of unlabeled training samples using the plurality of training samples received by the receiving module; The target detection module is configured to train an image target detection model using the target categories contained in the images corresponding to the multiple unlabeled training samples extracted from the category extraction module, and the limited number of labeled training samples received by the receiving module together with the images of the multiple unlabeled training samples, and to extract the positions and categories of the targets contained in the images of the multiple unlabeled training samples using the trained image target detection model; An image and category dual clustering module is configured to cluster the images and categories of all training samples using the locations and categories of the targets contained in the images of the multiple unlabeled training samples obtained by the image target detection model, and the locations and categories of the targets contained in the images of the limited number of labeled training samples received by the receiving module, so as to obtain clustering results of the images of the multiple training samples, clustering results of the categories, and image and category co-occurrence clustering results; as well as The visualization module is configured to display a representative image of each cluster in the clustering results of the images of the plurality of training samples, a name of each cluster in the clustering results of the categories, and each co-occurrence cluster in the image and category co-occurrence clustering results.

2. The system as claimed in claim 1, wherein the image and category dual clustering module is further configured to calculate the co-occurrence relationship between each image and each category by using the target position and target category contained in the image of each training sample in the obtained multiple training samples, and obtain the clustering results of the images of the multiple training samples, the clustering results of the categories, and the image and category co-occurrence clustering results.

3. The system as claimed in claim 2, wherein the co-occurrence relationship between each image and each category is related to the co-occurrence frequency between each image and each category, the co-occurrence relationship between multiple images and multiple categories is represented by a co-occurrence frequency matrix, and the image and category co-occurrence clustering result is obtained by minimizing the sum of the distances between each parameter in the co-occurrence frequency matrix and the parameter corresponding to the co-occurrence cluster center to which it belongs.

4. The system of claim 1, wherein the category extraction module is further configured to train a category extraction model using the limited number of labeled training samples, and to extract the category of the target contained in the image of the unlabeled training sample using the category extraction model.

5. The system according to any one of claims 1-4, wherein the visualization module is further configured to display text of unlabeled training samples corresponding to a certain category cluster name displayed by the visualization module in response to the receiving module receiving the category cluster name.

6. A system as claimed in claim 5, wherein in response to the receiving module receiving modifications to the text of the partial unlabeled training samples displayed by the visualization module, the category extraction module uses the modifications to the text of the partial unlabeled training samples displayed by the visualization module to update the unlabeled training samples and obtain an updated category extraction model, thereby enabling the object detection module to obtain an updated image object detection model.

7. A system as described in any one of claims 1-4, wherein the visualization module is further configured to display the location and category of the target contained in the image of the unlabeled training sample corresponding to the image cluster in response to the receiving module receiving a certain image cluster displayed by the visualization module.

8. A system as claimed in claim 7, wherein in response to the receiving module receiving a modification to the target position or category contained in the image of a portion of the unlabeled training samples, the image target detection model uses the modification to the target position or category contained in the image of the unlabeled training samples to obtain an updated image target detection model.

9. A method for visualizing the output of an object detection model trained using text corresponding to an image, comprising: Receive multiple training samples, wherein the multiple training samples include a limited number of labeled training samples and multiple unlabeled training samples, an unlabeled training sample includes an image and its corresponding text, and a labeled training sample includes an image and its corresponding text, as well as an object contained in the image and its position; Extracting the category of the target contained in the image corresponding to the plurality of unlabeled training samples from a section of text contained in each training sample of the plurality of unlabeled training samples using the received plurality of training samples; Training an image object detection model using the categories of objects contained in the images corresponding to the extracted multiple unlabeled training samples, and the received limited number of labeled training samples and the images of the multiple unlabeled training samples; Extracting the positions and categories of the objects contained in the images of the plurality of unlabeled training samples using the trained image object detection model; Using the positions and categories of the objects contained in the images of the plurality of unlabeled training samples obtained, and the positions and categories of the objects contained in the images of the limited number of labeled training samples received, clustering the images and categories of all the training samples to obtain clustering results of the images of the plurality of training samples, clustering results of the categories, and image and category co-occurrence clustering results; as well as A representative image of each cluster in the clustering results of the images of the plurality of training samples, a name of each cluster in the clustering results of the categories, and each co-occurrence cluster in the image and category co-occurrence clustering results are displayed.

10. The method of claim 9, wherein clustering the images and categories of all training samples using the positions and categories of the objects contained in the images of the plurality of unlabeled training samples and the positions and categories of the objects contained in the images of the limited number of labeled training samples received to obtain clustering results of the images of all training samples, clustering results of the categories, and image and category co-occurrence clustering results further comprises: Using the target position and target category contained in the image of each training sample among the obtained multiple training samples, calculate the co-occurrence relationship between each image and each category; as well as The clustering results of the images of the plurality of training samples, the clustering results of the categories, and the co-occurrence clustering results of the images and the categories are obtained.

11. The method as claimed in claim 10, wherein the co-occurrence relationship between each image and each category is related to the co-occurrence frequency between each image and each category, the co-occurrence relationship between multiple images and multiple categories is represented by a co-occurrence frequency matrix, and the image and category co-occurrence clustering result is obtained by minimizing the sum of the distances between each parameter in the co-occurrence frequency matrix and the parameter corresponding to the co-occurrence cluster center to which it belongs.

12. The method according to claim 9, wherein extracting the category of the target contained in the image corresponding to the plurality of unlabeled training samples from a section of text contained in each of the plurality of unlabeled training samples using the received plurality of training samples further comprises: Using the limited number of labeled training samples to train a category extraction model; as well as The category extraction model is used to extract the category of the target contained in the image of the unlabeled training sample.

13. The method according to any one of claims 9 to 12, further comprising: In response to receiving a displayed category cluster name, text of unlabeled training samples corresponding to the category cluster name is displayed.

14. The method of claim 13, further comprising: In response to receiving a modification to the text of the displayed portion of unlabeled training samples, updating the unlabeled training samples using the modification to the text of the displayed portion of unlabeled training samples; Obtain an updated category extraction model; as well as Get an updated image object detection model.

15. The method according to any one of claims 9 to 12, further comprising: In response to receiving a displayed image cluster, the location and category of the target contained in the image of the unlabeled training sample corresponding to the image cluster are displayed.

16. The method of claim 15, further comprising: In response to receiving a modification to a target position or category included in an image of a portion of unlabeled training samples, an updated image target detection model is obtained using the modification to the target position or category included in the image of the unlabeled training samples.

17. A computer storage medium for visualizing the output of an image object detection model trained using text corresponding to an image, on which a computer readable code is stored, and when the computer readable code is executed by a processor, the method according to any one of claims 9 to 16 is performed.

Images