Image-based detection method and related equipment

Abstract

The embodiment of the invention provides an image-based detection method and related equipment, and the method comprises the steps: obtaining a first sample set which comprises N first sample images which are not marked; calling a first detection model to perform target detection on each first sample image to obtain prediction classification information of an instance included in each first sample image, the prediction classification information being used for representing classification confidence of a prediction category to which the instance belongs and positioning confidence of a position of the instance in the corresponding first sample image; sampling the first sample set according to the prediction classification information of the instances included in the first sample images and a Gaussian mixture model associated with the N first sample images to obtain sampled sample images; training the first detection model according to the sampling sample image and annotation information annotated for the sampling sample image to obtain a second detection model; according to the embodiment of the invention, the sample image for model training can be better determined, and the model training efficiency is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image-based detection and processing method and related equipment. Background Technology

[0002] Object detection refers to the process of automatically identifying and locating specific targets in images or videos. To achieve object detection, a detection model is typically trained using a deep neural network, which is then used to detect targets in images. However, the performance of current detection models largely depends on the size of the labeled dataset; that is, increasing the amount of labeled data improves model performance. But this process requires a large amount of manual labeling of the data. For example, in object detection, manual time is needed to label the bounding boxes corresponding to targets in a large number of sample images, resulting in low model training efficiency. Summary of the Invention

[0003] This application provides an image-based detection method and related equipment, which can better determine the sample images used for model training and effectively improve the model training efficiency.

[0004] On one hand, embodiments of this application provide an image-based detection method, which includes:

[0005] Obtain the first sample set, which includes N unlabeled first sample images, where N is a positive integer;

[0006] The first detection model is invoked to perform target detection on each first sample image, and the predicted classification information of the instances included in each first sample image is obtained. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0007] Based on the predicted classification information of the instances included in each first sample image, and the Gaussian mixture model associated with N first sample images, the first sample set is sampled to obtain sampled images;

[0008] The first detection model is trained based on the sampled images and the annotation information used to annotate them, to obtain the second detection model.

[0009] On one hand, embodiments of this application provide an image-based detection device, which includes:

[0010] The acquisition unit is used to acquire a first sample set, which includes N unlabeled first sample images, where N is a positive integer;

[0011] The processing unit is used to call the first detection model to perform target detection on each first sample image and obtain the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0012] The processing unit is also used to sample the first sample set based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images.

[0013] The processing unit is also used to train the first detection model based on the sampled sample image and the annotation information for the sampled sample image to obtain the second detection model.

[0014] In one implementation, when the processing unit samples the first sample set based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images, it can specifically be used for:

[0015] Obtain the number of instances and the number of predicted categories in each first sample image;

[0016] The information content of each first sample image is determined based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories.

[0017] Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sample images.

[0018] In one implementation, when the processing unit determines the information content of each first sample image based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories, it may specifically be used to:

[0019] The information content of each instance is determined based on the predicted classification information of the instances included in each first sample image;

[0020] The information content of each first sample image is determined based on the information content of each instance, the number of instances in each first sample image, and the number of predicted categories.

[0021] In one implementation, when the processing unit samples the first sample set based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images, it can be specifically used for:

[0022] Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the N first sample images are clustered to obtain T cluster sets, each cluster set corresponding to a cluster center; T is a positive integer;

[0023] The first sample image corresponding to each cluster center is determined as the sampling sample image.

[0024] In one implementation, each first sample image includes image features, and the Gaussian mixture model includes T Gaussian distributions, each Gaussian distribution corresponding to a cluster set, where T is an integer greater than 1;

[0025] When the processing unit clusters the N first sample images based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain T cluster sets, it can be specifically used for:

[0026] Based on the information content and image features of each first sample image, determine the probability that each first sample image belongs to each Gaussian distribution;

[0027] Based on the probability that each first sample image belongs to each Gaussian distribution, the parameters of T Gaussian distributions are updated to obtain T updated Gaussian distributions;

[0028] When the parameter updates of T Gaussian distributions meet the update conditions, the probability that each first sample image belongs to each updated Gaussian distribution is determined based on the image features of each first sample image.

[0029] For a target sample image among N first sample images, determine the maximum probability from the probability of the target sample image belonging to each updated Gaussian distribution, and determine the target Gaussian distribution corresponding to the maximum probability;

[0030] Cluster the target sample images into the target cluster set corresponding to the target Gaussian distribution.

[0031] In one implementation, when the processing unit determines the probability that each first sample image belongs to each Gaussian distribution based on the information content and image features of each first sample image, it can specifically be used for:

[0032] For the target sample image among the N first sample images, the information content of the target sample image is used as the prior probability of the Gaussian mixture model.

[0033] Based on the prior probability and the image features of the target sample image, determine the probability that the target sample image belongs to each Gaussian distribution.

[0034] The processing unit is also used for:

[0035] Obtain a second sample set, which includes multiple second sample images and annotation information for each second sample image; the annotation information for the second sample images includes: the location information of the ground bounding box of each sample instance in the second sample image;

[0036] The initial detection model is invoked to perform target detection on each second sample image, and the predicted classification information of the sample instances contained in each second sample image and the position information of the predicted bounding box of each sample instance are obtained.

[0037] The quality of the bounding box corresponding to each sample instance is determined based on the location information of the ground bounding box and the location information of the predicted bounding box of each sample instance.

[0038] The loss of the initial detection model is determined based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance.

[0039] Based on the loss of the initial detection model, the parameters of the initial detection model are adjusted to obtain the first detection model.

[0040] The annotation information also includes the category labels of the sample instances; when the processing unit determines the loss of the initial detection model based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, it can be specifically used for:

[0041] Based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, the classification and localization loss of the initial detection model is determined.

[0042] Cross-entropy is calculated based on the predicted classification information of the sample instances contained in each second sample image and the category label of each sample instance to determine the cross-entropy loss of the initial detection model.

[0043] The loss of the initial detection model is determined based on the classification localization loss and cross-entropy loss of the initial detection model.

[0044] In one implementation, when the processing unit trains the first detection model based on the sampled image and the annotation information used to annotate the sampled image to obtain the second detection model, it can be specifically used for:

[0045] Update the second sample set based on the sampled images and the annotation information used to annotate them;

[0046] The first detection model is trained using the updated second sample set;

[0047] If the training of the first detection model meets the training conditions, then the trained first detection model will be determined as the second detection model.

[0048] If the training conditions for the first detection model are not met, then unlabeled first sample images are acquired to obtain a new first sample set, so that the first detection model can be trained based on the new first sample set.

[0049] The processing unit is also used for:

[0050] Based on the information content of each instance, each image region of each first sample image is distinguished and displayed;

[0051] In particular, the greater the information content of an instance, the more prominent the image region where the instance is located will be in the corresponding first sample image.

[0052] On one hand, embodiments of this application provide a computer device, the computer device comprising:

[0053] A processor is used to execute computer programs;

[0054] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image-based detection method described above.

[0055] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is loaded by a processor and executed as described above for the image-based detection method.

[0056] On the one hand, embodiments of this application provide a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, they implement the above-described image-based detection method.

[0057] In this embodiment, a first sample set is obtained, comprising N unlabeled first sample images, where N is a positive integer. A first detection model is invoked to perform object detection on each first sample image, obtaining the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image. By uniformly representing the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image using the predicted classification information, it is convenient to consider both the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image when sampling the first sample set, thus providing a sampling basis for subsequent sampling of the first sample set. Further, the first sample set is sampled according to the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images. The first detection model is trained according to the sampled sample images and the annotation information labeled for the sampled sample images to obtain a second detection model. As can be seen, by combining the Gaussian mixture model and the predicted classification information of the instance to sample the first sample set, representative sample images can be obtained, which can better determine the sample images used for model training. Compared with adding sample images to the model training, the sample images for model training can be better determined through the embodiments of this application, which can reduce the manual annotation time to a certain extent and improve the model training efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A schematic diagram of an image-based detection framework provided in an embodiment of this application;

[0060] Figure 2 An architecture diagram of an image-based detection system provided in an embodiment of this application;

[0061] Figure 3 A schematic flowchart of an image-based detection method provided in an embodiment of this application;

[0062] Figure 4 A schematic flowchart of an image-based detection method provided in an embodiment of this application;

[0063] Figure 5A schematic diagram of another image-based detection framework provided for an embodiment of this application;

[0064] Figure 6 This is a schematic diagram showing the image area where an example is located, provided in an embodiment of this application.

[0065] Figure 7a A schematic diagram comparing an image-based detection method provided in this application with other active learning methods;

[0066] Figure 7b A schematic diagram comparing another image-based detection method provided in this application embodiment with other active learning methods;

[0067] Figure 7c A schematic diagram comparing another image-based detection method provided in this application embodiment with other active learning methods;

[0068] Figure 8 A schematic diagram of the structure of an image-based detection device provided in an embodiment of this application;

[0069] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] First, the technical terms involved in the embodiments of this application will be explained.

[0072] I. Artificial Intelligence (AI)

[0073] AI (Artificial Intelligence) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0074] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0075] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0076] This application relates to computer vision (CV), which can be used to detect objects in images to obtain instances in the images. These instances can include people, animals, etc., and this application does not limit them in any way.

[0077] II. Target Detection

[0078] Object detection is one of the core problems in computer vision (CV). Object detection refers to finding objects of interest in an image and determining their category and location. In other words, object detection includes two parts, such as... Figure 1 The first part is to classify and predict the target (hereinafter referred to as instance) in the image to obtain the predicted category to which the instance belongs; the second part is to perform regression prediction on the instance in the image to determine the location of the instance. Specifically, the location of the instance includes the regression bounding box corresponding to the instance.

[0079] III. Active Learning (AL)

[0080] Active learning aims to select fewer samples from an unlabeled dataset for annotation. Specifically, active learning can annotate the most informative and diverse samples in the unlabeled dataset. Active learning can include the following processes: (1) Object detector training: training the object detector using an annotated sample set; (2) Evaluating unlabeled sample images; (3) Selecting valuable images from the unlabeled sample images; (4) Manually annotating the selected sample images and iteratively training the object detector based on the annotated sample images. Through these four processes, the accuracy of sample image selection can be improved.

[0081] In this embodiment, the classification confidence and localization accuracy of instances in the sample image can be considered in process (2), so that classification uncertainty and regression uncertainty are considered simultaneously in process (2), and then the classification uncertainty and regression uncertainty are unified as the uncertainty of the sample image. In process (3), when selecting valuable images, a Gaussian mixture model can be used to unify the uncertainty and diversity of sample images to accurately represent the value of unlabeled sample images, so as to select valuable sample images. The so-called valuable sample images refer to sample images with information content and diversity.

[0082] It should be understood that classification uncertainty can be used as an indicator of the confidence level of an instance's predicted category, while regression uncertainty can be used as an indicator of the confidence level of an instance's location in the first sample image. Generally, the lower the uncertainty, the higher the confidence level. For example, for an image, if the classification confidence level of an instance belonging to predicted category 1 (e.g., cat) is 0.5, and the classification confidence level of an instance belonging to predicted category 2 (e.g., dog) is also 0.5, because the classification confidence levels are close, it is impossible to determine whether the instance belongs to predicted category 1 or predicted category 2. This means that the classification uncertainty of the instance's predicted category is high, and correspondingly, the confidence level of the instance belonging to either predicted category 1 or predicted category 2 is low. On the other hand, if the classification confidence level of an instance belonging to predicted category 1 (e.g., cat) is 0.99, and the classification confidence level of an instance belonging to predicted category 2 (e.g., dog) is 0.01, it means that there is a high probability that the instance belongs to predicted category 1. This means that the classification uncertainty of the instance's predicted category is low, and the confidence level of the instance belonging to predicted category 1 is high.

[0083] IV. Predictive classification information of instances and information content of sample images

[0084] The predicted classification information of an instance is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence (i.e., localization accuracy) of the instance's position in the corresponding sample image (such as the first sample image). In other words, this embodiment of the application jointly represents the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding sample image as the predicted classification information of the instance. The predicted classification information may include a predicted classification score or a predicted classification confidence.

[0085] Among them, the classification confidence of the predicted category to which the instance belongs reflects the ability of the detection model to accurately predict the category of the instance, and the location confidence of the instance's position in the corresponding sample image reflects the ability of the detection model to accurately locate the instance.

[0086] The information content of a sample image is determined based on the predicted classification information of each instance in the sample image. For example, if a sample image contains 5 instances, its information content can be determined based on the predicted classification information of these 5 instances. In this embodiment, the information content of each instance can be determined based on its predicted classification information. This information content indicates the uncertainty of the corresponding instance, which includes the classification uncertainty of the predicted category to which the instance belongs and the regression uncertainty of the instance's position in the sample image. Therefore, the information content of the sample image corresponding to each instance is determined based on its information content. This information content of the sample image can be used to represent the uncertainty of the sample image.

[0087] For example, for a sample image, the predicted classification information 1 of an instance represents the classification confidence of the instance belonging to predicted category 1 and the location confidence of the instance at position 1 in the corresponding sample image (such as the first sample image); the predicted classification information 2 of an instance represents the classification confidence of the instance belonging to predicted category 2 and the location confidence of the instance at position 2 in the corresponding sample image (such as the first sample image). If the predicted classification information 1 of an instance is 0.5, the predicted classification information 2 of the same instance is also 0.5. Since the two predicted classification information values ​​of the instance are close, it can be determined that the uncertainty of the instance is high. As another example, if the predicted classification information 1 of an instance is 0.99, the predicted classification information 2 of the same instance is 0.01. Since the predicted scores of the two instances differ significantly, it is highly likely that the instance belongs to predicted category 1, which means that the uncertainty of the instance is low.

[0088] V. Gaussian Mixture Model (GMM)

[0089] Gaussian Mixture Models (GMMs) are a clustering method based on probability density functions. A GMM is a linear combination of multiple Gaussian distributions. The goal of a GMM is to find a set of parameters, including the mean, variance, and weights of each Gaussian distribution, estimated by maximizing the likelihood function of the data. Unlike some other clustering methods, GMMs assign data points to different cluster sets based on probability.

[0090] In this embodiment, the information content of the first sample image can be used as the prior probability (i.e., the weight of the Gaussian distribution) of the Gaussian mixture model. The Gaussian mixture model is then used to assign the first sample image to the corresponding cluster set according to the probability that it belongs to each Gaussian distribution. The first sample image is then selected from the cluster set for annotation. This results in a more diverse and information-rich selection of the first sample image. In one implementation, the first sample image corresponding to the cluster center can be selected for annotation.

[0091] This application provides an image-based detection scheme, which can be a Unified Active Object Detector (UAOD) based scheme. Specifically, the image-based detection scheme can include: selecting sample images for training the detection model based on unified uncertainty and unified sampling, labeling the selected sample images, and training the detection model based on the labeled sample images. The image-based detection scheme provided by this application can select a small number of valuable sample images from a large number of unlabeled sample images, meaning that the selected sample images have relatively rich information content. This can reduce labeling costs and improve model training efficiency. Furthermore, this selection method can achieve higher model accuracy compared to random selection or all selections, thus improving the performance of the finally trained detection model.

[0092] Please see Figure 1 This is a schematic diagram of an image-based detection framework provided in an embodiment of this application. Figure 1In this paper, two key modules are introduced into the image-based detection framework: a unified uncertainty module and a unified sampling module. The unified uncertainty module is responsible for determining the predicted classification information of each instance in the sample image by using a joint representation of instance existence confidence (i.e., the classification confidence of the predicted category to which the instance belongs) and location confidence, thus simultaneously considering regression uncertainty and classification uncertainty. The unified sampling module combines uncertainty- and diversity-based sampling into a unified sampling strategy. Specifically, it uses a Gaussian mixture model to identify informative and representative instances from dense regions in the feature space (i.e., dense regions constructed from image features corresponding to multiple sample images), thereby identifying informative and representative sample images.

[0093] In one embodiment, N unlabeled first sample images can be obtained, where N is a positive integer; the first detection model (such as...) can be called. Figure 1 The detection model in the first sample image performs target detection on each first sample image, obtaining the predicted classification information of the instances included in each first sample image. This predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the position confidence of the instance in the corresponding first sample image. Then, the unified uncertainty module determines the information degree of each first sample image based on the predicted classification information of the instances included in each first sample image. The information degree of the first sample image is used to represent the uncertainty of the first sample image, which includes regression uncertainty and classification uncertainty. The unified sampling module samples the N first sample images based on the information degree of each first sample image and the Gaussian mixture model associated with the N first sample images, obtaining sampled sample images. Based on the sampled sample images and the annotation information for the sampled sample images, the first detection model is trained to obtain the second detection model. Specifically, the N first sample images are clustered based on the information degree of each first sample image and the Gaussian mixture model associated with the N first sample images, resulting in T cluster sets. Each cluster set corresponds to a cluster center, and the first sample image corresponding to the cluster center is used as the sampled sample image. Figure 1 In this process, N first sample images can be clustered to obtain 3 cluster sets, and the first sample image corresponding to the cluster center of the cluster set is used as the sampling sample image.

[0094] In summary, after determining the predicted classification information, this embodiment can determine the information content of the first sample image based on the predicted classification information. This information content is used to represent the uncertainty of the first sample image, thereby simultaneously considering both the regression uncertainty and classification uncertainty of the sample image and unifying them into the uncertainty of the first sample image. This information content of the first sample image provides a basis for selecting sample images to be labeled later. Furthermore, when sampling the first sample image, N first sample images can be uniformly sampled based on both their information content and diversity. This solves the problem of sampling redundancy that may result from solely relying on uncertainty, or the problem of overemphasizing cluster centers and neglecting truly representative instances when relying solely on diversity. This allows for the selection of fewer information-rich sample images from the first sample image for labeling, reducing labeling costs and improving model training efficiency. Simultaneously, this embodiment eliminates the risk of inconsistency by combining predicted classification information and the diversity sampling brought by Gaussian distribution, and accurately describes the value of unlabeled sample images. This enables the identification of information-rich sample images for labeling, which to some extent improves model accuracy and enhances the target detection accuracy of the detection model.

[0095] The image-based detection system provided in the embodiments of this application will be described in detail below.

[0096] Please see Figure 2 This application provides an architecture diagram of an image-based detection system. It includes a terminal device 201 and a server 202. It should be understood that this application does not limit the number of terminal devices and servers. The terminal device 201 and the server 202 can be directly or indirectly connected via wired or wireless communication.

[0097] Terminal device 201 can provide a target detection interface, where an image to be detected can be input. The image to be detected includes at least one instance, such as an animal, a person, etc. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart vehicle, smart wearable device, etc.

[0098] Server 202 can be used to perform object detection on the image to be detected. A second detection model, trained on the first detection model, can be deployed on server 202. Server 202 can call the second detection model to perform object detection on the image to be detected. Server 202 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining the required resources in an on-demand and easily scalable manner through the network; broadly speaking, cloud computing refers to the delivery and usage model of services, meaning obtaining the required services in an on-demand and easily scalable manner through the network. These services can be IT and software, internet-related, or other services. Cloud computing is a product of the convergence and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly evolved. Unlike previous parallel and distributed computing, cloud computing will fundamentally revolutionize the entire internet model and enterprise management model. Cloud storage is a new concept that extends and develops from cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of various types of storage devices (also called storage nodes) in a network through application software or application interfaces to work collaboratively and provide data storage and business access functions.

[0099] In one embodiment, combined Figure 2 The diagram illustrates the image-based detection process between the terminal device and the server. This process includes model training and model application.

[0100] (I) Model Training Process: Specifically, a first detection model can be trained using labeled second sample images. Then, this first detection model is used to find a small number of valuable sample images from a large number of unlabeled first sample images for labeling. The model training process includes at least the following four steps:

[0101] (1) Server 202 uses the labeled second sample image to train the initial detection model to obtain the first detection model.

[0102] (2) Server 202 calls the first detection model to perform inference on the unlabeled first sample image: Specifically, server 202 can call the first detection model to perform target detection on the unlabeled first sample image and obtain the predicted classification information of the instances included in the first sample image. The predicted classification score is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0103] (3) Sampling N first sample images based on a unified sampling strategy: Specifically, server 202 can sample N first sample images based on the predicted classification information of the instances included in the first sample images and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images. It should be understood that server 202 can determine the information degree of each instance based on the predicted classification information of the instances included in the first sample images through the Gaussian mixture model. The information degree of the instance is used to represent the uncertainty of the instance. The uncertainty of the instance includes the classification uncertainty of the predicted category to which the instance belongs and the regression uncertainty of the instance's position in the first sample image. Further, the information degree of the corresponding first sample image can be determined according to the information degree of each instance. The information degree of the first sample image is used to represent the uncertainty of the first sample image. Then, the first sample set is sampled according to the information degree of the first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images. In this way, the sampling based on uncertainty and the sampling based on diversity are unified through the Gaussian mixture model to obtain sampled sample images with information content and diversity.

[0104] (4) The sampled images are labeled. The server 202 updates the second sample set according to the labeled sampled information and trains the first detection model according to the updated second sample set to obtain the second detection model.

[0105] (II) Application Phase

[0106] ① The object inputs the image to be detected through the terminal device 201. Correspondingly, the terminal device 201 can obtain the input image to be detected and generate a detection request based on the image to be detected. The detection request is used to request the server 202 to perform target detection on the image to be detected.

[0107] ② The server 202 receives the detection request sent by the terminal device 201 and responds to the detection request by calling the second detection model to perform target detection on the image to be detected.

[0108] In summary, the embodiments of this application can jointly represent the classification confidence of the predicted category to which an instance belongs and the location confidence of the instance in the corresponding first sample image as the predicted classification information of the instance included in the first sample image. This can simultaneously consider the predicted category to which the instance belongs and the location of the instance, providing a basis for subsequent sampling of the first sample image. Furthermore, based on the predicted classification information of the instances included in the first sample image and the Gaussian mixture model associated with the N first sample images, sampling of the N first sample images is achieved, enabling the selection of sample images for annotation from the N unannotated first sample images. This allows for better determination of the sample images used for model training, which also reduces annotation costs to a certain extent and improves model training efficiency. In addition, by using the predicted classification information of the instances included in the first sample image and the Gaussian mixture model, it is possible to select sample images with information content from the N first sample images, which can achieve higher model accuracy compared to selecting all N first sample images.

[0109] It should be noted that the above image-based detection interaction process is for illustrative purposes only and does not limit the specific execution process of the terminal device and server. Optionally, the terminal device 201 can execute the above model training process and application process. Alternatively, the server 202 can execute the above model training process, deploy a second detection model on the terminal device 201, and execute the application process.

[0110] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0111] The image-based detection method provided in the embodiments of this application will be described in detail below.

[0112] Please see Figure 3 This is a flowchart illustrating an image-based detection method provided in an embodiment of this application. The image-based detection method can be executed by a computer device, which can be a terminal device or a server in the aforementioned system. The image-based detection method may include the following steps S301-S304:

[0113] S301. Obtain the first sample set, which includes N unlabeled first sample images, where N is a positive integer. The first sample set can come from an object detection benchmark dataset, such as MSCOCO or Pascal VOC.

[0114] S302. Call the first detection model to perform target detection on each first sample image, and obtain the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0115] The instance (also referred to as the target) can include people, animals, etc., and the first sample image can include at least one instance. It should be understood that the location of the instance in the corresponding first sample image can include the regression bounding box (or predicted bounding box) of the instance in the corresponding first sample image.

[0116] S303. Based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sampled images.

[0117] Here, the sampled image refers to the sample image that needs to be labeled for training the first detection model. In one implementation, step S303 may specifically include the following steps s1-s3:

[0118] s1. Obtain the number of instances and the number of predicted categories in each first sample image. For example, the first sample image includes 4 instances, the number of instances in the first sample image is 4, and the number of predicted categories for class prediction of the instances in the first sample image can be, for example, 5.

[0119] s2. Determine the information content of each first sample image based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories. In one implementation, the information content of each instance can be determined based on the predicted classification information of the instances included in each first sample image, and then the information content of each first sample image can be determined based on the information content of each instance, the number of instances in each first sample image, and the number of predicted categories. The information content of an instance is used to represent the uncertainty of the instance. This uncertainty can include the uncertainty of the predicted category to which the instance belongs and the uncertainty of the instance in the first sample image. Therefore, in this embodiment, the uncertainty of the predicted category to which the instance belongs and the uncertainty of the instance in the first sample image are uniformly represented by the uncertainty of the instance. This facilitates subsequent sampling of the first sample images to obtain the most informative first sample images. It should be understood that after the first sample image is input into the first detection model, due to the uncertainty of the predicted category and the uncertainty of the position of a certain instance in the first sample image, the first sample image output by the first detection model will ultimately contain multiple predicted classification information for the instances. In this case, the formula for calculating the information content of the first sample image is as follows:

[0120]

[0121] In formula (1), U(I) represents the information content of the first sample image, and M I Let p represent the number of instances in the first sample image, i represent the i-th instance obtained from object detection in the first sample image, C' represent the number of predicted categories, j represent the j-th predicted category, and p represent the number of instances in the first sample image. ij This represents the predicted classification information corresponding to the i-th instance belonging to the j-th predicted category. `log()` represents the logarithmic operation. `-p` ij ·log(p ij ) represents the information level of the i-th instance.

[0122] s3. Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, sample the first sample set to obtain sampled images.

[0123] In this implementation, when sampling the first sample images in the first sample set, this embodiment uses a Gaussian mixture model combined with the information content of each first sample image to accurately represent the value of the first sample image. In one implementation, sampling the first sample set based on the information content of each first sample image and the Gaussian mixture model associated with N first sample images yields sampled sample images. This includes the following steps: clustering the N first sample images based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain T cluster sets, each cluster set corresponding to a cluster center; T is an integer greater than 1; and determining the first sample image corresponding to each cluster center as the sampled sample image. It can be seen that by using the Gaussian mixture model and the information content of each first sample image, the first sample images can be clustered. The first sample image corresponding to the cluster center can be considered the most representative and diverse sample image in the cluster set. The first sample image corresponding to the cluster center can be used as the sampled sample image to train the model. Thus, the diverse sampled sample images selected from multiple cluster sets can improve the model's performance to a certain extent when used for model training.

[0124] The first sample image includes image features. In this embodiment, the EM algorithm can be used to iteratively solve the problem based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to achieve clustering of the N first sample images. In one implementation, the Gaussian mixture model includes T Gaussian distributions; the N first sample images are clustered based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain T cluster sets, including the following steps 1-4:

[0125] Step 1: Based on the information content and image features of each first sample image, determine the probability that each first sample image belongs to each Gaussian distribution.

[0126] In this embodiment, the information content of a single sample image can be used as the prior probability of the Gaussian mixture model. Specifically, taking any first sample image as an example, for a target sample image among N first sample images, the information content of the target sample image is used as the prior probability of the Gaussian mixture model. Based on the prior probability and the image features of the target sample image, the probability that the target sample image belongs to each Gaussian distribution is determined. The modeling of using the information content of a single first sample image as the prior probability of the Gaussian mixture model is as follows:

[0127]

[0128] In formula (2), P(X; {U i μ i, ∑ i}) represents the probability of belonging to the i-th Gaussian distribution in the Gaussian mixture model, X represents the image feature of the first sample image, and N(X|μ) i , ∑ i ) refers to the i-th Gaussian distribution with mean μ and variance ∑. Ui is the prior probability of the Gaussian mixture model. T represents the number of Gaussian distributions in the Gaussian mixture model, and i represents the i-th Gaussian distribution.

[0129] Step 2: Update the parameters of the T Gaussian distributions based on the probability that each first sample image belongs to each Gaussian distribution, and obtain the T updated Gaussian distributions.

[0130] In this embodiment, the likelihood function is obtained by updating the parameters of T Gaussian distributions based on the likelihood function and the probability that each first sample image belongs to each Gaussian distribution. Specifically, the parameters of the T Gaussian distributions are updated based on the probability that each first sample image belongs to each Gaussian distribution to maximize the likelihood function. The formula for maximizing the likelihood function is as follows:

[0131]

[0132] In formula (3), Dt represents the first sample set, Xk represents the k-th first sample image in the first sample set, and argmax represents the maximization of the likelihood function. Let T represent the parameters of the T Gaussian distributions in the Gaussian mixture model when the likelihood function is maximized. Let be the likelihood function.

[0133] Step 3: When the parameter updates of the T Gaussian distributions meet the update conditions, determine the probability that each first sample image belongs to each updated Gaussian distribution based on the image features of each first sample image.

[0134] Specifically, if the updated T Gaussian distributions converge or the number of updates reaches the preset number of updates, then the parameter update of the T Gaussian distributions is determined to meet the update condition. Otherwise, the parameter update of the T Gaussian distributions is determined to not meet the update condition. In this case, it is necessary to continue to determine the probability that each first sample image belongs to the updated Gaussian distribution based on the image features of each first sample image, and continue to update the parameters of the updated Gaussian distribution based on the probability that each first sample image belongs to the updated Gaussian distribution, until the updated T Gaussian distributions converge or the number of updates reaches the preset number of updates.

[0135] Step 4: For the target sample image among the N first sample images, determine the maximum probability from the probability of the target sample image belonging to each updated Gaussian distribution, determine the target Gaussian distribution corresponding to the maximum probability, and cluster the target sample images into the target cluster set corresponding to the target Gaussian distribution. Finally, the N first sample images can be clustered into the cluster sets corresponding to the T Gaussian distributions.

[0136] For illustration, the probability of a target sample image belonging to updated Gaussian distribution 1 is 0.2, and the probability of a target sample image belonging to updated Gaussian distribution 2 is 0.4. Therefore, the target sample images can be clustered into the cluster set corresponding to updated Gaussian distribution 2. It should be understood that the clustering process for each first sample image can refer to the clustering process for the target sample images, and will not be repeated here.

[0137] S304. Based on the sampled images and the annotation information for the sampled images, train the first detection model to obtain the second detection model.

[0138] Specifically, the sampled images can be manually labeled to obtain the labeling information for the sampled images. Then, the sampled images and the labeling information for the sampled images can be directly input into the first detection model to train the first detection model and obtain the second detection model.

[0139] In this embodiment, a first sample set is obtained, comprising N unlabeled first sample images, where N is a positive integer. A first detection model is invoked to perform object detection on each first sample image, obtaining predicted classification information for the instances included in each first sample image. This predicted classification information represents the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image. Therefore, this embodiment considers both the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image. By jointly representing these two aspects with predicted classification information, a basis can be provided for subsequent sampling of the first sample set, effectively ensuring that suitable sample images can be sampled to a certain extent. Further, the first sample set is sampled based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images, resulting in sampled sample images. The first detection model is trained based on the sampled sample images and the annotation information used to annotate them, resulting in a second detection model. As can be seen, by combining the Gaussian mixture model and the predicted classification information of the instances to sample the first sample set, representative sample images can be obtained, which can better determine the sample images used for model training. This can reduce the cost of manual annotation and improve the efficiency of model training.

[0140] Please see Figure 4 This is a schematic flowchart of an image-based detection method provided in an embodiment of this application. The image-based detection method can be executed by a computer device, which can be a terminal device or a server in the aforementioned system. The image-based detection method may include the following steps S401-S409:

[0141] S401. Obtain the second sample set, which includes multiple second sample images and annotation information for each second sample image.

[0142] The second sample image includes at least one sample instance. The annotation information for the second sample image includes the location information of the ground truth bounding box of each sample instance in the second sample image. Furthermore, the annotation information for the second sample image may also include the category label of the sample instance. (Illustrative example, such as...) Figure 5 As shown, the category label of a sample instance can be represented in binary, such as... Figure 5 The values ​​are represented as 0…1…0, where “0” indicates that the sample instance does not belong to a certain category, and “1” indicates that the sample instance belongs to a certain category.

[0143] S402. Call the initial detection model to perform target detection on each second sample image, and obtain the predicted classification information of the sample instances contained in each second sample image and the position information of the predicted bounding box of each sample instance.

[0144] S403. Determine the quality of the bounding box corresponding to each sample instance based on the location information of the true bounding box of each sample instance and the location information of the predicted bounding box of each sample instance.

[0145] Specifically, for any sample instance, the Intersection over Union (IOU) can be calculated based on the location information of the ground truth bounding box and the location information of the predicted bounding box of the sample instance to obtain the quality of the bounding box corresponding to the sample instance.

[0146] S404. Determine the loss of the initial detection model based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance.

[0147] In this embodiment, a classification localization loss is designed to assist in supervising the initial detection model. Specifically, the loss of the initial detection model is determined based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, including steps (1) to (3):

[0148] Step (1): Based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, determine the classification localization loss of the initial detection model. The formula for determining the classification localization loss is as follows:

[0149]

[0150] In formula (4), L uul (p,q) represents the classification localization loss, where p represents the predicted classification information and q represents the bounding box quality.

[0151] Step (2): Calculate the cross-entropy based on the predicted classification information of the sample instances contained in each second sample image and the category label of each sample instance, and determine the cross-entropy loss of the initial detection model.

[0152] Step (3): Determine the loss of the initial detection model based on the classification localization loss and cross-entropy loss of the initial detection model. In one implementation, weights can be introduced, and the loss of the initial detection model can be determined based on the weights, the classification localization loss, and the cross-entropy loss of the initial detection model. The formula for calculating the loss of the initial detection model is as follows:

[0153] L=αL uul +L ce Formula (5)

[0154] In formula (5), L represents the loss of the initial detection model, L ce α represents the cross-entropy loss of the initial detection model, and α represents the weight. α can be used to balance the cross-entropy loss and the classification and localization loss.

[0155] S405. Based on the loss of the initial detection model, adjust the parameters of the initial detection model to obtain the first detection model.

[0156] S406. Obtain the first sample set, which includes N unlabeled first sample images, where N is a positive integer.

[0157] S407. Call the first detection model to perform target detection on each first sample image, and obtain the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0158] S408. Based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sampled images.

[0159] The specific implementation methods of steps S406-S408 can be found in the specific implementation methods of steps S301-S303 above, and will not be repeated here.

[0160] S409. Based on the sampled images and the annotation information for the sampled images, train the first detection model to obtain the second detection model.

[0161] Specifically, step S409 may include: updating the second sample set based on the sampled images and the annotation information for the sampled images; and training the first detection model using the updated second sample set to obtain the second detection model.

[0162] In one implementation, the first detection model is trained using the updated second sample set to directly obtain the second detection model. In another implementation, to obtain a second detection model with better performance, sample images can be continuously selected from unlabeled sample images in this embodiment. Specifically, training the first detection model using the updated second sample set to obtain the second detection model may include: determining whether the training of the first detection model meets the training conditions; if the training of the first detection model meets the training conditions, the trained first detection model is determined as the second detection model; if the training of the first detection model does not meet the training conditions, unlabeled first sample images are obtained to obtain a new first sample set, so as to complete the training of the first detection model based on the new first sample set. Specifically, after obtaining the new first sample set, new sample images can be obtained again according to the specific implementation of steps S406-S409, and the trained first detection model can be trained based on the new sample images and the annotation information for the new sample images to obtain the second detection model. It should be understood that the training of the first detection model based on the new sampled images and the annotation information for the new sampled images to obtain the second detection model can be found in the specific implementation process of training the first detection model based on the sampled images and the annotation information for the sampled images, which will not be repeated here.

[0163] The determination of whether the training of the first detection model meets the training conditions can be made by: determining whether the trained first detection model converges, or determining whether the number of training iterations has reached the target number. If the trained first detection model converges or the number of training iterations reaches the target number, then the training of the first detection model is determined to meet the training conditions; if the trained first detection model does not converge or the number of training iterations does not reach the target number, then the training of the first detection model is determined to not meet the training conditions.

[0164] In summary, the embodiments of this application can continuously sample unlabeled first sample images based on the predicted classification information of instances in each first sample image and the Gaussian mixture model, thereby continuously sampling effective and representative sample images. In this way, the first detection model can be trained based on the continuously sampled sample images, which can improve the performance of the final second detection model.

[0165] It should be understood that the initial detection model provided in this application embodiment can perform target detection on the second sample image to obtain the predicted classification information of the sample instance in the second sample image and the location information of the bounding box of the instance in the second sample image. Furthermore, by introducing the localization and classification loss function shown in the above formula (4) during the training process of the initial detection model, this application embodiment can enable the first detection model obtained by training the initial detection model using the second sample image to perform target detection on the first sample image. The predicted classification information of the instance output after performing target detection on the first sample image integrates the classification uncertainty and the location uncertainty. In this way, the sampled image used for model training can be better selected by combining the predicted classification information, so that the second detection model finally trained can be used to detect instances in any image. In one implementation, after obtaining the second detection model, a detection request can be received, which includes the image to be detected. In response to the detection request, the second detection model is called to perform target detection on the image to be detected to obtain the instance in the image to be detected. Specifically, calling the second detection model to perform object detection on the image to be detected and obtaining the instances in the image to be detected may include: calling the second detection model to perform object detection on the image to be detected and obtaining the predicted classification information of the instances in the image to be detected, as well as the location information of the bounding boxes of the instances; and determining the predicted category of each instance based on the predicted classification information of the instances contained in the image to be detected.

[0166] Optionally, embodiments of this application can visualize the image region where each instance is located based on the information level of each instance. Specifically, embodiments of this application can determine the information level of each instance based on the predicted classification information of the instances contained in each first sample image, and distinguish and display each image region of each first sample image according to the information level of each instance; wherein, the distinguished display includes: if the information level of an instance is higher, the image region where the instance is located in the corresponding first sample image is highlighted. If the information level of an instance is lower, the image region where the instance is located in the corresponding first sample image is displayed normally. Highlighting may include, for example, highlighting, magnification, etc. It should be understood that the information level of an instance is used to represent the uncertainty of the instance. The higher the uncertainty, the more the image region where the instance is located will be highlighted, for example, if Figure 6As shown, this is a schematic diagram of the image area display of an example provided in an embodiment of this application; the greater the information content of the horse in the first sample image 61, the greater the uncertainty of the horse, so the image area 62 where the horse is located in the first sample image 61 is highlighted (e.g., Figure 6 The diagonal box indicates that the image area is highlighted.

[0167] In this embodiment, a second sample set is obtained, which includes multiple second sample images and annotation information for each second sample image. Then, an initial detection model is invoked to perform object detection on each second sample image, obtaining the predicted classification information of the sample instances contained in each second sample image and the position information of the predicted bounding boxes of each sample instance. Based on the position information of the ground truth bounding boxes of each sample instance and the position information of the predicted bounding boxes of each sample instance, the quality of the bounding boxes corresponding to each sample instance is determined. Based on the predicted classification information of each sample instance and the quality of the bounding boxes corresponding to each sample instance, the loss of the initial detection model is determined. Based on the loss of the initial detection model, the parameters of the initial detection model are adjusted to obtain a first detection model. As can be seen, the embodiments of this application calculate the model loss by considering the predicted classification information of each sample instance and the bounding box quality corresponding to each sample instance. This enables the injection of the positional quality (i.e., bounding box quality) of the sample instance into the predicted category of the sample instance. This allows the first detection model, which is trained on the initial detection model, to output the predicted classification information of the instance after performing object detection on the first sample image, to integrate the classification uncertainty and the positional uncertainty. This allows the predicted classification information of the first detection model to be better used for subsequent sample image sampling. Further, a first sample set is obtained, comprising N unlabeled first sample images, where N is a positive integer. A first detection model is invoked to perform object detection on each first sample image, obtaining predicted classification information for the instances included in each first sample image. This predicted classification information represents the classification confidence of the instance's predicted category and the location confidence of the instance's position in the corresponding first sample image. Based on the predicted classification information of the instances included in each first sample image, and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sampled images. Based on the sampled images and the annotation information used to annotate them, the first detection model is trained to obtain a second detection model. It is evident that by combining the Gaussian mixture model and the predicted classification information of the instances to sample the first sample set, a small number of representative sampled images can be obtained, thereby reducing manual annotation costs and improving model training efficiency.

[0168] In an alternative embodiment, please refer to Figure 5 This is a schematic diagram of another image-based detection framework provided in an embodiment of this application. Next, we will combine... Figure 5 The image-based detection framework shown illustrates the image-based detection process provided in the embodiments of this application:

[0169] (1) Use the labeled second sample image to test the initial detection model (i.e. Figure 5 The detection model 51) is trained. During the training process, specifically, such as... Figure 5 In this process, the second sample image can be input into the detection model 51 to obtain the predicted classification information of each sample instance in the second sample image output by the detection model 51 and the position information of the predicted bounding box of each sample instance; then, the intersection-union ratio (IUU) between the predicted bounding box of the sample instance contained in the second sample image and the bounding box label (i.e., the ground truth bounding box) is calculated as the bounding box quality corresponding to the sample instance; then, based on the predicted classification information of each sample instance and the bounding box quality corresponding to each sample instance, the loss of the detection model 51 is determined, and based on the loss of the detection model 51, the parameters of the detection model 51 are adjusted to obtain the first detection model (i.e., as shown in the figure). Figure 5 Detection model 52 in the middle.

[0170] (2) The first detection model is inferred using an unlabeled first sample set. Specifically, the first sample images from the first sample set are input into the first detection model (e.g., ...). Figure 5 In the detection model 52), target detection is performed on the first sample image to obtain the predicted classification information of the instances contained in the first sample image, such as... Figure 8 In this process, based on the predicted classification information of each instance, the information degree of each instance can be determined. The information degree of an instance represents its uncertainty, which includes the classification uncertainty of the predicted category to which the instance belongs and the regression uncertainty of the instance's position in the corresponding first sample image. By unifying the classification uncertainty and regression uncertainty to obtain the uncertainty of the instance, a basis can be provided for subsequent sampling of the first sample set, thereby extracting sample images with richer information.

[0171] (3) Sample image selection based on unified uncertainty and unified sampling: such as Figure 5 In this context, "unified uncertainty" refers to the fact that, based on the predicted classification information of the instances contained in each first sample image, the information content of each first sample image can be determined, and this information content is used to indicate the uncertainty of the first sample image. "Unified sampling" refers to using a Gaussian mixture model and the information content of each sample image to cluster N first sample images, resulting in T cluster sets, such as... Figure 5 In this embodiment of the application, N first sample images (such as those from an unlabeled sample image pool) can be clustered, such as... Figure 5 In this process, the N first sample images are clustered into two cluster sets respectively. Figure 5 (As shown in 53 and 54, this is the process of clustering the first sample images), which ultimately yields the following results: Figure 5 The two cluster sets shown are cluster set A and cluster set B, where cluster set A includes sample image A and sample image C; and cluster set B includes sample image B and sample image C.

[0172] Furthermore, a first sample image that is diverse and informative can be selected from the T cluster sets as the sampling sample image. For example, a first sample image with low weight but high diversity can be selected as the sampling sample image, rather than a first sample image with high weight but low diversity. Specifically, the cluster centers corresponding to the cluster sets generally have information and diversity, and the first sample image corresponding to the cluster center can be sampled as the sampling sample image.

[0173] It should be understood that by unifying uncertainty-based sampling and diversity sampling through Gaussian mixture models, information-rich and diverse sample images can be selected from regions with dense image features formed by the first sample images, thereby improving model performance and accuracy.

[0174] (4) Manual annotation: After sampling is completed, the sampled images can be manually annotated to obtain the annotation information of the sampled images.

[0175] (5) Update the second sample set by updating the second sample set according to the annotation information of the sampled sample image and the sampled sample image.

[0176] (6) Based on the updated second sample set, retrain the first detection model. At this time, the trained first detection model can be regarded as the new first detection model. Repeat steps (2)-(6) until the model converges or the number of training times of the model reaches the target number, and obtain the second detection model.

[0177] It should be understood that in this embodiment, a preliminary first detection model is first trained using a labeled second sample set. This allows the predicted classification information output by the first detection model to incorporate classification uncertainty and location uncertainty. Then, the first detection model is used to search for a small number of valuable sample images from a large number of unlabeled second sample images for manual annotation of the labeled objects. The sample images with a small number of valuable annotations are mixed with the labeled second sample set to continue training the first detection model. This method achieves higher model accuracy and lower annotation cost compared to randomly selecting sample images or selecting all sample images. Furthermore, by unifying uncertainty and sampling, sample images with diversity and information content can be obtained. By iterating through the above operations, the model accuracy can be gradually improved, and valuable sample images can be selected in each round, further enhancing the model accuracy.

[0178] It should be understood that the image-based detection method provided in this application can be applied to sample image annotation in various object detection tasks, and is more suitable for scenarios where valuable sample images are selected for annotation in object detection tasks. The application process of the image-based detection method provided in this application will be illustrated below through two specific scenarios:

[0179] (1) Industrial quality inspection

[0180] In industrial quality inspection scenarios, the second detection model can be a model for detecting defects in industrial parts, and correspondingly, target detection can be for detecting defects in industrial parts. In this case, the first sample image and the second sample image can be images of industrial parts containing defects. Using the method provided in this application, information-rich sample images can be selected from unannotated images containing defective industrial parts, and these sample images can be annotated. Then, the first detection model is trained based on the selected sample images and the annotation information to obtain the second detection model. By selecting information-rich sample images, the accuracy of the second detection model in detecting defects in industrial parts can be improved. In application, an image containing an industrial part can be acquired as the image to be detected and input into the second detection model to obtain the defect category (such as surface cracks, dents, etc.) of the industrial part and the location of the defect in the image to be detected, as output by the second detection model.

[0181] (2) Object recognition

[0182] In object recognition scenarios, the second detection model can be a model for detecting objects (such as cats, dogs, etc.) in an image. In this case, the first and second sample images can be images containing various objects. Using the method provided in this application, information-rich sample images can be selected from unannotated images containing various objects, and these sample images can be annotated. Then, the first detection model is trained based on the selected sample images and the annotation information to obtain the second detection model. By selecting information-rich sample images, the accuracy of the second detection model in detecting objects in the image can be improved. In application, an image containing objects can be used as the image to be detected, and the image to be detected can be input into the second detection model to obtain the category (such as cats, dogs, etc.) of each object in the image to be detected, and the position of each object in the image to be detected.

[0183] It should be understood that the image-based detection method proposed in this application, compared with previous algorithms, has the following characteristics: UniUncertainty combines the classification confidence and localization accuracy of the instance, simultaneously considering classification uncertainty and regression uncertainty as the sample uncertainty. UniSampling uses a Gaussian mixture model to unify uncertainty-based sampling and diversity-based sampling to accurately represent the value of unlabeled second sample images. Extensive experiments on mainstream benchmark sets Pascal VOC and MS-COCO show that the image-based detection method (i.e., UAOD) provided in this application outperforms existing active learning-based detectors. UAOD using Faster RCNN (region Convolutional Neural Networks) on COCO achieves a mean accuracy (mAP) of 33.67% and surpasses supervised performance by 91.5% while reducing labeling costs by 60%. Figures 7a-7b The diagram shown is a comparison between the image-based detection method provided in this application and other target detection methods. The specific experimental design is as follows:

[0184] The proposed method in this application was evaluated on two object detection benchmark datasets: MS COCO and Pascal VOC. MS COCO contains 80 object categories, including 118,287 training images and 5,000 validation images. On MS COCO, the average precision (AP) metric was evaluated using an IOU threshold range from 0.5 to 0.95. Figure 7aIn the MS COCO benchmark, average accuracy was used within an IOU threshold range of 0.5 to 0.95. Pascal VOC includes 20 object categories, comprising the training dataset of VOC 2007 & 2012 (16551 images) and the test dataset of VOC 2007 (4952 images). On PascalVOC, performance was evaluated using average accuracy (mAP) at an IOU threshold of 0.5. Figure 7b As shown.

[0185] Active learning is configured for large-scale MS COCO datasets, such as... Figure 7a In this study, evaluation was performed using different numbers of subsets of the training set, specifically 20%, 25%, 30%, 35%, and 40%. The first 20% subset (i.e., the second sample set) was randomly selected. In each acquisition period (or sampling period), after training the detection model, active learning was used to acquire and annotate 5% of the total sample images from the remaining unlabeled training set (i.e., the aforementioned sampled images). Furthermore, another active learning setting for COCO was explored, such as... Figure 7b In this process, only 2.0% of the images randomly selected from the training set are used to initialize the labeled training set (i.e., the second sample set mentioned above). In each acquisition cycle, an additional 2.0% of the sample images are selected from the remaining unlabeled training set (i.e., the sampled images mentioned above), until 10.0% of the training set (i.e., the second sample set) is reached. For Pascal VOC, as... Figure 7c As shown, the trainval set is used as the unlabeled sample images, and 1000 sample images are randomly selected as the initial labeled training set (i.e., the second sample set mentioned above). The budget for each acquisition cycle is fixed at 1000 sample images.

[0186] For the MS COCO dataset, Faster R-CNN and RetinaNet (a detection model) were used as the base detection models. Both of these models utilize ResNet-50 and FPN (feature pyramid networks). Figure 7a In the MS COCO dataset, Faster R-CNN is used as the basic detection model; Figure 7b In the MS COCO dataset, RetinaNet is used as the base detection model. The default training set consists of 26 epochs of training the detection model with a batch size of 8 in all active learning epochs. The learning rate is initialized to 0.01 and decays by a factor of 0.1 at the 20th epoch. On the Pascal VOC dataset, as... Figure 7c As shown, SSD (SingleShotMultiBoxDetector) and VGG (Visual Geometry Group)-16 were used as the base detection models, and the models were trained for 300 epochs with a batch size of 32, following the previous method. After 240 epochs, the learning rate was reduced from 0.001 to 0.0001. Furthermore, all model training was performed using eight NVIDIA 2080Ti GPUs (graphics processing units).

[0187] In the three settings mentioned above, the image-based detection method provided in this application is compared with existing active learning methods. For example, diversity-based methods include Core-Set and CDAL (Contextual diversity for active learning); uncertainty-based methods include Learn Loss and MIAL (Multiple instance active learning); and methods combining diversity and uncertainty include DivProto and PPAL (Plug and play active learning). On the challenging MS COCO dataset, the comparison results with other active learning methods are summarized, such as... Figure 7b As shown, the method provided in this application consistently achieves best performance across all active learning cycles, demonstrating the superiority of the proposed method. In the final cycle, using 40% of the labeled sample images, the method proposed in this application achieves 33.67% mAP, representing improvements of 0.8% and 0.57% compared to DivProto (32.87%) and PPAL (33.1%), respectively. The method used in this application achieves 90% supervised performance using only 40% of the bounding box annotations. Furthermore, results with a lower budget are reported, with only 10% of the labeled sample images in the final cycle. Figure 7b In these embodiments, the method used in this application continues to significantly outperform other methods in these settings, achieving an mAP of 23.48%, which is an improvement of 1.18% and 0.69% compared to DivProto (22.3%) and PPAL (22.79%), respectively. This indicates that the method used in this application can achieve greater benefits when dealing with smaller budget costs.

[0188] The image-based detection device provided in the embodiments of this application will be described next.

[0189] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an image-based detection device provided in an embodiment of this application. The image-based detection device can be a computer program (including program code) in a computer device; for example, the image-based detection device can be application software in a computer device. The image-based detection device can be used to execute... Figure 3 or Figure 4 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 8 The image-based detection device includes the following units:

[0190] The acquisition unit 801 is used to acquire a first sample set, which includes N unlabeled first sample images, where N is a positive integer;

[0191] The processing unit 802 is used to call the first detection model to perform target detection on each first sample image and obtain the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0192] The processing unit 802 is further configured to sample the first sample set based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images.

[0193] The processing unit 802 is also used to train the first detection model based on the sampled sample image and the annotation information for the sampled sample image to obtain the second detection model.

[0194] In one implementation, when the processing unit 802 samples the first sample set based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images, it can specifically be used for:

[0195] Obtain the number of instances and the number of predicted categories in each first sample image;

[0196] The information content of each first sample image is determined based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories.

[0197] Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sample images.

[0198] In one implementation, when the processing unit 802 determines the information content of each first sample image based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories, it may specifically be used to:

[0199] The information content of each instance is determined based on the predicted classification information of the instances included in each first sample image;

[0200] The information content of each first sample image is determined based on the information content of each instance, the number of instances in each first sample image, and the number of predicted categories.

[0201] In one implementation, when the processing unit samples the first sample set based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images, it can be specifically used for:

[0202] Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the N first sample images are clustered to obtain T cluster sets, each cluster set corresponding to a cluster center; T is a positive integer;

[0203] The first sample image corresponding to each cluster center is determined as the sampling sample image.

[0204] In one implementation, each first sample image includes image features, and the Gaussian mixture model includes T Gaussian distributions, each Gaussian distribution corresponding to a cluster set, where T is an integer greater than 1;

[0205] When processing unit 802 clusters N first sample images based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain T cluster sets, it can specifically be used for:

[0206] Based on the information content and image features of each first sample image, determine the probability that each first sample image belongs to each Gaussian distribution;

[0207] Based on the probability that each first sample image belongs to each Gaussian distribution, the parameters of T Gaussian distributions are updated to obtain T updated Gaussian distributions;

[0208] When the parameter updates of T Gaussian distributions meet the update conditions, the probability that each first sample image belongs to each updated Gaussian distribution is determined based on the image features of each first sample image.

[0209] For a target sample image among N first sample images, determine the maximum probability from the probability of the target sample image belonging to each updated Gaussian distribution, and determine the target Gaussian distribution corresponding to the maximum probability;

[0210] Cluster the target sample images into the target cluster set corresponding to the target Gaussian distribution.

[0211] In one implementation, when the processing unit 802 determines the probability that each first sample image belongs to each Gaussian distribution based on the information content and image features of each first sample image, it can specifically be used to:

[0212] For the target sample image among the N first sample images, the information content of the target sample image is used as the prior probability of the Gaussian mixture model.

[0213] Based on the prior probability and the image features of the target sample image, determine the probability that the target sample image belongs to each Gaussian distribution.

[0214] The processing unit 802 is also used for:

[0215] Obtain a second sample set, which includes multiple second sample images and annotation information for each second sample image; each second sample image includes at least one sample instance, and the annotation information for each second sample image includes: the location information of the ground bounding box of each sample instance in the second sample image;

[0216] The initial detection model is invoked to perform target detection on each second sample image, and the predicted classification information of the sample instances contained in each second sample image and the position information of the predicted bounding box of each sample instance are obtained.

[0217] The quality of the bounding box corresponding to each sample instance is determined based on the location information of the ground bounding box and the location information of the predicted bounding box of each sample instance.

[0218] The loss of the initial detection model is determined based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance.

[0219] Based on the loss of the initial detection model, the parameters of the initial detection model are adjusted to obtain the first detection model.

[0220] The annotation information includes the category labels of the sample instances; when the processing unit 802 determines the loss of the initial detection model based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, it can be specifically used for:

[0221] Based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, the classification and localization loss of the initial detection model is determined.

[0222] Cross-entropy is calculated based on the predicted classification information of the sample instances contained in each second sample image and the category label of each sample instance to determine the cross-entropy loss of the initial detection model.

[0223] The loss of the initial detection model is determined based on the classification localization loss and cross-entropy loss of the initial detection model.

[0224] In one implementation, when the processing unit 802 trains the first detection model based on the sampled sample image and the annotation information annotated for the sampled sample image to obtain the second detection model, it can be specifically used for:

[0225] Update the second sample set based on the sampled images and the annotation information used to annotate them;

[0226] The first detection model is trained using the updated second sample set;

[0227] If the training of the first detection model meets the training conditions, then the trained first detection model will be determined as the second detection model.

[0228] If the training conditions for the first detection model are not met, then unlabeled first sample images are acquired to obtain a new first sample set, so that the first detection model can be trained based on the new first sample set.

[0229] The processing unit 802 is also used for:

[0230] Based on the information content of each instance, each image region of each first sample image is distinguished and displayed;

[0231] In particular, the greater the information content of an instance, the more prominent the image region where the instance is located will be in the corresponding first sample image.

[0232] In this embodiment, a first sample set is obtained, comprising N unlabeled first sample images, where N is a positive integer. A first detection model is invoked to perform object detection on each first sample image, obtaining the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image. By uniformly representing the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image using the predicted classification information, it is convenient to consider both the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image when sampling the first sample set, thus providing a sampling basis for subsequent sampling of the first sample set. Further, the first sample set is sampled according to the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images. The first detection model is trained according to the sampled sample images and the annotation information labeled for the sampled sample images to obtain a second detection model. As can be seen, by combining the Gaussian mixture model and the predicted classification information of the instance to sample the first sample set, representative sample images can be obtained, which can better determine the sample images used for model training. Compared with adding sample images to the model training, the sample images for model training can be better determined through the embodiments of this application, which can reduce the manual annotation time to a certain extent and improve the model training efficiency.

[0233] The computer device provided in the embodiments of this application will be described next.

[0234] Furthermore, this application also provides a schematic diagram of the structure of a computer device, which can be found in [reference needed]. Figure 9 The computer device may include a processor 901, an input device 902, an output device 903, and a memory 904. The processor 901, input device 902, output device 903, and memory 904 are connected via a bus. The memory 904 stores computer programs, which include program instructions, and the processor 901 executes the program instructions stored in the memory 904.

[0235] The processor 901 executes the following operations by running program instructions stored in memory 904:

[0236] Obtain the first sample set, which includes N unlabeled first sample images, where N is a positive integer;

[0237] The first detection model is invoked to perform target detection on each first sample image, and the predicted classification information of the instances included in each first sample image is obtained. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image.

[0238] Based on the predicted classification information of the instances included in each first sample image, and the Gaussian mixture model associated with N first sample images, the first sample set is sampled to obtain sampled images;

[0239] The first detection model is trained based on the sampled images and the annotation information used to annotate them, to obtain the second detection model.

[0240] In one implementation, when the processor 901 samples the first sample set based on the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images, it may specifically perform the following operations:

[0241] Obtain the number of instances and the number of predicted categories in each first sample image;

[0242] The information content of each first sample image is determined based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories.

[0243] Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sample images.

[0244] In one implementation, when the processor 901 determines the information content of each first sample image based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories, it may specifically perform the following operations:

[0245] The information content of each instance is determined based on the predicted classification information of the instances included in each first sample image;

[0246] The information content of each first sample image is determined based on the information content of each instance, the number of instances in each first sample image, and the number of predicted categories.

[0247] In one implementation, when the processor 901 samples the first sample set based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images, it may specifically perform the following operations:

[0248] Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the N first sample images are clustered to obtain T cluster sets, each cluster set corresponding to a cluster center; T is a positive integer;

[0249] The first sample image corresponding to each cluster center is determined as the sampling sample image.

[0250] In one implementation, each first sample image includes image features, and the Gaussian mixture model includes T Gaussian distributions, each Gaussian distribution corresponding to a cluster set, where T is an integer greater than 1;

[0251] When processor 901 clusters N first sample images based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain T cluster sets, it can specifically perform the following operations:

[0252] Based on the information content and image features of each first sample image, determine the probability that each first sample image belongs to each Gaussian distribution;

[0253] Based on the probability that each first sample image belongs to each Gaussian distribution, the parameters of T Gaussian distributions are updated to obtain T updated Gaussian distributions;

[0254] When the parameter updates of T Gaussian distributions meet the update conditions, the probability that each first sample image belongs to each updated Gaussian distribution is determined based on the image features of each first sample image.

[0255] For a target sample image among N first sample images, determine the maximum probability from the probability of the target sample image belonging to each updated Gaussian distribution, and determine the target Gaussian distribution corresponding to the maximum probability;

[0256] Cluster the target sample images into the target cluster set corresponding to the target Gaussian distribution.

[0257] In one implementation, when the processor 901 determines the probability that each first sample image belongs to each Gaussian distribution based on the information content and image features of each first sample image, it may specifically perform the following operations:

[0258] For the target sample image among the N first sample images, the information content of the target sample image is used as the prior probability of the Gaussian mixture model.

[0259] Based on the prior probability and the image features of the target sample image, determine the probability that the target sample image belongs to each Gaussian distribution.

[0260] In one implementation, processor 901 may also perform the following operations:

[0261] Obtain a second sample set, which includes multiple second sample images and annotation information for each second sample image; each second sample image includes at least one sample instance, and the annotation information for each second sample image includes: the location information of the ground bounding box of each sample instance in the second sample image;

[0262] The initial detection model is invoked to perform target detection on each second sample image, and the predicted classification information of the sample instances contained in each second sample image and the position information of the predicted bounding box of each sample instance are obtained.

[0263] The quality of the bounding box corresponding to each sample instance is determined based on the location information of the ground bounding box and the location information of the predicted bounding box of each sample instance.

[0264] The loss of the initial detection model is determined based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance.

[0265] Based on the loss of the initial detection model, the parameters of the initial detection model are adjusted to obtain the first detection model.

[0266] In one implementation, the annotation information includes the category label of the sample instance. When the processor 901 determines the loss of the initial detection model based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, it can specifically perform the following operations:

[0267] Based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, the classification and localization loss of the initial detection model is determined.

[0268] Cross-entropy is calculated based on the predicted classification information of the sample instances contained in each second sample image and the category label of each sample instance to determine the cross-entropy loss of the initial detection model.

[0269] The loss of the initial detection model is determined based on the classification localization loss and cross-entropy loss of the initial detection model.

[0270] In one implementation, when the processor 901 trains the first detection model based on the sampled sample images and the annotation information for the sampled sample images to obtain the second detection model, it may specifically perform the following operations:

[0271] Update the second sample set based on the sampled images and the annotation information used to annotate them;

[0272] The first detection model is trained using the updated second sample set;

[0273] If the training of the first detection model meets the training conditions, then the trained first detection model will be determined as the second detection model.

[0274] If the training conditions for the first detection model are not met, then unlabeled first sample images are acquired to obtain a new first sample set, so that the first detection model can be trained based on the new first sample set.

[0275] In one implementation, processor 901 may also perform the following operations:

[0276] Based on the information content of each instance, each image region of each first sample image is distinguished and displayed;

[0277] In particular, the greater the information content of an instance, the more prominent the image region where the instance is located will be in the corresponding first sample image.

[0278] In this embodiment, a first sample set is obtained, comprising N unlabeled first sample images, where N is a positive integer. A first detection model is invoked to perform object detection on each first sample image, obtaining the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image. By uniformly representing the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image using the predicted classification information, it is convenient to consider both the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance's position in the corresponding first sample image when sampling the first sample set, thus providing a sampling basis for subsequent sampling of the first sample set. Further, the first sample set is sampled according to the predicted classification information of the instances included in each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images. The first detection model is trained according to the sampled sample images and the annotation information labeled for the sampled sample images to obtain a second detection model. As can be seen, by combining the Gaussian mixture model and the predicted classification information of the instance to sample the first sample set, representative sample images can be obtained, which can better determine the sample images used for model training. Compared with adding sample images to the model training, the sample images for model training can be better determined through the embodiments of this application, which can reduce the manual annotation time to a certain extent and improve the model training efficiency.

[0279] In this application, the term "unit" refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more units. Furthermore, each unit can be part of an overall unit that includes the functionality of that unit.

[0280] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When a processor executes these program instructions, it can execute the aforementioned... Figure 3 or Figure 4 The methods described in the corresponding embodiments are therefore not repeated here. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions may be deployed on a computer device, executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0281] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, enabling the computer device to perform the aforementioned... Figure 3 or Figure 4 The methods described in the corresponding embodiments are therefore not repeated here.

[0282] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0283] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image-based detection method, characterized in that, include: Obtain a first sample set, which includes N unlabeled first sample images, where N is a positive integer; The first detection model is invoked to perform target detection on each first sample image to obtain the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image. Based on the predicted classification information of the instances included in each of the first sample images, and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sampled sample images; The first detection model is trained based on the sampled images and the annotation information used to annotate the sampled images to obtain the second detection model.

2. The method as described in claim 1, characterized in that, The step of sampling the first sample set based on the predicted classification information of the instances included in each of the first sample images and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images includes: Obtain the number of instances and the number of predicted categories in each of the first sample images; The information content of each first sample image is determined based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories. Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the first sample set is sampled to obtain sampled images.

3. The method as described in claim 2, characterized in that, The step of determining the informatics of each first sample image based on the predicted classification information of the instances included in each first sample image, the number of instances in each first sample image, and the number of predicted categories includes: The information degree of each instance is determined based on the predicted classification information of the instances included in each of the first sample images; The information content of each first sample image is determined based on the information content of each instance, the number of instances in each first sample image, and the number of predicted categories.

4. The method as described in claim 2, characterized in that, The step of sampling the first sample set based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images to obtain sampled sample images includes: Based on the information content of each first sample image and the Gaussian mixture model associated with the N first sample images, the N first sample images are clustered to obtain T cluster sets, each cluster set corresponding to a cluster center; T is a positive integer; The first sample image corresponding to each cluster center is determined as the sampling sample image.

5. The method as described in claim 4, characterized in that, Each first sample image includes image features, and the Gaussian mixture model includes T Gaussian distributions, each Gaussian distribution corresponding to a cluster set, where T is an integer greater than 1; The step involves clustering the N first sample images based on their information content and the Gaussian mixture model associated with them, resulting in T cluster sets, including: Based on the information content and image features of each first sample image, determine the probability that each first sample image belongs to each Gaussian distribution; Based on the probability that each first sample image belongs to each Gaussian distribution, the parameters of the T Gaussian distributions are updated to obtain T updated Gaussian distributions; When the parameter update of the T Gaussian distributions meets the update condition, the probability that each first sample image belongs to each updated Gaussian distribution is determined based on the image features of each first sample image. For a target sample image among N first sample images, determine the maximum probability from the probability that the target sample image belongs to each updated Gaussian distribution, and determine the target Gaussian distribution corresponding to the maximum probability; The target sample images are clustered into the target cluster set corresponding to the target Gaussian distribution.

6. The method as described in claim 5, characterized in that, The step of determining the probability that each first sample image belongs to each Gaussian distribution based on the information content and image features of each first sample image includes: For the target sample image among the N first sample images, the information content of the target sample image is used as the prior probability of the Gaussian mixture model. Based on the prior probability and the image features of the target sample image, the probability that the target sample image belongs to each Gaussian distribution is determined.

7. The method as described in claim 1, characterized in that, The method further includes: Obtain a second sample set, which includes multiple second sample images and annotation information for each second sample image; each second sample image includes at least one sample instance, and the annotation information for the second sample image includes: the location information of the ground bounding box of each sample instance in the second sample image; The initial detection model is invoked to perform target detection on each of the second sample images, thereby obtaining the predicted classification information of the sample instances contained in each second sample image and the position information of the predicted bounding box of each sample instance. Based on the location information of the ground truth bounding box of each sample instance and the location information of the predicted bounding box of each sample instance, the quality of the bounding box corresponding to each sample instance is determined. The loss of the initial detection model is determined based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance. Based on the loss of the initial detection model, the parameters of the initial detection model are adjusted to obtain the first detection model.

8. The method as described in claim 7, characterized in that, The annotation information also includes the category labels of the sample instances; determining the loss of the initial detection model based on the predicted classification information of each sample instance and the quality of the bounding boxes corresponding to each sample instance includes: Based on the predicted classification information of each sample instance and the quality of the bounding box corresponding to each sample instance, the classification and localization loss of the initial detection model is determined. Cross-entropy is calculated based on the predicted classification information of the sample instances contained in each second sample image and the category label of each sample instance to determine the cross-entropy loss of the initial detection model. The loss of the initial detection model is determined based on the classification and localization loss of the initial detection model and the cross-entropy loss.

9. The method as described in claim 7, characterized in that, The step of training the first detection model to obtain a second detection model based on the sampled image and the annotation information annotated for the sampled image includes: The second sample set is updated based on the sampled images and the annotation information used to annotate the sampled images; The first detection model is trained using the updated second sample set; If the training of the first detection model meets the training conditions, then the trained first detection model is determined as the second detection model. If the training conditions for the first detection model are not met, then unlabeled first sample images are acquired to obtain a new first sample set, so as to complete the training of the first detection model based on the new first sample set.

10. The method as described in claim 3, characterized in that, The method further includes: Based on the information content of each instance, each image region of each first sample image is distinguished and displayed; Wherein, the greater the information content of the instance, the more prominent the image area where the instance is located in the corresponding first sample image is.

11. An image-based detection device, characterized in that, include: The acquisition unit is used to acquire a first sample set, which includes N unlabeled first sample images, where N is a positive integer; The processing unit is used to call the first detection model to perform target detection on each first sample image and obtain the predicted classification information of the instances included in each first sample image. The predicted classification information is used to represent the classification confidence of the predicted category to which the instance belongs and the location confidence of the instance in the corresponding first sample image. The processing unit is further configured to sample the first sample set based on the predicted classification information of the instances included in each of the first sample images and the Gaussian mixture model associated with the N first sample images, to obtain sampled sample images; The processing unit is further configured to train the first detection model based on the sampled image and the annotation information annotated for the sampled image to obtain a second detection model.

12. A computer device, characterized in that, include: A processor is used to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, performs the image-based detection method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, performs the image-based detection method as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image-based detection method as described in any one of claims 1-10.

Images