A recognition method, apparatus and related device

Abstract

The application provides a recognition method, device and related equipment. The method comprises: inputting an image to be recognized into a plurality of trained recognition models by a computing device, wherein each trained recognition model comprises an image language pre-training model and a prompt template comprising a plurality of prompt words, and the prompt templates of the trained recognition models are different from each other; determining, by the computing device, probabilities that the image to be recognized belongs to respective categories based on the plurality of trained recognition models, wherein an i-th trained recognition model outputs probabilities that the image to be recognized belongs to a plurality of categories corresponding to the i-th trained recognition model; and finally determining a category of the image to be recognized according to the probabilities that the image to be recognized belongs to the respective categories. Through grouping of a data set and combination of a visual language pre-training model and a prompt template, a model for unknown class detection is trained, so that the detection accuracy of the model for unknown classes outside a closed set is improved, and the classification accuracy of the model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an identification method, apparatus and related equipment. Background Technology

[0002] Object recognition refers to the use of computer technology to identify and classify objects in a scene, and it is one of the important research areas in computer vision. In recent years, object recognition has made rapid progress in closed-set scenes, with models trained on a finite number of classes within the closed set demonstrating excellent performance on test sets of these classes. However, in real-world open-set tests, the number of data categories far exceeds the finite number of categories in the training set. In practical applications, models trained on a finite number of classes often exhibit severe misclassification of data outside the closed set, identifying data of unknown classes as known classes. Therefore, how to accurately classify closed-set data while also identifying open-set data that does not belong to the closed set is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a recognition method, apparatus, and related equipment that can improve the model's detection accuracy for unknown classes outside of closed sets and improve the model's classification accuracy.

[0004] In a first aspect, this application provides a recognition method, which includes: a computing device inputting an image to be recognized into multiple trained recognition models, wherein each trained recognition model includes an image-language pre-trained model and a prompt template including multiple prompt words, and the prompt templates of each trained recognition model are different; the computing device determines the probability of the image to be recognized belonging to each category based on the multiple trained recognition models, wherein the i-th trained recognition model outputs the probability of the image to be recognized belonging to the multiple categories corresponding to the i-th trained recognition model; and finally, the category of the image to be recognized is determined according to the probability of the image to be recognized belonging to each category.

[0005] When training multiple recognition models for joint classification, for each subset of the training model, a set of prompt templates including multiple prompt words is set. A corresponding recognition model for this subset, including a visual language pre-trained model and the prompt templates, is constructed. Then, the corresponding recognition models are trained on each subset using the same method. This transforms the training of open-set recognition models on large datasets into training on multiple small datasets based on visual language pre-trained models, allowing the training method for small datasets to be reused for open-set recognition on large datasets. When recognizing the image to be recognized, joint prediction using multiple trained recognition models improves the model's accuracy. By using the aforementioned joint prediction of multiple trained recognition models, the prediction based on a large dataset is transformed into joint prediction based on multiple visual language pre-trained models. Predicting the image to be recognized using multiple trained recognition models separately improves the final prediction accuracy.

[0006] In one possible implementation, the multiple categories corresponding to the i-th trained recognition model include closed-set categories and open-set categories. The closed-set categories are the categories included in the i-th subset used to train the i-th recognition model, and the open-set categories are the categories other than those included in the training set. The training set is a dataset used to train multiple recognition models, and the i-th subset belongs to the training set.

[0007] It should be understood that a well-trained recognition model can output the probabilities of multiple categories, which are the categories corresponding to the category labels used when training the recognition model. In other words, when these multiple categories include open-set categories, when training the constructed recognition model based on the training set, in addition to using category labels from the training set, category labels unrelated to the downstream task are also used in the model training. This enables the trained model to reduce label bias during classification, i.e., reduce the probability of identifying an unknown category in the open set as a category in the closed set, thereby improving recognition accuracy.

[0008] In one possible implementation, determining the category of the image to be identified based on the probability of the image belonging to each category includes: obtaining the probability of each category in the closed set of categories corresponding to each recognition model from the probabilities of multiple categories output by each recognition model; if the maximum value of the probability of each category in the closed set of categories corresponding to each recognition model is greater than or equal to a preset threshold, determining the category corresponding to the maximum value as the category of the image to be identified; or, if the maximum value is less than the preset threshold, determining that the image to be identified belongs to an unknown class.

[0009] By using multiple models to identify the image to be identified, and determining the category of the image to be identified based on the category with the highest probability in the closed set of categories, the occurrence of false identifications can be reduced and the accuracy of identification can be improved.

[0010] In one possible implementation, the above method also includes training the various recognition models, specifically including:

[0011] The computing device divides the training set into G subsets, each subset containing different image categories; the recognition model corresponding to the i-th subset is trained using the i-th subset to obtain the i-th trained recognition model, where the recognition model corresponding to the i-th subset includes an image-language pre-trained model and a prompt template corresponding to the i-th subset, and i is a positive integer less than or equal to G.

[0012] In one possible implementation, the above-mentioned method of training the recognition model corresponding to the i-th subset using the i-th subset to obtain the i-th trained recognition model includes: determining the negative category label corresponding to the i-th subset, which belongs to the label corresponding to the above-mentioned open set category; and training the recognition model corresponding to the i-th subset based on the i-th subset, the negative category label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset to obtain the i-th trained recognition model corresponding to the i-th subset.

[0013] In one possible implementation, the above-mentioned training of the recognition model corresponding to the i-th subset based on the i-th subset, the negative category label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset includes:

[0014] The recognition model corresponding to the i-th subset is trained based on the first sample image in the i-th subset, the negative class label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, and the word vector embedding corresponding to the prompt template corresponding to the i-th subset is updated.

[0015] For the training set, a set of prompt templates including multiple prompt words is set up, and a corresponding recognition model including a visual language pre-trained model and prompt templates is constructed. When training the constructed recognition model based on the training set, in addition to the category labels in the training set participating in the model training, the labels of categories unrelated to the downstream task are also participating in the model training. This enables the trained model to reduce label bias when performing classification and recognition (that is, to identify unknown categories in the open set as categories in the closed set). Furthermore, when training a model capable of open-set recognition on a large-scale dataset, the large-scale dataset is divided into several subsets, each of which includes a portion of the training data from the large-scale dataset. For each subset, a set of prompt templates containing multiple prompt words is set, and a corresponding recognition model including a visual language pre-trained model and the prompt templates is constructed. Then, the corresponding recognition models are trained on each subset using the same method. During the training process, the parameters of the visual language pre-trained model are not updated; only the corresponding prompt templates are updated based on each subset. This allows the training of the open-set recognition model on the large-scale dataset to be transformed into training on multiple small datasets based on the visual language pre-trained model. This enables the training method for small-scale datasets to be reused for open-set recognition on large-scale datasets, reducing the misidentification problem caused by the small difference in probability between classes due to the large number of categories when the model trained on the large-scale dataset makes predictions more accurate.

[0016] Secondly, embodiments of this application provide a recognition device, which includes: an acquisition module, configured to acquire an image to be recognized, input the image to be recognized into G trained recognition models, wherein each trained recognition model includes an image-language pre-trained model and a prompt template including multiple prompt words, the prompt templates of each trained recognition model are different, and G is an integer greater than 1; a recognition module, configured to determine the probability of the image to be recognized belonging to each category based on the multiple trained recognition models, wherein the i-th trained recognition model outputs the probability of the image to be recognized belonging to the multiple categories corresponding to the i-th trained recognition model; the recognition module is further configured to determine the category of the image to be recognized based on the probability of the image to be recognized belonging to each category.

[0017] In one possible implementation, the multiple categories corresponding to the i-th trained recognition model include closed-set categories and open-set categories. The closed-set categories are the categories included in the i-th subset used to train the i-th recognition model, and the open-set categories are the categories other than those included in the training set. The training set is a dataset used to train multiple recognition models, and the i-th subset belongs to the training set.

[0018] In one possible implementation, the identification module is specifically used for:

[0019] The probability of each category in the closed set of categories corresponding to each recognition model is obtained from the probabilities of multiple categories output by each recognition model. If the maximum value of the probability of each category in the closed set of categories corresponding to each recognition model is greater than or equal to a preset threshold, the category corresponding to the maximum value is determined as the category of the image to be recognized. Alternatively, if the maximum value is less than the preset threshold, the image to be recognized is determined to belong to an unknown class.

[0020] In one possible implementation, the device further includes a training module for dividing the training set into G subsets, each subset containing different image categories; training the recognition model corresponding to the i-th subset using the i-th subset to obtain the i-th trained recognition model, wherein the recognition model corresponding to the i-th subset includes an image-language pre-trained model and a prompt template corresponding to the i-th subset, and i is a positive integer less than or equal to G.

[0021] In one possible implementation, the training module is specifically used to: determine the negative class label corresponding to the i-th subset, where the negative class label belongs to the label corresponding to the open set category; and train the recognition model corresponding to the i-th subset based on the i-th subset, the negative class label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, to obtain the i-th trained recognition model corresponding to the i-th subset.

[0022] In one possible implementation, the training device is specifically used to: train the recognition model corresponding to the i-th subset based on the first sample image in the i-th subset, the negative class label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, and update the word vector embedding corresponding to the prompt template corresponding to the i-th subset.

[0023] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation thereof.

[0024] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method described in the first aspect or any possible implementation thereof.

[0025] Fifthly, this application provides a computer program product comprising computer instructions that, when executed by a computing device, enable the computing device to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating one classification method provided in an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of a model training process provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of a model training method provided in an embodiment of this application;

[0030] Figure 4 This is a flowchart illustrating an identification method provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of an identification process provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of an identification device provided in an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of a computing device provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0035] Figure 9 This is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions in this application will now be described with reference to the accompanying drawings. For ease of understanding, the terminology used in this application will first be explained.

[0037] A closed set refers to the target categories in the training set.

[0038] An open set is a target category in the test set that does not belong to a closed set.

[0039] Open set recognition refers to a classification model trained on a training set of finite categories. During testing, it is necessary not only to accurately classify targets belonging to the closed set, but also to identify targets belonging to the open set as unknown classes.

[0040] Public cloud, the core attribute of which is shared resource service, refers to cloud infrastructure and services provided by third-party providers to tenants that can be used through public networks (such as the Internet). Tenants obtain access to cloud infrastructure and services by paying or applying.

[0041] A Virtual Private Cloud (VPC) is an isolated, user-configurable, and managed virtual network environment built for resources such as cloud servers, cloud containers, and cloud databases. VPCs enhance the security of users' cloud resources and simplify network deployment. VPCs enable users to flexibly manage their cloud networks, including creating subnets, managing routing tables, and requesting elastic public IP addresses and bandwidth.

[0042] A region is defined by its geographical location and network latency. Within the same region, the same resource pool is used (which can be understood as sharing public services such as elastic computing, block storage, object storage, VPC network, elastic public IP, and images).

[0043] An availability zone (AZ) is a collection of one or more physical data centers. An AZ has its own water and electricity supply. Within an AZ, computing, network, storage, and other resources are logically divided into multiple clusters. Multiple AZs in a region are connected by high-speed fiber optic cables to meet the needs of users to build high-availability systems across AZs.

[0044] Elastic cloud servers (ECS), also known as cloud instances, are basic computing components consisting of a central processing unit (CPU), memory, operating system, and cloud disks, such as virtual machines (VMs) and containers. After successfully purchasing an Elastic cloud server, tenants can use it on the public cloud just like using their own local personal computer (PC) or physical server to create a high-efficiency, reliable, and secure computing environment. A container is a group of resource-limited, isolated processes, while a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment.

[0045] Cloud services refer to services provided by public clouds that users can apply for or purchase to use, such as ECS, model training cloud services, data storage cloud services, and image recognition cloud services mentioned above. It should be noted that different cloud service providers may use different names for cloud services with the same functionality. Therefore, the cloud services listed above are merely examples and do not constitute a specific limitation on the cloud services provided by any particular cloud service provider.

[0046] A cloud service portfolio includes multiple cloud services that work together to achieve a specific business function. For example, ECS, data storage cloud services, and data search cloud services can be combined to provide data (such as logs, documents, images, etc.) storage and query services.

[0047] In traditional recognition tasks, the target categories in the training and test sets are identical. Therefore, models trained on the training set exhibit good recognition performance on the test set. However, in real-world scenarios, such as autonomous driving and product classification, the number of real-world target categories exceeds the finite number of categories in the training set. When facing real-world recognition tasks, models trained on a limited set of categories may misclassify data deployed on the training set into categories already in the training set, leading to recognition errors. Figure 1 As shown, Figure 1 This is a schematic diagram of an identification task provided in an embodiment of this application. Figure 1 1(a) in the diagram shows the true categories of the data. Categories 1 to 5 are a finite set of categories included in the training set, i.e., closed-set categories. However, real-world scenarios also include unknown categories such as categories 6 to 8, i.e., open-set categories. A model trained on a finite set of categories will identify unknown categories as known categories within the closed set when recognizing targets of the aforementioned eight categories. For example, it might identify a target of category 6 as category 3, a target of category 7 as category 1, and a target of category 8 as category 4. Figure 1 As shown in 1(b) of the diagram. Therefore, open set identification was proposed to address the aforementioned misclassification phenomenon, as shown in... Figure 1 As shown in 1(c), open set recognition not only requires accurately identifying targets belonging to closed sets, but also requires identifying targets belonging to open sets as unknown classes. Therefore, open set recognition is a technical problem that urgently needs to be solved.

[0048] This application provides a target recognition method based on a visual language pre-trained model, capable of achieving open-set recognition on large-scale datasets. The following first describes the model training method for target recognition provided in the embodiments of this application. Figure 2 As shown, Figure 2This is a schematic diagram of a model training process provided in an embodiment of this application. The model training method includes the following steps S201 to S203.

[0049] S201. Divide the training set data into one or more subsets.

[0050] In this embodiment, for images of multiple target categories in the training set, the images in the training set are first divided into one or more subsets. When the training set is divided into multiple subsets, the categories of any two subsets do not overlap; that is, images of a category in one subset will not appear in other subsets. For example, the training set includes N... c The images of the target class are divided into G subsets. The class of the images in the k-th subset is different from the classes of the images in the other G-1 subsets, where G is a positive integer.

[0051] In one possible implementation, each subset includes a number of categories less than or equal to a preset category threshold N. max When classifying images of multiple categories in the training set, the number of categories N c Number of subsets G, preset category threshold N max and the number of categories in the r-th subset It satisfies the following (Formula 1).

[0052]

[0053] That is, the number of categories in the last subset is less than or equal to the number of categories in the other subsets.

[0054] For example, if the training set includes 1000 categories of targets and the preset category threshold is 100, the training set can be divided into 10 subsets, each of which includes 100 categories of targets; if the training set includes 978 categories of targets and the preset category threshold is 100, the training set can be divided into 10 subsets, where 9 subsets can each include 100 categories of targets, and the other subset includes 78 categories of targets.

[0055] In one possible implementation, N may not be set. max Instead, the training set data is divided into multiple subsets on an average basis according to the categories. For example, if the training set includes 978 categories of targets, and the training set is divided into 10 subsets, then 9 of the subsets can each include 98 categories of targets, and the other subset includes 96 categories of targets.

[0056] S202. For each subset, determine a prompt template and multiple negative category labels.

[0057] In this embodiment, for any subset of the G subsets mentioned above, such as the i-th subset, n negative category labels corresponding to that subset are selected from the dictionary. Here, n is a positive integer, and this embodiment does not limit the number of negative category labels. Negative category labels refer to words unrelated to the category of the target in the training set and the target category to be identified in the downstream task; that is, labels corresponding to the categories in the open set. For example, if the target category in the training set and the target category of the downstream task are both animals, the negative category label could be a plant name. Furthermore, for each subset, a prompt template is constructed. The prompt template includes word embeddings of L prompt words, which are used to form the recognition model provided in this embodiment of the application with the visual language pre-trained model. The prompt template corresponding to the i-th subset can be expressed as follows (Formula 2).

[0058]

[0059] Where i is a positive integer less than or equal to G. Let C represent the word vector of the k-th prompt word in the prompt template corresponding to the i-th subset, and let C represent the category label.

[0060] In one possible implementation, m negative category labels can be randomly selected from a dictionary to construct a negative category word set. When training based on the training data from each subset, n words are randomly selected from that negative category set as the negative category labels for that subset. The dictionary mentioned above can be a dictionary such as WordNet, where n is less than or equal to m.

[0061] S203. Train the recognition model corresponding to the i-th subset based on the i-th subset, the negative category label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset to obtain the trained recognition model corresponding to the i-th subset.

[0062] In this embodiment of the application, the recognition model corresponding to the i-th subset includes a visual language pre-trained model and a prompt template corresponding to the i-th subset. For example... Figure 3 As shown, when training the recognition model corresponding to the i-th subset, the label of the j-th sample image in the i-th subset is first filled into the above prompt template, and the prompt template with the category index is encoded by a text encoder. Then, the sample image x to be trained, the negative category label corresponding to the i-th subset, and the category labels included in the i-th subset are input into the recognition model including the visual language pre-trained model and the prompt template encoded by the encoder, and the recognition model is trained. For a sample image x in a training set, the probability that the sample image x belongs to the target category satisfies the following (Formula 3).

[0063]

[0064] Among them, E I E represents the visual encoder. T Indicates a text encoder. The label represents the sample image x. N represents the category label of the j-th category in the i-th subset. O The value of T represents the number of negative category labels. T is a constant. In this embodiment of the application, the value of T is not specifically limited. Preferably, the value of T is 1.

[0065] After obtaining the probability that the sample image x belongs to the target category, the loss value corresponding to the sample image is calculated, and then the embedding of the prompt template is updated according to the loss value. Optionally, the loss value in this embodiment can be determined as follows (Formula 4).

[0066]

[0067] It should be noted that the parameters included in the visual language pre-trained model are not updated during training. During training based on the sample images of the i-th subset, only the embedding of the prompt template corresponding to the i-th subset is updated. The recognition model corresponding to the i-th subset is trained iteratively multiple times using the sample images of the i-th subset until the trained recognition model corresponding to the i-th subset is obtained.

[0068] For each of the G subsets mentioned above, the training is performed using the methods described in S202 and S203 to obtain a trained recognition model for each subset, resulting in G trained recognition models. It should be understood that the negative class labels for each subset can be the same or different, and the prompt templates for each subset can be different.

[0069] It should be noted that the aforementioned visual language pre-training models are models trained on datasets that include a large number of image-text pairs, such as contrastive language-image pre-training (CLIP) models or ALIGN models trained on the large-scale artificial intelligence open network (LAION) dataset (LAION-400M) which includes 400 million image-text pairs.

[0070] For the training set, a set of prompt templates including multiple prompt words is set up, and a corresponding recognition model including a visual language pre-trained model and prompt templates is constructed. When training the constructed recognition model based on the training set, in addition to the category labels in the training set participating in the model training, the labels of categories unrelated to the downstream task are also participating in the model training. This enables the trained model to reduce label bias when performing classification and recognition (that is, to identify unknown categories in the open set as categories in the closed set). Furthermore, when training a model capable of open-set recognition on a large-scale dataset, the large-scale dataset is divided into several subsets, each of which includes a portion of the training data from the large-scale dataset. For each subset, a set of prompt templates containing multiple prompt words is set, and a corresponding recognition model including a visual language pre-trained model and the prompt templates is constructed. Then, the corresponding recognition models are trained on each subset using the same method. During the training process, the parameters of the visual language pre-trained model are not updated; only the corresponding prompt templates are updated based on each subset. This allows the training of the open-set recognition model on the large-scale dataset to be transformed into training on multiple small datasets based on the visual language pre-trained model. This enables the training method for small-scale datasets to be reused for open-set recognition on large-scale datasets, reducing the misidentification problem caused by the small difference in probability between classes due to the large number of categories when the model trained on the large-scale dataset makes predictions more accurate.

[0071] The training method of the recognition model according to the embodiments of this application has been described above. The following describes a recognition method for image recognition using G recognition models trained according to the above training method. Figure 4 As shown, Figure 4 This is a flowchart illustrating an identification method provided in an embodiment of this application, which includes the following steps S401 to S402.

[0072] S401. Input the image to be recognized into multiple trained recognition models.

[0073] In this embodiment of the application, multiple trained recognition models are based on the above... Figure 2 The corresponding embodiment trains G recognition models. The i-th recognition model includes a visual language pre-trained model and a prompt template trained based on data from the i-th subset. The image to be recognized is then input into each of the G trained recognition models.

[0074] S402. Based on multiple trained recognition models, identify the image to be identified, and determine the category to which the image to be identified belongs based on the multiple recognition results.

[0075] like Figure 5As shown, after inputting the image to be recognized into G trained recognition models, each model can output the probability that the target in the image belongs to multiple categories. These multiple categories include a subset of closed categories and the category corresponding to the negative category label of that subset. For example, if the i-th subset includes 5 closed categories (i.e., 5 category labels) and has 10 negative category labels, then the recognition model corresponding to the i-th subset can output 15 probabilities.

[0076] For G trained recognition models, each model can output... There are probabilities, where This represents the number of categories included in the i-th subset, and n represents the number of negative class labels. This is used to obtain the output of each model. After determining the probabilities, the maximum probability that the target in the image to be identified belongs to the closed set category is determined. The subset containing the category corresponding to the maximum probability is then defined as the target subset. The maximum probability corresponding to each trained recognition model can be expressed as follows (Formula 5).

[0077]

[0078] Where i∈[1,G], Let I represent the maximum probability value output by the recognition model corresponding to the i-th subset. Then, the index I of the target subset... opt The corresponding expression can be represented as:

[0079]

[0080] In determining the target in the image to be identified, in the I... opt After finding the maximum probability in the j-th category of the closed set of each subset, if the maximum probability is greater than or equal to a preset threshold, then the target in the image to be identified is determined to belong to the j-th category; if the maximum probability is less than the preset threshold, then the target in the image to be identified is determined to belong to an unknown category.

[0081] Through the above Figure 2 After training multiple recognition models, the method described in the corresponding embodiment can be used to jointly predict an image to be recognized when the image needs to be recognized. This transforms the prediction of the recognition model based on a large-scale dataset into a joint prediction based on multiple visual language pre-trained models, thereby improving the accuracy of the prediction.

[0082] As shown in Table 1 below, Table 1 compares the performance of the open set unknown class detection on a small dataset after introducing labels of open set categories into the training of the recognition model provided in this application. NR indicates no results.

[0083] Table 1 lists the accuracy and standard deviation of the identification results of the nine methods on four datasets. As can be seen from Table 1, the method provided in this application embodiment has a higher accuracy in identifying the unknown class on the following four datasets than other methods.

[0084] Table 1

[0085]

[0086]

[0087] Table 2 below shows the comparison results of F1 scores for open set recognition using different methods on images of 11 categories. Among these 11 categories, 10 are closed set categories and 1 is an open set category. The images for the 10 closed set categories are selected from the CIFAR10 dataset. The images of these 11 categories were processed in different ways, including ImageNet-crop, ImageNet-resize, LSUN-crop, and LSUN-resize, resulting in four sets of images covering the 11 categories. The comparison of the recognition results (F1 scores) of different methods on these four sets of images of different categories is shown in Table 2 below.

[0088] As can be seen from the analysis in Table 2, the open set recognition method provided in this application outperforms other methods in recognizing images from the same dataset that have been processed by different methods.

[0089] Table 2

[0090] OpenMax 66.0 68.4 65.7 66.8 OSRCI 63.6 63.5 65.0 64.8 LadderNet+Openmax 65.3 67.0 65.2 65.9 DHRNet+Openmax 65.5 67.5 65.6 66.4 CROSR 72.1 73.5 72.0 74.9 C2AE 83.7 82.6 80.6 80.1 CGDL 84.0 83.2 80.6 81.2 PROSER 84.9 82.4 87.5 85.6 The method of this application 87.0 84.2 87.5 88.5

[0091] Table 3 shows the performance comparison results of five open-set recognition methods for detecting unknown classes on three large-scale datasets. The analysis in Table 3 demonstrates that the method provided in this application outperforms other methods on various large-scale datasets.

[0092] Table 3

[0093] Softmax 79.7 78.4 53.3 CPN 82.3 79.5 54.5 RPL 81.2 80.2 55.1 PMAL 94.9 93.9 71.7 The method of this application 98.1 95.5 81.9

[0094] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments. Other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application.

[0095] like Figure 6 As shown, Figure 6 This is a schematic diagram of a recognition device 600 provided by this application. The recognition device 600 includes an acquisition module 610 and a recognition module 620. The acquisition module 610 is used to acquire an image to be recognized by inputting the image to be recognized into multiple pre-trained recognition models. After acquiring the image to be recognized, the recognition module 620 uses the multiple pre-trained recognition models to recognize the image to obtain the category corresponding to the image to be recognized.

[0096] In one possible implementation, the recognition device further includes a training module 630, which is used to construct multiple different recognition models and train the different recognition models based on the data in the training set to obtain the multiple trained recognition models.

[0097] Specifically, the operation implemented by the aforementioned identification module 620 can be referred to the above. Figure 4 The operations described in the corresponding embodiments, and the operations implemented by the training module 630 described above, can be referred to the above. Figure 2 The operations described in the corresponding embodiments will not be repeated here.

[0098] In this embodiment, each module of the identification device 600 can be implemented in software or in hardware. For example, the implementation of identification module 620 will be described below.

[0099] As an example of a software functional module, the identification module 620 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the identification module 620 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0100] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0101] As an example of a hardware functional unit, module 620 may include at least one computing device, such as a server. Alternatively, module A may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0102] The multiple computing devices included in the identification module 620 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in module A can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in module A can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0103] It should be noted that, in other embodiments, the recognition module 620 can be used to execute any step in the image recognition method, and the training module 630 can be used to execute any step in the image recognition method. The steps that each module is responsible for implementing can be specified as needed. By implementing different steps in the image recognition method through each module, all functions of the recognition device can be realized.

[0104] See Figure 7 , Figure 7This is a schematic diagram of a computing device 700 provided in an embodiment of this application. The computing device 700 includes one or more processors 710, a communication interface 720, and a memory 730. The processors 710, the communication interface 720, and the memory 730 are interconnected via a bus 740.

[0105] The specific implementation of various operations performed by the processor 710 can be found in the above description. Figure 2 and Figure 4 The specific operations in the method embodiments shown will not be repeated here.

[0106] The processor 710 can be implemented in various ways. For example, it can be a central processing unit (CPU), a single-core processor, or a multi-core processor. The processor 710 can be a combination of a CPU and hardware chips. These hardware chips can be ASICs, PLDs, or combinations thereof. The PLDs can be complex programmable logic devices (CPLDs), FPGAs, generic array logic (GALs), or any combination thereof. The processor 710 can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs).

[0107] The communication interface 720 can be a wired interface or a wireless interface, used for communication with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., while the wireless interface can be a cellular network interface or a wireless LAN interface, etc. In this embodiment, the communication interface 720 can specifically be used for training data or to acquire images to be recognized, etc.

[0108] Memory 730 may be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 730 may also be volatile memory, which may be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0109] The memory 730 can also be used to store program code and data, such as a trained recognition model, so that the processor 710 can call the program code stored in the memory 730 to execute the operation steps of recognizing the image to be recognized in the above method embodiment. Furthermore, the computing device 700 may include, compared to... Figure 7 The number of components displayed may be more or less, or there may be different component configurations.

[0110] Bus 740 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 740 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0111] Optionally, the computing device 700 may also include an input / output interface 750, which is connected to an input / output device for receiving user-input data, displaying recognition results, etc.

[0112] Specifically, the specific implementation of the various operations performed by the aforementioned computing device 700 can be referred to the above. Figure 2 and Figure 4 The specific operations in the method embodiments shown will not be repeated here.

[0113] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0114] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 730 in one or more computing devices 700 in the computing device cluster may store the same instructions for executing the routing method described in the above method embodiments.

[0115] In some possible implementations, the memory 730 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the identification method described in the above method embodiments. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for the identification method described in the above method embodiments.

[0116] It should be noted that the memory 730 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned identification method. That is, the instructions stored in the memory 730 of different computing devices 700 can implement the functions of one or more modules among the acquisition module 610, the identification module 620, or the training module 630.

[0117] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, the two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 730 in computing device 700A stores instructions for performing the functions of the identification module 620. Simultaneously, the memory 730 in computing device 700B stores instructions for performing the functions of the training module 630.

[0118] It should be understood that Figure 9The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0119] This application also provides a computer-readable storage medium storing instructions that, when executed on a processor, can implement the method steps performed by the authentication platform in the above method embodiments. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can refer to the specific operation of the above method embodiments, and will not be repeated here.

[0120] This application also provides a computer-readable storage medium storing instructions that, when executed on a processor, can implement the method steps described in the above method embodiments. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can be referred to the specific operation of the above method embodiments, and will not be repeated here.

[0121] This application also provides a computer program product, which includes computer instructions. When executed by a computing device, the computing device performs the method as described in the above method embodiments.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0124] The steps in the method of this application embodiment can be adjusted, merged, or deleted in order according to actual needs; the modules in the device of this application embodiment can be divided, merged, or deleted according to actual needs.

[0125] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A recognition method, characterized in that, include: The image to be recognized is input into G trained recognition models. Each trained recognition model includes an image language pre-trained model and a prompt template containing multiple prompt words. The prompt templates of each trained recognition model are different, and G is an integer greater than 1. Based on the G trained recognition models, the probability of the image to be recognized belonging to each category is determined, wherein the i-th trained recognition model outputs the probability of the image to be recognized belonging to multiple categories corresponding to the i-th trained recognition model; The category of the image to be identified is determined based on the probability that the image belongs to each category.

2. The method according to claim 1, characterized in that, The multiple categories corresponding to the i-th trained recognition model include closed-set categories and open-set categories. The closed-set categories are the categories included in the i-th subset used to train the i-th recognition model, and the open-set categories are the categories other than those included in the training set. The training set is a dataset used to train multiple recognition models, and the i-th subset belongs to the training set.

3. The method according to claim 2, characterized in that, The category of the image to be identified is determined based on the probability that the image belongs to each category, including: The probability of each category in the closed set category corresponding to each trained recognition model is obtained from the probabilities of multiple categories output by each trained recognition model. If the maximum probability of each category in the closed set corresponding to each trained recognition model is greater than or equal to a preset threshold, the category corresponding to the maximum probability is determined as the category of the image to be recognized; or, if the maximum probability is less than the preset threshold, the image to be recognized is determined to belong to an unknown class.

4. The method according to claim 2 or 3, characterized in that, The method further includes: The training set is divided into G subsets, each subset containing images of a different category; The recognition model corresponding to the i-th subset is trained by the i-th subset to obtain the i-th trained recognition model, wherein the recognition model corresponding to the i-th subset includes the image language pre-trained model and the prompt template corresponding to the i-th subset, and i is a positive integer less than or equal to G.

5. The method according to claim 4, characterized in that, The step of training the recognition model corresponding to the i-th subset using the i-th subset to obtain the i-th trained recognition model includes: Determine the negative category label corresponding to the i-th subset, wherein the negative category label belongs to the label corresponding to the open set category; The recognition model corresponding to the i-th subset is trained based on the i-th subset, the negative category label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, to obtain the i-th trained recognition model corresponding to the i-th subset.

6. The method according to claim 5, characterized in that, The step of training the recognition model corresponding to the i-th subset based on the i-th subset, the negative class label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset includes: The recognition model corresponding to the i-th subset is trained based on the first sample image in the i-th subset, the negative class label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, and the word vector embedding corresponding to the prompt template corresponding to the i-th subset is updated.

7. An identification device, characterized in that, include: The acquisition module is used to acquire the image to be recognized and input the image to be recognized into G trained recognition models. Each trained recognition model includes an image language pre-trained model and a prompt template including multiple prompt words. The prompt templates of each trained recognition model are different, and G is an integer greater than 1. The recognition module is used to determine the probability that the image to be recognized belongs to each category based on the G trained recognition models, wherein the i-th trained recognition model outputs the probability that the image to be recognized belongs to multiple categories corresponding to the i-th trained recognition model; The recognition module is further configured to determine the category of the image to be recognized based on the probability that the image to be recognized belongs to each category.

8. The apparatus according to claim 7, characterized in that, The multiple categories corresponding to the i-th trained recognition model include closed-set categories and open-set categories. The closed-set categories are the categories included in the i-th subset used to train the i-th recognition model, and the open-set categories are the categories other than those included in the training set. The training set is a dataset used to train multiple recognition models, and the i-th subset belongs to the training set.

9. The apparatus according to claim 8, characterized in that, The identification module is specifically used for: The probability of each category in the closed set category corresponding to each trained recognition model is obtained from the probabilities of multiple categories output by each trained recognition model. If the maximum probability of each category in the closed set corresponding to each trained recognition model is greater than or equal to a preset threshold, the category corresponding to the maximum probability is determined as the category of the image to be recognized; or, if the maximum probability is less than the preset threshold, the image to be recognized is determined to belong to an unknown class.

10. The apparatus according to claim 8 or 9, characterized in that, The device further includes a training module for dividing the training set into G subsets, each subset containing images of different categories; The recognition model corresponding to the i-th subset is trained by the i-th subset to obtain the i-th trained recognition model, wherein the recognition model corresponding to the i-th subset includes the image language pre-trained model and the prompt template corresponding to the i-th subset, and i is a positive integer less than or equal to G.

11. The apparatus according to claim 10, characterized in that, The training module is specifically used for: Determine the negative category label corresponding to the i-th subset, wherein the negative category label belongs to the label corresponding to the open set category; The recognition model corresponding to the i-th subset is trained based on the i-th subset, the negative category label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, to obtain the i-th trained recognition model corresponding to the i-th subset.

12. The apparatus according to claim 11, characterized in that, The training device is specifically used for: The recognition model corresponding to the i-th subset is trained based on the first sample image in the i-th subset, the negative class label corresponding to the i-th subset, and the prompt template corresponding to the i-th subset, and the word vector embedding corresponding to the prompt template corresponding to the i-th subset is updated.

13. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the method as described in any one of claims 1 to 6.

Images