Method, device, electronic device and storage medium for recognizing object in image

Abstract

The application discloses a kind of query vector corresponding object feature, bounding box and confidence;The bounding box of each query vector indicates that the object detection is carried out to the image to be detected in the image to be detected, and the object feature corresponding to each query vector, the bounding box and the confidence are obtained;Based on the confidence corresponding to each query vector, target object feature is determined from the object feature corresponding to multiple query vectors;Attribute feature extraction is carried out to target object feature, and target object attribute feature is obtained;Based on target object attribute feature, the name of target object located in target region is generated, and target object name is obtained.The method according to the application has higher accuracy of target object name.

Description

Technical Field

[0001] This application relates to the field of electronic information technology, and more specifically, to a method, apparatus, electronic device, and storage medium for recognizing objects in an image. Background Technology

[0002] Generative object detection tasks refer to the process of detecting objects in an image, outputting the location of the objects in the image and generating their names simultaneously. Related technologies utilize object detection models to detect objects in the image to obtain their location information and names.

[0003] However, when the object is one that was not covered during the training of the object detection model, the accuracy of the name generated for the object is poor. Summary of the Invention

[0004] In view of this, embodiments of this application propose a method, apparatus, electronic device, and storage medium for identifying objects in an image.

[0005] In a first aspect, embodiments of this application provide a method for identifying objects in an image. The method includes: performing object detection on an image to be detected based on multiple query vectors to obtain object features, detection boxes, and confidence scores corresponding to each query vector; the detection box of each query vector indicates a region in the image to be detected; the object features are feature representations of the region indicated by the detection box of the corresponding query vector in the image to be detected; the confidence score is used to indicate the accuracy of the detection box of the corresponding query vector; determining target object features from the object features corresponding to multiple query vectors based on the confidence scores of each query vector; extracting attribute features from the target object features to obtain target object attribute features; generating a name for the target object located in the target region based on the target object attribute features to obtain a target object name; the target region is the region indicated by the detection box of the query vector corresponding to the target object features in the image to be detected.

[0006] Secondly, embodiments of this application provide an apparatus for identifying objects in an image. The apparatus includes: a detection module, configured to perform object detection on an image to be detected based on multiple query vectors, obtaining object features, detection boxes, and confidence scores corresponding to each query vector; the detection box of each query vector indicates a region in the image to be detected; the object features are feature representations of the region indicated by the detection box of the corresponding query vector in the image to be detected; the confidence score is used to indicate the accuracy of the detection box of the corresponding query vector; a feature determination module, configured to determine target object features from the object features corresponding to multiple query vectors based on the confidence scores corresponding to each query vector; an attribute extraction module, configured to extract attribute features from the target object features to obtain target object attribute features; and a name generation module, configured to generate a name for the target object located in a target region based on the target object attribute features to obtain a target object name; the target region is the region indicated by the detection box of the query vector corresponding to the target object features in the image to be detected.

[0007] Optionally, the object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open-set detector. The device further includes a training module for performing object detection on the first sample image based on multiple initial query vectors using the open-set detector to obtain a first sample detection box corresponding to each initial query vector. The first sample detection box is used to indicate a region in the first sample image. An initial query vector that matches each first labeled detection box in the first sample image is determined from the multiple initial query vectors and used as the first matching query vector corresponding to each first labeled detection box. The number of first labeled detection boxes in the first sample image does not exceed the number of multiple initial query vectors. Each first labeled detection box is used to indicate the location of at least one first sample object in the first sample image. Based on the first sample detection box corresponding to each first matching query vector and the difference between the first labeled detection boxes, the parameters of the open-set detector are adjusted, and the multiple initial query vectors are updated to obtain multiple query vectors.

[0008] Optionally, the open set detector also outputs the first sample object features and sample confidence scores corresponding to each initial query vector; the sample confidence scores of the initial query vectors are used to indicate the accuracy of the first sample detection boxes of the initial query vectors; the training module is also used to determine a first loss based on the difference between the first sample detection boxes and the first labeled detection boxes corresponding to each first matching query vector; determine a second loss based on the sample confidence scores of each of the multiple initial query vectors; and adjust the parameters of the open set detector based on the first loss and the second loss, and update the multiple initial query vectors to obtain multiple query vectors.

[0009] Optionally, the training module is further configured to determine the intersection-union ratio loss based on the intersection of the first sample detection box and the first labeled detection box corresponding to each first matching query vector, and the minimum closure region of the first sample detection box and the first labeled detection box corresponding to each first matching query vector; determine the positional loss based on the positional difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector; and determine the first loss based on the intersection-union ratio loss and the positional loss.

[0010] Optionally, the attribute features of each target object are extracted using a descriptor; the training module is further used to perform object detection on the second sample image based on multiple query vectors, obtaining the second sample object features and second sample detection boxes corresponding to each query vector; the second sample detection box of each query vector indicates a region in the second sample image; the second sample object features are the feature indicators of the region indicated by the second sample detection box of the corresponding query vector in the second sample image; a query vector that matches each second labeled detection box in the second sample image is determined from multiple query vectors, serving as the second matching query vector corresponding to each second labeled detection box; the number of second labeled detection boxes in the second sample image does not exceed the number of multiple query vectors; each second labeled detection box is used to indicate the location of at least one second sample object in the second sample image; attribute features are extracted by the descriptor based on the second sample object features of the second matching query vectors corresponding to each second labeled detection box, obtaining the predicted object attribute features corresponding to the second sample object indicated by each second labeled detection box; the parameters of the descriptor are adjusted based on the differences between the predicted object attribute features and the labeled attribute features corresponding to each second sample object.

[0011] Optionally, the training module is also used to construct reference prompt information corresponding to each second sample object based on the labeled object name and description information of the second sample object; generate object attribute features based on the reference prompt information corresponding to the second sample object by referencing the large language model, and obtain the initial attribute features corresponding to the second sample object; and determine the labeled attribute features corresponding to the second sample object based on the difference between the labeled object name and the initial attribute features corresponding to the second sample object.

[0012] Optionally, the initial attribute features corresponding to each second sample object include multiple single-dimensional attribute features of each dimension. The training module is further configured to construct new prompt information corresponding to the second sample object based on the labeled object name corresponding to the second sample object and the new descriptive information if the number of single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches a similarity threshold is less than a preset number. The module then returns to execute the step of generating object attribute features based on the reference prompt information corresponding to the second sample object through a reference large language model to obtain the initial attribute features corresponding to the second sample object, until the cumulative number of single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches a similarity threshold reaches a preset number. Finally, the module determines the labeled attribute features corresponding to the second sample object from the single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches a similarity threshold.

[0013] Optionally, the target object name is generated using a large language model; the training module is also used to perform object detection on the third sample image based on multiple query vectors, obtaining a third sample detection box corresponding to each query vector; the third sample detection box of each query vector indicates a region in the third sample image; a query vector that matches each third labeled detection box in the third sample image is determined from multiple query vectors, serving as the third matching query vector corresponding to each third labeled detection box; the number of third labeled detection boxes in the third sample image does not exceed the number of multiple query vectors; each third labeled detection box is used to indicate the location of at least one third sample object in the third sample image; attribute features are extracted based on the third sample object features of the third matched query vectors corresponding to each third labeled detection box, obtaining the predicted object attribute features of the third sample object indicated by each third labeled detection box; the name is generated by the large language model based on the predicted object attribute features corresponding to each third labeled detection box, obtaining the predicted object name corresponding to the third sample object indicated by each third labeled detection box; the parameters of the large language model are adjusted based on the differences between the predicted object names corresponding to each third sample object and the labeled object names.

[0014] Optionally, the object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open-set detector; the attribute features of each target object are extracted by a descriptor; the target object name is generated by a large language model; the training module is also used to perform object detection on the fourth sample image based on multiple query vectors using an open-set detector, obtaining the fourth sample detection box corresponding to each query vector; the fourth sample detection box of each query vector indicates a region in the fourth sample image; a query vector that matches each fourth labeled detection box in the fourth sample image is determined from multiple query vectors, serving as the fourth matching query vector corresponding to each fourth labeled detection box; the number of fourth labeled detection boxes in the fourth sample image does not exceed the number of multiple query vectors; each fourth labeled detection box is used to indicate at least one fourth sample pair in the fourth sample image. The location of the object is determined; attribute features are extracted from the fourth sample object features of the fourth matching query vector corresponding to each fourth labeled detection box using a descriptor, resulting in the predicted object attribute features of the fourth sample object indicated by each fourth labeled detection box; the name is generated by the large language model based on the predicted object attribute features of each fourth sample object, resulting in the predicted object name corresponding to each fourth sample object; based on the differences between the fourth sample detection box and the fourth labeled detection box corresponding to each fourth matching query vector, the differences between the predicted object attribute features and the labeled attribute features corresponding to each fourth sample object, and the differences between the predicted object name and the labeled object name corresponding to each fourth sample object, the parameters of the open set detector, descriptor, and large language model are adjusted, and each query vector is updated.

[0015] Optionally, the detection module is also used to extract image features from the image to be detected using the image encoder in the open set detector, to obtain the image features corresponding to the image to be detected; and to perform object detection based on the image features and multiple query vectors using the detection head in the open set detector, to obtain the object features, detection boxes and confidence scores corresponding to each query vector.

[0016] Optionally, the attribute extraction module is further configured to project the target object features through the first projection layer to map the target object features to a feature space aligned with the input of the descriptor, thereby obtaining the target object projected features; and to extract attribute features from the target object projected features through the descriptor, thereby obtaining the target object attribute features.

[0017] Optionally, the name generation module is also used to project the target object attribute features through the second projection layer to map the target object attribute features to a feature space aligned with the input of the large language model, thereby obtaining the target object attribute projection features; and to generate the name of the target object based on the target object attribute projection features through the large language model.

[0018] Optionally, the device further includes a result determination module for determining the object detection result of the image to be detected based on the target object name and the detection box of the target query vector.

[0019] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described method.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the above-described method.

[0021] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the method described above.

[0022] This application provides a method, apparatus, electronic device, and storage medium for identifying objects in an image. In this application, object detection is performed on the image to be detected based on multiple query vectors, obtaining object features, detection boxes, and confidence scores corresponding to each query vector. Then, based on the confidence scores corresponding to each query vector, target object features are determined from the object features corresponding to the multiple query vectors. Next, attribute features are extracted from the target object features to obtain target object attribute features. Finally, based on the target object attribute features, a name is generated for the target object located in the target region to obtain the target object name. Thus, the generation of target object names based on target object attribute features is achieved. The goal is no longer to directly rely on the image features of the image to be detected to generate the target object name. When the target object features are determined based on confidence, the target object features with higher accuracy can be identified. This results in higher accuracy of the target object attribute features determined based on the target object features, which can accurately indicate the attributes of the target object in the image to be detected. Consequently, the target object name generated for the target object is more accurate. Therefore, even if the target object is a target object that has never appeared before, a name can be generated based on the target object attribute features. This overcomes the problem of poor accuracy in generating names for target objects that have never appeared before and improves the accuracy of the target object name generated for the target object. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram is shown illustrating the application scenarios applicable to the embodiments of this application;

[0025] Figure 2 A flowchart illustrating a method for identifying objects in an image according to an embodiment of this application is shown;

[0026] Figure 3 It shows Figure 2 A flowchart of the steps preceding step S110 in the corresponding embodiment is shown in one embodiment;

[0027] Figure 4 A flowchart of a method for identifying objects in an image according to yet another embodiment of this application is shown;

[0028] Figure 5 A flowchart illustrating the process of determining a labeled attribute feature in an embodiment of this application is shown;

[0029] Figure 6 A schematic diagram illustrating the process of determining a labeled attribute feature in an embodiment of this application is shown;

[0030] Figure 7 A flowchart of a method for identifying objects in an image according to another embodiment of this application is shown;

[0031] Figure 8 It shows Figure 2 A flowchart of step S110 in one embodiment is shown in the corresponding example;

[0032] Figure 9 A schematic diagram illustrating a process for determining the name of a target object in an embodiment of this application is shown;

[0033] Figure 10 A block diagram of an apparatus for recognizing objects in an image according to an embodiment of this application is shown;

[0034] Figure 11 A structural block diagram of an electronic device for performing a method for recognizing objects in an image according to embodiments of this application is shown. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0036] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the application. It should be noted that "multiple" as used herein refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0038] Please refer to Figure 1 The diagram illustrates an application scenario applicable to the embodiments of this application. This application scenario includes a terminal 110, a server 120, and a database 130.

[0039] Terminal 110 includes devices such as smartphones, tablets, e-book readers, music players, wearable devices, smart home devices, and in-vehicle terminals. Terminal 110 contains a client for recognizing objects in images. This client can generate the name of the target object based on the image to be detected, or it can send an object recognition request so that server 120 can recognize the image to be detected based on the object recognition request and obtain the name of the target object.

[0040] The server 120 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0041] The remote database 130 can be an independent storage device, a cluster or distributed system consisting of multiple storage devices, or a cloud storage device providing cloud services. The remote database 130 can be used to train open set detectors, train large language models, train descriptors, train the first projection layer, train the second projection layer, and update various sample data for query vectors.

[0042] In this embodiment, the Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text.

[0043] Terminal 110 can generate and send an object recognition request to server 120 (through a client that identifies objects in an image). In response to the object recognition request, server 120 determines the name of the target object based on the image to be detected. Finally, server 120 returns the name of the target object to the terminal so that terminal 110 can display the name of the target object in the image to be detected.

[0044] Of course, the server 120 can also continue to determine the object detection result of the image to be detected and return the object detection result to the terminal 110 so that the terminal 110 can display the object detection result of the image to be detected.

[0045] In some implementations, the server 120 itself can respond to a local object recognition request, directly determine the target object name and / or object detection result based on the image to be detected, and then the server 120 can directly display the target object name and / or object detection result.

[0046] Alternatively, the terminal 110 itself can respond to a local object recognition request, directly determine the target object name and / or object detection result based on the image to be detected, and then the terminal 110 can directly display the target object name and / or object detection result.

[0047] In addition, the server 120 can obtain sample data from the remote database 130, and train an open set detector, a large language model, a descriptor, a first projection layer, a second projection layer, and update the query vector based on the obtained sample data. The trained open set detector, the trained large language model, the trained descriptor, the trained first projection layer, the trained second projection layer, and the updated query vector are stored locally for the purpose of determining the target object name and the object detection result.

[0048] Of course, when the terminal 110 implements the process of recognizing objects in the image according to this application, after the server 120 obtains the trained open set detector, the trained large language model, the trained descriptor, the trained first projection layer, the trained second projection layer, and the updated query vector, it can send the trained open set detector, the trained large language model, the trained descriptor, the trained first projection layer, the trained second projection layer, and the updated query vector to the terminal 110. The terminal 110 stores the trained open set detector, the trained large language model, the trained descriptor, the trained first projection layer, the trained second projection layer, and the updated query vector so that the terminal can recognize objects in the image.

[0049] It is easy to understand that terminal 110 can also obtain sample data from remote database 130 through server 120 to train open set detector, train large language model, train descriptor, train first projection layer, train second projection layer and update query vector, and store the trained open set detector, trained large language model, trained descriptor, trained first projection layer, trained second projection layer and updated query vector locally.

[0050] In some other implementations, the remote database 130 may be a storage module integrated within the server 120 to store the required sample data.

[0051] Of course, in order to efficiently train the open set detector, train the large language model, train the descriptor, train the first projection layer, train the second projection layer, and update the query vector, the terminal 110 and the server 120 can have high-performance GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units) to improve the efficiency of training the open set detector, training the large language model, training the descriptor, training the first projection layer, training the second projection layer, and updating the query vector.

[0052] To more clearly explain the solution of this application, the following embodiments will use an electronic device as the execution subject of the method for recognizing objects in an image.

[0053] Please see Figure 2 , Figure 2 This application illustrates a flowchart of a method for identifying objects in an image according to an embodiment of the present application. The method is applied to an electronic device, which may be... Figure 1 The terminal 110 or server 120 in the middle, the method may include:

[0054] S110. Perform object detection on the image to be detected based on multiple query vectors, and obtain the object features, detection box and confidence score corresponding to each query vector.

[0055] In this context, the detection box for each query vector indicates a region in the image to be detected; the object feature is the feature representation of the region indicated by the detection box of the corresponding query vector in the image to be detected; and the confidence score is used to indicate the accuracy of the detection box of the corresponding query vector.

[0056] Generally, the accuracy of the bounding box of a query vector is positively correlated with the confidence level of the query vector. That is, the higher the accuracy of the bounding box, the higher the confidence level of the query vector, and the higher the probability that the region within the bounding box includes the target object. The more accurate and higher quality the object features corresponding to the query vector, the more accurately the object features can indicate the target object. Conversely, the lower the accuracy of the bounding box, the lower the confidence level of the query vector, and the lower the probability that the region within the bounding box includes the target object. The lower the accuracy and lower the quality of the object features corresponding to the query vector, the less accurately the object features can indicate the target object.

[0057] The confidence level of the query vector can be a value in the range [0,1]. A confidence level of 0 means that the accuracy of the detection box of the query vector is the lowest, and a confidence level of 1 means that the accuracy of the detection box of the query vector is the highest. Of course, the confidence level of the query vector can also be in other numerical forms, which are not limited in this application.

[0058] The image to be detected can be an image that needs to be named. It can include at least one target object. The target object can be any independent object. For example, the target object can be a person, animal, tree, hill, river, vehicle and building in the real environment, or a virtual character, virtual tree, virtual hill and virtual building in the virtual environment.

[0059] The image to be detected can be an image captured by a camera (such as a mobile phone or camera) or an image generated by an image generation model. The image generation model can be, for example, a generative adversarial network or a large language model.

[0060] In this embodiment, multiple query detection boxes can be set, and then each query detection box is vector-encoded to obtain a query vector corresponding to each query detection box. The query detection boxes can be rectangular, circular, or diamond-shaped, etc.

[0061] The query detection box can be feature-encoded using a visual encoder or a fully connected layer (a network layer consisting of one or more fully connected networks) to obtain the encoded result of the query detection box as the query vector. Here, the visual encoder can be a model with a certain feature encoding capability (i.e., pre-trained) such as the visual modality encoder in a large language model, BERT model, or Clip model (Contrastive Language-Image Pre-training, a multimodal pre-trained model).

[0062] Optionally, object detection can be performed based on the image to be detected using the first detection model to obtain the object features, detection boxes, and confidence scores output by the first detection model for each query vector. The shape of the detection boxes output by the first detection model is not limited in this application.

[0063] The training process of the first detection model can include: training a parameter-initialized neural network model based on a fifth sample image including the fifth sample object, the fifth object features corresponding to each query vector, the fifth detection box, and the fifth confidence score, to obtain the first detection model. The fifth object features, the fifth detection box, and the fifth confidence score can be labeled by experts or technicians. The fifth object features corresponding to the query vector can be obtained by extracting features from the region where a labeled fifth sample object is located in the fifth sample image through a visual encoder. The fifth detection box corresponding to the query vector can indicate the region where a labeled fifth sample object is located. The fifth confidence score corresponding to the query vector can be used to indicate the accuracy of the fifth detection box corresponding to the query vector. The higher the accuracy of the fifth detection box, the higher the probability that the region within the fifth detection box includes the fifth sample object; the higher the accuracy and quality of the fifth object features, the more accurately the fifth object features indicate the fifth sample object. Conversely, the lower the accuracy of the fifth detection box, the lower the probability that the region within the fifth detection box includes the fifth sample object; the lower the accuracy and quality of the fifth object features, the less accurately the fifth object features indicate the fifth sample object.

[0064] Of course, the accuracy of the fifth detection box varies depending on the query vector, therefore, the confidence score of the fifth detection box can be different for different query vectors.

[0065] S120. Based on the confidence level corresponding to each query vector, determine the target object features from the object features corresponding to multiple query vectors.

[0066] In this application, the target query vector can be determined from multiple query vectors based on the confidence level corresponding to each query vector, and then the object features corresponding to the target query vector can be determined as the target object features.

[0067] As mentioned earlier, the accuracy of the detection bounding boxes in the query vector is positively correlated with the confidence level of the query vector. Therefore, the target query vector can be selected from the query vectors corresponding to the detection bounding boxes with confidence levels higher than a first threshold, or a specified number of the detection bounding boxes with confidence levels from high to low can be selected as the target query vector, or the query vector corresponding to the detection bounding box with the highest confidence level can be selected as the target query vector. The first threshold can be, for example, 0.6, and the specified number can be, for example, 5.

[0068] Of course, it's easy to understand that the first threshold and the specified number can differ for different images to be detected. For example, for an image to be detected that includes multiple target objects, the first threshold can be set lower or the specified number higher to identify as many target object features as possible, thereby generating as many names as possible for the target objects included in the image to be detected. Conversely, for high-quality or high-resolution images to be detected, the information they contain is more complete and clearer, so the first threshold can be set lower to generate as many names as possible for the target objects included in the image to be detected. For low-quality or low-resolution images to be detected, the information they contain is less complete or clearer, so the first threshold can be set higher to ensure that the generated names are more accurate and reliable.

[0069] S130. Extract the attribute features of the target object to obtain the attribute features of the target object.

[0070] After determining the target object features, attribute features can be extracted from the target object features to obtain the target object attribute features. For each target object feature, the corresponding target object attribute features can be extracted. For ease of understanding, in the following embodiments of this application, the number of target object features is explained as 1.

[0071] The attribute features of the target object can be extracted using an attribute extraction model. This model can be trained on a parameter-initialized neural network using the object features and labeled attribute features of the sixth sample object. The object features of the sixth sample object can be obtained by extracting features from the sixth sample image, including the sixth sample object, using the aforementioned visual encoder. The labeled attribute features of the sixth sample object can be attribute features labeled by experts or technicians. Alternatively, the labeled attribute features of the sixth sample object can be obtained by extracting the attribute information of the sixth sample object using a text encoder. Here, the text encoder can be a model with certain feature encoding capabilities, such as a BERT model, a large language model, or a text modality encoder in a clip model.

[0072] The attribute information of a target object can refer to text describing the attributes of the target object. Attributes of a target object can include shape, color, category, size, density, location, and origin, etc. For example, when the target object is a dog, the dog's attributes can include its size, eye color, coat color, weight, breed, and ancestry.

[0073] S140. Based on the target object's attribute characteristics, generate a name for the target object located in the target area to obtain the target object name.

[0074] Here, the target region is the area in the image to be detected indicated by the detection box of the target query vector. As mentioned earlier, each query vector yields a detection box; therefore, the target region is the area in the image to be detected selected by the detection box of the target query vector.

[0075] After obtaining the target object attribute features corresponding to the target object features of each target query vector, a name can be generated based on the target object attribute features. This name serves as the target object name for the target object in the target region. This can be achieved by using a large language model to generate the name based on the target object attribute features, and then using this name as the name of the target object located in the target region. This fulfills the purpose of generating target object names for target objects located in the target region.

[0076] Alternatively, a name generation model can be used to generate the name of the target object based on its attribute features. Here, the name generation model can be obtained by training a parameter-initialized neural network using the target object attribute features of the seventh sample object and its labeled name. The target object attribute features of the seventh sample object can be attribute features labeled by experts or technicians, or they can be obtained by feature extraction from the attribute information of the seventh sample object using a text encoder.

[0077] In some implementations, after S140, the method may further include: determining the object detection result of the image to be detected based on the target object name and the detection box of the target query vector.

[0078] In other words, after performing object detection on the image to be detected using the first detection model, the object features, detection boxes, and confidence scores of the query vector are obtained. Therefore, for the target query vector, the object features, detection boxes, and confidence scores of the detection boxes can also be obtained. The region indicated by the detection boxes of the target query vector in the image to be detected is the target region. At this point, the detection box corresponding to the target query vector can be drawn in the target region indicated by the target query vector in the image to be detected, and the generated target object name can be labeled on the target region. This allows the target object name and the detection box to jointly indicate the target object within the target region, thus obtaining the object detection result for the image to be detected.

[0079] Of course, in order to more clearly indicate the target object within the target area, different prompting effects can be added to the target area when drawing the detection box in the target area of ​​the image to be detected. The prompting effects can be, for example, adding a transparent layer, drawing an outline of the target object within the target area, and blinking the detection box drawn in the target area.

[0080] In this embodiment, object detection is performed on the image to be detected based on multiple query vectors, obtaining the object features, detection boxes, and confidence scores corresponding to each query vector. Then, based on the confidence scores corresponding to each query vector, the target object features are determined from the object features corresponding to the multiple query vectors. Attribute features are then extracted from the target object features to obtain the target object attribute features. Finally, based on the target object attribute features, the name of the target object located in the target region is generated to obtain the target object name. Thus, the goal of generating the target object name based on the target object attribute features is achieved, instead of directly relying on the image features of the image to be detected to generate the target object name. Furthermore, when the target object features are determined based on the confidence scores, the target object features with higher accuracy can be determined, resulting in higher accuracy of the target object attribute features determined based on the target object features. This accurately indicates the attributes of the target object in the image to be detected, thereby making the generated target object name more accurate. Therefore, even if the target object is a target object that has never appeared before, the name can still be generated based on the target object attribute features, overcoming the problem of poor accuracy in generating names for target objects that have never appeared before, and improving the accuracy of the generated target object name.

[0081] Secondly, after obtaining the target object name, the detection result of the image to be detected is determined based on the target object name and the detection box of the target query vector. The object detection result indicates the location and name of the target object, making it easier for users to intuitively view the target object detected in the image to be detected, thus improving the target detection effect.

[0082] In some embodiments, the aforementioned S110 is implemented using an open-set detector, that is, the object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open-set detector, such as... Figure 3 As shown, before S110, the method further includes:

[0083] S210. The first sample image is detected by an open set detector based on multiple initial query vectors to obtain the first sample detection box corresponding to each initial query vector.

[0084] The first sample detection box is used to indicate a region in the first sample image.

[0085] The first sample image can refer to an image that includes the first sample object. The first sample object can be any type of target object, and there is no limitation here. The initial query vector can be an initialized query vector. It can be obtained by encoding the features of the query detection box through a visual encoder or a fully connected layer to obtain the parameter-initialized initial query detection box. The encoded result is used as the initial query vector corresponding to the initial query detection box. Thus, multiple initial query detection boxes are set to obtain multiple initial query vectors.

[0086] An open-set detector may include an image encoder (which could refer to a convolutional neural network or a visual transformer, etc.) and a detection head (e.g., the DETR algorithm). Thus, a first sample image is input into the image encoder to obtain encoded image features: m image tokens. Then, the detection head decodes each initial query vector based on these image features to obtain a first sample detection box corresponding to each initial query vector. The first sample detection box then indicates a region predicted by the open-set detector for the first sample image; this region is the area selected by the first sample detection box.

[0087] As mentioned above, when performing object detection on an image to be detected, the output includes the object features of the query vector, the detection box, and the confidence score. Therefore, during the training process of the open set detector, when detecting the first sample image, the open set detector also outputs the first sample object features and sample confidence scores corresponding to each initial query vector. The sample confidence score of the initial query vector is used to indicate the accuracy of the first sample detection box of the initial query vector. The higher the accuracy of the first sample detection box, the higher the probability that the region within the first sample detection box includes the first sample object. The higher the accuracy and quality of the first sample object features, the more accurately the first sample object features can indicate the first sample object. Conversely, the lower the accuracy of the first sample detection box, the lower the probability that the region within the first sample detection box includes the first sample object. The lower the accuracy and quality of the first sample object features, the less accurately the first sample object features can indicate the first sample object.

[0088] Therefore, the data processing process of the image encoder can be summarized by Formula 1, as follows:

[0089] f1,f2,…,f m =Encoder(I),f i ∈R d (one)

[0090] Correspondingly, the data processing process of the detection head can be simplified to Formula 2, as follows:

[0091] o1,o2,…,o n =Decoder(Q=q,KV=F),q∈R n×d ,F∈{f1,f2,…,f m} (two)

[0092] Where q represents n initial query vectors, F represents the extracted image features (m image tokens, d-dimensional vectors), and o represents a first sample object feature determined for each initial query vector. Encoder refers to the aforementioned image encoder, and Decoder here is the detection head, which can be a decoder with a cross-attention mechanism.

[0093] S220. Determine an initial query vector from multiple initial query vectors that matches each first labeled detection box in the first sample image, and use it as the first matching query vector corresponding to each first labeled detection box.

[0094] In this process, the number of first labeled detection boxes in the first sample image does not exceed the number of initial query vectors; each first labeled detection box is used to indicate the location of at least one first sample object in the first sample image. That is, the number of initialized initial query detection boxes is not less than the number of first labeled detection boxes. Therefore, in order to ensure that the number of initialized initial query detection boxes is not less than the number of first labeled detection boxes, as many initial query detection boxes as possible can be determined, for example, the number of initialized initial query detection boxes can be 200 or 300, etc.

[0095] Generally, the region indicated by each first bounding box in the first sample image includes at least one first sample object. However, in order to ensure that the features of the region indicated by the first bounding box are accurate and clear, the region indicated by each first bounding box in the first sample image includes one first sample object, so that the first sample object corresponds one-to-one with the first bounding box. This avoids the situation where the features of the region indicated by the first bounding box are chaotic and inaccurate when the region indicated by each first bounding box in each first sample image includes multiple first sample objects.

[0096] Experts or technicians can manually add corresponding first-label detection boxes to each first-sample object in the first-sample image to ensure that the first-label detection boxes are accurate and effective.

[0097] In this embodiment, Hungarian matching can be performed based on the positions of each initial query detection box and each first annotation detection box, so that each first annotation detection box is matched with an initial query detection box, and the initial query vector corresponding to the initial query detection box matched by each first annotation detection box is used as the first matching query vector corresponding to each first annotation detection box.

[0098] S230. Based on the difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector, the parameters of the open set detector are adjusted, and multiple initial query vectors are updated to obtain multiple query vectors.

[0099] After determining multiple first matching query vectors (the same number as the first labeled detection boxes) from multiple initial query vectors, the loss can be determined based on the difference between the first sample detection box corresponding to each first matching query vector and the first labeled detection box (e.g., using cross-entropy loss function, mean squared error loss function, and absolute value error loss function). The determined loss is then used to adjust the parameters of the open set detector, and the multiple initial query vectors are updated. Each updated initial query vector serves as a query vector. In other words, the initial query vectors include adjustable parameters; updating these adjustable parameters yields query vectors that correspond one-to-one with the initial query vectors.

[0100] Correspondingly, the query vector in S110 can be the query vector obtained after updating the initial query vector according to the method of S210-S230 in this embodiment. The query vector has a high selection capability, so when performing object detection on the image to be detected based on the query vector, the object features and confidence (which may also include the detection box) determined by each query vector are more accurate, and the generated target object name is also more accurate.

[0101] In some implementations, S230 may further include: determining a first loss based on the difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector; determining a second loss based on the sample confidence of each of the multiple initial query vectors; and adjusting the parameters of the open set detector based on the first loss and the second loss, and updating the multiple initial query vectors to obtain multiple query vectors.

[0102] In other words, when training the open set detector and updating multiple initial query vectors, the loss determined based on the difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector is used as the first loss, and the second loss is determined by the sample confidence of the initial query vector. Then, the first loss and the second loss are summed, weighted, etc., to obtain the detection box loss. Then, the parameters of the open set detector are adjusted according to the detection box loss, and multiple initial query vectors are updated to obtain multiple query vectors.

[0103] In some implementations, the loss is determined based on the positional difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector (e.g., by determining the loss through the cross-entropy loss function, the mean squared error loss function, and the absolute value error loss function), thus obtaining the first loss.

[0104] The positional difference between the first sample detection box and the first labeled detection box can refer to the difference between their positional information. The positional information of the first sample detection box refers to its location within the first sample image; similarly, the positional information of the first labeled detection box refers to its location within the first sample image. For example, when both the first sample detection box and the first labeled detection box are rectangles, their positional information can be the coordinates of the rectangle's corner points.

[0105] In some other implementations, the intersection-union loss can be determined based on the intersection of the first sample detection box and the first labeled detection box corresponding to each first matching query vector, and the minimum closure region of the first sample detection box and the first labeled detection box corresponding to each first matching query vector; the positional loss can be determined based on the positional difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector; and the first loss can be determined based on the intersection-union loss and the positional loss. Here, the minimum closure region can refer to the minimum bounding rectangle of the first sample detection box and the first labeled detection box.

[0106] Here, the positional loss can be determined based on the positional difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector (for example, by determining the loss through the cross-entropy loss function, the mean squared error loss function, and the absolute value error loss function).

[0107] Meanwhile, for any first matching query vector, the intersection of the predicted detection box and the query detection box corresponding to the first matching query vector, as well as the minimum closure region formed by the predicted detection box and the query detection box corresponding to the first matching query vector, can be determined. Then, based on the intersection and the minimum closure region corresponding to the first matching query vector, the sub-intersection over union ratio loss of the first matching query vector is determined. The sub-intersection over union ratio losses of each first matching query vector are summed to obtain the intersection over union ratio loss.

[0108] For example, the sub-intersection over union ratio loss corresponding to the first matching query vector can be determined according to Formula 3, as follows:

[0109]

[0110] Among them, L zGiou Let Gi be the sub-intersection over union ratio (IoU) loss corresponding to the first matching query vector, Giou be the IoU ratio of the first sample detection box and the first labeled detection box corresponding to the first matching query vector, iou be the intersection of the first sample detection box and the first labeled detection box corresponding to the first matching query vector, C be the minimum closure region corresponding to the first matching query vector, A and B be the first sample detection box and the first labeled detection box corresponding to the first matching query vector, respectively, and C / (A∪B) represents the part of the minimum closure region not covered by A or B.

[0111] The second loss can be obtained by calculating cross-entropy loss, mean squared error loss, and absolute value error loss based on the differences between the sample confidence scores corresponding to multiple initial query vectors and the preset confidence scores. The preset confidence score of the initial query vectors indicates the accuracy of the detection boxes corresponding to the initial query vectors. Detection boxes corresponding to initial query vectors that match the first labeled detection boxes are more accurate, while detection boxes corresponding to initial query vectors that do not match any of the first labeled detection boxes are less accurate. For example, when using values ​​between [0,1] to indicate confidence, a detection box corresponding to an initial query vector that matches the first labeled detection box is more accurate, meaning that the region within the detection box is more likely to include the first sample object, and the first sample object features are more indicative of the first sample object. In this case, the preset confidence score corresponding to the initial query vector that matches the first labeled detection box can be 1. Similarly, a detection box corresponding to an initial query vector that does not match any of the first labeled detection boxes is less accurate, meaning that the region within the detection box is less likely to include the first sample object, and the first sample object features are less indicative of the first sample object. In this case, the preset confidence score corresponding to the initial query vector that does not match any of the first labeled detection boxes can be 0.

[0112] Next, the intersection-union loss and the location loss are summed or weighted to obtain the first loss. For example, the detection box loss determined based on the first loss and the second loss can be denoted as `loss_{undefined}`. detection=l Giou +l l1 +l z , where l Giou To compare the loss between intersection and union, l l1 For position loss, l z This is the second loss.

[0113] It is easy to understand that, when the open set detector includes an image encoder and a detection head, S110 may also include: extracting image features from the image to be detected by the image encoder in the open set detector to obtain the image features corresponding to the image to be detected; and performing object detection based on the image features and multiple query vectors by the detection head in the open set detector to obtain the object features, detection boxes and confidence scores corresponding to each query vector.

[0114] In other words, the image to be detected can be input into an image encoder to obtain encoded image features: m image tokens. Then, the detection head decodes each query vector based on these image features to obtain the object features, detection boxes, and confidence scores corresponding to each query vector. Thus, the detection boxes indicate the region predicted by the open-set detector for the image to be detected; this region is the area selected by the detection box.

[0115] In this embodiment, the open set detector is trained and the initial query vector is updated to obtain the query vector. The open set detector has a high feature extraction capability and the query vector has a high selection capability. This makes it more accurate to obtain the object features and confidence (which may also include the detection box) corresponding to the query vector when performing object detection on the image to be detected based on the query vector and the open set detector. As a result, the target object name generated for the target object is more accurate, and the determined object detection result is also more accurate.

[0116] In some embodiments, the attribute features of each target object are extracted using a descriptor; such as... Figure 4 As shown, before S130, the method further includes:

[0117] S310. Perform object detection on the second sample image based on multiple query vectors to obtain the second sample object features and the second sample detection box corresponding to each query vector.

[0118] In this context, the second sample detection box of each query vector indicates a region in the second sample image; the second sample object feature is the feature indication of the region indicated by the second sample detection box of the corresponding query vector in the second sample image.

[0119] The second sample image can refer to an image that includes a second sample object. The second sample object can be any type of target object, and there is no limitation here.

[0120] Here, object detection can be performed on the second sample image based on multiple query vectors using the aforementioned open-set detector or the first detection model. This yields the second sample object features and the second sample detection box corresponding to each query vector. Additionally, the second sample confidence score for each query vector can also be obtained. Similarly, the second sample confidence score indicates the accuracy of the second sample detection box for the query vector. A higher accuracy score means a higher probability that the region within the second sample detection box includes the second sample object, and a higher accuracy and better quality of the second sample object features, making the second sample object features more indicative of the second sample object. Conversely, a lower accuracy score means a lower probability that the region within the second sample detection box includes the second sample object, and a lower accuracy and worse quality of the second sample object features, making the second sample object features less indicative of the second sample object. However, this second sample confidence score is not involved in the training process; therefore, in this embodiment, it can be left unprocessed.

[0121] Additionally, it is worth mentioning that S310 can be executed before training the open set detector, in which case the query vector is obtained by encoding a query detection box. Of course, S310 can also be executed after training the open set detector, in which case the query vector is the result of updating the initial query vector.

[0122] S320. Determine the query vector that matches each of the second labeled detection boxes in the second sample image from multiple query vectors, and use it as the second matching query vector corresponding to each of the second labeled detection boxes.

[0123] The number of second-labeled detection boxes in the second sample image does not exceed the number of multiple query vectors; each second-labeled detection box is used to indicate the location of at least one second sample object in the second sample image.

[0124] Generally, each region indicated by a second bounding box in a second sample image includes at least one second sample object. However, in order to ensure that the features of the region indicated by the second bounding box are accurate and clear, each region indicated by a second bounding box in a second sample image includes one second sample object, so that the second sample object corresponds one-to-one with the second bounding box. This avoids the situation where the features of the region indicated by the second bounding box are chaotic and inaccurate when multiple second sample objects are included in the region indicated by each second bounding box in each second sample image.

[0125] Experts or technicians can manually add corresponding second-label detection boxes to each second-sample object in the second-sample image to ensure that the second-label detection boxes are accurate and effective.

[0126] In some implementations, the query vector is obtained by encoding a query detection box. The query detection box corresponding to each query vector can be obtained. Then, based on the position of each query detection box and the position of each second labeled detection box, Hungarian matching is performed so that each second labeled detection box is matched with a query detection box. The query vector corresponding to the query detection box matched by each second labeled detection box is used as the second matching query vector corresponding to each second labeled detection box.

[0127] Of course, in some other implementations, the query vector is the result of updating the initial query vector. In this case, the position of each query vector can be decoded to obtain the target query detection box for each query vector. Then, based on the position of each target query detection box and the position of each second-labeled detection box, Hungarian matching is performed to match each second-labeled detection box with a target query detection box. The query vector corresponding to the target query detection box matched by each second-labeled detection box is used as the second matching query vector corresponding to each second-labeled detection box. Here, the query vector can be decoded by a multilayer perceptron (MLP) or a fully connected layer (one or more fully connected layers).

[0128] S330. The attribute features of the second sample object are extracted by the descriptor based on the second matching query vector corresponding to each second labeled detection box, so as to obtain the predicted object attribute features corresponding to the second sample object indicated by each second labeled detection box.

[0129] The descriptor can be an encoder for the text modality in a large language model or clip model, an autoencoder, or a variational autoencoder, etc.

[0130] The second sample object features corresponding to the second matching query vector of the second labeled detection box can be input into the descriptor. The descriptor extracts the attribute features and outputs the attribute features of the second sample object indicated by each of the second labeled detection boxes, which are used as the predicted object attribute features of the second sample object.

[0131] It is easy to understand that, as mentioned above, the second labeled detection box indicates at least one second sample object. Therefore, after the descriptor extracts the attribute features of the second sample object features of the second matching query vector corresponding to the second labeled detection box, the obtained attribute feature extraction result is used as the predicted object attribute feature of each second sample object indicated by the second labeled detection box.

[0132] In some implementations, a first projection layer may also be provided. Thus, S330 may include: projecting the second sample object features through the first projection layer to map the second sample object features to a feature space aligned with the input of the descriptor, obtaining a second sample object projection feature; and extracting attribute features from the second sample object projection feature using the descriptor to obtain a predicted object attribute feature. That is, for each second labeled detection box, the second sample object features of the second matching query vector corresponding to the second labeled detection box are used to obtain a second sample object projection feature corresponding to the second labeled detection box. Then, attribute features are extracted from this second sample object projection feature to obtain the predicted object attribute feature corresponding to the second sample object indicated by the second labeled detection box. The first projection layer may be a multilayer perceptron or a fully connected layer (one or more fully connected layers).

[0133] In this embodiment, the features of the second sample object are projected through a first projection layer to change the feature space of the second sample object features. This maps the features of the second sample object to a feature space aligned with the input of the descriptor, resulting in features that can be used by the descriptor. This allows the descriptor to better understand the projected features of the second sample object, making the predicted object attribute features generated by the descriptor based on the projected features of the second sample object more accurate, thereby improving the training effect of the descriptor.

[0134] For example, when both the second annotation detection box and the second sample detection box are rectangular boxes, the process described above in S320 and S330 can be summarized as Formula 4 when indicating the detection box through the four corner points of the rectangle. Formula 4 is as follows:

[0135]

[0136] Among them, o i 2 Let $\mathbf{i}$ be the second matching query vector, $k$ be the number of second labeled detection boxes, $GT$ be the set of $k$ second labeled detection boxes, $s$ be the number of query vectors, $proj1$ be the data processing of the first projection layer, $captioner$ be the data processing process of the descriptor, and $P$ be the second matching query vector. y It is the set of attribute features of the predicted object.

[0137] Correspondingly, in the case of a first projection layer, step S130 may further include projecting the target object features through the first projection layer to map the target object features to a feature space aligned with the input of the descriptor, thereby obtaining the target object projected features; and then extracting attribute features from the target object projected features using the descriptor to obtain the target object attribute features. That is, for each target object feature, the target object feature can be input into the first projection layer to obtain the target object projected features of that feature, and then the target object projected features of that feature can be input into the descriptor for attribute feature extraction to obtain the target object attribute features corresponding to that feature.

[0138] S340. Based on the differences between the predicted object attribute features and the labeled attribute features corresponding to each second sample object, adjust the parameters of the descriptor.

[0139] The labeled attribute features corresponding to the second sample object refer to the features labeled by experts or technicians. These features can be obtained by extracting features from the attribute information (the attribute information labeled by experts or technicians) of the second sample object through a text encoder.

[0140] Based on the differences between the predicted object attribute features and the labeled attribute features corresponding to each second sample object, cross-entropy loss, mean squared error loss, and absolute value error loss can be determined as the prediction loss for each second sample object. The prediction losses of each second sample object are summed to obtain the attribute prediction loss. The parameters of the descriptor are adjusted through the attribute prediction loss to achieve the training of the descriptor.

[0141] For example, when the loss function is cross-entropy loss, the attribute prediction loss is loss. attribute =BCE(P1,D), where P1 is the predicted object attribute feature corresponding to the second sample object, and D is the labeled attribute feature corresponding to the second sample object.

[0142] In addition, when setting the first projection layer, it is necessary to adjust the parameters of the first projection layer based on the attribute prediction loss so that the projection effect of the first projection layer after training is better, thereby making the accuracy of the extracted target object attribute features higher.

[0143] In this embodiment, the descriptor is trained to have a high attribute feature extraction capability, which makes the extracted target object attribute features more accurate when performing attribute feature extraction based on the target object features. As a result, the target object name generated for the target object is more accurate, and the determined object detection result is also more accurate.

[0144] In some embodiments, such as Figure 5As shown, the process of determining the aforementioned labeled attribute features includes:

[0145] S410. For each second sample object, construct reference prompt information corresponding to the second sample object based on the target object name and description information of the second sample object.

[0146] The descriptive information of the second sample object refers to the information used to describe the second sample object. In some embodiments, the descriptive information of the second sample object may be information related to the second sample object in the image description information of the second sample image. The image description information of an image refers to information describing the content of the image. It may be the image description information created by experts or technicians, or the image description information generated by a large language model based on the content of the image.

[0147] Of course, in some other implementations, the descriptive information of the second sample object can also be information describing the attributes and name of the second sample object. Specifically, the descriptive information of the second sample object can be created by experts or technicians, or it can be created by a large language model based on the region where the second sample object is located in the second sample image. Thus, the descriptive information of the second sample object can be different for different second sample objects.

[0148] For ease of understanding, the following embodiments will be explained using the second sample object as a single entity. When there are multiple second sample objects, the method for determining the labeled attribute features of each second sample object is consistent.

[0149] The annotation target object name and description information of the second sample object can be combined into a prompt message, which is then input into the first large language model (this first large language model may be different from the large language model used for name generation mentioned above). Based on this prompt message, the first large language model determines the first prompt message corresponding to the second sample object.

[0150] Alternatively, the name and description of the target object of the second sample object can be directly concatenated into a single piece of information, which can then be used as a reference prompt for the second sample object.

[0151] S420. By referencing the large language model, object attribute features are generated based on the reference prompt information corresponding to the second sample object, and the initial attribute features corresponding to the second sample object are obtained.

[0152] The reference large language model here can be different from or the same as the aforementioned large language model and the first large language model. For each second sample object, the reference prompt information corresponding to the second sample object can be input into the reference large language model. The reference large language model then generates object attribute features based on the reference prompt information corresponding to the second sample object, and outputs the generated attribute features as the initial attribute features corresponding to the second sample object. In this way, each second sample object is traversed to obtain its own initial attribute features.

[0153] This process can be expressed as D = (d1, d2, ..., d...). c ),d1,d2,…,d c =LLM(input text), where input text is the reference prompt information corresponding to the second sample object, LLM is the reference large language model, and D is the c-dimensional attribute of the second sample object.

[0154] S430. Based on the difference between the labeled target object name and the initial attribute features corresponding to the second sample object, determine the labeled attribute features corresponding to the second sample object.

[0155] Based on the difference between the name of the labeled target object corresponding to the second sample object and the initial attribute features, the initial attribute features corresponding to the second sample object can be used as the labeled attribute features. Alternatively, the adjusted result obtained after adjusting the initial attribute features corresponding to the second sample object can be used as the labeled attribute features.

[0156] Here, the cosine similarity, Euclidean distance, and Manhattan distance between the labeled target object name and the initial attribute features corresponding to the second sample object can be determined as the difference between the labeled target object name and the initial attribute features corresponding to the second sample object. If the difference is less than the difference threshold, the initial attribute features corresponding to the second sample object can be used as the labeled attribute features. If the difference is not less than the difference threshold, the initial attribute features corresponding to the second sample object can be adjusted, and the adjusted result can be used as the labeled attribute features.

[0157] In this embodiment, the weighted sum of the target object name and the initial attribute features corresponding to the second sample object can be used as the attribute feature corresponding to the second sample object. Here, the weight of the target object name can be higher than the weight of the initial attribute features so that the obtained attribute feature matches the target object name as closely as possible.

[0158] In some other implementations, the initial attribute features corresponding to each second sample object include multiple single-dimensional attribute features for each dimension; S430 may include:

[0159] A1. If the number of single-dimensional attribute features whose similarity to the labeled target object name of the second sample object reaches the similarity threshold is less than the preset number, construct new prompt information corresponding to the second sample object based on the labeled target object name and new description information.

[0160] Here, the similarity between the labeled target object name of the second sample object and the multiple single-dimensional attribute features included in the initial attribute features can be evaluated using cosine similarity, Jaccard similarity, or an embedding-based similarity calculation method (such as BERT embedding). Therefore, the similarity between the labeled target object name of the second sample object and the initial attribute features is score = sim(Y,D), where Y is the labeled target object name of the second sample object. In this case, the obtained similarity includes the similarity between the labeled target object name and each single-dimensional attribute feature.

[0161] The preset quantity can be a value set based on requirements, and the preset quantity should not exceed the number of dimensions included in the initial attribute features. For example, if the initial attribute features include 20 dimensions, then the preset quantity can be 15. It is understandable that a larger preset quantity can be set to more accurately indicate the attribute features.

[0162] The higher the similarity between the labeled target object name and the single-dimensional attribute feature, the higher the fit between the labeled target object name and the single-dimensional attribute feature, and the more accurately the single-dimensional attribute feature can indicate the labeled target object name. Conversely, the lower the similarity between the labeled target object name and the single-dimensional attribute feature, the lower the fit between the labeled target object name and the single-dimensional attribute feature, and the less accurately the single-dimensional attribute feature can indicate the labeled target object name. Therefore, when the similarity between the labeled target object name and the single-dimensional attribute feature reaches the similarity threshold, it means that the labeled target object name and the single-dimensional attribute feature have a high fit, and the single-dimensional attribute feature can accurately indicate the labeled target object name.

[0163] Therefore, if the number of single-dimensional attribute features whose similarity to the target object name of the second sample object reaches a similarity threshold is not less than a preset number, it means that the preset number of single-dimensional attribute features in the obtained initial attribute features can accurately indicate the target object name. In this case, a preset number of single-dimensional attribute features are directly selected from the single-dimensional attribute features whose similarity reaches the similarity threshold to form a total feature, which is then used as the annotation attribute feature of the second sample object. This can be done by selecting a preset number of single-dimensional attribute features in descending order of similarity, or by randomly selecting a preset number of single-dimensional attribute features. In this case, the annotation attribute feature has a high degree of fit with the target object name, can serve as the true attribute feature of the second sample object, and has good distinguishability between different target objects.

[0164] It is easy to understand that if the number of single-dimensional attribute features whose similarity to the target object name of the second sample object reaches the similarity threshold is less than the preset number, it means that the initial attribute features obtained do not have the preset number of single-dimensional attribute features that can accurately indicate the target object name. If the preset number of single-dimensional attribute features is determined to form a labeling attribute feature, the resulting labeling attribute feature will have single-dimensional attribute features with low similarity, resulting in inaccurate labeling attribute features. Therefore, the description information of the second sample object can be re-determined as new description information.

[0165] Then, based on the labeled target object name and the new description information corresponding to the second sample object, a new prompt message corresponding to the second sample object is constructed. Then, the process returns to step S420 until the cumulative number of single-dimensional attribute features whose similarity to the labeled target object names of the second sample object reaches the similarity threshold reaches the preset number, and then A2 is executed.

[0166] It's worth noting that the multiple dimensions included in each generated initial attribute feature are actually abstract dimensions, not fixed to any specific attribute. For example, both dimensions 1 and 2 indicate color, but their single-dimensional attribute features are different, meaning that dimensions 1 and 2 represent color from different perspectives. Therefore, the newly generated single-dimensional attribute features in each loop can all be different. The number of single-dimensional attribute features in each generated single-dimensional attribute feature that indicates the similarity between the target object names reaches a similarity threshold can be accumulated. If a preset number is reached, then A2 is executed.

[0167] A2. Determine the annotation attribute features corresponding to the second sample object from the single-dimensional attribute features whose similarity to the annotation target object name of the second sample object reaches the similarity threshold.

[0168] Similarly, a predetermined number of single-dimensional attribute features can be selected from high to low similarity to form a labeled attribute feature, or a predetermined number of single-dimensional attribute features can be randomly selected to form a labeled attribute feature. In this case, the labeled attribute feature has a high degree of fit with the name of the labeled target object and can be used as the real attribute feature of the second sample object.

[0169] For example, the process of determining the labeled attribute features is as follows: Figure 6 As shown, the constructed reference prompt information includes "valid features describing the object name in the detection box of the image" and "some attributes already exist to describe the object name, please provide more attributes." These are used to prompt the reference large language model to generate initial attribute features. After generating the initial attribute features, the similarity between the initial attribute features and the labeled target object name is determined. If the number of single-dimensional attribute features whose similarity to the labeled target object name reaches the similarity threshold does not reach the preset number, the similarity is determined to be low, the reference description information is updated, new reference description information is obtained, and the initial attribute features are regenerated based on the new reference description information. This process continues until the cumulative number of single-dimensional attribute features whose similarity reaches the similarity threshold reaches the preset number, thus obtaining the labeled attribute features.

[0170] In this embodiment, users do not need to manually label the corresponding attribute features of the second sample object, thereby improving the generation efficiency of the attribute features of the second sample object and reducing the labeling cost of the attribute features of the second sample object. At the same time, it reduces the occurrence of labeling errors when manually labeling the attribute features of the second sample object, improves the accuracy of the attribute features of the second sample object, and thus makes the model of the descriptor trained based on the attribute features of the second sample object perform better.

[0171] In some embodiments, the target object name is generated using a large language model; such as Figure 7 As shown, before S140, the method further includes:

[0172] S510. Perform object detection on the third sample image based on multiple query vectors to obtain the third sample detection box corresponding to each query vector; determine the query vector that matches each third labeled detection box in the third sample image from the multiple query vectors, and use it as the third matching query vector corresponding to each third labeled detection box; extract attribute features based on the third sample object features corresponding to the third matched query vector of each third labeled detection box to obtain the predicted object attribute features of the third sample object indicated by each third labeled detection box.

[0173] In this context, the third sample detection box of each query vector indicates a region in the third sample image; the number of third annotation detection boxes in the third sample image does not exceed the number of multiple query vectors; and each third annotation detection box is used to indicate the location of at least one third sample object in the third sample image.

[0174] Here, the principle of S510 is similar to that of S310-S330 mentioned above, and will not be repeated here. It is worth mentioning that the descriptor (which may include the first projection layer) or the attribute extraction model can be used to extract the predicted object attribute features of the third sample objects indicated by the third labeled detection boxes. When extracting the predicted object attribute features of the third sample objects indicated by the third labeled detection boxes through the attribute extraction model, the specific principle is described in S130, and will not be repeated here.

[0175] S520. The name is generated by the large language model based on the attribute features of the predicted object corresponding to each third labeled detection box, so as to obtain the name of the predicted target object corresponding to the third sample object indicated by each third labeled detection box.

[0176] The large language model generates names based on the attribute features of the predicted object corresponding to each third-labeled detection box, thus obtaining the name of the predicted target object corresponding to the third sample object indicated by each third-labeled detection box.

[0177] In other words, the predicted object attribute features of each third-labeled detection box can be input into the large language model. The large language model generates a name based on the predicted object attribute features of the third-labeled detection box, and obtains a target object name, which serves as the predicted target object name corresponding to the third sample object indicated by the third-labeled detection box.

[0178] Of course, in this embodiment, a second projection layer can also be set. Therefore, S520 may include: projecting the predicted object attribute features corresponding to the third labeled detection box through the second projection layer to map the predicted object attribute features corresponding to the third labeled detection box to a feature space aligned with the input of the large language model, thereby obtaining the predicted target object attribute projection features corresponding to the third labeled detection box; and generating a name based on the predicted target object attribute projection features corresponding to the third labeled detection box through the large language model, thereby obtaining the predicted target object name corresponding to the third sample object indicated by the third labeled detection box.

[0179] The second projection layer projects the predicted object attribute features corresponding to the third labeled detection box, thereby changing the feature space of the predicted object attribute features corresponding to the third labeled detection box. This results in predicted target object attribute projection features that can be used by the large language model. As a result, the predicted target object name of the third sample object generated by the large language model based on the predicted target object attribute projection features corresponding to the third labeled detection box is more consistent with the predicted object attribute features corresponding to the third labeled detection box, thus improving the name generation effect.

[0180] For example, step S520 can be briefly described as: P x =LLA 0 (proj2(P z ), where P z To predict object attribute features, proj2 is the data processing procedure for the second projection layer, LLA. 0 For the data processing of large language models, P x To predict the name of the target object.

[0181] It is worth mentioning that, with a second projection layer, S140 may include: projecting the target object attribute features through the second projection layer to map the target object attribute features to a feature space aligned with the input of the large language model, obtaining target object attribute projection features; and generating names for the target objects based on the target object attribute projection features using the large language model, obtaining the target object name. For each target object feature, the second projection layer projects the corresponding target object attribute features to obtain a target object attribute projection feature, and the large language model generates a name based on this target object attribute projection feature, obtaining the target object name of the target object within the target region indicated by the detection box of the target query vector corresponding to the target object feature.

[0182] S530. Based on the differences between the predicted target object name and the labeled target object name corresponding to each third sample object, the parameters of the large language model are adjusted.

[0183] Among them, the labeling of the third sample object; the name can refer to the name of the labeling of the third sample object by experts or technicians, and the name of the target object of the labeling of the third sample object is the real name of the third sample object, which is accurate and appropriate.

[0184] Based on the difference between the predicted target object name and the labeled target object name corresponding to each third sample object, cross-entropy loss, mean squared error loss, and absolute value error loss can be determined as the sub-name prediction loss for each third sample object. The sub-name prediction losses of each third sample object are summed to obtain the name prediction loss. The parameters of the large language model are adjusted through the name prediction loss to achieve the training of the large language model.

[0185] For example, when the loss function is cross-entropy loss, the name prediction loss is loss. object =BCE(P2,Y3), where P2 is the name of the predicted target object corresponding to the second sample object, and Y3 is the name of the labeled target object corresponding to the third sample object.

[0186] In addition, when setting a second projection layer, it is necessary to adjust the parameters of the second projection layer based on the name prediction loss so that the projection effect of the trained second projection layer is better, thereby making the accuracy of the generated target object name higher.

[0187] In this embodiment, a large language model is trained. The large language model has a high name generation capability, which makes the generated target object name more accurate when generating names based on the target object attribute features, and thus the determined object detection results are also more accurate.

[0188] In addition, when training the open set detector, descriptor extractor, and large language model, the parameters of the other two can be fixed while training one of them to ensure targeted training and thus improve the performance of the trained open set detector, descriptor extractor, and large language model.

[0189] In some embodiments, the object features, bounding boxes, and confidence scores corresponding to each query vector are determined using an open-set detector; the attribute features of each target object are extracted using a descriptor; and the target object name is generated using a large language model. Figure 8 As shown, before S110, the method further includes:

[0190] S610. The fourth sample image is detected by an open set detector based on multiple query vectors to obtain the fourth sample detection box corresponding to each query vector; the query vector that matches each fourth labeled detection box in the fourth sample image is determined from the multiple query vectors and used as the fourth matching query vector corresponding to each fourth labeled detection box.

[0191] S620. The descriptor extracts the attribute features of the fourth sample object based on the fourth matching query vector corresponding to each fourth labeled detection box, and obtains the predicted object attribute features of the fourth sample object indicated by each fourth labeled detection box.

[0192] S630. Generate names based on the predicted object attribute features of each fourth sample object using a large language model, and obtain the predicted target object name corresponding to each fourth sample object.

[0193] The principle of S610 is the same as that of S210-S230, the principle of S620 is the same as that of S330, and the principle of S630 is the same as that of S520. They will not be repeated here.

[0194] S640. Based on the differences between the fourth sample detection box and the fourth labeled detection box corresponding to each fourth matching query vector, the differences between the predicted object attribute features and labeled attribute features corresponding to each fourth sample object, and the differences between the predicted target object name and labeled target object name corresponding to each fourth sample object, the parameters of the open set detector, descriptor, and large language model are adjusted, and each query vector is updated.

[0195] In other words, based on the differences between the fourth sample detection box and the fourth labeled detection box corresponding to each fourth matching query vector, the differences between the predicted object attribute features and the labeled attribute features corresponding to each fourth sample object, and the differences between the predicted target object name and the labeled target object name corresponding to each fourth sample object, an open set detector, a descriptor, and a large language model are trained simultaneously, and the query vector is continuously updated. This allows the trained open set detector, descriptor, and large language model, along with the updated query vector, to generate object names and detect target objects in the image to be detected.

[0196] It is worth mentioning that the first update loss can be determined here based on the difference between the fourth sample detection box and the fourth labeled detection box corresponding to each matching query vector. The process of determining the first update loss is the same as the principle of determining the detection box loss mentioned above, and will not be repeated here.

[0197] Meanwhile, based on the difference between the predicted object attribute features and the labeled attribute features corresponding to each fourth sample object, the second update loss is determined. The determination process of the first update loss is the same as the determination principle of the aforementioned attribute prediction loss, and will not be repeated here.

[0198] Furthermore, based on the difference between the predicted target object name and the labeled target object name corresponding to each fourth sample object, the third update loss is determined. The determination process of the third update loss is consistent with the determination principle of the aforementioned name prediction loss, and will not be repeated here.

[0199] Then, the first update loss, the second update loss, and the third update loss are summed or weighted and summed to obtain the total update loss. The open set detector, descriptor, and large language model are then trained using the total update loss, and the query vector is updated.

[0200] In this embodiment, the open set detector, descriptor, large language model, and query vector with certain capabilities are further updated to improve their capabilities. This makes the generated target object names more accurate and the obtained object detection results more accurate when detection is performed based on the open set detector, descriptor, large language model, and query vector.

[0201] For example, after obtaining the open-set detector, descriptor, large language model, and query vector, the process of determining the name of the target object in the image to be detected using the open-set detector, descriptor, large language model, and query vector is as follows: Figure 9 As shown.

[0202] The image encoder of the open set detector extracts features from the image to be detected, and the open set detector head determines the object features, detection box, and confidence level of the query vector based on the image features and the query vector.

[0203] The target query vector is determined from the query vector by confidence score. Then, the target object features corresponding to the target query vector are input into the descriptor to extract attribute features and obtain target object attribute features. Finally, the target object name is generated by the large language model based on the target object attribute features. The target object name is the name of the target object in the region selected by the detection box of the target query vector in the image to be detected.

[0204] Of course, a first projection layer and a second projection layer can also be set. Before the target object features of the target query vector are input into the descriptor for attribute feature extraction, the target object features of the target query vector are first projected through the first projection layer. Similarly, when the name is generated by the large language model based on the target object attribute features, the target object attribute features are first projected through the second projection layer. Thus, through the projection processing of the first and second projection layers, the target object features of the target query vector are projected onto the feature space targeted by the large language model, so that the result after the projection processing of the second projection layer can be better understood by the large language model, making the generated target object name more accurate.

[0205] After testing, the method of this application significantly outperforms existing target detection models. In certain target object domains, the detection accuracy is improved by more than 2%. This means that the method of this application not only enhances the model's generalization ability when dealing with diverse and unknown objects, but also significantly improves the overall accuracy and reliability of detection and recognition.

[0206] Please see Figure 10 , Figure 10A block diagram of an apparatus for identifying objects in an image according to an embodiment of this application is shown. The apparatus 1000 includes:

[0207] The detection module 1010 is used to perform object detection on the image to be detected based on multiple query vectors, and obtain the object features, detection boxes, and confidence scores corresponding to each query vector; the detection box of each query vector indicates a region in the image to be detected; the object features are the feature representations of the region indicated by the detection box of the corresponding query vector in the image to be detected; the confidence score is used to indicate the accuracy of the detection box of the corresponding query vector.

[0208] The feature determination module 1020 is used to determine the target object features from the object features corresponding to multiple query vectors based on the confidence level corresponding to each query vector.

[0209] The attribute extraction module 1030 is used to extract attribute features from the target object to obtain the target object attribute features;

[0210] The name generation module 1040 is used to generate a name for a target object located in a target region based on the target object's attribute features, thereby obtaining the target object's name; the target region is the area indicated by the detection box of the query vector corresponding to the target object's features in the image to be detected.

[0211] Optionally, the object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open-set detector. The device further includes a training module for performing object detection on the first sample image based on multiple initial query vectors using the open-set detector to obtain a first sample detection box corresponding to each initial query vector. The first sample detection box is used to indicate a region in the first sample image. An initial query vector that matches each first labeled detection box in the first sample image is determined from the multiple initial query vectors and used as the first matching query vector corresponding to each first labeled detection box. The number of first labeled detection boxes in the first sample image does not exceed the number of multiple initial query vectors. Each first labeled detection box is used to indicate the location of at least one first sample object in the first sample image. Based on the first sample detection box corresponding to each first matching query vector and the difference between the first labeled detection boxes, the parameters of the open-set detector are adjusted, and the multiple initial query vectors are updated to obtain multiple query vectors.

[0212] Optionally, the open set detector also outputs the first sample object features and sample confidence scores corresponding to each initial query vector; the sample confidence scores of the initial query vectors are used to indicate the accuracy of the first sample detection boxes of the initial query vectors; the training module is also used to determine a first loss based on the difference between the first sample detection boxes and the first labeled detection boxes corresponding to each first matching query vector; determine a second loss based on the sample confidence scores of each of the multiple initial query vectors; and adjust the parameters of the open set detector based on the first loss and the second loss, and update the multiple initial query vectors to obtain multiple query vectors.

[0213] Optionally, the training module is further configured to determine the intersection-union ratio loss based on the intersection of the first sample detection box and the first labeled detection box corresponding to each first matching query vector, and the minimum closure region of the first sample detection box and the first labeled detection box corresponding to each first matching query vector; determine the positional loss based on the positional difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector; and determine the first loss based on the intersection-union ratio loss and the positional loss.

[0214] Optionally, the attribute features of each target object are extracted using a descriptor; the training module is further used to perform object detection on the second sample image based on multiple query vectors, obtaining the second sample object features and second sample detection boxes corresponding to each query vector; the second sample detection box of each query vector indicates a region in the second sample image; the second sample object features are the feature indicators of the region indicated by the second sample detection box of the corresponding query vector in the second sample image; a query vector that matches each second labeled detection box in the second sample image is determined from multiple query vectors, serving as the second matching query vector corresponding to each second labeled detection box; the number of second labeled detection boxes in the second sample image does not exceed the number of multiple query vectors; each second labeled detection box is used to indicate the location of at least one second sample object in the second sample image; attribute features are extracted by the descriptor based on the second sample object features of the second matching query vectors corresponding to each second labeled detection box, obtaining the predicted object attribute features corresponding to the second sample object indicated by each second labeled detection box; the parameters of the descriptor are adjusted based on the differences between the predicted object attribute features and the labeled attribute features corresponding to each second sample object.

[0215] Optionally, the training module is also used to construct reference prompt information corresponding to each second sample object based on the labeled object name and description information of the second sample object; generate object attribute features based on the reference prompt information corresponding to the second sample object by referencing the large language model, and obtain the initial attribute features corresponding to the second sample object; and determine the labeled attribute features corresponding to the second sample object based on the difference between the labeled object name and the initial attribute features corresponding to the second sample object.

[0216] Optionally, the initial attribute features corresponding to each second sample object include multiple single-dimensional attribute features of each dimension. The training module is further configured to construct new prompt information corresponding to the second sample object based on the labeled object name corresponding to the second sample object and the new descriptive information if the number of single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches a similarity threshold is less than a preset number. The module then returns to execute the step of generating object attribute features based on the reference prompt information corresponding to the second sample object through a reference large language model to obtain the initial attribute features corresponding to the second sample object, until the cumulative number of single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches a similarity threshold reaches a preset number. Finally, the module determines the labeled attribute features corresponding to the second sample object from the single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches a similarity threshold.

[0217] Optionally, the target object name is generated using a large language model; the training module is also used to perform object detection on the third sample image based on multiple query vectors, obtaining a third sample detection box corresponding to each query vector; the third sample detection box of each query vector indicates a region in the third sample image; a query vector that matches each third labeled detection box in the third sample image is determined from multiple query vectors, serving as the third matching query vector corresponding to each third labeled detection box; the number of third labeled detection boxes in the third sample image does not exceed the number of multiple query vectors; each third labeled detection box is used to indicate the location of at least one third sample object in the third sample image; attribute features are extracted based on the third sample object features of the third matched query vectors corresponding to each third labeled detection box, obtaining the predicted object attribute features of the third sample object indicated by each third labeled detection box; the name is generated by the large language model based on the predicted object attribute features corresponding to each third labeled detection box, obtaining the predicted object name corresponding to the third sample object indicated by each third labeled detection box; the parameters of the large language model are adjusted based on the differences between the predicted object names corresponding to each third sample object and the labeled object names.

[0218] Optionally, the object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open-set detector; the attribute features of each target object are extracted by a descriptor; the target object name is generated by a large language model; the training module is also used to perform object detection on the fourth sample image based on multiple query vectors using an open-set detector, obtaining the fourth sample detection box corresponding to each query vector; the fourth sample detection box of each query vector indicates a region in the fourth sample image; a query vector that matches each fourth labeled detection box in the fourth sample image is determined from multiple query vectors, serving as the fourth matching query vector corresponding to each fourth labeled detection box; the number of fourth labeled detection boxes in the fourth sample image does not exceed the number of multiple query vectors; each fourth labeled detection box is used to indicate at least one fourth sample pair in the fourth sample image. The location of the object is determined; attribute features are extracted from the fourth sample object features of the fourth matching query vector corresponding to each fourth labeled detection box using a descriptor, resulting in the predicted object attribute features of the fourth sample object indicated by each fourth labeled detection box; the name is generated by the large language model based on the predicted object attribute features of each fourth sample object, resulting in the predicted object name corresponding to each fourth sample object; based on the differences between the fourth sample detection box and the fourth labeled detection box corresponding to each fourth matching query vector, the differences between the predicted object attribute features and the labeled attribute features corresponding to each fourth sample object, and the differences between the predicted object name and the labeled object name corresponding to each fourth sample object, the parameters of the open set detector, descriptor, and large language model are adjusted, and each query vector is updated.

[0219] Optionally, the detection module 1010 is further configured to extract image features from the image to be detected using the image encoder in the open set detector, thereby obtaining the image features corresponding to the image to be detected; and to perform object detection based on the image features and multiple query vectors using the detection head in the open set detector, thereby obtaining the object features, detection boxes, and confidence scores corresponding to each query vector.

[0220] Optionally, the attribute extraction module 1030 is further configured to perform projection processing on the target object features through the first projection layer to map the target object features to a feature space aligned with the input of the descriptor, thereby obtaining the target object projection features; and to extract attribute features from the target object projection features through the descriptor, thereby obtaining the target object attribute features.

[0221] Optionally, the name generation module 1040 is further configured to project the target object attribute features through the second projection layer to map the target object attribute features to a feature space aligned with the input of the large language model, thereby obtaining the target object attribute projection features; and generate the name of the target object based on the target object attribute projection features through the large language model, thereby obtaining the target object name.

[0222] Optionally, the device further includes a result determination module for determining the object detection result of the image to be detected based on the target object name and the detection box of the target query vector.

[0223] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0224] Figure 11 A structural block diagram of an electronic device for performing a method for recognizing objects in an image according to embodiments of this application is shown. The electronic device may be... Figure 1 The terminal (110) or server (120), etc., should be noted. Figure 11 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0225] like Figure 11 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0226] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0227] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0228] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0229] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0230] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0231] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0232] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the electronic device to perform the methods of any of the above embodiments.

[0233] In the embodiments of this application, the terms "module" or "unit" refer to a part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0234] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0235] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0236] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for identifying objects in an image, characterized in that, The method includes: Object detection is performed on the image to be detected based on multiple query vectors, resulting in object features, detection boxes, and confidence scores for each query vector. The detection box for each query vector indicates a region in the image to be detected. The object features are feature representations of the region indicated by the detection box of the corresponding query vector in the image to be detected. The confidence score is used to indicate the accuracy of the detection box of the corresponding query vector. Based on the confidence level corresponding to each query vector, the target object features are determined from the object features corresponding to the plurality of query vectors; The target object features are extracted by extracting attribute features to obtain the target object attribute features; Based on the target object attribute features, a name is generated for the target object located in the target region to obtain the target object name; the target region is the area indicated by the detection box of the target query vector corresponding to the target object features in the image to be detected.

2. The method according to claim 1, characterized in that, The object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open set detector; Before performing object detection on the image to be detected based on multiple query vectors and obtaining the object features, detection bounding boxes, and confidence scores corresponding to each query vector, the method further includes: The open set detector performs object detection on the first sample image based on multiple initial query vectors to obtain a first sample detection box corresponding to each initial query vector; the first sample detection box is used to indicate a region in the first sample image. An initial query vector is determined from the plurality of initial query vectors that matches each first labeled detection box in the first sample image one by one, and is used as the first matching query vector corresponding to each first labeled detection box; the number of first labeled detection boxes in the first sample image does not exceed the number of the plurality of initial query vectors; each first labeled detection box is used to indicate the location of at least one first sample object in the first sample image; Based on the difference between the first sample detection box and the first labeled detection box corresponding to each of the first matching query vectors, the parameters of the open set detector are adjusted, and the multiple initial query vectors are updated to obtain multiple query vectors.

3. The method according to claim 2, characterized in that, The open set detector also outputs the first sample object features and sample confidence scores corresponding to each initial query vector; the sample confidence scores of the initial query vectors are used to indicate the accuracy of the first sample detection boxes of the initial query vectors. The step of adjusting the parameters of the open set detector based on the difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector, and updating the multiple initial query vectors to obtain multiple query vectors, includes: The first loss is determined based on the difference between the first sample detection box and the first labeled detection box corresponding to each of the first matching query vectors; The second loss is determined based on the sample confidence scores of the multiple initial query vectors. Based on the first loss and the second loss, the parameters of the open set detector are adjusted, and the multiple initial query vectors are updated to obtain multiple query vectors.

4. The method according to claim 3, characterized in that, The step of determining the first loss based on the difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector includes: Based on the intersection of the first sample detection box and the first labeled detection box corresponding to each first matching query vector, and the minimum closure region of the first sample detection box and the first labeled detection box corresponding to each first matching query vector, the intersection-union ratio loss is determined; Based on the positional difference between the first sample detection box and the first labeled detection box corresponding to each first matching query vector, the positional loss is determined; The first loss is determined based on the intersection-union ratio loss and the position loss.

5. The method according to any one of claims 1-4, characterized in that, Each of the target object attribute features is extracted using an object descriptor; before extracting the attribute features of the target objects to obtain the target object attribute features, the method further includes: Based on the multiple query vectors, object detection is performed on the second sample image to obtain the second sample object features and the second sample detection box corresponding to each query vector; the second sample detection box of each query vector indicates a region in the second sample image; the second sample object features are the feature indicators of the region indicated by the second sample detection box of the corresponding query vector in the second sample image. From the plurality of query vectors, a query vector that matches each of the second annotation detection boxes in the second sample image is determined, and used as the second matching query vector corresponding to each of the second annotation detection boxes; the number of second annotation detection boxes in the second sample image does not exceed the number of the plurality of query vectors; each second annotation detection box is used to indicate the location of at least one second sample object in the second sample image; The descriptor extracts attribute features based on the second sample object features of the second matching query vector corresponding to each second labeled detection box, thereby obtaining the predicted object attribute features corresponding to the second sample object indicated by each second labeled detection box. Based on the differences between the predicted object attribute features and the labeled attribute features corresponding to each of the second sample objects, the parameters of the descriptor are adjusted.

6. The method according to claim 5, characterized in that, The method further includes: For each of the second sample objects, reference prompt information is constructed based on the labeled object name and description information of the second sample object; By referencing the large language model and generating object attribute features based on the reference prompt information corresponding to the second sample object, the initial attribute features corresponding to the second sample object are obtained. Based on the difference between the labeled object name and the initial attribute features corresponding to the second sample object, the labeled attribute features corresponding to the second sample object are determined.

7. The method according to claim 6, characterized in that, The initial attribute features corresponding to each of the second sample objects include multiple single-dimensional attribute features of each dimension. The step of determining the labeled attribute features corresponding to the second sample object based on the difference between the labeled object name and the initial attribute features corresponding to the second sample object includes: If the number of single-dimensional attribute features whose similarity to the labeled object name of the second sample target object reaches the similarity threshold is less than the preset number, a new prompt message corresponding to the second sample object is constructed based on the labeled object name corresponding to the second sample object and the new description information. Return to the step of generating object attribute features based on the reference prompt information corresponding to the second sample object by referring to the large language model, and obtaining the initial attribute features corresponding to the second sample object, until the cumulative number of single-dimensional attribute features whose similarity to the labeled object name of the second sample object reaches the similarity threshold reaches the preset number. The annotation attribute features corresponding to the second sample object are determined from the single-dimensional attribute features whose similarity to the annotated object name of the second sample object reaches a similarity threshold.

8. The method according to claim 1, characterized in that, The target object name is generated through a large language model; before generating the name of the target object located in the target region based on the target object attribute features, the method further includes: Object detection is performed on the third sample image based on the multiple query vectors to obtain a third sample detection box corresponding to each query vector; the third sample detection box of each query vector indicates a region in the third sample image; From the plurality of query vectors, a query vector that matches each third labeled detection box in the third sample image is determined as the third matching query vector corresponding to each third labeled detection box; the number of third labeled detection boxes in the third sample image does not exceed the number of the plurality of query vectors; each third labeled detection box is used to indicate the location of at least one third sample target object in the third sample image; Based on the third sample object features corresponding to the third matching query vector of each of the third labeled detection boxes, attribute features are extracted to obtain the predicted object attribute features of the third sample target object indicated by each of the third labeled detection boxes. The large language model generates names based on the attribute features of the predicted objects corresponding to each of the third labeled detection boxes, thereby obtaining the predicted object name corresponding to the third sample target object indicated by each of the third labeled detection boxes; Based on the differences between the predicted object name and the labeled object name corresponding to each of the third sample objects, the parameters of the large language model are adjusted.

9. The method according to claim 1, characterized in that, The object features, detection boxes, and confidence scores corresponding to each query vector are determined by an open-set detector; the attribute features of each target object are extracted by a descriptor; and the target object name is generated by a large language model. Before performing object detection on the image to be detected based on multiple query vectors to obtain the object features and confidence scores corresponding to each query vector, the method further includes: The open set detector performs object detection on the fourth sample image based on the multiple query vectors to obtain a fourth sample detection box corresponding to each query vector; the fourth sample detection box of each query vector indicates a region in the fourth sample image. From the plurality of query vectors, a query vector is determined that matches each fourth annotation detection box in the fourth sample image one by one, and is used as the fourth matching query vector corresponding to each fourth annotation detection box; the number of fourth annotation detection boxes in the fourth sample image does not exceed the number of the plurality of query vectors; each fourth annotation detection box is used to indicate the location of at least one fourth sample object in the fourth sample image; The descriptor extracts attribute features based on the fourth sample object features of the fourth matching query vector corresponding to each of the fourth labeled detection boxes, thereby obtaining the predicted object attribute features of the fourth sample object indicated by each of the fourth labeled detection boxes. The name is generated by the large language model based on the predicted object attribute features of each of the fourth sample objects, and the predicted object name corresponding to each of the fourth sample objects is obtained. Based on the differences between the fourth sample detection box and the fourth labeled detection box corresponding to each fourth matching query vector, the differences between the predicted object attribute features and labeled attribute features corresponding to each fourth sample object, and the differences between the predicted object name and labeled object name corresponding to each fourth sample object, the parameters of the open set detector, the descriptor, and the large language model are adjusted, and each query vector is updated.

10. The method according to claim 1, characterized in that, The process of object detection based on multiple query vectors in the image to be detected, obtaining the object features, detection bounding box, and confidence score corresponding to each query vector, includes: The image features corresponding to the image to be detected are obtained by extracting image features from the image encoder in the open set detector. The open set detector performs object detection based on the image features and the multiple query vectors using its detection head, thereby obtaining the object features, detection box, and confidence score corresponding to each query vector.

11. The method according to claim 1, characterized in that, The step of extracting attribute features from the target object features to obtain target object attribute features includes: The target object features are projected through the first projection layer to map the target object features to a feature space aligned with the input of the descriptor, thereby obtaining the target object projected features. The attribute features of the target object are obtained by extracting attribute features from the projection features of the target object using the descriptor.

12. The method according to claim 1, characterized in that, The step of generating a name for a target object located in the target region based on the target object's attribute characteristics, to obtain the target object name, includes: The target object attribute features are projected through a second projection layer to map them onto a feature space aligned with the input of the large language model, thus obtaining the target object attribute projection features. The name of the target object is obtained by generating the name based on the projection features of the target object's attributes using the large language model.

13. The method according to claim 1, characterized in that, After generating names for target objects located in the target region based on the target object attribute characteristics, and obtaining the target object names, the method further includes: Based on the target object name and the detection bounding box of the target query vector, the object detection result of the image to be detected is determined.

14. An apparatus for identifying objects in an image, characterized in that, The device includes: The detection module is used to perform object detection on an image to be detected based on multiple query vectors, and obtain the object features, detection boxes, and confidence scores corresponding to each query vector; the detection boxes of each query vector indicate a region in the image to be detected; the object features are the feature representations of the region indicated by the detection boxes of the corresponding query vectors in the image to be detected; the confidence scores are used to indicate the accuracy of the detection boxes of the corresponding query vectors. The feature determination module is used to determine the target object features from the object features corresponding to the plurality of query vectors based on the confidence level corresponding to each of the query vectors; The attribute extraction module is used to extract attribute features from the target object features to obtain the target object attribute features; The name generation module is used to generate a name for the target object located in the target region based on the target object attribute features, thereby obtaining the target object name; the target region is the area indicated by the detection box of the query vector corresponding to the target object features in the image to be detected.

15. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-13.

17. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the method of any one of claims 1-13.

Images