Method, apparatus, device and storage medium for object detection

Abstract

The present disclosure provides a method, device, equipment and storage medium for object detection. The method comprises: extracting a set of visual feature representations of a target image by using an object detection model, the set of visual feature representations comprising respective visual feature representations of at least one object region in the target image; and generating a set of text sequences based on the set of visual feature representations by using a language model, each text sequence describing at least one category to which an object in an object region corresponding to a visual feature representation belongs. Thus, by virtue of the generation capability and semantic analysis capability of the language model, the object in the image can be aligned with the text information of the category without relying on a model for object classification trained based on text associated with the target image. Not only can the dependence on large-scale training data with category annotations be reduced, but also the comprehensiveness and accuracy of object detection can be improved in model application.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, devices, and computer-readable storage media for object detection. Background Technology

[0002] With the development of internet technology, an increasing number of applications and platforms offer image processing capabilities, bringing great convenience to users. Applications and platforms with image processing functions can provide image processing services to users based on trained machine learning models. Image recognition is a crucial task within image processing, and object detection is a key component of image recognition. The aim is to improve the accuracy of detection results generated by machine learning models performing object detection tasks. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for object detection is provided. The method includes: extracting a set of visual feature representations of a target image using an object detection model, the set of visual feature representations including visual feature representations of at least one object region in the target image; and generating a set of text sequences based on the set of visual feature representations using a language model, each text sequence describing at least one category to which an object in the object region corresponding to the visual feature representation belongs.

[0004] In a second aspect of this disclosure, an apparatus for object detection is provided. The apparatus includes: a visual feature extraction module configured to extract a set of visual feature representations of a target image using an object detection model, the set of visual feature representations including visual feature representations of at least one object region in the target image; and a text sequence generation module configured to generate a set of text sequences based on the set of visual feature representations using a language model, each text sequence describing at least one category to which an object in the object region corresponding to the visual feature representation belongs.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores a computer program that, when executed by a processor, implements the method of the first aspect.

[0007] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2A and Figure 2B Schematic diagrams illustrating several examples of object detection are shown;

[0011] Figure 3 A schematic diagram of an example architecture for object detection according to some embodiments of the present disclosure is shown;

[0012] Figure 4A and Figure 4B Schematic diagrams illustrating several examples of model training according to some embodiments of the present disclosure are shown;

[0013] Figure 5 A flowchart of an object detection process according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A schematic structural block diagram of an apparatus for object detection according to some embodiments of the present disclosure is shown; and

[0015] Figure 7 An electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0021] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0024] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0025] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, environment 100 may include electronic device 110.

[0027] Electronic device 110 can acquire target image 102. Target image 102 can be an image of any suitable size, resolution, and type (e.g., RGB image, grayscale image, etc.). Target image 102 can be an image stored locally by electronic device 110, or an image captured in real-time by electronic device 110 (e.g., an image captured by a camera). Target image 102 can also be an image acquired by electronic device 110 from other electronic devices (e.g., remote devices). Electronic device 110 can perform an object detection task (also called a region detection task, object recognition task, etc.) on target image 102 to determine object detection result 112 for target image 102.

[0028] Electronic device 110 can, for example, utilize a trained target model 115 to perform an object detection task. The target model 115 can be, for example, any suitable model including but not limited to, a Transformer model, a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Neural Network (DNN), etc. The target model 115 can be a model native to electronic device 110 or a model installed on another electronic device 110 (e.g., installed on a remote device). It should be noted that the target model 115 can include multiple models. For example, the target model 115 can include an object detection model and a language model. Depending on the specific scenario, the target model 115 can also include any other suitable model.

[0029] Electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side equipment may include computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] As briefly mentioned earlier, applications or platforms with image processing capabilities can provide image processing services to users based on trained machine learning models. Image recognition is an important task within image processing, and object detection is a crucial task within image recognition. With the continuous advancement and popularization of artificial intelligence technology, expectations for the effectiveness of object detection are also increasing.

[0032] Traditionally, the image to be detected and the text associated with the object are fed together into a machine learning model used to perform object detection tasks. (Reference) Figure 2A and Figure 2B . Figure 2A and Figure 2B Schematic diagrams of several examples of object detection (i.e., Examples 200A and 200B) are shown. In Example 200A, an electronic device performing the object detection task (hereinafter referred to as the detection device) can acquire an image 202 to be detected and text 201 indicating a predefined category of the image 202. The text 201 may include multiple predefined categories (e.g., the categories "person", "tennis racket", and "cat" shown in the figure), and these multiple predefined categories may correspond to different objects in the image 202. The detection device can provide the text 201 to a trained text encoder 220-1 and the image 202 to a trained object detection model 210-1.

[0033] The text encoder 220-1 can output a text feature representation 221 of text 201. The model output of the object detection model 210-1 can be, for example, a visual feature representation of image 202. The model output of the object detection model 210-1 is provided to the classifier 230-1. The classifier 230-1 can perform a classification operation on the model output of the object detection model 210-1 to output a visual feature representation (i.e., visual feature representation 231) corresponding to each object in image 202. The detection device 110 can perform an alignment operation on the text feature representation 221 and the visual feature representation 231. This process can also be referred to as the alignment process between predefined categories in text 201 and objects in image 202. The detection device 110 can then determine a detection result 241 corresponding to image 202 based on the alignment result. The detection result 241 includes the predicted category corresponding to at least one object in image 202.

[0034] Taking an image 202 containing four objects and text 201 containing three predefined categories as an example, the detection device 110 can identify the objects among the four objects that match these three predefined categories. For instance, for an object with a corresponding predefined category, the detection device can determine the predefined category in text 201 corresponding to that object as the detection result for that object (i.e., the predicted category of that object). For an object that does not have a predefined category, the detection device cannot determine the detection result for that object; the detection device can determine that there is no detection result for that object and output the text "<No object>".

[0035] In Example 200B, the detection device can acquire an image 202 to be detected and text 203 matching the image 202. The text 203 may be, for example, a caption for the image 202. The detection device can provide the text 203 to a trained text encoder 220-2 and the image 202 to a trained object detection model 210-2. The text encoder 220-2 can output a text feature representation 222 of the text 201. The model output of the object detection model 210-2 can also be, for example, a visual feature representation of the image 202. The model output of the object detection model 210-2 is provided to a classifier 230-2. The classifier 230-2 can perform a classification operation on the model output of the object detection model 210-2 to output a visual feature representation (i.e., visual feature representation 232) corresponding to each object in the image 202.

[0036] The detection device 110 can perform an alignment operation on the text feature representation 222 and the visual feature representation 232. This process can also be referred to as the alignment process between phrases / words in text 201 and objects in image 202. The detection device 110 can then determine the detection result 242 corresponding to image 202 based on the alignment result. Taking text 201 as "A man and a little boy are playing tennis together on a court." as an example, the detection device 110 can determine that there are two objects in image 202 that correspond to the text "man" and the text "little boy" in text 201, respectively.

[0037] As can be seen, in Examples 200A and 200B, the detection device relies on the acquired text when performing object detection on image 202. The object detection process shown in Examples 200A and 200B can be understood as the process of detecting objects from image 202 that belong to a known category. If the text lacks a category corresponding to an object, the detection device cannot determine the object from image 202, leading to missed detections and false detections. This affects the accuracy of object detection.

[0038] In view of this, according to embodiments of the present disclosure, an improved object detection scheme is proposed. In this scheme, an object detection model is used to extract a set of visual feature representations of a target image. The set of visual feature representations includes visual feature representations of at least one object region in the target image. A language model is used to generate a set of text sequences based on the set of visual feature representations. Each text sequence describes at least one category to which the object in the object region corresponding to the visual feature representation belongs.

[0039] In embodiments of this disclosure, an object detection model is applied to extract visual features and locate object regions in an image without requiring specific analysis and determination of the object's category. Object category determination is accomplished using a language model. In this way, leveraging the generative and semantic analysis capabilities of the language model, the model for object classification can be trained to align objects in the image with their category textual information without relying on text associated with the target image. According to embodiments of this disclosure, the object detection model can identify more objects in the target image, and the language model generates a classification result for each object based on visual features. This approach not only reduces the reliance on large-scale training data with category annotations but also improves the comprehensiveness and accuracy of object detection in model applications.

[0040] Some exemplary embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0041] Figure 3 A schematic diagram of an example architecture 300 for object detection according to some embodiments of the present disclosure is shown. Architecture 300 can be implemented in Figure 1 There are 110 electronic devices. For example... Figure 3 As shown, architecture 300 includes an object detection model 310 and a language model 320. The object detection model 310 and the language model 320 can be... Figure 1 The target model 115 includes the following models. For ease of discussion, reference will be made to... Figure 1 The environment 100 is used to describe the architecture 300.

[0042] After acquiring the target image 102, the electronic device 110 can determine the object detection result 112 for the target image 102 using a trained object detection model 310 and a trained language model 320. Specifically, the electronic device 110 can provide the target image 102 to the object detection model 310. The object detection model 310 can include any suitable model. The object detection model 310 can be, for example, including but not limited to, the Transformer model, convolutional neural network (CNN), recurrent neural network (RNN), deep neural network (DNN), etc. In some embodiments, the object detection model 310 can be, for example, a DETR model.

[0043] It is important to note that the object detection model 310 can only identify the location information of at least one object in the target image 102, but cannot identify the specific category of each object. That is, the object detection model 310 can identify at least one object region in the target image 102 (each object region includes only one object). Taking image 202 as the target image 102, and image 202 including four objects as an example, the object detection model 310 can determine the four object regions of image 202. The object detection model 310 can extract a set of visual feature representations of the target image 102. This set of visual feature representations includes at least one visual feature representation, and each visual feature representation corresponds to an object region. In other words, the set of visual feature representations includes the visual feature representation of at least one object region in the target image 102. Each visual feature representation, for example, can indicate the location information of the corresponding object region.

[0044] Electronic device 110 can then provide a set of visual feature representations output by object detection model 310 to trained language model 320. In some embodiments, architecture 300 may further include a visual-linguistic feature adapter (not shown) associated with language model 320. The visual-linguistic feature adapter can perform a transformation on the set of visual feature representations output by object detection model 310 to obtain a set of transformed feature representations. The visual-linguistic feature adapter can perform any suitable transformation operation on the set of visual feature representations (e.g., dimensional transformation can be performed), which is not limited in this disclosure. The set of transformed feature representations are feature representations that satisfy the model input requirements of language model 320. The model input requirements of language model 320 indicate at least one condition that needs to be satisfied for the model input of language model 320. For example, the dimensions of the set of transformed feature representations satisfy the dimensional condition indicated by the model input requirements of language model 320. Electronic device 110 and / or visual-linguistic feature adapter can then provide the set of transformed feature representations to language model 320. Therefore, the speech model 320 can be provided with model input that meets the model input requirements, which can reduce the possibility of errors in the language model 320 and improve the accuracy of object detection.

[0045] The language model 320 can be, for example, a conventional language model (LM), a large language model (LLM), etc. The language model 320, learned and trained from a large corpus, possesses powerful semantic understanding capabilities. The language model 320 can generate object detection results 112 based on a set of visual feature representations. The object detection results 112 can include a set of text sequences 301. The number of text sequences 301 can be equal to the number of objects included in the target image 102. For example, if the target image 102 includes 4 objects, the set of text sequences 301 output by the language model 320 can include text sequences 301-1, 301-2, 301-3, and 301-4. It can be understood that if the target image 102 includes other numbers of objects, the set of text sequences 301 output by the language model 320 can also include other numbers of text sequences. For ease of description, one or more text sequences can be collectively referred to as text sequences 301.

[0046] Each text sequence describes visual features representing at least one category to which an object in the corresponding object region belongs. For example, each text sequence may include at least one text unit, which may be, for example, a word or phrase. Each text unit corresponds to a category of the object in the object region corresponding to the text sequence (also referred to as the object region corresponding to the text sequence, the object corresponding to the text sequence, etc.). For example, taking target image 102 as image 202, and image 202 including 4 objects, the detection result 302 output by model 320 for image 202 may include 4 text sequences, that is... Figure 3 The text sequences shown are ["person", "man"...], ["person", "boy"...], ["tennis racket", "racket"...] and ["fence", "chain fence"...].

[0047] In some embodiments, at least one category included in the text sequence corresponding to each object can be referred to as at least one detection result for that object. That is, each text sequence may include at least one detection result for the corresponding object. In some embodiments, each output sequence in a set of output sequences has a preset length (e.g., K), and at least one detection result corresponding to each object includes a preset number (e.g., K) of detection results matching the preset length. The preset length can be any length, and the corresponding preset number can be any appropriate positive integer. Taking a preset length of 2 as an example (i.e., K=2), the length of each output sequence output by the language model 320 is 2, that is, each output sequence includes a preset number (i.e., 2) of detection results matching the preset length (i.e., each output sequence includes 2 detection results, or each output sequence includes 2 categories).

[0048] In some embodiments, for each object in the target image 102, the language model 320 can determine multiple categories that match the object. The language model 320 can also determine the degree of association between each of these multiple categories and the object, and generate a text sequence for the object based on the multiple degrees of association corresponding to the multiple categories. Specifically, the language model 320 can determine the multiple degrees of association corresponding to the multiple categories by determining the semantic relevance between the text feature representations of each of these multiple categories and the visual feature representations of the object.

[0049] For example, language model 320 can determine multiple categories (e.g., categories A, B, C, and D) that match object A based on the acquired visual feature representation of object A. Language model 320 can determine the text feature representation of each category. Language model 320 can determine the semantic relevance between the text feature representation of each category and the acquired visual feature representation of object A. Language model 320 can, for example, sort the multiple categories in descending order based on the semantic relevance corresponding to the text feature representation of each category. Language model 320 can generate a text sequence corresponding to object A based on the sorting of multiple categories. For example, if the semantic relevance of category A > the semantic relevance of category B > the semantic relevance of category C > the semantic relevance of category D, the text sequence output by language model 320 for object A could be ["category A", "category B", "category C", "category D"].

[0050] In some embodiments, if each output sequence has a preset length of 2, the text sequence output by the language model 320 may include the two categories that rank first or second among multiple categories. That is, in this case, the text sequence output by the language model 320 for object A may be ["Category A", "Category B"].

[0051] In some embodiments, the electronic device 110 may also provide object detection results 112 for the target image 102. Object detection results 112 may indicate at least one object region located in the target object 102 by the object detection model 310 and the text sequences corresponding to each object region generated by the language model 320. Exemplarily, the electronic device 110 may provide object detection results 112 by identifying at least one object region in the target object 102 and presenting its corresponding text sequence associated with each object region. The electronic device 110 may identify at least one object region in any suitable manner (e.g., by adding a border to each object region), which is not limited in this disclosure. For example, for object A of the target object, whose corresponding text sequence is ["Category A", "Category B"], the electronic device 110 may identify the object region of object A in the target image 102 and present the text sequence ["Category A", "Category B"] associated with that object region. The text sequence may, for example, be overlaid on the object region or presented at any location near the object region.

[0052] The above describes in detail the process by which electronic device 110 uses an object detection model and a language model to determine the object detection result corresponding to the target image. The training method of the object detection model and the language model will be described below. In some embodiments, the object detection model and the language model can be trained at electronic device 110 or at other electronic devices (e.g., remote electronic devices). For ease of description, the electronic device that trains the object detection model and the language model will be referred to as the training device. That is, the object detection model and the language model are trained at the training device, and electronic device 110 can obtain the trained object detection model and language model from the training device to perform the above-described object detection process.

[0053] In some embodiments, the training device needs to train at least an object detection model (i.e., the language model can be pre-trained or jointly trained with the object detection model). Figure 4A and Figure 4B Schematic diagrams of several examples (i.e., examples 400A and 400B) of model training according to some embodiments of the present disclosure are shown. Both examples 400A and 400B can be implemented at a training device. Example 400A illustrates a case where both object detection model 410 and language model 430 are trained models. Example 400B illustrates a case where object detection model 440 and language model 450 are jointly trained.

[0054] In Example 400A, the object detection model 410 includes an image encoder 411 and an image decoder 412. The training device can fix the parameters of the image encoder 411 and the parameters of the trained language model 430 in the object detection model 410, and can train the object detection model 410 by updating only the parameters of the image decoder 412. In Example 400A, the training device can use the object detection model 410 to extract a set of sample visual feature representations 402 for a sample image 401 (e.g., any sample image in the training dataset). Specifically, the parameter-fixed image encoder 411 can output a visual feature representation for the sample image 401. The image decoder 412 can decode a set of sample visual feature representations 402 based on the visual feature representation for the sample image 401 and an object query 413 determined based on the sample image 401.

[0055] A set of sample visual feature representations 402 includes sample visual feature representations for at least one object region in the sample image 401. The training device can utilize a visual-linguistic feature adapter 420 associated with the language model 430 to perform a transformation on the set of sample visual feature representations, resulting in a set of transformed sample visual feature representations 421, and provide the transformed set of sample visual feature representations 421 to the language model 430. The visual-linguistic feature adapter 420 can also be trained, and its parameters are fixed during the training of the object detection model 410. The training device can utilize the language model 430 to generate a set of predicted text sequences based on the set of sample visual feature representations 402, each predicted text sequence describing at least one predicted category to which an object in the object region corresponding to the sample visual feature representation belongs.

[0056] In Example 400A, the object detection model 440 includes an image encoder 441 and an image decoder 442. The training device can co-train the object detection model 440 and the language model 450. In Example 400B, the training device can utilize the object detection model 440 to extract a set of sample visual feature representations 402 for a sample image 401 (e.g., any sample image in the training dataset). Specifically, the image encoder 441 can output the visual feature representation for the sample image 401. The image decoder 442 can decode the set of sample visual feature representations 402 based on the visual feature representation for the sample image 401 and an object query 443. Here, the object query 443 is determined based on the sample image 401.

[0057] A set of sample visual feature representations 402 includes sample visual feature representations of at least one object region in the sample image 401, respectively. The training device can directly provide the set of sample visual feature representations 402 to the language model 450. The training device can utilize the language model 450 to generate a set of predicted text sequences based on the set of sample visual feature representations 402, each predicted text sequence describing at least one predicted category to which the object in the object region corresponding to the sample visual feature representation belongs.

[0058] In some embodiments, regardless of whether a language model needs to be trained, the training device can, for example, employ a two-stage training approach to at least train the object detection model. In the first training stage, due to the incompleteness of manually labeled data and the high labor costs, the training device can obtain a first training dataset with less labeled information. The first training dataset includes a first sample image, ground truth location information of object regions in the first sample image (which may include ground truth location information corresponding to at least one object region in the first sample image), and the sample category to which the object belongs (which may include at least one sample category to which at least one object in the first sample image belongs, and each object may belong to at least one sample category). Here, the first sample image can be any sample image in the first training dataset. The first training dataset includes multiple sample images, and the training device can employ the following training method for each sample image. The training device can determine the first predicted location information and the first predicted category of the object region in the first sample image by providing the first sample image to the object detection model and the language model.

[0059] For example, the training device may provide a first sample image to an object detection model to extract a first set of sample visual feature representations from the first sample image. The first set of sample visual feature representations includes sample visual feature representations of at least one object region in the first sample image. The training device and / or the object detection model may, for example, determine first predicted location information corresponding to at least one object region in the first sample image based on the first set of sample visual feature representations. For example, the training device and / or the object detection model may perform a decoding operation on the first set of sample visual feature representations to determine the first predicted location information corresponding to at least one object region. The first predicted location information may include multiple predicted location information sets, and each predicted location information set is the predicted location information corresponding to one object region in the first sample image.

[0060] The training device can then provide the first set of sample visual feature representations to the language model, which, based on these representations, generates a first set of predicted text sequences. The first set of predicted text sequences includes at least one predicted text sequence, and each predicted text sequence describes at least one expected category to which an object in the object region corresponding to the sample visual feature representation belongs. The training device can determine the at least one expected category corresponding to the first set of predicted text sequences (i.e., the total number of at least one expected category described by each predicted text sequence in the first set of predicted text sequences) as the first expected category.

[0061] The training device can then update the object detection model based on a first distance loss between the first predicted location information and the ground truth location information, and a first semantic loss between the sample category and the first predicted category. Specifically, for each object in the first sample image, the training device can update the object detection model based on a first distance loss between the ground truth location information and the predicted location information of the object's object region, and a first semantic loss between at least one predicted category and at least one sample category to which the object belongs. The training device can determine the distance loss and semantic loss in any suitable manner, and this disclosure does not limit the specific manner in which these two losses are determined. For example, if an object A corresponds to a predicted category of "person" and a corresponding sample category of "man", the training device can determine the semantic loss between the two categories based on the semantics of "person" and "man" respectively. The training device can at least train the object detection model by reducing the first distance loss and the first semantic loss. The training device can, for example, determine that the first stage of training is complete in response to both the first distance loss and the first semantic loss reaching the first predicted target.

[0062] In the second training phase, the training device can again provide the first sample image to the object detection model and language model trained in the first phase. The training device can acquire a second set of predicted text sequences output by the object detection model and language model from the first training phase. Each predicted text sequence in the second set describes at least one expected category to which the object in the corresponding object region belongs based on the visual features of the sample. The training device can determine the at least one expected category corresponding to the second set of predicted text sequences (i.e., the total number of at least one expected category described by each predicted text sequence in the second set of predicted text sequences) as a pseudo-label category for the first sample image. The pseudo-label category differs from the sample category. For example, the pseudo-label category includes at least one expected category that differs from the at least one sample category included in the sample category.

[0063] Regarding the specific method for determining pseudo-label categories different from sample categories, in some embodiments, the language model can learn the categories corresponding to the same type of objects in different sample images during training. The language model can determine multiple categories corresponding to a certain type of object through extensive learning. For example, if sample image A includes object A, sample image B includes object B, and objects A and B belong to the same type (or the similarity between objects A and B is greater than a threshold), with object A corresponding to category A and object B corresponding to category B, then the trained language model can determine that both object A and object B can correspond to categories A and B, respectively. The language model can, for example, generate a text sequence that at least describes category B based on the visual feature representation of object A.

[0064] The training device can again determine the second predicted location information and the second predicted category of the object region in the first sample image by feeding the first sample image to the updated object detection model and language model. The method by which the training device determines the second predicted location information and the second predicted category can refer to the method described above for determining the first predicted location information and the first predicted category, and will not be repeated here. The training device can update the object detection model based on the second distance loss between the second predicted location information and the ground truth location information, and the second semantic loss between the sample category and the pseudo-label category and the second predicted category. It should be noted that if the sample category and the pseudo-label category are referred to as the updated sample category, the training device can determine the second semantic loss between the updated sample category and the second predicted category. Similarly, the training device can at least train the object detection model by reducing the second distance loss and the second semantic loss. The training device can, for example, determine that the second stage of training is complete in response to both the second distance loss and the second semantic loss reaching the second prediction objective.

[0065] In summary, in the embodiments of this disclosure, by leveraging the generative and semantic analysis capabilities of a language model, it is possible to train a model for object classification to align objects in an image with textual information about their categories without relying on text associated with the target image. According to embodiments of this disclosure, the object detection model can identify more objects in the target image, and the language model generates classification results for each object based on visual features. This approach not only reduces the reliance on large-scale training data with category annotations but also improves the comprehensiveness and accuracy of object detection in model applications.

[0066] Figure 5 A flowchart of an object detection process 500 according to some embodiments of the present disclosure is shown. Process 500 can be implemented at electronic device 110. Reference is made below. Figure 1 Describe the process 500.

[0067] In box 510, electronic device 110 uses an object detection model to extract a set of visual feature representations of a target image, the set of visual feature representations including visual feature representations of at least one object region in the target image.

[0068] In box 520, electronic device 110 uses a language model to generate a set of text sequences based on a set of visual feature representations, each text sequence describing at least one category to which an object in the object region corresponding to the visual feature representation belongs.

[0069] In some embodiments, each output sequence in a set of output sequences has a preset length, and at least one detection result for each object includes a preset number of detection results that match the preset length.

[0070] In some embodiments, the language model generates a set of output sequences by generating a text sequence corresponding to each visual feature representation based on the semantic relevance between the visual feature representation and the text feature representation of the category.

[0071] In some embodiments, generating a set of text sequences based on a set of visual feature representations using a language model includes: performing a transformation on a set of visual feature representations using a visual-language feature adapter associated with the language model to obtain a set of transformed feature representations; and obtaining a set of text sequences generated by the language model by providing the set of transformed feature representations to the language model.

[0072] In some embodiments, the language model and the visual-language feature adapter are trained, and the parameters of the trained language model and the visual-language feature adapter are fixed during the training of the object detection model.

[0073] In some embodiments, the object detection model includes an image encoder and an image decoder, and the image encoder is trained and its parameters are fixed during the training of the object detection model.

[0074] In some embodiments, the object detection model and the language model are jointly trained.

[0075] In some embodiments, at least the object detection model is trained by: acquiring a first training dataset, the first training dataset including a first sample image, ground truth location information of an object region in the first sample image, and the sample category to which the object belongs; determining a first predicted location information and a first predicted category of the object region in the first sample image by providing the first sample image to the object detection model and the language model; and updating the object detection model based on a first distance loss between the first predicted location information and the ground truth location information and a first semantic loss between the sample category and the first predicted category.

[0076] In some embodiments, at least the object detection model is trained by: providing a first sample image to an updated object detection model and a language model to obtain a pseudo-label category output by the language model, the pseudo-label category being different from the sample category; providing the first sample image to the updated object detection model and the language model to determine second predicted location information and a second predicted category of object regions in the first sample image; and updating the object detection model based on a second distance loss between the second predicted location information and the ground truth location information, and a second semantic loss between the sample category and the pseudo-label category and the second predicted category.

[0077] In some embodiments, process 500 further includes: providing object detection results for a target image, the object detection results indicating at least one object region located from a target object by an object detection model and text sequences corresponding to each object region generated by a language model.

[0078] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an apparatus 600 for object detection according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in... Figure 1 In the electronic device 110. The various modules / components in the device 600 can be implemented by hardware, software, firmware, or any combination thereof.

[0079] As shown in the figure, the device 600 includes a visual feature extraction module 610, configured to extract a set of visual feature representations of a target image using an object detection model. The set of visual feature representations includes visual feature representations of at least one object region in the target image. The device 600 also includes a text sequence generation module 620, configured to generate a set of text sequences based on the set of visual feature representations using a language model. Each text sequence describes at least one category to which an object in the corresponding object region belongs.

[0080] In some embodiments, each output sequence in a set of output sequences has a preset length, and at least one detection result for each object includes a preset number of detection results that match the preset length.

[0081] In some embodiments, the language model generates a set of output sequences by generating a text sequence corresponding to each visual feature representation based on the semantic relevance between the visual feature representation and the text feature representation of the category.

[0082] In some embodiments, the text sequence generation module 620 includes: a transformation feature acquisition module configured to perform a transformation on a set of visual feature representations using a visual-language feature adapter associated with a language model to obtain a set of transformed feature representations; and a feature providing module configured to obtain a set of text sequences generated by the language model by providing a set of transformed feature representations to the language model.

[0083] In some embodiments, the language model and the visual-language feature adapter are trained, and the parameters of the trained language model and the visual-language feature adapter are fixed during the training of the object detection model.

[0084] In some embodiments, the object detection model includes an image encoder and an image decoder, and the image encoder is trained and its parameters are fixed during the training of the object detection model.

[0085] In some embodiments, the object detection model and the language model are jointly trained.

[0086] In some embodiments, at least the object detection model is trained by: acquiring a first training dataset, the first training dataset including a first sample image, ground truth location information of an object region in the first sample image, and the sample category to which the object belongs; determining a first predicted location information and a first predicted category of the object region in the first sample image by providing the first sample image to the object detection model and the language model; and updating the object detection model based on a first distance loss between the first predicted location information and the ground truth location information and a first semantic loss between the sample category and the first predicted category.

[0087] In some embodiments, at least the object detection model is trained by: providing a first sample image to an updated object detection model and a language model to obtain a pseudo-label category output by the language model, the pseudo-label category being different from the sample category; providing the first sample image to the updated object detection model and the language model to determine second predicted location information and a second predicted category of object regions in the first sample image; and updating the object detection model based on a second distance loss between the second predicted location information and the ground truth location information, and a second semantic loss between the sample category and the pseudo-label category and the second predicted category.

[0088] In some embodiments, the apparatus 600 further includes a result providing module configured to provide object detection results for a target image, the object detection results indicating at least one object region located from a target object by an object detection model and text sequences corresponding to each object region generated by a language model.

[0089] The units and / or modules included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 600 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0090] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 6 Device 600.

[0091] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose computing device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0092] Electronic device 700 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 700.

[0093] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0094] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0095] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0096] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0097] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0099] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0101] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An object detection method, comprising: A set of visual feature representations of a target image is extracted using an object detection model, wherein the set of visual feature representations includes visual feature representations of at least one object region in the target image; Using a visual-language feature adapter associated with the language model, the set of visual feature representations is transformed to obtain a set of transformed feature representations that meet the model input requirements of the language model. as well as Using the language model, a set of text sequences is generated based on the set of transformed feature representations, each text sequence describing at least one category to which the object in the corresponding object region belongs; Generating the set of text sequences based on the transformed feature representations includes: For each object region in the at least one object region, multiple categories corresponding to the object region are determined based on the transformed feature representation in the set of transformed feature representations; Determine multiple text feature representations, wherein the multiple text feature representations correspond to the multiple categories; and Based on the semantic correlation between the transformed feature representation and the plurality of text feature representations, a text sequence is generated, the text sequence corresponding to the transformed feature representation.

2. The method according to claim 1, wherein each output sequence in a set of output sequences has a preset length, and at least one detection result corresponding to each object includes a preset number of detection results matching the preset length.

3. The method according to claim 1, wherein the language model and the visual-language feature adapter are trained, and during the training of the object detection model, the parameters of the trained language model and the visual-language feature adapter are fixed.

4. The method of claim 3, wherein the object detection model comprises an image encoder and an image decoder, and the image encoder is trained and the parameters of the image encoder are fixed during the training of the object detection model.

5. The method according to claim 1, wherein the object detection model and the language model are jointly trained.

6. The method of claim 1, wherein at least the object detection model is trained in the following manner: Obtain a first training dataset, which includes a first sample image, ground truth location information of object regions in the first sample image, and the sample category to which the object belongs; By providing the first sample image to the object detection model and the language model, the first predicted location information and the first predicted category of the object region in the first sample image are determined; as well as The object detection model is updated based on the first distance loss between the first predicted location information and the ground truth location information, and the first semantic loss between the sample category and the first predicted category.

7. The method of claim 6, wherein at least the object detection model is trained in the following manner: By providing the first sample image to the updated object detection model and the language model, a pseudo-label category output by the language model is obtained, which is different from the sample category; By feeding the first sample image to the updated object detection model and the language model, a second predicted location information and a second predicted category of the object region in the first sample image are determined; as well as The object detection model is updated based on a second distance loss between the second predicted location information and the ground truth location information, and a second semantic loss between the sample category and the pseudo-label category and the second predicted category.

8. The method according to claim 1, further comprising: Provide object detection results for the target image, the object detection results indicating at least one object region located from the target object by the object detection model and the text sequence corresponding to each object region generated by the language model.

9. An apparatus for object detection, comprising: The visual feature extraction module is configured to extract a set of visual feature representations of a target image using an object detection model, the set of visual feature representations including visual feature representations of at least one object region in the target image; The transformation feature acquisition module is configured to perform transformation on the set of visual feature representations using a visual-language feature adapter associated with the language model, to obtain a set of transformed feature representations that meet the model input requirements of the language model. as well as The text sequence generation module is configured to use the language model to generate a set of text sequences based on the set of transformed feature representations, each text sequence describing at least one category to which an object in the corresponding object region belongs; The text sequence generation module is further configured as follows: For each object region in the at least one object region, multiple categories corresponding to the object region are determined based on the transformed feature representation in the set of transformed feature representations; Determine multiple text feature representations, wherein the multiple text feature representations correspond to the multiple categories; and Based on the semantic correlation between the transformed feature representation and the plurality of text feature representations, a text sequence is generated, the text sequence corresponding to the transformed feature representation.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images