Target detection method, method and device for training target detection model

Abstract

The present disclosure provides a target detection method and a target detection model training method and device, relates to the technical field of artificial intelligence, in particular to the technical field of augmented reality, virtual reality, computer vision, deep learning and the like, and can be applied to scenarios such as smart cities. The implementation scheme is as follows: image data including a target object is obtained, and a target category feature corresponding to a target category of the target object is obtained, wherein the target category feature is fed to a decoder of a target detection model used for target detection; the target category of the target object is predicted based on the target category feature, to obtain a predicted category and a predicted category feature corresponding to the predicted category; the consistency of the predicted category and the target category is determined based on the similarity between the predicted category feature and the target category feature; and the predicted category is taken as a result of target detection on the target object according to the consistency of the predicted category and the target category.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of augmented reality, virtual reality, computer vision and deep learning, and can be applied to scenarios such as smart cities, and specifically relates to a target detection method and a method, device, electronic device, computer readable storage medium and computer program product for training a target detection model. BACKGROUND

[0002] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of humans, and includes both hardware and software technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0003] In computer vision technology, target detection has a wide range of use scenarios. Image segmentation, object tracking, key point detection, etc. usually rely on target detection. Target detection is generally implemented through a trained target detection model. In the training of the target detection model, the image data used for training needs to be labeled. The labeling process is generally implemented manually, for example, manually labeling the target to be detected in the form of a detection box. However, in actual applications, there may be a lack of such labeled training data due to factors such as labor costs, etc., thus causing certain limitations on the training of the model.

[0004] The methods described in this section can not necessarily be the methods previously conceived or employed. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because it is included in this section. Similarly, unless otherwise indicated, matters discussed in this section should not be assumed to be prior to the application. SUMMARY

[0005] The present disclosure provides a target detection method and a method, device, electronic device, computer readable storage medium and computer program product for training a target detection model.

[0006] According to an aspect of the present disclosure, a target detection method is provided, including: obtaining image data including a target object, and a target category feature corresponding to a target category of the target object, wherein the target category feature is fed to a decoder of a target detection model used for target detection; predicting the target category of the target object based on the target category feature to obtain a predicted category and a predicted category feature corresponding to the predicted category; determining consistency of the predicted category and the target category based on similarity between the predicted category feature and the target category feature; and taking the predicted category as a result of target detection on the target object according to the consistency of the predicted category and the target category.

[0007] According to another aspect of the present disclosure, a method for training a target detection model is provided, including: obtaining sample image data including a sample target object, and a sample target category feature corresponding to a sample target category of the sample target object, wherein the sample target category feature is fed to a decoder of a target detection model used for target detection; predicting the sample target category of the sample target object based on the sample target category feature to obtain a sample predicted category and a sample predicted category feature corresponding to the sample predicted category; determining consistency of the sample predicted category and the sample target category based on similarity between the sample predicted category feature and the sample target category feature; and obtaining a trained target detection model according to the consistency of the sample predicted category and the sample target category.

[0008] According to another aspect of the present disclosure, a target detection device is provided, including: an obtaining module configured to obtain image data including a target object, and a target category feature corresponding to a target category of the target object, wherein the target category feature is fed to a decoder of a target detection model used for target detection; a first predicting module configured to predict the target category of the target object based on the target category feature to obtain a predicted category and a predicted category feature corresponding to the predicted category; a first determining module configured to determine consistency of the predicted category and the target category based on similarity between the predicted category feature and the target category feature; and a first processing module configured to take the predicted category as a result of target detection on the target object according to the consistency of the predicted category and the target category.

[0009] According to another aspect of the present disclosure, there is provided an apparatus for training a target detection model, comprising: a sample obtaining module configured to obtain sample image data comprising a sample target object, and sample target category features corresponding to a sample target category of the sample target object, wherein the sample target category features are fed to a decoder of a target detection model for target detection; a sample predicting module configured to predict the sample target category of the sample target object based on the sample target category features to obtain a sample predicted category and sample predicted category features corresponding to the sample predicted category; a sample determining module configured to determine consistency of the sample predicted category and the sample target category based on similarity between the sample predicted category features and the sample target category features; and a sample processing module configured to obtain a trained target detection model according to the consistency of the sample predicted category and the sample target category.

[0010] According to another aspect of the present disclosure, there is provided an electronic device comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as provided above.

[0011] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as provided above.

[0012] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method as provided above.

[0013] According to one or more embodiments of the present disclosure, the target detection model can be guided to perform target detection on a target object in image data without the need for labeling the target object, reducing the labeling process for new target categories, and improving the universality of the target detection method.

[0014] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the embodiments. The illustrated embodiments are merely examples and do not limit the scope of the claims. In all the drawings, like reference numerals refer to like elements throughout the various drawings. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0016] Figure 1 A schematic diagram illustrating an exemplary system in which various methods described herein can be implemented according to an embodiment of the present disclosure is shown;

[0017] Figure 2 A flowchart illustrating a target detection method according to an embodiment of the present disclosure is shown;

[0018] Figure 3 A flowchart illustrating another target detection method according to an embodiment of the present disclosure is shown;

[0019] Figure 4 A structural schematic diagram of a target detection model according to an embodiment of the present disclosure is shown;

[0020] Figure 5 A flowchart illustrating a method of training a target detection model according to an embodiment of the present disclosure is shown;

[0021] Figure 6 A structural schematic diagram of a target detection model decoder according to an embodiment of the present disclosure is shown;

[0022] Figure 7 A structural block diagram of a target detection apparatus according to an embodiment of the present disclosure is shown;

[0023] Figure 8 A structural block diagram of a target detection apparatus according to another embodiment of the present disclosure is shown;

[0024] Figure 9 A structural block diagram of an apparatus for training a target detection model according to an embodiment of the present disclosure is shown;

[0025] Figure 10 A structural block diagram of an apparatus for training a target detection model according to another embodiment of the present disclosure is shown;

[0026] Figure 11 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary and not limiting. As such, various changes and modifications can be suggested to one ordinarily skilled in the art, and it is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims. Furthermore, in the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order to not unnecessarily obscure the present disclosure.

[0028] In the present disclosure, the terms "first", "second", etc. used in the description of various described examples are not intended to limit the positional relationship, timing relationship or importance relationship of the elements, and such terms are only used to distinguish one element from another. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0029] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing the specific examples, and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations.

[0030] In related technologies of target detection, in order to ensure the effect of the target detection model, all targets of interest classes in the data set are often fully labeled. When a new class is added, there are often targets of existing classes in the new data. If all targets of interest are labeled, the labeling cost is high. If only the new class is labeled, the detection model will bring ambiguity in the training process. And there may also be new classes in the original data set, which also need to be supplemented, and the labeling cost is huge.

[0031] Usually, a trained target detection model is used to predict new data and pre-label the existing class targets, thereby reducing the labeling cost. Pre-labeling can reduce some labeling cost, but still needs to be audited and modified, and the cost is still high.

[0032] To solve the above technical problems, the present disclosure provides a target detection method and a method for training a target detection model.

[0033] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0035] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable a method of object detection and a method of training an object detection model to be performed.

[0036] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0037] In Figure 1 In the illustrated configuration, the server 120 can include one or more components that implement the functionality performed by the server 120. These components can include software components that are executable by one or more processors, hardware components, or combinations thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by the components. It should be understood that a wide variety of different system configurations are possible, which can differ from system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0038] Users can use the client devices 101, 102, 103, 104, 105, and / or 106 to conduct object detection and to train object detection models. The client devices can provide interfaces that enable users of the client devices to interact with the client devices. The client devices can also output information to the users via the interfaces. Although Figure 1 Only six client devices are depicted, but one of skill in the art will appreciate that the present disclosure can support any number of client devices.

[0039] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or including various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. Client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0040] Network 110 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a local area network (LAN), a wide area network (WAN), a wireless network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth network), and / or any combination of these and / or other networks.

[0041] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, large mainframe computers, server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can run one or more services or software applications that provide the functionality described below.

[0042] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0043] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and / or 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and / or 106.

[0044] In some embodiments, the server 120 can be a server of a distributed system, or a server combined with a blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0045] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in a variety of locations. For example, databases used by the server 120 can reside locally to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network- or application-specific connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0046] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores backed by file systems.

[0047] Figure 1The system 100 can be configured and operated in various ways to enable application of various methods and apparatuses described according to the present disclosure. The following detailed description is directed to a target detection method according to an embodiment of the present disclosure.

[0048] Figure 2 A flowchart of a target detection method 200 according to an embodiment of the present disclosure is shown. As shown, the method 200 includes steps S201, S202, S203 and S204. Figure 2

[0049] In step S201, image data including a target object is acquired, and a target class feature corresponding to a target class of the target object is acquired, wherein the target class feature is fed into a decoder of a target detection model used for target detection.

[0050] In an example, the target detection model can be a DETR (Bidirectional Encoder Representations from Transformer) model, and the DETR model includes an encoder and a decoder based on a Transformer structure, i.e., a Transformer encoder and a Transformer decoder. The target class feature can be fed into the Transformer decoder of the DETR model used for target detection.

[0051] In an example, the target class feature corresponding to the target class of the target object can be in the form of a feature vector.

[0052] In an example, the target class feature corresponding to the target class of the target object can be extracted based on a neural network for natural language processing or a cross-modal neural network such as CLIP (Contrastive Language-Image Pretraining).

[0053] In step S202, the target class of the target object is predicted based on the target class feature to obtain a predicted class and a predicted class feature corresponding to the predicted class.

[0054] In an example, based on the target class feature corresponding to the target class of the target object being fed into the decoder of the target detection model used for target detection, the target class of the target object in the image data including the target object can be predicted to obtain the predicted class and the predicted class feature corresponding to the predicted class.

[0055] In an example, the predicted class feature corresponding to the predicted class can be in the form of a feature vector.

[0056] ​At step S203, consistency between the predicted category and the target category is determined based on similarity between the predicted category feature and the target category feature.

[0057] In an example, similarity calculation is performed between the predicted category feature and the target category feature, and when the similarity between the predicted category feature and the target category feature is greater than or equal to a preset threshold, it is determined that the predicted category is consistent with the target category; and when the similarity between the predicted category feature and the target category feature is less than the preset threshold, it is determined that the predicted category is inconsistent with the target category.

[0058] At step S204, the predicted category is taken as a result of target detection on the target object according to the consistency between the predicted category and the target category.

[0059] In an example, when the predicted category is consistent with the target category, the predicted category is the target category, and the predicted category can be taken as a result of target detection on the target object.

[0060] According to the target detection method of the embodiments of the present disclosure, by feeding the target category feature corresponding to the target category of the target object to the decoder of the target detection model used for target detection, the target detection model can be guided to perform target detection on the target object without the need for labeling the target object in the image data. Thus, for any newly added target category, the labeling process of the newly added target category can be reduced, and the target categories of target detection can be more conveniently extended, thereby reducing the labeling cost in target detection and improving the universality of the target detection method.

[0061] The various aspects of the target detection method according to the embodiments of the present disclosure are further described below.

[0062] According to some embodiments, the target category feature can be spliced with the inherent target query feature used for target detection in the decoder as a new target query feature.

[0063] For example, the DETR target detection model can take a fixed number N of inherent target query features (also commonly referred to as object query) as input, each query feature adaptively samples a pixel point from the image data through the network, and information interaction between the query features is performed through self-attention. Finally, each query feature is used to predict the position and category of a detection box. In target detection, a fixed number of detection targets can be predicted through the encoder-decoder manner.

[0064] Exemplarily, both the target category feature and the intrinsic target query feature can be in the form of a feature vector, and the target category feature corresponding to the target category of the target object is spliced with the intrinsic target query feature used for target detection in the decoder to be used as a new target query feature.

[0065] According to an embodiment of the present disclosure, by splicing the target category feature with the intrinsic target query feature used for target detection in the decoder to be used as a new target query feature, the target category feature can be included in the new target query feature, so that the decoder can be more targeted when decoding, and thus the prediction category corresponding to the target category is obtained.

[0066] According to some embodiments, obtaining the target category feature corresponding to the target category of the target object in step S201 can include: obtaining sub-image data corresponding to the target object from the image data; and extracting the target category feature based on the sub-image data.

[0067] Exemplarily, when the image data includes multiple objects, when the target category feature of the target object among the multiple objects needs to be extracted, sub-image data corresponding to the target object can be obtained from the image data, and the target category feature corresponding to the target object can be extracted based on the sub-image data. For example, the image data includes two objects of "person" and "dog", when the "dog" is the target object, sub-image data corresponding to the "dog" can be obtained from the image data, and the target category feature of the "dog" can be extracted from the sub-image data corresponding to the "dog".

[0068] According to an embodiment of the present disclosure, by obtaining sub-image data corresponding to the target object from the image data and extracting the target category feature of the target object based on the sub-image data, the extraction of the target category feature can be simpler and more accurate.

[0069] Figure 3 A flowchart of another target detection method 300 according to an embodiment of the present disclosure is shown, as shown in Figure 3 As shown, the method 300 can include steps S301, S302, S303, S304, S305, S306 and S307. Steps S301, S302, S303 and S304 can correspond to steps S201, S202, S203 and S204 as shown in Figure 2 Therefore, the details thereof will not be described here.

[0070] According to some embodiments, the image data can further include other objects, and the other objects have known categories and known category features corresponding to the known categories.

[0071] At step S305, the known category of the other object is predicted to obtain a predicted known category and a predicted known category feature corresponding to the predicted known category.

[0072] For example, the known category feature corresponding to the known category of the other object can be extracted based on a neural network for natural language processing or a cross-modal neural network such as CLIP from the image data including the target object and the other object.

[0073] For example, when the target detection model performs target detection, the known category of the other object is predicted to obtain a predicted known category and a predicted known category feature corresponding to the predicted known category.

[0074] At step S306, the consistency between the predicted known category and the known category is determined based on the similarity between the predicted known category feature and the known category feature.

[0075] For example, the similarity between the predicted category feature and the target category feature of the known target is calculated, and when the similarity between the predicted category feature and the known category feature is greater than or equal to a preset threshold, it is determined that the predicted category is consistent with the known category; and when the similarity between the predicted category feature and the known category feature is less than the preset threshold, it is determined that the predicted category is inconsistent with the known category.

[0076] At step S307, the predicted known category is taken as the result of target detection of the known object according to the consistency between the predicted known category and the known category.

[0077] For example, when the predicted category is consistent with the known category, the predicted category is the known category, and the predicted category can be taken as the result of target detection of the other object.

[0078] According to the embodiments of the present disclosure, by predicting the known category of the other object to obtain a predicted known category and a predicted known category feature corresponding to the predicted known category, determining the consistency between the predicted known category and the known category based on the similarity between the predicted known category feature and the known category feature, and taking the predicted known category as the result of target detection of the other object according to the consistency between the predicted known category and the known category, the target detection method can simultaneously detect the target category and the known category, so that the target detection model can more accurately understand part of the annotation.

[0079] Figure 4 A structural schematic diagram of a target detection model according to an embodiment of the present disclosure is shown.

[0080] As Figure 4As shown, the target detection model includes a convolutional neural network (CNN) 404, a position embedding 405, an encoder and a decoder based on a Transformer structure, i.e., a Transformer encoder 406 and a Transformer decoder 407, an intrinsic target query feature 409 inputted into the decoder, and a prediction layer composed of a feedforward neural network (FNN), which includes branch one 410, branch two 411, and branch three 412.

[0081] In the following, a method of training a target detection model will be described in conjunction with Figure 4 and Figure 5 .

[0082] Figure 5 A flowchart of the method 500 of training a target detection model according to an embodiment of the present disclosure is shown.

[0083] As shown, the method 500 of training a target detection model includes steps S501, S502, S503, and S504. Figure 5

[0084] In step S501, sample image data including a sample target object is obtained, and a sample target class feature corresponding to a sample target class of the sample target object is obtained, wherein the sample target class feature is fed into a decoder of a target detection model used for target detection.

[0085] For example, as shown, the sample image data 401 can be image data containing a "person" and a "dog", and the sample target object 402 can be a "dog". A sample target class feature corresponding to the target class of the "dog" can be extracted by the CLIP 403. The sample target class feature of the "dog" is fed into the Transformer decoder 407. Figure 4

[0086] In step S502, based on the sample target class feature, a sample target class of the sample target object is predicted to obtain a sample predicted class and a sample predicted class feature corresponding to the sample predicted class.

[0087] For example, based on the sample target class feature of the "dog", the target detection model predicts the "dog" to obtain a sample predicted class and a sample predicted class feature corresponding to the sample predicted class.

[0088] In step S503, based on the similarity between the sample predicted class feature and the sample target class feature, the consistency of the sample predicted class and the sample target class is determined. ​​

[0089] like Figure 4 As shown, the detection head consists of three branches composed of three FFNs: Branch 1 (410) is responsible for regressing the target box, trained using L1 loss and GIoU (Generalized Intersection over Union) loss to obtain the target box of the detected object; Branch 2 (411) is responsible for predicting the consistency between the category of the output box and the category of the sample target, trained using cross-entropy loss, and can be labeled when the category of the output box is consistent with the category of the sample target, for example, labeled with "1"; Branch 3 (412) is responsible for determining the similarity between the predicted category features of the sample and the category features of the sample target, obtaining the predicted category of the sample through similarity calculation, and trained using cross-entropy loss.

[0090] For example, the similarity between the predicted category features of the sample and the target category features of the "dog" sample is calculated. When the similarity between the predicted category features of the sample and the target category features of the "dog" sample is greater than or equal to a threshold, it is determined that the predicted category features of the sample and the target category features of the "dog" sample are consistent. When the similarity between the predicted category features of the sample and the target category features of the "dog" sample is less than the threshold, it is determined that the predicted category features of the sample and the target category features of the "dog" sample are inconsistent.

[0091] In step S504, the target detection model is obtained after training, based on the consistency between the predicted category of the sample and the target category of the sample.

[0092] In the example, when the predicted category features of the sample match the target category features of the sample "dog", a target detection model that can detect "dog" is obtained.

[0093] The method for training an object detection model according to embodiments of this disclosure feeds sample object category features of a sample object to the decoder of the object detection model used for object detection. This allows the object detection model to be guided to perform object detection on the sample object without requiring annotation of the sample object in the sample image data. Therefore, for any newly added sample object category, the annotation process for the new sample object category can be reduced, and the object detection model can be more easily expanded to detect a wider range of object categories, thereby reducing annotation costs and improving the versatility of the object detection model.

[0094] Figure 6 This is a schematic diagram illustrating a target detection model decoder according to an embodiment of the present disclosure.

[0095] like Figure 6As shown, the decoder can include a multi-head self-attention module 606, a multi-head cross-attention module 604, a normalization layer 603, and a normalization layer 605. The normalization layer 603 and the normalization layer 605 can use common implementation manners. The input of the decoder can be a fixed number of intrinsic target query features 607, the multi-head self-attention module 606 performs non-dense attention processing on the intrinsic target query features 607, and the multi-head cross-attention module 604 performs attention processing on the intrinsic target query features 607 processed by the multi-head self-attention module 606 and the features from the encoder 601.

[0096] According to some embodiments, the sample target class features are spliced with the intrinsic target query features used for target detection in the decoder as new target query features.

[0097] Exemplarily, Figure 6 The intrinsic target query features 607 in the encoder can be a vector with a dimension of N*d. Figure 6 The sample target class features 602 in the decoder can be a vector with a dimension of K*d, where K is the number of sample target classes of the sample target object. The sample target class features 602 (K*d) are spliced with the intrinsic target query features 607 (N*d) used for target detection as new target query features, which are a vector with a dimension of (K+N)*d. The new target query features with a dimension of (K+N)*d are taken as a value vector (V) and a key vector (K), and the intrinsic target query features N*d are still taken as a query vector (Q).

[0098] According to the embodiments of the present disclosure, by splicing the sample target class features with the intrinsic target query features used for target detection in the decoder as new target query features, the decoder can be more targeted when decoding, and thus the prediction class corresponding to the sample target class is obtained.

[0099] According to some embodiments, as Figure 5 As shown, the step S501 can include: obtaining sub-sample image data corresponding to the sample target object from the sample image data; and extracting sample target class features corresponding to the sample target class of the sample target object based on the sub-sample image data.

[0100] Exemplarily, when the target object is a "dog", as Figure 6 As shown, sub-sample image data corresponding to the "dog" can be obtained from the sample image data, and then the sample target class features corresponding to the sample target class of the sample target object are extracted through the CLIP model.

[0101] Exemplarily, there can be multiple sub-sample image data corresponding to the target object.

[0102] According to the embodiments of the present disclosure, by obtaining the sub-sample image data corresponding to the sample target object from the sample image data, and extracting the sample target class feature corresponding to the sample target class of the sample target object based on the sub-sample image data, the sample target class feature can be obtained simply and accurately.

[0103] According to another aspect of the present disclosure, a target detection apparatus is also provided.

[0104] Figure 7 A structural block diagram of a target detection apparatus 700 according to an embodiment of the present disclosure is shown.

[0105] As shown in Figure 7 The target detection apparatus 700 includes an obtaining module 710, a first predicting module 720, a first determining module 730 and a first processing module 740.

[0106] The obtaining module 710 is configured to obtain image data including a target object, and a target class feature corresponding to a target class of the target object, wherein the target class feature is fed to a decoder of a target detection model used for target detection.

[0107] The first predicting module 720 is configured to predict the target class of the target object based on the target class feature, to obtain a predicted class and a predicted class feature corresponding to the predicted class.

[0108] The first determining module 730 is configured to determine the consistency of the predicted class and the target class based on the similarity between the predicted class feature and the target class feature.

[0109] The first processing module 740 is configured to take the predicted class as the result of target detection on the target object according to the consistency of the predicted class and the target class.

[0110] Thus, by feeding the target class feature corresponding to the target class of the target object to the decoder of the target detection model used for target detection, the target detection model can be guided to perform target detection on the target object without the need for labeling the target object in the image data. Thus, for any newly added target class, the labeling process of the newly added target class can be reduced, and the target classes of the target detection can be more conveniently expanded, thereby reducing the labeling cost in target detection and improving the universality of the target detection method.

[0111] According to some embodiments, the target class feature is concatenated with the inherent target query feature used for target detection in the decoder as a new target query feature.

[0112] Therefore, by concatenating the target category features with the inherent target query features used in target detection in the decoder as new target query features, the new target query features can contain target category features, thus enabling the decoder to be more targeted during decoding and obtain the predicted category corresponding to the target category.

[0113] Figure 8 A structural block diagram of a target detection apparatus 800 according to another embodiment of the present disclosure is shown. Figure 8 As shown, the target detection device 800 includes an acquisition module 810, a first prediction module 820, a first determination module 830, and a first processing module 840. The acquisition module 810, the first prediction module 820, the first determination module 830, and the first processing module 840 can be integrated with... Figure 7 The acquisition module 710, the first prediction module 720, the first determination module 730 and the first processing module 740 shown correspond to each other, so the details will not be repeated here.

[0114] According to some embodiments, the acquisition module 810 may include: a sub-image acquisition module 811 configured to acquire sub-image data corresponding to a target object from image data; and an extraction module 812 configured to extract target category features based on the sub-image data.

[0115] Therefore, by obtaining sub-image data corresponding to the target object from the image data, and extracting the target category features of the target object based on the sub-image data, the extraction of target category features can be made simpler and more accurate.

[0116] According to some embodiments, the image data may further include other objects having known categories and known category features corresponding to those categories, wherein the target detection device 800 may further include:

[0117] The second prediction module 850 is configured to predict the known categories of other objects to obtain the predicted known categories and the predicted known category features corresponding to the predicted known categories; the second determination module 860 is configured to determine the consistency between the predicted known categories and the known categories based on the similarity between the predicted known category features and the known category features; and the second processing module 870 is configured to use the predicted known categories as the result of target detection for other objects based on the consistency between the predicted known categories and the known categories.

[0118] Thus, by predicting the known class of the other object to obtain a predicted known class and a predicted known class feature corresponding to the predicted known class, determining the consistency of the predicted known class and the known class based on the similarity between the predicted known class feature and the known class feature, and taking the predicted known class as the result of target detection on the other object according to the consistency of the predicted known class and the known class, the target detection method can simultaneously detect the target class and the other class, so that the target detection model is more accurate in understanding part of the label.

[0119] According to another aspect of the present disclosure, an apparatus for training a target detection model is also provided.

[0120] Figure 9 A structural block diagram of the apparatus 900 for training a target detection model according to one embodiment of the present disclosure is shown.

[0121] As shown in Figure 9 The apparatus 900 for training a target detection model can include a sample acquisition module 910, a sample prediction module 920, a sample determination module 930, and a sample processing module 940.

[0122] The sample acquisition module 910 is configured to acquire sample image data including a sample target object, and sample target class features corresponding to a sample target class of the sample target object, wherein the sample target class features are fed to a decoder of a target detection model for target detection.

[0123] The sample prediction module 920 is configured to predict the sample target class of the sample target object based on the sample target class features to obtain a sample predicted class and sample predicted class features corresponding to the sample predicted class.

[0124] The sample determination module 930 is configured to determine the consistency of the sample predicted class and the sample target class based on the similarity between the sample predicted class features and the sample target class features.

[0125] The sample processing module 940 is configured to obtain a trained target detection model according to the consistency of the sample predicted class and the sample target class.

[0126] Thus, by feeding the sample target class feature of the sample target object to the decoder of the target detection model for target detection, the target detection model can be guided to perform target detection on the sample target object without the need for labeling the sample target object in the sample image data. Thus, for any newly added sample target class, the labeling process of the newly added sample target class can be reduced, and the target classes that the target detection model can detect can be more conveniently expanded, thereby reducing the labeling cost in target detection and improving the universality of the target detection model.

[0127] According to some embodiments, the sample target class feature can be spliced with the inherent target query feature used for target detection in the decoder as a new target query feature.

[0128] Thus, by splicing the sample target class feature with the inherent target query feature used for target detection in the decoder as a new target query feature, the decoder can be more targeted when decoding, and thus obtain the prediction class corresponding to the target class.

[0129] Figure 10 A structural block diagram of an apparatus 1000 for training a target detection model according to another embodiment of the present disclosure is shown. As shown, the apparatus 1000 for training a target detection model can include a sample obtaining module 1010, a sample prediction module 1020, a sample determination module 1030, and a sample processing module 1040. The sample obtaining module 1010, the sample prediction module 1020, the sample determination module 1030, and the sample processing module 1040 can correspond to the sample obtaining module 910, the sample prediction module 920, the sample determination module 930, and the sample processing module 940 shown in FIG. 9, and thus details will not be repeated here. Figure 10 Figure 9

[0130] According to some embodiments, the sample obtaining module 1010 can include a sample sub-image obtaining sub-module 1011 configured to obtain sub-sample image data corresponding to the sample target object from the sample image data, and a sample extraction module 1012 configured to extract a sample target class feature corresponding to a sample target class of the sample target object based on the sub-sample image data.

[0131] Thus, by obtaining sub-sample image data corresponding to the sample target object from the sample image data, and extracting a sample target class feature corresponding to a sample target class of the sample target object based on the sub-sample image data, the sample target class feature can be obtained simply and accurately.

[0132] ​​According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in the above embodiments.

[0133] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause a computer to perform the method in the above embodiments.

[0134] According to another aspect of the present disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method in the above embodiments.

[0135] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0136] Reference Figure 11 A block diagram of an electronic device 1100, which can be used as a server or a client of the present disclosure, will now be described, which is an example of a hardware device that can be applied to aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computing devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent various forms of mobile devices such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components, their connections, and their functions as shown in the figures, and their functions, are by way of example only, and are not intended to limit implementations of the present disclosure described and / or claimed herein.

[0137] As Figure 11 shown, the electronic device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0138] The plurality of components in the electronic device 1100 are connected to the I / O interface 1105, including an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device that can input information to the electronic device 1100, can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1107 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0139] The computing unit 1101 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above, such as the object detection method and the method of training the object detection model. For example, in some embodiments, the object detection method and the method of training the object detection model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded onto the RAM 1103 and executed by the computing unit 1101, one or more steps of the object detection method and the method of training the object detection model described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the object detection method and the method of training the object detection model by any other appropriate means, such as by means of firmware.

[0140] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0141] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0144] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0145] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0146] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0147] While embodiments or examples of this disclosure have been described with reference to the figures, it will be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the application is not limited to these embodiments or examples. Various elements of the embodiments or examples can be omitted or substituted by equivalents thereof. Furthermore, the steps can be performed in a different order than described in the disclosure. Further, various elements of the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described herein can be substituted by equivalents which serve the same function.

Claims

1. A method for object detection, comprising: obtaining image data comprising a target object, and a target category feature corresponding to a target category of the target object, wherein the target category feature is fed into a decoder of an object detection model used for object detection; predicting the target category of the target object based on the target category feature to obtain a predicted category and a predicted category feature corresponding to the predicted category; determining consistency of the predicted category and the target category based on similarity between the predicted category feature and the target category feature; and regarding the predicted category as a result of object detection on the target object according to the consistency of the predicted category and the target category, wherein the target category feature is used in the decoder by being concatenated with intrinsic target query features used by the object detection as new target query features, wherein the target category feature is obtained by a multi-modal neural network, and wherein the new target query features are used as value vectors and key vectors of the decoder, and the intrinsic target query features are used as query vectors of the decoder.

2. The method of claim 1, wherein, The obtaining of the target category feature corresponding to the target category of the target object comprises: obtaining sub-image data corresponding to the target object from the image data; and extracting the target category feature based on the sub-image data.

3. The method of claim 1 or 2, wherein, The image data further comprises other objects having known categories and known category features corresponding to the known categories, wherein the method further comprises: predicting the known categories of the other objects to obtain predicted known categories and predicted known category features corresponding to the predicted known categories; determining consistency of the predicted known categories and the known categories based on similarity between the predicted known category features and the known category features; and regarding the predicted known categories as results of the object detection on the other objects according to the consistency of the predicted known categories and the known categories. 4.A method for training an object detection model, comprising: obtaining sample image data comprising sample target objects, and sample target category features corresponding to sample target categories of the sample target objects, wherein the sample target category features are fed into a decoder of an object detection model used for object detection; predicting the sample target categories of the sample target objects based on the sample target category features to obtain sample predicted categories and sample predicted category features corresponding to the sample predicted categories; determining consistency of the sample predicted categories and the sample target categories based on similarity between the sample predicted category features and the sample target category features; and obtaining a trained object detection model according to the consistency of the sample predicted categories and the sample target categories. The sample target class feature is obtained by a multi-modal neural network, and the new target query feature is used as a value vector and a key vector of the decoder, and the inherent target query feature is used as a query vector of the decoder.

5. The method of claim 4, wherein, The obtaining of the sample target class feature corresponding to the sample target class of the sample target object comprises: obtaining sub-sample image data corresponding to the sample target object from the sample image data, and extracting the sample target class feature corresponding to the sample target class of the sample target object based on the sub-sample image data.

6. A target detection apparatus, comprising: an obtaining module configured to obtain image data comprising a target object, and a target class feature corresponding to a target class of the target object, wherein the target class feature is fed to a decoder of a target detection model used for target detection; a first predicting module configured to predict the target class of the target object based on the target class feature to obtain a predicted class and a predicted class feature corresponding to the predicted class; a first determining module configured to determine consistency of the predicted class and the target class based on similarity between the predicted class feature and the target class feature; and a first processing module configured to take the predicted class as a result of target detection on the target object according to the consistency of the predicted class and the target class, wherein the target class feature is used in the decoder together with an inherent target query feature used for the target detection as a new target query feature, wherein the target class feature is obtained by a multi-modal neural network, and wherein the new target query feature is used as a value vector and a key vector of the decoder, and the inherent target query feature is used as a query vector of the decoder.

7. The apparatus of claim 6, wherein, The obtaining module comprises: a sub-image obtaining module configured to obtain sub-image data corresponding to the target object from the image data; and an extracting module configured to extract the target class feature based on the sub-image data.

8. The apparatus of claim 6 or 7, wherein, The image data further comprises other objects, the other objects having known classes and known class features corresponding to the known classes, wherein the apparatus further comprises: a second predicting module configured to predict the known classes of the other objects to obtain predicted known classes and predicted known class features corresponding to the predicted known classes; a second determining module configured to determine consistency of the predicted known classes and the known classes based on similarity between the predicted known class features and the known class features; and a second processing module configured to take the predicted known classes as results of the target detection on the other objects according to the consistency of the predicted known classes and the known classes.

9. An apparatus for training a target detection model, comprising: a sample obtaining module configured to obtain sample image data comprising a sample target object, and sample target class features corresponding to a sample target class of the sample target object, wherein the sample target class features are fed to a decoder of a target detection model used for target detection; a sample predicting module configured to predict the sample target class of the sample target object based on the sample target class features to obtain a sample predicted class and sample predicted class features corresponding to the sample predicted class; a sample determining module configured to determine consistency of the sample predicted class and the sample target class based on similarity between the sample predicted class features and the sample target class features; and a sample processing module configured to obtain the trained target detection model according to the consistency of the sample predicted class and the sample target class, wherein the sample target class features are used in the decoder by being spliced with intrinsic target query features used by the target detection as new target query features, wherein the sample target class features are obtained by a multi-modal neural network, and wherein the new target query features are used as value vectors and key vectors of the decoder, and the intrinsic target query features are used as query vectors of the decoder.

10. The apparatus of claim 9, wherein, The sample obtaining module comprises: a sample sub-image obtaining sub-module configured to obtain sub-sample image data corresponding to the sample target object from the sample image data; and a sample extracting module configured to extract sample target class features corresponding to the sample target class of the sample target object based on the sub-sample image data.

11. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5.

13. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-5. The computer program, when executed by a processor, implements the method of any one of claims 1-5.

Images