Image feature extraction method, open set target detection method, device and program product

Abstract

The present disclosure provides an image feature extraction method, an open set target detection method, device and program product, and relates to the technical field of image recognition. The image feature extraction method in the present disclosure comprises: obtaining first image features corresponding to each sample image, each sample image containing image elements associated with the same target category; calculating an average image feature based on each first image feature; performing feature fusion on the average image feature and the first image feature based on an attention mechanism to obtain a second image feature; and determining a target image feature based on the second image feature, the target image feature being used to represent the target category.

Description

Technical Field

[0001] This specification relates to the field of image recognition technology, and in particular to an image feature extraction method, an open set target detection method, an apparatus, a device, a storage medium, and a program product. Background Technology

[0002] In the field of image recognition, image recognition models can be programmed with cues of a specific category to identify images of that category. These cues can include visual cues, which are images containing elements of a specific category, such as a picture of a dog or a house. Visual cues typically consist of multiple images containing elements of the same category. Features from these images can be combined to create a new cue feature, which the image recognition model can then learn from the visual cues to identify the category. However, if the visual cues contain mislabeled images or images with weak relevance to the specific category, the category features will not accurately represent the category, thus reducing the accuracy of the image recognition model in identifying that category.

[0003] In view of this, embodiments of this specification provide an image feature extraction method, an open set target detection method, an apparatus, a device, a storage medium, and a program product, in order to improve the accuracy of image recognition models in recognizing images of specific categories. Summary of the Invention

[0004] This specification provides one or more embodiments of an image feature extraction method, the method comprising: acquiring first image features corresponding to each sample image, each sample image containing image elements associated with the same target category; calculating an average image feature based on each first image feature; performing feature fusion on the average image feature and the first image feature based on an attention mechanism to obtain a second image feature; and determining a target image feature based on the second image feature, the target image feature being used to characterize the target category.

[0005] According to one or more embodiments of this specification, the method further includes: performing a residual concatenation between an average image feature and a second image feature; determining a target image feature based on the second image feature, including: using the sum of the residual concatenations of the average image feature and the second image feature as the target image feature.

[0006] According to one or more embodiments of this specification, image elements in a sample image that are associated with the same target category are labeled using at least one of a mask, a bounding box, coordinate points, or a hand-drawn sketch; obtaining a first image feature corresponding to each sample image includes: obtaining a first image feature corresponding to each sample image based on each sample image and the labeled image elements in the sample image.

[0007] According to one or more embodiments of this specification, the method further includes: using target image features as visual cue features for open-set target detection to identify image elements in the image to be detected that correspond to the target category based on the target image features.

[0008] According to one or more embodiments of this specification, the third image features obtained based on the image to be detected have the same feature vector dimension as the target image features.

[0009] According to one or more embodiments of this specification, a method based on an attention mechanism is used to fuse average image features and a first image feature to obtain a second image feature, including: mapping the average image feature to a query vector space; mapping the first image feature to a key vector space and a value vector space respectively; and fusing the average image feature mapped to the query vector space and the first image feature mapped to the key vector space and the value vector space based on an attention mechanism to obtain the second image feature.

[0010] According to one or more embodiments of this specification, a method based on an attention mechanism is used to fuse average image features mapped to a query vector space and first image features mapped to a key vector space and a value vector space to obtain second image features. This includes: obtaining a query matrix corresponding to the average image features; obtaining a key matrix corresponding to the first image features; obtaining a value matrix corresponding to the first image features; obtaining the similarity between the average image features and the first image features based on the query matrix and the key matrix to generate attention weights for the average image features and the first image features; and determining the second image features based on the attention weights and the value matrix.

[0011] One or more embodiments of this specification also provide an open set target detection method, the method comprising: acquiring sample images and an image to be detected, wherein each sample image contains image elements associated with the same target category; acquiring first image features corresponding to each sample image, and calculating an average image feature based on each first image feature; performing feature fusion on the average image feature and the first image feature based on an attention mechanism to obtain a second image feature, and determining a target image feature based on the second image feature, wherein the target image feature is used to characterize the target category; and identifying image elements in the image to be detected corresponding to the target category based on the target image feature.

[0012] According to one or more embodiments of this specification, a method for identifying image elements in an image to be detected that correspond to a target category based on target image features includes: obtaining third image features based on each candidate region of the image to be detected; calculating the similarity between each third image feature and the target image features; and determining image elements in candidate regions whose similarity reaches a preset condition as the corresponding target category.

[0013] One or more embodiments of this specification also provide an image feature extraction apparatus, the apparatus comprising: a sample feature acquisition module, configured to acquire first image features corresponding to each sample image, wherein each sample image contains image elements associated with the same target category; an average feature calculation module, configured to calculate an average image feature based on each first image feature; a feature fusion module, configured to perform feature fusion on the average image feature and the first image feature based on an attention mechanism to obtain a second image feature; and a target feature determination module, configured to determine a target image feature based on the second image feature, wherein the target image feature is used to characterize a target category.

[0014] One or more embodiments of this specification also provide an open set target detection device, the device comprising: an image acquisition module for acquiring sample images and an image to be detected, wherein each sample image contains image elements associated with the same target category; a sample feature fusion module for acquiring first image features corresponding to each sample image and calculating an average image feature based on each first image feature; performing feature fusion on the average image feature and the first image feature based on an attention mechanism to obtain a second image feature, and determining a target image feature based on the second image feature, wherein the target image feature is used to characterize the target category; and an image recognition module for recognizing image elements in the image to be detected corresponding to the target category based on the target image feature.

[0015] One or more embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the image feature extraction method and / or open set target detection method described in some embodiments of this specification.

[0016] One or more embodiments of this specification also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, can implement the image feature extraction method and / or open set target detection method described in some embodiments of this specification.

[0017] One or more embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement the image feature extraction method and / or open set target detection method described in some embodiments of this specification.

[0018] The effective effects that the embodiments of this specification may bring include, but are not limited to: obtaining first image features of each sample image containing image elements of the same target category, calculating average image features based on each first image feature, fusing the average image features and the first image features based on an attention mechanism to obtain second image features, and determining target image features representing the target category based on the second image features. Through the attention mechanism, the second image features that fuse the average image features and the first image features can pay more attention to features related to the target category and reduce attention to features that are weakly related to the target category. This can reduce the impact of sample images with labeling errors or weak correlation with the target category on the representation of the target category, thereby improving the accuracy of the second image features in representing the target category, and thus improving the accuracy and robustness of subsequent image identification of the target category based on the target image features.

[0019] Since the average image features retain the complete features in the sample images, by concatenating the residuals of the average image features and the second image features as the target image features, even if the second image features obtained based on the attention mechanism lose some features, the target image features can retain the complete features of the sample images while paying more attention to features that are strongly related to the target category.

[0020] By annotating image elements in a sample image that are associated with the target category using at least one of the following methods: mask, bounding box, coordinate points, or hand-drawn illustration, the extraction of the first image feature from the sample image can be determined based on the annotated image elements, thus avoiding the influence of image elements or background elements in the sample image that are not associated with the target category on the extraction of the first image feature.

[0021] By using target image features as visual cue features for open-set object detection, image elements in the image to be detected that correspond to the target category can be identified based on the target category represented by the target image features. For any newly added target category, by providing sample images containing image elements of the newly added target category, the open-set object detection can be made capable of recognizing image elements of the newly added target category based on the determined target image features.

[0022] By setting the third image features obtained from the image to be detected and the target image features to the same feature vector dimension, it is easier to calculate the similarity between the third image features and the target image features.

[0023] Since the feature vector dimension of the first image feature is equal to the image feature dimension of each sample image multiplied by the number of sample images, and the average image feature can be calculated by summing the vector elements of each corresponding dimension in the first image feature and dividing by the number of sample images, by mapping the average image feature to the query vector space and mapping the first image feature to the key vector space and value vector space respectively, the dimension of the query vector can be reduced. In the subsequent calculation of attention weights based on the query vector and key vector, and the determination of the output second image feature based on the attention weight and value vector, the overall computational load of the attention mechanism can be reduced, realizing lightweight extraction of the second image feature, thereby improving the speed of image feature extraction and making it more suitable for real-time image recognition scenarios.

[0024] By obtaining third image features from each candidate region of the image to be detected, and based on the similarity between each third image feature and the target image feature, candidate regions that meet the preset conditions of similarity with the target image feature can be determined, such as the candidate region with the highest similarity. The image elements in the candidate region can be determined as the corresponding target category, thereby realizing the rapid identification of image elements of the target category in the image to be detected.

[0025] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects may be any one or a combination of the above, or any other possible beneficial effects. Attached Figure Description

[0026] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0027] Figure 1 This is a schematic diagram illustrating an application scenario of image feature extraction based on some embodiments of this specification.

[0028] Figure 2 This is an exemplary flowchart of an image feature extraction method according to some embodiments of this specification.

[0029] Figure 3 This is an exemplary flowchart illustrating image feature fusion according to some embodiments of this specification.

[0030] Figure 4 This is an exemplary flowchart illustrating an image feature fusion calculation according to some embodiments of this specification.

[0031] Figure 5 This is a schematic diagram illustrating an image feature extraction method according to some embodiments of this specification.

[0032] Figure 6 This is an exemplary flowchart of an open set target detection method according to some embodiments of this specification.

[0033] Figure 7 This is an exemplary flowchart illustrating image element recognition according to some embodiments of this specification.

[0034] Figure 8 This is an exemplary block diagram of an image feature extraction apparatus according to some embodiments of this specification.

[0035] Figure 9 This is an exemplary block diagram of an open set target detection device according to some embodiments of this specification. Detailed Implementation

[0036] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0037] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0038] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0039] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Such adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0040] In the field of image recognition, image recognition models can be trained to identify images of specific categories by inputting cues of a particular category. These cues can include visual cues, i.e., images containing elements of a specific category, such as a picture of a dog or a house. In some embodiments, visual cues typically include multiple images containing elements of the same specific category. The features of these multiple images can be combined to synthesize cue features, allowing the image recognition model to learn category features related to that category from the visual cues. However, if the visual cues contain mislabeled images or images with weak relevance to the specific category, the category features will not accurately represent the specific category, thus reducing the accuracy of the image recognition model in identifying that category. In some embodiments, to obtain image features that more accurately represent the specific category based on visual cues and improve the accuracy of the image recognition model in identifying that category, complex network structures are often designed to enhance the image features. This undoubtedly increases the computational load during image feature extraction, making it unsuitable for real-time image recognition scenarios.

[0041] Therefore, some embodiments of this specification propose an image feature extraction method, an open set target detection method, an apparatus, a device, a storage medium, and a program product, aiming to improve the accuracy of image recognition models in identifying specific categories of images. It is understood that in the specific implementations of this specification, the collection, use, or processing of data (e.g., sample images, images to be detected, etc.) is involved. When one or more embodiments of this specification are applied to specific products or technical implementations, permission or consent from the data subject is required. Furthermore, the collection, use, or processing of related data must strictly comply with the relevant laws, regulations, and standards of the data source country, implementation country, and other relevant countries and regions. De-identification technology ensures that the final data used is securely processed de-identified data, protecting the rights and interests of the data subject and data security.

[0042] In some embodiments, an image recognition model can refer to a computational system based on machine learning that can automatically perform semantic understanding and category determination of visual content in digital images or video frames. In some embodiments, image feature extraction can refer to the process of automatically learning and extracting a digital representation with strong discriminative, high-dimensional semantic information from the pixel data of an input image using an image recognition model. This process can map the input image from a high-dimensional, redundant, and noisy pixel space to a low-dimensional, compact feature space containing essential structure. In some embodiments, open-set object detection can refer to a computer vision method capable of identifying and locating objects with open-ended category definitions that have not been seen during the training phase, during the inference phase. Open-set object detection can achieve object detection of new categories through external information cues (textual or visual cues). In some embodiments, open-set object detection can be implemented based on the architecture of a real-time object detection and segmentation model, such as YOLOE (You Only Look Once Enhanced, an efficient visual detection model for open sets).

[0043] Figure 1 This is a schematic diagram illustrating an application scenario of image feature extraction according to some embodiments of this specification. In some embodiments, such as Figure 1 As shown, application scenario 100 may include a client 110, a server 120, and a network 130. The client 110 and server 120 can transmit data through the network 130. The client 110 can provide an upload interface and upload entry for sample images and / or images to be detected, and send the user-uploaded sample images and / or images to be detected to the server 120. Multiple sample images may contain image elements associated with the same target category. The server 120 can determine the target image features used to characterize the target category based on multiple sample images, and can also identify image elements in the image to be detected corresponding to the target category based on the target image features.

[0044] In some embodiments, client 110 may provide an upload interface and upload entry for sample images and images to be detected. In some embodiments, client 110 may be a terminal device including, but not limited to, a desktop computer, smartphone, laptop, VR device, tablet computer, smart TV, in-vehicle terminal, etc. Client 110 may include a display screen and a processor. The display screen may be used to present a graphical user interface. For example, client 110 may present an upload interface and upload entry for sample images and images to be detected through the graphical user interface. In some embodiments, the display screen may be separate from the human-machine interface device. Users may operate on the graphical user interface through the human-machine interface device. The processor of client 110 may receive operation instructions generated by operating on the graphical user interface through the human-machine interface device. For example, the processor may generate an image recognition request based on the sample image and image to be detected uploaded by the user, and receive the image recognition result fed back by server 120. The image recognition result may include the annotation result of image elements labeled with the target category for the image to be detected. In some embodiments, the display screen may be a touch screen, which may receive operation instructions input by the user based on the graphical user interface. In some embodiments, client 110 may include a memory for storing the received image recognition results.

[0045] The server 120 may be a high-performance computer device used to receive image recognition requests sent by the client 110. The image recognition request may include an image to be detected and multiple sample images containing image elements associated with the same target category. The server 120 can determine target image features representing the target category based on the multiple sample images, and can also identify image elements corresponding to the target category in the image to be detected based on the target image features. In some embodiments, the server 120 may include a local server or a cloud server. Depending on different service requirements, a local server or a cloud server corresponding to that region may be deployed in one or more regions. In some embodiments, the server 120 may include a background processing server. The background processing server can determine target image features representing the target category based on the multiple sample images containing image elements associated with the same target category sent by the client 110, and can also identify image elements corresponding to the target category in the image to be detected sent by the client 110 based on the target image features. In some embodiments, the server 120 may be a single computer device or a computing cluster composed of multiple computer devices, thereby providing more powerful computing power and more efficient data processing, such as quickly determining target image features for characterizing target categories based on multiple sample images, and quickly identifying image elements in the image to be detected that correspond to the target category based on the target image features.

[0046] Network 130 can be any form of wired or wireless network, or any combination thereof. As an example, network 130 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, internal networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth networks. Network 130 can have multiple access points, and clients 110 and servers 120 can access network 130 through these access points.

[0047] It should be noted that, Figure 1 The illustrated application scenario diagram for image feature extraction is merely an example. The application scenarios for image feature extraction described in the embodiments of this specification are intended to more clearly illustrate the technical solutions of the embodiments of this specification and do not constitute a limitation on the technical solutions provided in the embodiments of this specification. For example, Figure 1 The number of client 110 and server 120 is merely illustrative and not intended to limit the scope of patent protection of this application. Depending on the actual situation, there can be any number of client 110 and server 120. For example, client 110 and server 120 can be implemented as a single device, allowing the uploading and image processing of sample images and / or images to be detected to be completed on the same device. As will be apparent to those skilled in the art, with the development of data processing technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this specification are also applicable to similar technical problems.

[0048] To improve the accuracy of identifying specific categories of images, this specification provides an image feature extraction method. Figure 2 This is an exemplary flowchart of an image feature extraction method according to some embodiments of this specification. Figure 2 The process 200 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown is executed. In some embodiments, process 200 can be implemented by an image feature extraction device 800 deployed on a processing device. The image feature extraction device 800 may include a sample feature acquisition module 810, an average feature calculation module 820, a feature fusion module 830, and a target feature determination module 840. In some embodiments, such as Figure 2 As shown, process 200 may include the following steps.

[0049] Step 210: Obtain the first image features corresponding to each sample image, wherein each sample image contains image elements associated with the same target category. In some embodiments, step 210 may be implemented by the sample feature acquisition module 810.

[0050] In some embodiments, the sample images may refer to classified images, and the classification of sample images may refer to grouping sample images belonging to the same target category into an image set. In some embodiments, the sample images in step 210 may belong to the same image set.

[0051] In some embodiments, the first image feature can refer to a high-dimensional semantic representation extracted from the sample image by an image encoder, used to represent the visual features of the sample image. In some embodiments, since multiple sample images are input, the first image feature can be the original set of visual features corresponding to each sample image. For example, the first image feature can be represented in the form of a feature vector. After inputting four sample images, if a 1*256-dimensional feature vector is extracted based on each sample image, and this feature vector corresponds to the original visual features of the sample image, then the first image feature can represent the original set of visual features of the four sample images by a 4*256-dimensional feature vector. In some embodiments, the image encoder can extract a high-dimensional semantic representation from the image, which can be implemented using a pre-trained visual model, such as the CLIP (Contrastive Language–Image Pre-training) model.

[0052] In some embodiments, an image element may refer to a visual constituent unit in a sample image that carries semantic information. A sample image may include at least one image element. For example, a sample image may include an image element corresponding to a house, an image element corresponding to a road, an image element corresponding to a human body, etc.

[0053] In some embodiments, a category may refer to a visual type with specific semantics, and a target category may refer to a visual type with specific semantics that is to be identified. An image set may correspond to a target category. Each sample image in the image set is required to contain image elements associated with the target category, that is, the semantic information carried by the image elements common to all sample images in the image set must have the specific semantics represented by the target category. By obtaining the first image features based on each sample image containing image elements associated with the same target category, the first image features can contain the original visual features of the image elements associated with the target category in each sample image.

[0054] In some embodiments, image elements in a sample image that are associated with the same target category are labeled using at least one of a mask, a bounding box, coordinate points, or a hand-drawn sketch.

[0055] In some embodiments, a mask is a pixel-level binary or probabilistic labeling method. On a sample image, each pixel can be labeled as whether it belongs to the target category of the image element; if it belongs, the pixel value is 1, and if it doesn't, the pixel value is 0. The mask can outline the shape and contour of the image element. In some embodiments, by labeling image elements of the target category using a mask, the image encoder can extract the first image features only from the pixels within the masked area, thereby obtaining a feature representation unaffected by background interference.

[0056] In some embodiments, the bounding box is typically a regular geometric shape, such as a rectangle. The bounding box completely encloses image elements of the target category and is a compact representation of the spatial location and approximate extent of these elements. In some embodiments, bounding box annotation can reduce interference from image elements unrelated to the target category in the sample image. Because bounding boxes do not require attention to the shape contours of image elements, annotation is faster than masking. In some embodiments, the image encoder can extract features from image elements within the bounding box region to obtain a feature vector representing the overall visual content within that region.

[0057] In some embodiments, coordinate points can refer to one or more discrete two-dimensional coordinates labeled on an image element of the target category, representing a sparse labeling form. In some embodiments, a single coordinate point is typically labeled at the center or most discriminative location of the image element of the target category, such as the tip of the nose on a face; multiple coordinate points are typically labeled at multiple key points of the image element of the target category, such as human joints. Coordinate point labeling is fast and easy to interact with, and can often be converted into positional codes, fused with image features, and used to guide the image encoder to focus on the coordinate point and its surrounding area.

[0058] In some embodiments, hand-drawn sketches can roughly indicate the position and shape of image elements of a target category using irregular, continuous lines or areas freely drawn by the user. This is a sketch-like annotation between coordinate points and masks, combining annotation efficiency with some shape guidance, but containing a certain amount of noise. In some embodiments, hand-drawn sketches can be processed as a sparse soft mask, and the image encoder can learn the intent implied by this rough sketch and generalize it to the complete region containing image elements of the target category.

[0059] In some embodiments, step 210 may further include: obtaining a first image feature corresponding to each sample image based on each sample image and the image elements annotated in the sample images. In some embodiments, after annotating the image elements in the sample images that are associated with the target category through annotation forms such as masks, bounding boxes, coordinate points, and hand-drawn drawings, when the image encoder obtains the first image feature corresponding to each sample image, it can pay more attention to the image elements in the sample images that are related to the target category, reduce the interference of image elements in the sample images that are unrelated to the target category, and further improve the accuracy of the target image feature representation of the target category obtained subsequently.

[0060] Step 220: Calculate the average image features based on each of the first image features. In some embodiments, step 220 can be implemented by the average feature calculation module 820.

[0061] In some embodiments, the first image feature includes a set of visual features corresponding to each sample image. Therefore, the feature vector dimension of the first image feature is equal to the image feature dimension of each sample image multiplied by the number of sample images. The average image feature, when calculated, can be obtained by summing the vector elements of each corresponding dimension in the first image feature and then dividing by the number of sample images. Therefore, the average image feature reduces the feature dimension compared to the first image feature. For example, for four sample images, if a 1*256 dimensional image feature is extracted from each sample image, the first image feature can be a 4*256 dimensional image feature, and the average image feature can be obtained by dividing by the number of features (4) to get a 1*256 dimensional image feature. In the subsequent calculation of the attention mechanism, by mapping the average image feature to the query vector space, the dimension of the query vector can be reduced. Furthermore, when calculating attention weights based on the query vector and key vector, and calculating the second image feature based on the attention weights and value vector, the overall computational cost of the attention mechanism can be reduced, thus achieving lightweight feature extraction and feature fusion.

[0062] Step 230: Based on an attention mechanism, feature fusion is performed on the average image features and the first image features to obtain the second image features. In some embodiments, step 230 can be implemented by the feature fusion module 830.

[0063] In some embodiments, the attention mechanism can refer to a computational module that simulates selective information focusing in biological cognition, enabling the enhancement of information of interest and the suppression of information of non-interest. In some embodiments, the attention mechanism may include at least one of cross-attention mechanism and self-attention mechanism.

[0064] In some embodiments, feature fusion can refer to the process of integrating different features into a unified, enhanced new feature representation through specific computational rules.

[0065] In some embodiments, the second image feature is the image feature obtained by feature fusion based on an attention mechanism. The second image feature obtained by fusing the average image feature and the first image feature based on the attention mechanism can enhance the image features in the sample image that are associated with the target category and suppress the image features that are weakly associated with the target category. Therefore, in cases where some sample images are improperly selected, resulting in weak association with the target category or incorrect labeling of the sample images, the impact of these abnormal sample images on the representation of the target category can be reduced, thereby improving the robustness of the image feature extraction process for representing the target category.

[0066] Step 240: Determine target image features based on the second image features, whereby the target image features are used to characterize the target category. In some embodiments, step 240 may be implemented by the target feature determination module 840.

[0067] In some embodiments, target image features refer to the image features ultimately used to characterize the target category. Since the second image features enhance image features in the sample image that are associated with the target category and suppress image features in the sample image that are unrelated to the target category based on an attention mechanism, the target image features determined based on the second image features can more accurately characterize the target category.

[0068] In some embodiments, by providing multiple image sets associated with different target categories (each image set corresponding to one target category), the first image features of each sample image in each image set can be obtained separately. For example, a total image set containing all target categories can be provided. ,in, Let C represent a set of images containing all target categories and all sample images. C can represent the set of target categories, c can represent a target category within the set of target categories C, N can represent the number of sample images for target category c, and i can represent the index of a sample image, i = 1, 2, ..., N, used to index the i-th sample image in target category c. This can be represented as the i-th sample image belonging to target category c. It can represent the set of N sample images corresponding to target category c. This can be represented as the union of the sets of sample images for all target categories. The first image feature obtained based on the sample image associated with target category c is denoted as... ,in, It can represent the feature set of all sample images associated with target category c, and N can represent the number of sample images of target category c. The subscripts 1, 2, ..., N are used as indices for the sample images in target category c, representing the 1st sample image, the 2nd sample image, ..., the Nth sample image. The image features obtained from the first sample image after being encoded by the image encoder, the second sample image after being encoded by the image encoder, ..., the Nth sample image after being encoded by the image encoder, and the set of these image features constitutes the first image feature. Based on each sample image corresponding to each target category, the image feature extraction methods in steps 210 to 240 are executed respectively to determine each target image feature used to characterize different target categories (each target image feature corresponds to one target category). In some embodiments, for multiple image sets, the corresponding target image features can be extracted sequentially for the sample images in each image set according to the image feature extraction methods provided in steps 210 to 240, or the corresponding target image features can be extracted in parallel for the sample images in each image set.

[0069] Figure 3 This is an exemplary flowchart illustrating image feature fusion according to some embodiments of this specification. Figure 3 The process 300 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown executes this. In some embodiments, process 300 may be a further description of step 230. In some embodiments, process 300 may be implemented by a feature fusion module 830 in an image feature extraction apparatus 800 deployed on a processing device. Figure 3 As shown, in some embodiments, process 300 may include the following steps.

[0070] Step 310: Map the average image features to the query vector space.

[0071] In some embodiments, mapping can refer to the transformation process of converting features from one feature representation space to another feature representation space.

[0072] In some embodiments, the query vector space, key vector space, and value vector space together constitute the computational framework of the attention mechanism. A query vector can refer to an instance or member in the query vector space, a key vector can refer to an instance or member in the key vector space, and a value vector can refer to an instance or member in the value vector space. In some embodiments, the data in the query vector space is mapped from average image features. The query vector space is used to query the correlation between the features (key vectors) of all sample images and the common features (query vectors) of the samples.

[0073] Step 320: Map the first image features to the key vector space and the value vector space respectively.

[0074] In some embodiments, the data in the key vector space is derived from the first image feature mapping, representing the features of each sample image, and is used to calculate the similarity with the query vector. The size of the dot product between the key vector and the query vector determines the similarity between the image features of the corresponding sample image and the average image features corresponding to the query vector.

[0075] In some embodiments, the data in the value vector space is derived from the first image feature mapping, preserving complete semantic information of each sample image that can be fused.

[0076] Step 330: Based on the attention mechanism, the average image features mapped to the query vector space and the first image features mapped to the key vector space and the value vector space are fused to obtain the second image features.

[0077] In some embodiments, the feature fusion process based on the attention mechanism may include applying the attention weights calculated based on the query vector and the key vector to the value vector for weighted summation.

[0078] Figure 4 This is an exemplary flowchart illustrating an image feature fusion calculation according to some embodiments of this specification. Figure 4 The process 400 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown executes this. In some embodiments, process 400 may be a further description of step 330. In some embodiments, process 400 may be implemented by a feature fusion module 830 in an image feature extraction apparatus 800 deployed on a processing device. Figure 4 As shown, in some embodiments, process 400 may include the following steps.

[0079] Step 410: Obtain the query matrix corresponding to the average image features.

[0080] In some embodiments, the query matrix can refer to the data corresponding to the query vector, which is derived from the average image features. In some embodiments, assuming that the dimension of the image features obtained by the image encoder based on each sample image is d (e.g., d=256), then the dimension of the average image features is also d, and the query matrix can be a 1*d dimension matrix. In some embodiments, the query matrix can serve as a query vector representing the commonalities of the target category.

[0081] For example, the first image feature obtained based on each sample image in target category c is denoted as... Then the average image features can be expressed as ,in, It can represent average image features; Let N represent the image features of the i-th sample image in target category c; N represents the number of sample images in target category c; the formula for calculating the average image features represents the arithmetic mean of the first image features of each sample image in target category c. For example, assuming the dimension of the average image features is d, the query vector (query matrix) can be obtained using the following formula: ,in, , It can represent a query vector (query matrix). It represents the set of real numbers (the value of each element in the vector is a real number). This indicates that the query vector is a 1*d dimensional vector, where each element is a real number. Representation layer normalization is used to average image features. Each feature dimension is normalized to have a mean of 0 and a variance of 1, in order to stabilize the training process and accelerate convergence. This represents the fully connected network corresponding to the query vector space, used to transform the normalized average image features to the query vector space.

[0082] Step 420: Obtain the key matrix corresponding to the first image features.

[0083] In some embodiments, the key matrix can refer to the data corresponding to the key vectors, and the data originates from the first image features. In some embodiments, assuming the number of sample images in the target category is N, and the dimension of the image features obtained by the image encoder based on each sample image is d, then the key matrix can be an N*d dimension matrix. In some embodiments, the key matrix can serve as a set of vectors for similarity comparison with the query matrix.

[0084] For example, assuming the dimension of the first image feature obtained based on each sample image in target category c is N*d, the key vector (key matrix) can be obtained by the following formula: ,in, , It can represent a key vector (key matrix). It represents the set of real numbers (the value of each element in the vector is a real number). This indicates that the key vector is an N*d dimensional vector, and the value of each element in the vector is a real number; Representation layer normalization is used to normalize the first image features. Normalization is performed on each feature dimension (i.e., each column); This represents the fully connected network corresponding to the key vector space, used to transform the normalized first image features to the key vector space.

[0085] Step 430: Obtain the value matrix corresponding to the first image feature.

[0086] In some embodiments, the value matrix can refer to the data corresponding to the value vector, and the data originates from the first image features. In some embodiments, assuming the number of sample images in the target category is N, and the dimension of the image features obtained by the image encoder based on each sample image is d, then the value matrix can be an N*d dimension matrix. In some embodiments, the value matrix can serve as a set of vectors carrying the original feature semantic information of the sample images for weighted fusion.

[0087] For example, assuming the dimension of the first image feature obtained based on each sample image in target category c is N*d, the value vector (value matrix) can be obtained using the following formula: ,in, , It can represent value vectors (value matrices). It represents the set of real numbers (the value of each element in the vector is a real number). This indicates that the value vector is an N*d dimensional vector, where each element has a real number value. Representation layer normalization is used to normalize the first image features. Normalization is performed on each feature dimension (i.e., each column); This represents a fully connected network corresponding to the value vector space, used to transform the normalized first image features to the value vector space.

[0088] Step 440: Based on the query matrix and the key matrix, obtain the similarity between the average image features and the first image features to generate attention weights for the average image features and the first image features.

[0089] In some embodiments, attention weights can be calculated using the following formula: ,in, , Indicates attention weights, It represents the set of real numbers (the value of each element in the vector is a real number). This indicates that the attention weights are a 1*N dimensional vector, where each element is a value between 0 and 1, and the sum of all elements is 1. This represents the transpose of the key vector, which interchanges the row and column elements of the key vector (key matrix) to obtain a d*N dimensional vector (matrix) that can be used with the query vector (query matrix). Perform matrix multiplication; This can refer to calculating the dot product of the query vector and each key vector, representing the original relevance score between the query and each sample image; d represents the feature dimension, and the value of d is usually large, such as 256. This is used to scale the dot product result. Since the dot product value increases with increasing dimension d, it is scaled by dividing by... This prevents the dot product result from being too large and affecting subsequent calculations; softmax represents normalization, used to convert the original relevance score into a probability distribution. In the calculation of attention weights, the weaker the relevance, the higher the probability that the sample image belongs to an anomaly; the stronger the relevance, the more accurately the sample image can represent the semantic information of the target category c.

[0090] Step 450: Determine the second image features based on the attention weights and value matrix.

[0091] In some embodiments, the second image feature is the output feature of the attention mechanism module. In some embodiments, the second image feature can be calculated using the following formula: ,in, , This represents the second image feature of the output. It represents the set of real numbers (the value of each element in the vector is a real number). This indicates that the second image feature is a 1*d dimensional vector, where each element has a real number value; This represents the matrix multiplication between the vector corresponding to the attention weight and the value vector; the specific calculation is a weighted summation.

[0092] In some embodiments, the image feature extraction method may further include: performing a residual concatenation between the average image features and the second image features. In some embodiments, step 240 may further include: using the sum of the residual concatenations of the average image features and the second image features as the target image features.

[0093] In some embodiments, residual connection is a network structure design in deep neural networks, which refers to the operation of adding the input of a layer or module, bypassing intermediate transformations, to the output processed by that layer or module. In some embodiments, the module corresponding to residual connection can refer to an attention mechanism module that performs feature fusion based on an attention mechanism. The operation of adding the average image features as input to the attention mechanism module to the second image features output by the attention mechanism module is a residual connection. For example, the target image features obtained by residual connection can be calculated using the following formula: ,in, This indicates that the average image features are residually concatenated (added) with the second image features output by the attention mechanism module; This represents the vector corresponding to the generated target image features, due to the average image features. Second image features They are all 1*d dimension vectors, therefore the generated target image features It is also a vector of 1*d dimensions; This represents the normalization function, usually referring to L2 normalization, which normalizes the vector after residual concatenation, converting it into a unit vector (with a magnitude of 1), so that the target image features of all target categories can be normalized to the same scale.

[0094] In some embodiments, the sum of the residual connections between the average image features and the second image features is used as the target image feature, which is equivalent to adding the average image features and the second image features together. Since the average image features calculate the average of the first image features for each sample image, although they may be affected by some abnormal sample images, they can completely encompass the basic image features corresponding to the sample images. Therefore, performing a residual connection between the average image features and the second image features ensures that the image features lost by the second image features compared to the first image features during the attention mechanism transformation can be provided by the average image features, improving the stability and completeness of image feature extraction. The residual connection between the average image features and the second image features can achieve the effect of combining the original features and enhanced features for the sample images. This not only preserves the stability of the original features but also incorporates the attention mechanism to improve the discriminative power regarding the target category, thereby obtaining a more optimized target category representation.

[0095] Figure 5 This is a schematic diagram illustrating an image feature extraction method according to some embodiments of this specification. For example... Figure 5 As shown, in the image feature extraction framework, the provided sample images can include image sets corresponding to N target categories. For each sample image in each category's image set, the first image feature V can be obtained. 1 V 2 ... V N For example, a row of squares in the diagram represents the image features obtained for a single sample image. Assuming each target category's image set contains four sample images, the first image features will contain four image features corresponding to each sample image. For any target category c, the first image features... Calculate the average image features of the first image features. And it is processed through the fully connected network corresponding to the query vector space. Mapping the first image features into the query vector space Q Through the fully connected network corresponding to the key vector space Mapped into the key vector space K, and through the corresponding fully connected network in the value vector space. Mapped into the value vector space V, attention weights are calculated based on the attention mechanism. Based on attention weights and the first image features The value vectors (value matrices) obtained by mapping to the value vector space V can be used to calculate the output of the attention mechanism module, i.e., the second image features. Average image features Second image features After performing residual connections, the target image features p corresponding to each target category can be obtained separately. 1 p 2 ... p N .

[0096] In some embodiments, the image feature extraction scheme may further include: using target image features as visual cue features for open-set target detection, so as to identify image elements in the image to be detected that correspond to the target category based on the target image features.

[0097] In some embodiments, visual cue features can refer to high-dimensional semantic features used to define target categories in open-set object detection. In some embodiments, since target image features extract the visual commonalities from each sample image to characterize the target category, using target image features as visual cue features allows open-set object detection to learn the image features possessed by the target category and identify image elements in the image to be detected that correspond to the target category.

[0098] In some embodiments, when multiple image sets exist, each corresponding to a target category, the corresponding target image features can be determined based on the sample images in each image set. During the open-set target detection stage, multiple target image features can be used as visual cue features to identify image elements in the image to be detected that correspond to each target category based on each target image feature. In some embodiments, when outputting image recognition results for open-set target detection, image elements corresponding to each target category can be individually labeled on the image to be detected, and multiple labeled images corresponding to each target category can be output separately. Alternatively, image elements corresponding to each target category can be labeled together on the image to be detected, outputting a mixed labeled image corresponding to each target category. In some embodiments, the same or different labeling methods can be used for different categories in the mixed labeled image. In some embodiments, if an image set containing sample images of the same target category is used as a visual cue and the name of the corresponding target category is used as a text cue, then when outputting image recognition results for open-set target detection, not only can image elements associated with the target category be labeled in the image to be detected, but the name of the target category can also be labeled in the image to be detected or the output image file can be named using the name of the target category.

[0099] In some embodiments, the third image features obtained based on the image to be detected have the same feature vector dimension as the target image features.

[0100] In some embodiments, the third image feature may refer to the feature representation of the image to be detected or the feature representation of a local region in the image to be detected. In some embodiments, there may be multiple candidate regions in the image to be detected. For each candidate region, a corresponding third image feature is obtained. By comparing the similarity between each third image feature and the target image feature, the candidate region corresponding to the third image feature with the highest similarity can be determined, and the image elements within the candidate region are determined as the corresponding target category. In some embodiments, for the determined candidate region, the corresponding image elements can be labeled in the image to be detected using at least one of the aforementioned labeling methods: mask, bounding box, and coordinate points, and then output.

[0101] In some embodiments, in open-set object detection, by comparing the similarity between a third image feature obtained from the image to be detected and a target image feature used as a visual cue feature, it can be determined whether the candidate region corresponding to the third image feature includes image elements of the target category. In some embodiments, the third image feature and the target image feature have the same feature vector dimension, which facilitates the calculation of similarity during feature comparison. In some embodiments, when the feature vector dimensions of the third image feature and the target image feature are inconsistent, either image feature can be multiplied by an increasing-dimensional matrix or a decreasing-dimensional matrix to obtain a feature vector dimension with the same as the other image feature. In some embodiments, to reduce computational load, a decreasing-dimensional matrix can be used to reduce the feature vector dimension, achieving lightweight image detection. In some embodiments, when the feature vector dimensions of two features that need to be similarity calculated are inconsistent, they can be mapped to the same dimensional space through a linear transformation. The increasing-dimensional matrix or the decreasing-dimensional matrix are linear transformation parameters that can achieve this dimensional alignment. In some embodiments, when the source feature dimension is lower than the target feature dimension, the increasing-dimensional matrix can map the source feature to a higher-dimensional space through matrix multiplication, making the mapped feature dimension consistent with the target feature dimension. The mapping process of the increasing-dimensional matrix can introduce additional semantic expressions while preserving the core semantics of the source feature. In some embodiments, the lower-dimensional features among the third image features and the target image features can be used as source features in the dimension-upgrading matrix mapping process, and the higher-dimensional features among the third image features and the target image features can be used as target features in the dimension-upgrading matrix mapping process. In some embodiments, when the dimension of the source features is higher than the dimension of the target features, the dimension-reduction matrix can map the source features to a lower-dimensional space through matrix multiplication, so that the dimension of the mapped features is consistent with the dimension of the target features. The dimension-reduction matrix mapping process can reduce computational complexity while eliminating redundant information and retaining the most discriminative feature components in the source features. In some embodiments, the higher-dimensional features among the third image features and the target image features can be used as source features in the dimension-reduction matrix mapping process, and the lower-dimensional features among the third image features and the target image features can be used as target features in the dimension-reduction matrix mapping process.

[0102] This specification also provides an open set target detection method. Figure 6 This is an exemplary flowchart of an open set target detection method according to some embodiments of this specification. Figure 6 The process 600 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown executes the process. In some embodiments, process 600 may be implemented by an open-set target detection device 900 deployed on a processing device. The open-set target detection device 900 may include an image acquisition module 910, a sample feature fusion module 920, and an image recognition module 930. In some embodiments, such as Figure 6 As shown, process 600 may include the following steps.

[0103] Step 610: Acquire each sample image and the image to be detected, wherein each sample image contains image elements associated with the same target category. In some embodiments, step 610 may be implemented by the image acquisition module 910.

[0104] In some embodiments, the sample images can be used as visual cues for open-set object detection and input along with the image to be detected. In some embodiments, the sample images and the image to be detected can be processed in parallel by different network layers or modules in open-set object detection.

[0105] Step 620: Obtain first image features corresponding to each sample image, and calculate average image features based on each first image feature; perform feature fusion on the average image features and the first image features based on an attention mechanism to obtain second image features, and determine target image features based on the second image features, which are used to characterize the target category. In some embodiments, step 620 can be implemented by the sample feature fusion module 920.

[0106] In some embodiments, step 620 corresponds to the aforementioned image feature extraction method. For a detailed description of step 620, please refer to the foregoing... Figures 2 to 5 The relevant content will not be repeated here.

[0107] Step 630: Identify image elements in the image to be detected that correspond to the target category based on the target image features. In some embodiments, step 630 may be implemented by the image recognition module 930.

[0108] In some embodiments, since the target image features extract the visual commonalities in each sample image to characterize the target category, the open set target detection can learn the image features possessed by the target category in order to identify the image elements in the image to be detected that correspond to the target category.

[0109] In some embodiments, when multiple image sets exist, each corresponding to a target category, the corresponding target image features can be determined based on the sample images in each image set. During open-set target detection, image elements corresponding to each target category in the image to be detected can be identified based on each target image feature. In some embodiments, when outputting image recognition results for open-set target detection, image elements corresponding to each target category can be individually labeled on the image to be detected, and multiple labeled images corresponding to each target category can be output separately. Alternatively, image elements corresponding to each target category can be labeled together on the image to be detected, and a mixed labeled image corresponding to each target category can be output. In some embodiments, the same or different labeling methods can be used for different categories in the mixed labeled image. In some embodiments, if an image set containing sample images of the same target category is used as a visual cue and the name of the corresponding target category is used as a text cue, then when outputting image recognition results for open-set target detection, not only can image elements associated with the target category be labeled in the image to be detected, but the name of the target category can also be labeled in the image to be detected or the output image file can be named with the name of the target category.

[0110] Figure 7 This is an exemplary flowchart illustrating image element recognition according to some embodiments of this specification. Figure 7 The process 700 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown executes this. In some embodiments, process 300 may be a further description of step 630. In some embodiments, process 300 may be implemented by the image recognition module 930 in the open-set target detection device 900 deployed on the processing device. Figure 7 As shown, in some embodiments, process 700 may include the following steps.

[0111] Step 710: Obtain third image features based on each candidate region of the image to be detected.

[0112] In some embodiments, a candidate region may refer to a potential sub-region of the image in the image to be detected that may contain image elements of the target category to be detected. In some embodiments, a candidate region may be represented by a bounding box (e.g., a rectangle). In some embodiments, a third image feature may refer to a local region feature representation in the image to be detected that corresponds to the candidate region.

[0113] In some embodiments, a region proposal network (RPN) can be used to perform a preliminary scan of the image to be detected, generating a series of a small number of high-quality candidate regions.

[0114] In some embodiments, a dense anchor point mechanism can be used to predefine a set of reference anchor boxes, each with a specific aspect ratio and size, to match the shapes of common objects. Based on the predefined reference anchor boxes, the image to be detected is input into a visual network to obtain multi-layer feature maps. At each spatial location in each layer of the feature map, all reference anchor boxes are associated, ultimately generating a set of candidate regions at multiple scales covering all spatial locations in the entire image. These candidate regions can be fine-tuned according to the model's prediction process.

[0115] Step 720: Calculate the similarity between each third image feature and the target image feature, and determine the image elements in the candidate region whose similarity reaches the preset condition as the corresponding target category.

[0116] In some embodiments, the third image feature may refer to the local region feature representation corresponding to each candidate region. In some embodiments, by comparing the similarity between the third image feature and the target image feature, it can be determined whether the candidate region corresponding to the third image feature includes image elements associated with the target category. In some embodiments, the third image feature and the target image feature may have the same feature vector dimension to facilitate the calculation and comparison of similarity.

[0117] In some embodiments, the preset criteria may include the highest similarity value among the similarities to the target image features calculated based on each candidate region. The candidate region with the highest similarity can be determined as the one containing image elements most similar to the target image features, and therefore most likely associated with the target category.

[0118] In some embodiments, the preset conditions may include a similarity value reaching a preset threshold. In some embodiments, if multiple candidate regions have similarities reaching the preset threshold, multiple possible image elements related to the target category can be simultaneously marked in the image to be detected for user confirmation. In some embodiments, candidate regions corresponding to similarities reaching the preset threshold and arranged from high to low similarity can be marked in the image to be detected by varying the color of the bounding box from dark to light or the lines of the bounding box from solid to dashed. That is, candidate regions with high similarity are marked with dark colors or solid lines, and candidate regions with low similarity are marked with light colors or dashed lines.

[0119] In some embodiments, the target category may refer to the commonality of image elements in each sample image. Image elements in candidate regions where the similarity reaches a preset condition are determined as the corresponding target category, indicating that image elements in candidate regions where the similarity reaches a preset condition have the commonality of image elements in each sample image.

[0120] This specification also provides an image feature extraction device. Figure 8This is an exemplary block diagram of an image feature extraction apparatus according to some embodiments of this specification. In some embodiments, the image feature extraction apparatus 800 may be deployed on a server 120. Figure 8 As shown, in some embodiments, the image feature extraction device 800 may include a sample feature acquisition module 810, an average feature calculation module 820, a feature fusion module 830, and a target feature determination module 840.

[0121] In some embodiments, the sample feature acquisition module 810 can be used to acquire first image features corresponding to each sample image, wherein each sample image contains image elements associated with the same target category.

[0122] In some embodiments, image elements in the sample images that are associated with the same target category are labeled using at least one of the following: mask, bounding box, coordinate points, and hand-drawn illustrations. In some embodiments, the sample feature acquisition module 810 can also be used to acquire the first image features corresponding to each sample image based on the image elements labeled in each sample image and the sample images.

[0123] In some embodiments, the average feature calculation module 820 can be used to calculate average image features based on each first image feature.

[0124] In some embodiments, the feature fusion module 830 can be used to perform feature fusion on the average image features and the first image features based on an attention mechanism to obtain the second image features.

[0125] In some embodiments, the feature fusion module 830 can also be used to map the average image features to the query vector space; map the first image features to the key vector space and the value vector space respectively; and perform feature fusion on the average image features mapped to the query vector space and the first image features mapped to the key vector space and the value vector space based on the attention mechanism to obtain the second image features.

[0126] In some embodiments, the feature fusion module 830 may also be used to: obtain a query matrix corresponding to the average image features; obtain a key matrix corresponding to the first image features; obtain a value matrix corresponding to the first image features; obtain the similarity between the average image features and the first image features based on the query matrix and the key matrix, so as to generate attention weights between the average image features and the first image features; and determine the second image features based on the attention weights and the value matrix.

[0127] In some embodiments, the target feature determination module 840 can be used to determine target image features based on second image features, the target image features being used to characterize the target category.

[0128] In some embodiments, the target feature determination module 840 may also be used to take the sum of the residuals of the average image features and the second image features as the target image features.

[0129] This specification also provides an open set target detection device. Figure 9 This is an exemplary block diagram of an open-set target detection apparatus according to some embodiments of this specification. In some embodiments, the open-set target detection apparatus 900 may be deployed on server 120. Figure 9 As shown, in some embodiments, the open set target detection device 900 may include an image acquisition module 910, a sample feature fusion module 920, and an image recognition module 930.

[0130] In some embodiments, the image acquisition module 910 can be used to acquire each sample image and the image to be detected, wherein each sample image contains image elements associated with the same target category.

[0131] In some embodiments, the sample feature fusion module 920 can be used to obtain first image features corresponding to each sample image, and calculate average image features based on each first image feature; based on an attention mechanism, perform feature fusion on the average image features and the first image features to obtain second image features, and determine target image features based on the second image features, wherein the target image features are used to characterize the target category.

[0132] In some embodiments, the image recognition module 930 can be used to identify image elements in the image to be detected that correspond to the target category based on the target image features.

[0133] In some embodiments, the image recognition module 930 can also be used to obtain third image features based on each candidate region of the image to be detected; calculate the similarity between each third image feature and the target image feature; and determine the image element in the candidate region with the highest similarity as the corresponding target category.

[0134] For more information on each module, please refer to [link / reference]. Figures 2 to 7 The relevant explanations will not be repeated here. It should be understood that... Figure 8 and Figure 9The apparatus and modules shown can be implemented in various ways. For example, in some embodiments, the system and modules can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as on a media such as a disk, CD, or DVD-ROM, or in the memory of a programmable device. The systems and modules of this specification can be implemented not only with hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips or transistors, or programmable hardware devices such as field-programmable gate arrays or programmable logic devices, but also with software, for example, executed by various types of processors, or with a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0135] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules without departing from these principles to form subsystems connected to other modules. Alternatively, some modules may be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0136] Some embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement this specification. Figures 2 to 7 The image feature extraction method and / or open set target detection method shown are illustrated.

[0137] Some embodiments of this specification also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, can implement this specification. Figures 2 to 7 The image feature extraction method and / or open set target detection method shown are illustrated.

[0138] Some embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement this specification. Figures 2 to 7The image feature extraction method and / or open set target detection method are illustrated. In some embodiments, the computer program product may refer only to a computer program, which may be carried on a storage medium or processing device. In other embodiments, the computer program product may also be a storage medium or processing device containing the aforementioned computer program. The processing device may include one or more processors, and the storage medium.

[0139] In some embodiments, the processor may be a combination of one or more of the following processors: central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), graphics processing unit (GPU), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic device (PLD), programmable logic controller (PLC), reduced instruction set computer (RISC), and microprocessor.

[0140] In some embodiments, the storage medium may include one or more combinations of the following: mass storage, removable storage, volatile read-write memory, and read-only memory (ROM). Exemplary mass storage may include disks, optical disks, solid-state drives, etc. Exemplary removable storage may include flash drives, floppy disks, optical disks, memory cards, compressed hard disks, magnetic tapes, etc. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAM may include dynamic random access memory (DRAM), dual data rate synchronous dynamic random access memory (DDRSDRAM), static random access memory (SRAM), silicon controlled retrieval memory (T-RAM), and zero-capacitance memory (Z-RAM), etc. Exemplary read-only memory may include masked read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compressed hard disk read-only memory (CD-ROM), and digital multifunction hard disk read-only memory, etc.

[0141] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. An image feature extraction method, characterized in that, The method includes: Obtain the first image feature corresponding to each sample image, wherein each sample image contains image elements associated with the same target category; Calculate the average image feature based on each of the first image features; Based on the attention mechanism, the average image features and the first image features are fused to obtain the second image features; The target image features are determined based on the second image features, and the target image features are used to characterize the target category.

2. The method according to claim 1, characterized in that, The method further includes: performing a residual connection between the average image features and the second image features; The step of determining the target image features based on the second image features includes: The sum of the residuals of the average image features and the second image features is taken as the target image features.

3. The method according to claim 1 or 2, characterized in that, The image elements in the sample image that are associated with the same target category are labeled using at least one of the following: mask, bounding box, coordinate points, and hand-drawn illustration. The step of obtaining the first image feature corresponding to each sample image includes: Based on each of the sample images and the image elements labeled in the sample images, the first image feature corresponding to each of the sample images is obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The target image features are used as visual cue features for open-set target detection to identify image elements in the image to be detected that correspond to the target category based on the target image features.

5. The method according to claim 4, characterized in that, The third image features obtained based on the image to be detected have the same feature vector dimension as the target image features.

6. The method according to any one of claims 1 to 5, characterized in that, The method of fusing the average image features and the first image features based on the attention mechanism to obtain the second image features includes: Map the average image features to the query vector space; The first image features are mapped to the key vector space and the value vector space, respectively. Based on the attention mechanism, the average image features mapped to the query vector space and the first image features mapped to the key vector space and the value vector space are fused to obtain the second image features.

7. The method according to claim 6, characterized in that, The step of fusing the average image features mapped to the query vector space and the first image features mapped to the key vector space and the value vector space based on the attention mechanism to obtain the second image features includes: Obtain the query matrix corresponding to the average image features; Obtain the key matrix corresponding to the first image features; Obtain the value matrix corresponding to the first image feature; Based on the query matrix and the key matrix, the similarity between the average image feature and the first image feature is obtained to generate attention weights for the average image feature and the first image feature; The second image feature is determined based on the attention weights and the value matrix.

8. An open set target detection method, characterized in that, The method includes: Acquire each sample image and the image to be detected, wherein each sample image contains image elements associated with the same target category; First image features corresponding to each of the sample images are obtained, and average image features are calculated based on each of the first image features; based on an attention mechanism, the average image features and the first image features are fused to obtain second image features, and target image features are determined based on the second image features, wherein the target image features are used to characterize the target category; Based on the target image features, image elements in the image to be detected that correspond to the target category are identified.

9. The method according to claim 8, characterized in that, The step of identifying image elements in the image to be detected that correspond to the target category based on the target image features includes: The third image features are obtained based on each candidate region of the image to be detected; Calculate the similarity between each of the third image features and the target image features, and determine the image elements in the candidate region whose similarity reaches a preset condition as the corresponding target category.

10. An image feature extraction device, characterized in that, The device includes: The sample feature acquisition module is used to acquire the first image features corresponding to each sample image, wherein each sample image contains image elements associated with the same target category; The average feature calculation module is used to calculate the average image feature based on each of the first image features; The feature fusion module is used to fuse the average image features and the first image features based on an attention mechanism to obtain the second image features; The target feature determination module is used to determine target image features based on the second image features, wherein the target image features are used to characterize the target category.

11. An open set target detection device, characterized in that, The device includes: An image acquisition module is used to acquire each sample image and the image to be detected, wherein each sample image contains image elements associated with the same target category; The sample feature fusion module is used to acquire first image features corresponding to each of the sample images, and calculate average image features based on each of the first image features; based on an attention mechanism, the average image features and the first image features are fused to obtain second image features, and target image features are determined based on the second image features, wherein the target image features are used to characterize the target category; An image recognition module is used to identify image elements in the image to be detected that correspond to the target category based on the features of the target image.

12. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it is able to implement the method as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, enable the implementation of the method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes a computer program that, when at least a portion of the computer program is executed by a processor, enables the implementation of the method as described in any one of claims 1 to 9.

Images