Target determination method and electronic equipment

Abstract

The invention provides a target determination method and an electronic device, which are used for being compatible with two tasks of target detection and instance segmentation at the same time, do not need to perform network training again when a new category is expanded, and are more convenient to use. The method comprises the following steps: acquiring a retrieval image and a reference image; the retrieval image and the reference image are input into a multi-task model, a target object in the retrieval image is determined, and the target object and the reference image are the same in image category; wherein the multi-task model is used for extracting features of the retrieval image and the reference image, performing feature matching on the features of the retrieval image and the reference image to obtain matched features, and determining a target object in the retrieval image according to the matched features.

Description

Target determination method and electronic equipment Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a target determination method and electronic device. Background Art

[0002] Single-shot detection (OSD) uses only a single image of the category to be detected to detect objects of that category in an input image. This image of the category to be detected is called an OSD marker. Unlike general object detection, OSD can theoretically detect any user-specified category. This method allows for differences in perspective and a certain degree of appearance between the OSD marker and the object in the input image. OSD is not suitable for detecting 3D objects such as faces, animals, and cars. It is more suitable for 3D objects with flat patterns, such as iconic buildings and product packaging.

[0003] In the existing technology, single-sample detection and instance segmentation are implemented independently. The designed single-sample detection model can only realize target detection and cannot handle instance segmentation tasks. A separate segmentation model is required to realize instance segmentation tasks. One model structure cannot be compatible with both target detection and instance segmentation tasks at the same time.

[0004] Summary of the Invention

[0005] The present disclosure provides a target determination method and electronic device, which are used to simultaneously handle both target detection and instance segmentation tasks. When expanding new categories, there is no need to retrain the network, making it more convenient to use.

[0006] In a first aspect, an embodiment of the present disclosure provides a target determination method, the method comprising:

[0007] Obtaining a retrieval image and a reference image;

[0008] Inputting the retrieval image and the reference image into a multi-task model, determining a target object in the retrieval image, wherein the target object and the reference image have the same image category;

[0009] The multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image based on the matching features.

[0010] As an optional implementation, the multi-task model includes a first encoder, a second encoder, and a decoder, wherein:

[0011] Inputting the search image into a first encoder, extracting and outputting features of the search image, and inputting the reference image into a second encoder, extracting and outputting features of the reference image;

[0012] The features of the retrieval image and the features of the reference image are input into the decoder, and the features of the retrieval image and the reference image are matched to obtain matching features. The position information of the target object in the retrieval image is determined based on the matching features.

[0013] As an optional implementation, the first encoder is a first backbone network, the second encoder is a second backbone network, and the decoder is a neural network based on a self-attention mechanism.

[0014] As an optional implementation, the multi-task model is used to output the location information of the target object in the retrieval image, and implement different tasks based on the location information of the target object. The different tasks correspond to different levels of granularity of the location information of the target object in the output retrieval image.

[0015] As an optional implementation, the multi-task model is used to implement a target detection task; the output target object location information includes a detection box of the target object, and the detection box is used to represent the location information of the target object in the retrieved image.

[0016] As an optional implementation, the location information of the target object is output in the following manner:

[0017] The matching features are input into the fully connected layer of the multi-task model, and the detection box of the target object is output on the retrieval image.

[0018] As an optional implementation, the multi-task model is used to implement an instance segmentation task; the output target object location information includes a mask image of the target object, and the mask image is used to represent the pixel-level location information of the target object in the retrieval image.

[0019] As an optional implementation, the location information of the target object is output in the following manner:

[0020] Multiply the matching features and the retrieval image after feature matching to output the mask image of the target object.

[0021] As an optional implementation, the multi-task model is trained in the following manner:

[0022] Acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0023] Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0024] The multi-task model to be trained is trained according to the loss function value, and when the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0025] As an optional implementation, determining the loss function value based on the output result, the image category annotated by the reference image sample, and the position of the target object annotated by the retrieval image sample includes:

[0026] Inputting the retrieval image sample and the reference image sample into the multi-task model to be trained, and extracting the first feature of the retrieval image sample and the second feature of the reference image sample respectively;

[0027] Classify the second feature to output the category of the reference image sample, perform feature matching on the first feature and the second feature to obtain a matching feature, and output the location information and confidence level of the target object based on the matching feature;

[0028] Determine a first loss value according to the category of the output reference image sample and the labeled image category;

[0029] Determining a second loss value according to the marked position of the target object, the output position information of the target object, and the confidence level;

[0030] A loss function value is determined based on the first loss value and the second loss value.

[0031] As an optional implementation, the multi-task model includes a first encoder, a second encoder and a decoder;

[0032] Inputting the retrieval image sample into a first encoder and outputting a first feature of the retrieval image sample, and inputting the reference image sample into a second encoder and outputting a second feature of the reference image sample;

[0033] The second feature is input into the fully connected layer to output the category of the reference image sample, and the first feature and the second feature are input into the decoder to output the location information and confidence of the target object.

[0034] As an optional implementation, after outputting the first feature of the retrieved image sample and before inputting the first feature into the decoder, the method further includes:

[0035] A position code is added to the first feature, where the position code is used to represent the position information of each pixel in the retrieved image sample.

[0036] As an optional implementation manner, before the second feature is input to the decoder, the method further includes:

[0037] Concatenate the token parameter and the second feature, input the concatenated second feature and the first feature into the decoder, and output the location information and confidence of the target object;

[0038] The token parameter is used to distinguish the location information and confidence of the output target object.

[0039] As an optional implementation manner, determining the second loss value based on the marked position of the target object, the output position information of the target object, and the confidence level includes:

[0040] Determine a first sub-loss value according to the marked position of the target object and the output position information of the target object;

[0041] Calculate the intersection-over-union ratio of the marked position and the output position information, determine the confidence of the output target object based on the intersection-over-union ratio, and determine the second sub-loss value based on the confidence of the output target object and the confidence of the marked target object;

[0042] A second loss value is determined according to the first sub-loss value and the second sub-loss value.

[0043] As an optional implementation manner, the reference image includes one reference image or multiple reference images of the same image category.

[0044] In a second aspect, the present disclosure also provides a multi-task model training method, the model comprising:

[0045] Acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0046] Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0047] The multi-task model to be trained is trained according to the loss function value, and when the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0048] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0049] Obtaining a retrieval image and a reference image;

[0050] Inputting the retrieval image and the reference image into a multi-task model, determining a target object in the retrieval image, wherein the target object and the reference image have the same image category;

[0051] The multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image based on the matching features.

[0052] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0053] Acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0054] Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0055] The multi-task model to be trained is trained according to the loss function value, and when the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0056] In a fifth aspect, an embodiment of the present disclosure further provides a target determination device, the device comprising:

[0057] An image acquisition unit, configured to acquire a search image and a reference image;

[0058] A position determination unit is used to input the retrieval image and the reference image into a multi-task model to determine a target object in the retrieval image, where the target object and the reference image have the same image category; wherein the multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image based on the matching features.

[0059] In a sixth aspect, an embodiment of the present disclosure further provides a multi-task model training device, the device comprising:

[0060] A training set acquisition unit, configured to acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0061] a loss function determination unit, configured to input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine a loss function value based on the output result, the image category annotated by the reference image sample, and the position of the target object annotated by the retrieval image sample;

[0062] The training model unit is used to train the multi-task model to be trained according to the loss function value, and when the loss function value meets the preset requirements or the number of training iterations reaches a threshold, the trained multi-task model is obtained.

[0063] In a seventh aspect, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, which, when executed by a processor, is used to implement the steps of the method described in any one of the first aspect or the second aspect above.

[0064] In an eighth aspect, the present disclosure provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method of any one of the first aspect or the second aspect.

[0065] These and other aspects of the present disclosure will become more readily apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] FIG1 is a flowchart of a specific implementation of a target determination method provided by an embodiment of the present disclosure;

[0068] FIG2 is a schematic diagram of the architecture of a multi-task model provided by an embodiment of the present disclosure;

[0069] FIG3 is a schematic diagram of the architecture of a multi-task model for outputting detection frames provided by an embodiment of the present disclosure;

[0070] FIG4 is a schematic diagram of the architecture of a multi-task model for outputting a mask image provided by an embodiment of the present disclosure;

[0071] FIG5 is a flowchart of an implementation of a multi-task model training method provided by an embodiment of the present disclosure;

[0072] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0073] FIG7 is a schematic structural diagram of another electronic device provided by an embodiment of the present disclosure;

[0074] FIG8 is a schematic structural diagram of a target determination device provided by an embodiment of the present disclosure;

[0075] FIG9 is a schematic structural diagram of a multi-task model training device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the present disclosure will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present disclosure, rather than all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0077] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0078] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Persons skilled in the art will appreciate that, as new application scenarios emerge, the technical solutions provided by the embodiments of the present disclosure will also be applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, "multiple" means two or more.

[0079] Before introducing the target determination method provided by the embodiment of the present disclosure, for ease of understanding, the technical background of the embodiment of the present disclosure is first introduced in detail below.

[0080] Single-shot detection (OSD) uses only a single image of the category to be detected to detect objects of that category in an input image. This image of the category to be detected is called an OSD marker. Unlike general object detection, OSD can theoretically detect any user-specified category. This method allows for differences in perspective and a certain degree of appearance between the OSD marker and the object in the input image. OSD is not suitable for detecting 3D objects such as faces, animals, and cars. It is more suitable for 3D objects with flat patterns, such as iconic buildings and product packaging.

[0081] Single-shot learning can be achieved through methods such as the nearest neighbor method and siamese networks, but these methods can only be used for recognition (classification) and cannot be used for detection and segmentation. Few-shot learning methods such as TFA (Two-stage fine-tuning approach) and FSCE (Few-Shot Object Detection via Contrastive Proposal Encoding) are achieved through fine-tuning. Their performance is strongly correlated with the number of samples and requires training when applied to new categories. Meta-learning methods such as Meta R-CNN (small-shot detection / segmentation) are even less effective than fine-tuning-based methods in actual applications due to defects in their model structure. They also require iterative training, and their ease of use and model performance are inferior to the methods of this patent.

[0082] In existing technologies, single-shot detection and instance segmentation are implemented independently. The designed single-shot detection model can only perform object detection and cannot handle instance segmentation. This requires a separate segmentation model, and a single model architecture cannot simultaneously handle both object detection and instance segmentation. Furthermore, current single-shot detection cannot achieve few-shot learning, requiring network retraining when expanding to new categories. Single-shot learning techniques used in face recognition cannot generate location information or instance segmentation results, making them difficult to extend beyond face recognition.

[0083] Based on this, the present disclosure provides a target determination method that uses a multi-task model to output the location information of a target object of the same image category as a reference image from a retrieval image. Since the multi-task model can implement different tasks, it is compatible with both target detection and instance segmentation tasks, where target detection is used to output the location of the target object in the retrieval image, and instance segmentation is used to output a mask image of the target object in the retrieval image. Since the multi-task model extracts features from the retrieval image and the reference image and performs feature matching between the features of the reference image and the features of the retrieval image, the multi-task model can learn to find features from the retrieval image that match the features of the reference image, thereby finding the location of the target object based on the matched features. Therefore, when expanding new categories, there is no need to retrain the multi-task model, making it more convenient to use. During training, multiple different data sets can be combined for training, and the labeled categories of different data sets do not need to be consistent. It is also compatible with small sample learning.

[0084] As shown in FIG1 , the specific implementation steps of a target determination method provided in this embodiment are as follows:

[0085] Step 100: Acquire a search image and a reference image;

[0086] Step 101: Input the retrieval image and the reference image into a multi-task model to determine a target object in the retrieval image, where the target object has the same image category as the reference image.

[0087] The multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image based on the matching features.

[0088] In some embodiments, the multi-task model is used to output the location information of the target object in the retrieval image, and different tasks are implemented according to the location information of the target object. The different tasks correspond to different granularity levels of the output location information of the target object in the retrieval image.

[0089] It should be noted that the multi-task model in this embodiment can implement different tasks. Based on the image category of the reference image, the location information of target objects of the same image category can be determined from the search image. For example, if the image category of the reference image is vehicle, the location information of the vehicle can be determined from the search image. Alternatively, if the image category of the reference image is animal, the location information of the animal can be determined from the search image. Furthermore, the multi-task model can be compatible with different tasks and used to determine location information at different granularity levels.

[0090] Optionally, the position information of different granularity levels in this embodiment includes but is not limited to image-level position information, for example, the multi-task model can output a detection box of the target object, which is used to indicate the position of the target object of the same category as the reference image in the retrieval image; it may also include pixel-level position information, for example, the multi-task model can output a mask image of the target object, which is used to indicate the pixel-level position of the target object of the same category as the reference image in the retrieval image.

[0091] In some embodiments, as shown in FIG2 , a schematic diagram of the architecture of a multi-task model is provided. The multi-task model in this embodiment includes a first encoder, a second encoder, and a decoder, wherein:

[0092] Inputting the search image into a first encoder, extracting and outputting features of the search image, and inputting the reference image into a second encoder, extracting and outputting features of the reference image;

[0093] The features of the search image and the reference image are input to the decoder, and the features of the search image and the reference image are matched to obtain matching features. The target object in the search image is determined based on the matching features. Optionally, the location information of the target object in the search image is output.

[0094] Optionally, the reference image input to the second encoder includes one reference image or multiple reference images of the same image category, so that the multi-task model is compatible with single-sample learning and small-sample learning.

[0095] During implementation, the first encoder and the second encoder are used to perform feature extraction, and the decoder is used to perform feature fusion on the features of the extracted retrieval image, perform feature fusion on the features of the extracted reference image, and perform feature matching on the features of the fused retrieval image and the features of the reference image, thereby outputting the position information of the target object in the retrieval image.

[0096] In some embodiments, the first encoder is a first backbone network, the second encoder is a second backbone network, and the decoder is a neural network based on a self-attention mechanism.

[0097] Optionally, the first backbone network includes, but is not limited to, SwinTransformV2, ResNet, Vit, etc. The second backbone network includes, but is not limited to, ResNet, VGG, etc. It should be noted that the first backbone network and the second backbone network can be the same or different, and the second backbone network can use a lighter network than the first backbone network. The decoder includes, but is not limited to, transform, such as a two-layer transform structure network. This embodiment does not impose any further restrictions on this.

[0098] In implementation, the first backbone network is used to extract and output the features of the retrieval image in the following manner:

[0099] The retrieval image is input into the first backbone network for processing. The output features of the 1st to 4th layers of the backbone are sent to FPN (Feature Pyramid Networks) for processing to generate a new 4-layer feature. The feature of the retrieval image is recorded as F i , the width and height of these four layers of features are W i , H i , the number of channels is C f (The number of channels in each layer is the same, typically 256), the feature tensor is [B, C f , H i , W i ], where i∈[1, 2, 3, 4]. Among them, FPN can perform feature fusion from shallow to deep and from deep to shallow, and enhance feature extraction.

[0100] Because images contain objects of varying sizes, each with its own unique characteristics, shallow features can be used to distinguish simple objects, while deep features can be used to distinguish complex objects. FPN uses different levels of pyramidal layers (feature maps) for prediction, extracting features of varying sizes from different layers of the network for multi-scale fusion, and then predicting the fused features. FPN upsamples the lowest-level features and fuses them with the underlying features to produce high-resolution, semantically robust features, facilitating the detection of small objects and enhancing feature extraction.

[0101] In implementation, the second backbone network is used to extract and output the features of the reference image in the following manner:

[0102] The reference image is input into the second backbone network for processing, and the two-dimensional feature output of the last layer is obtained. It should be noted that the reference image can be of the same category as the reference image sample in the training dataset, or it can be a different category. The reference image can be a single image or multiple images of the same category. In this way, the multi-task model is compatible with single-sample learning and small-sample learning.

[0103] The two-dimensional features of the reference image output are averaged in the spatial dimension and batch size dimension (used to smooth the features of multiple inputs) and then L2 normalized to output the features of the reference image. The feature tensor of the reference image is [1, C f , 1].

[0104] In some embodiments, the multi-task model in this embodiment is used to implement the target detection task; the output target object location information includes a detection box of the target object, and the detection box is used to represent the location information of the target object in the retrieved image.

[0105] Optionally, when the target object location information output by the multi-task model includes a detection frame of the target object, the target object location information is output in the following manner:

[0106] The matching features are input into the fully connected layer of the multi-task model, and the detection box of the target object is output on the retrieval image.

[0107] In implementation, when outputting the detection frame of the target object on the retrieval image, the confidence of the detection frame is also output. The confidence of the detection frame is used to indicate the probability of whether the target object exists in the detection frame.

[0108] As shown in Figure 3, this embodiment provides a schematic diagram of the architecture of a multi-task model that outputs detection boxes. In the figure, a search image is input to a first encoder, and a reference image is input to a second encoder. The output of the first encoder and the output of the second encoder are input to a decoder, which outputs a detection box for the target object in the search image. The output is the detection box marked in the search image.

[0109] In some embodiments, the multi-task model of this embodiment is used to implement instance segmentation tasks; the output location information of the target object includes a mask image of the target object, and the mask image is used to represent the pixel-level location information of the target object in the retrieval image.

[0110] Optionally, when the location information of the target object output by the multi-task model includes a mask image of the target object, the location information of the target object is output in the following manner:

[0111] Multiply the matching features and the retrieval image after feature matching to output the mask image of the target object.

[0112] In implementation, while outputting the mask image of the target object, the confidence of the mask image is also output. The confidence of the mask image is used to indicate the probability of whether the target object exists in the mask image.

[0113] As shown in Figure 4, this embodiment provides an architectural diagram of a multi-task model that outputs a mask image. In the figure, the retrieval image is input into the first encoder, the reference image is input into the second encoder, the output result of the first encoder and the output result of the second encoder are input into the decoder together, and the mask image of the target object in the retrieval image is output.

[0114] In some embodiments, the multi-task model is trained by the following steps:

[0115] Step a: obtaining a training data set, wherein the training data set includes a plurality of search image samples and a plurality of reference image samples;

[0116] Randomly sample an image category from the gtlabels of the retrieval image sample Q, and then randomly crop one or more sub-images of the same category from the reference image sample in the training dataset based on the annotation information. The corresponding sub-image is the reference image sample R; where gtlabels represents the image category annotated in a retrieval image sample.

[0117] Step b: input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value according to the output result, the image category annotated by the reference image sample, and the position of the target object annotated by the retrieval image sample;

[0118] Optionally, determine the loss function value as follows:

[0119] Step b1: input the retrieval image sample and the reference image sample into the multi-task model to be trained, and extract the first feature of the retrieval image sample and the second feature of the reference image sample respectively;

[0120] In implementation, the search image sample is input into the first encoder to extract the first feature of the search image sample, and the reference image sample is input into the second encoder to extract the second feature of the reference image sample. In implementation, the first encoder is the first backbone network, and the second encoder is the second backbone network.

[0121] The retrieval image sample Q is input into the first backbone network for processing, and the output features of the 1st to 4th layers of the backbone are sent to FPN for processing to generate a new 4-layer feature, which is recorded as F i , the width and height of these four layers of features are W i , H i , the number of channels is C f (The number of channels in each layer is the same, typically 256), the feature tensor is [B, C f , H i , W i ], where i∈[1, 2, 3, 4]. Where B represents the number of images processed simultaneously in a batch. W i Indicates the output feature width of the backbone layer i, whose value is W / 2 i , H i Indicates the feature height of the backbone layer i output, its value is H / 2 i . C f Indicates the number of channels of the input image.

[0122] The reference image sample R is input into the second backbone network for processing, and the two-dimensional feature output of the last layer is obtained. It should be noted that the reference image sample R can be one or multiple images of the same category. In this way, the multi-task model is compatible with single-sample learning and small-sample learning.

[0123] The two-dimensional features output by the reference image sample R are averaged in the spatial dimension and the batch size dimension (used to smooth the features of multiple inputs) and then L2 normalized. The features of the reference image sample are output, denoted as E, and its feature tensor is [B, C f ,1]. Among them, averaging is to unify the size, and normalization is to normalize the feature intensity to make the feature expression more stable.

[0124] Step b2: classify the second feature to output the category of the reference image sample, perform feature matching on the first feature and the second feature to obtain matching features, and output the location information and confidence level of the target object based on the matching features;

[0125] In the implementation, the 4-layer feature F of the image sample will be retrieved i Scale to [W e , H e ], and then accumulate and record it as the feature F of the retrieved image sample; the scaling is used to unify features of different sizes, and the accumulation is used to fuse the four layers of features.

[0126] The feature E of the reference image sample is processed by a weight-normalized fully connected layer to generate a feature tensor of [B, num_classes]; where num_classes represents the number of all categories in the reference image sample.

[0127] Add position coding to the feature F of the retrieved image sample. The position coding is used to represent the position information of each pixel in the image. Expand the feature F of the retrieved image sample in the spatial dimension, and its tensor is [B, C f , W e ×H e ], abbreviated as [B, C f , L]; Transpose F to [B, L, C f ]; among them, H e Represents the height of the decoder input, W e Indicates the width of the decoder input.

[0128] Generate a learnable token parameter T, the feature tensor of the token parameter T is [1, C f , 1+N o ], broadcast T to shape [B, C f , 1+N o ]; that is, repeat B times. Among them, N o Indicates the maximum number of samples of the same category in a retrieved image.

[0129] The token parameter T is concatenated with the feature E of the reference image sample on the last dimension, where the last dimension is the length, that is, the concatenation is performed on the length, and the generated feature tensor is [B, C f , 1+N o +1], transposed to the tensor [B, 1+N o +1, C f ], the generated features are still recorded as R; the token parameter T is understood as a placeholder, which is used to output two features at corresponding positions on the feature E of the reference image sample. The token parameter T is the same for each reference image sample.

[0130] A multi-layer transform structure (self-attention) is used to process the concatenated feature R of the reference image sample and token T, and the transposed feature F of the retrieval image sample. First, R is used as the query, F is used as the key and value, and then F is used as the query and R is used as the key and value. This is mainly used to perform feature matching on R and F. The feature matching step can be repeated once or multiple times as needed.

[0131] It's important to note that algorithms based on the self-attention mechanism are primarily based on matrix multiplication and the softmax function. The self-attention mechanism converts the input sequence into three matrices: query, key, and value. It then calculates the similarity or correlation between the query and each key, obtains a weight coefficient for each key-value pair, and then takes the weighted sum of these values ​​to produce the final Attention score.

[0132] From the second dimension of the output R, two different features are sequentially divided, namely Obj and Reg. The feature tensor of Obj is [B, 1, C f ], the feature tensor of Reg is [B, N o , C f ], these two features are exactly the data corresponding to the token, and the redundant data (such as the data corresponding to R) are ignored; wherein, the output R refers to the two features corresponding to the token parameter T of the reference image sample after feature matching, and the second dimension is the position of the placeholder T.

[0133] After a layer of full connection processing, Obj generates a feature tensor of [B, 1, N o ] feature, which is used to indicate the confidence of the output target object, that is, Obj is used to indicate whether there is a target object.

[0134] Step b3: determining a first loss value according to the category of the output reference image sample and the labeled image category;

[0135] In the implementation, the cross entropy is used as the first loss function to calculate the loss value according to the category of the output reference image sample and the labeled image category. The first loss value is recorded as L cls .

[0136] Step b4: determining a second loss value based on the marked position of the target object, the output position information of the target object, and the confidence level;

[0137] Optionally, the second loss value is determined as follows:

[0138] 4i) determining a first sub-loss value according to the marked position of the target object and the output position information of the target object;

[0139] 4ii) calculating the intersection-over-union ratio of the marked position and the output position information, determining the confidence of the output target object based on the intersection-over-union ratio, and determining the second sub-loss value based on the confidence of the output target object and the confidence of the marked target object;

[0140] 4iii) Determine a second loss value based on the first sub-loss value and the second sub-loss value.

[0141] In implementation, different processes are executed according to the tasks of the multi-task model to calculate the corresponding second loss value, as shown below:

[0142] The first type, the multi-task model, is used to implement target detection tasks.

[0143] (1a) Feature Reg[B, N o , C f ] is multiplied by the feature F of the retrieval image sample after feature matching in the form of matrix multiplication, and the output feature shape is [B, N o , L], expand the last dimension, the feature tensor is [B, N o , H e , W e ], which is also the mask prediction value, directly represented by the symbol mask; L represents the number of target object categories to be predicted. The feature Reg is used to represent the location information of the target object in the retrieved image sample. In instance segmentation, mask refers to the mask of the target object, which is used to represent the pixel-level location information of the target object.

[0144] (1b) After sigmoid processing, if the IOU of the mask with the gtmask is greater than 0.5, the corresponding obj target is 1, and the remaining obj targets are set to 0. If all the predicted IOUs corresponding to a gtmask are less than 0.5, then the target with the largest IOU is set to 1; where gtmask represents the mask annotated in a retrieval image sample and is only used for training.

[0145] This step is to predict obj. The next step is to calculate the loss using the predicted value and labeled value of obj.

[0146] (1c) Both mask and Obj use cross entropy loss, and the corresponding loss is recorded as L det With L obj ;

[0147] Among them, the first sub-loss value is L det ; The second sub-loss value is L obj .

[0148] The second type, the multi-task model, is used to implement instance segmentation tasks.

[0149] (2a) Feature Reg[B,N o , C f ]After a fully connected layer processing, the output feature tensor is [B, N o , 4], its value is the value of the upper left corner and the lower right corner of the detection box [x0, y0, x1, y1], which is represented by the symbol bboxes. The feature Reg is used to represent the position information of the target object in the retrieved image sample.

[0150] (2b) If the IOU of bboxes and gtbboxes is greater than 0.5, the corresponding obj target is 1, and the remaining obj targets are set to 0. If all predicted IOUs corresponding to a gtbboxes are less than 0.5, then the target with the largest IOU is set to 1; where gtbboxes represents the bounding box of the target object annotated in a retrieval image sample and is only used for training.

[0151] (2c) bboxes uses GIOU loss, obj uses cross entropy loss, and the corresponding loss is recorded as L det With L obj Among them, the first sub-loss value is L det ; The second sub-loss value is L obj .

[0152] Step b5: Determine a loss function value based on the first loss value and the second loss value.

[0153] Step c: training the multi-task model to be trained according to the loss function value. When the loss function value meets the preset requirements or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0154] The steps of target detection and instance segmentation can be performed either at one or both, but usually only one of them is performed according to the needs; cls +L det +L obj Backpropagation is performed as the final loss; optimization is performed using the AdamW optimizer.

[0155] Among them, L cls Used to represent the classification loss of the reference image sample, L obj It is used to express the confidence of the target object, that is, the probability of whether there is a target object, L det Used to represent the position loss of the target object.

[0156] When calculating the loss function value based on the first loss value and the second loss value, different weight parameters can also be set to perform weighted summation of the first loss value and the second loss value to obtain the loss function value. This embodiment does not impose too many restrictions on the setting of the weight parameters.

[0157] In some embodiments, the multi-task model includes a first encoder, a second encoder, and a decoder;

[0158] Inputting the retrieval image sample into a first encoder and outputting a first feature of the retrieval image sample, and inputting the reference image sample into a second encoder and outputting a second feature of the reference image sample;

[0159] The second feature is input into the fully connected layer to output the category of the reference image sample, and the first feature and the second feature are input into the decoder to output the location information and confidence of the target object.

[0160] Optionally, after outputting the first feature of the retrieved image sample and before inputting the first feature into the decoder, it is further used to add a position code to the first feature, where the position code is used to represent the position information of each pixel in the retrieved image sample.

[0161] Optionally, before the second feature is input to the decoder, it is also used to:

[0162] The token parameter and the second feature are spliced ​​together, and the spliced ​​second feature and the first feature are input into the decoder to output the location information and confidence of the target object; wherein the token parameter is used to distinguish the output location information and confidence of the target object.

[0163] In implementation, the token parameter is understood as a placeholder for distinguishing two different features from the second feature of the reference image sample after feature matching output by the decoder, where one feature is used to determine the location information of the target object and the other feature is used to determine the confidence of the target object.

[0164] The multi-task model provided by the present disclosure is divided into two stages: training and testing. The training stage uses manually annotated data for training, and the testing stage only requires the input of the search image and the reference image to detect or segment the same categories as the reference image in the search image. The specific training process of the multi-task model provided by this embodiment is described below. First, the symbols involved are explained:

[0165] Q: Input retrieval image sample. The goal of the task is to perform detection or instance segmentation on the retrieval image sample.

[0166] R: Input reference image sample, used to represent the image of the category that needs to be detected or segmented.

[0167] H: Enter the image height.

[0168] W: Input image width.

[0169] C: Number of input image channels.

[0170] B: The number of images processed simultaneously in a batch.

[0171] W e : Decoder input width.

[0172] H i : The output feature height of the backbone layer i is H / 2 i .

[0173] W i : backbone layer i output feature width, its value is W / 2 i .

[0174] C i : The number of backbone layer i output feature channels.

[0175] L: The number of categories of target objects that need to be predicted.

[0176] N o : The maximum number of samples of the same category in a retrieved image.

[0177] gtlabels: The categories annotated in a retrieved image sample, used only for training.

[0178] gtbboxes: bounding boxes of target objects annotated in a retrieved image sample, used only for training.

[0179] gtmasks: masks annotated in a retrieved image sample, used only for training.

[0180] num_classes: The number of all categories of reference image samples in the training dataset.

[0181] The training method is as follows:

[0182] 1) Input the retrieval image sample Q into the first encoder backbone for processing. The backbone can use SwinTransformV2, ResNet, Vit, etc., and take the output features of the 1st to 4th layers of the backbone and send them to FPN for processing to generate a new 4-layer feature. Let the feature be recorded as F i , the width and height of these four layers of features are W i , H i , the number of channels is C f (The number of channels in each layer is the same, typically 256), the feature tensor is [B, C f , H i , W i ], where i∈[1,2,3,4];

[0183] 2) Fi Scale to [W e , H e ], add them together and record them as F;

[0184] 3) Randomly sample a category from the gtlabels of Q, and then randomly crop one or more sub-images of the same category in the reference image sample according to the annotation information. The corresponding sub-image is the reference image sample R;

[0185] 4) Input the reference image sample R into the second encoder backbone for processing. The backbone uses a lightweight model such as ResNet and VGG, and takes the two-dimensional feature output of the last layer;

[0186] 5) The two-dimensional features of R are averaged in the spatial dimension and then L2 normalized to output the final reference image sample features, denoted as E, whose tensor is [B, C f , 1];

[0187] 6) After feature E is processed by a weight-normalized fully connected layer, a feature of shape [B, num_classes] is generated. Based on the category of the reference image sample (i.e., the image category in this embodiment), cross entropy is used as the loss for classification, and the loss is recorded as L cls ;

[0188] 7) Add position encoding to the feature F of the retrieved image sample;

[0189] 8) Expand F in the spatial dimension, and its tensor is [B, C f , W e ×H e ], abbreviated as [B, C f , L];

[0190] 9) Transpose F to [B, L, C f ];

[0191] 10) Generate learnable token parameters T, tensor [1, C f , 1+N o ], broadcast T as a tensor [B, C f , 1+N o ];

[0192] 11) Concatenate T with the feature E of the reference image sample in the last dimension, and the generated feature tensor is [B, C f , 1+N o +1], transposed to the tensor [B, 1+N o +1, C f ], the generated features are still recorded as R;

[0193] 12) Use the decoder (two-layer transform structure) to process the concatenated feature R and the transposed feature F. First, use R as the query, F as the key and value, and then use F as the query, R as the key and value. This is mainly used to perform feature matching on R and F.

[0194] 13) The above process can be repeated one or more times as needed;

[0195] 14) After feature matching, two different features are separated from the second dimension of R. One is Obj, and the tensor is [B, 1, C f ], the other is Reg, the tensor is [B, N o , C f ], these two features are exactly the data corresponding to the token, and the extra data is ignored (the data corresponding to R);

[0196] 15) After a layer of full connection processing, Obj generates a size of [B, 1, N o ] feature, which is used to indicate the confidence of the output target object, that is, Obj is used to indicate whether there is a target object;

[0197] 16) Instance segmentation performs the following steps:

[0198] a) Reg is multiplied by the feature-matched F in the form of matrix multiplication, and the output feature shape is [B, N o , L], expand the last dimension, the feature tensor is [B, N o , H e , W e ], which is also the mask prediction value of the target object, directly represented by the symbol mask;

[0199] b) After sigmoid processing, if the IOU of the mask with the gtmask is greater than 0.5, the corresponding obj target is 1, and the remaining obj targets are set to 0. If all the predicted IOUs corresponding to a gtmask are less than 0.5, then the target with the largest IOU is set to 1;

[0200] c) Both mask and Obj use cross entropy loss, and the corresponding loss is recorded as L det With L obj ;

[0201] 17) Target detection performs the following steps:

[0202] a) After Reg is processed by a fully connected layer, the output feature tensor is [B, N o , 4], its value is the value of the upper left corner and the lower right corner of the detection box [x0, y0, x1, y1], represented by the symbol bboxes;

[0203] b) If the IOU of bboxes and gtbboxes is greater than 0.5, the corresponding obj target is 1, and the remaining obj targets are set to 0. If all predicted IOUs corresponding to a gtbboxes are less than 0.5, then the target with the largest IOU is set to 1;

[0204] c) bboxes uses GIOU loss, obj uses cross entropy loss, and the corresponding loss is recorded as L det With L obj .

[0205] Among them, the steps of target detection and instance segmentation can be performed only one or both, but generally speaking, only one of them can be performed according to the needs.

[0206] 18) L cls +L det +L obj Backpropagation is performed as the final loss;

[0207] 19) Use the optimizer AdamW for optimization.

[0208] The following describes the specific prediction process of the multi-task model provided in this embodiment. First, the symbols involved are explained:

[0209] Q: Input retrieval image. The goal of the task is to perform detection or instance segmentation on the retrieval image.

[0210] R: Input reference image, used to represent the image of the category that needs to be detected or segmented.

[0211] H: Enter the image height.

[0212] W: Input image width.

[0213] C: Number of input image channels.

[0214] B: The number of images processed simultaneously in a batch.

[0215] W e : Decoder input width.

[0216] H i : The output feature height of the backbone layer i is H / 2 i .

[0217] W i : backbone layer i output feature width, its value is W / 2 i .

[0218] C i: The number of backbone layer i output feature channels.

[0219] L: The number of categories of target objects that need to be predicted.

[0220] N o : The maximum number of samples of the same category in a retrieved image.

[0221] num_classes: The number of all categories of reference image samples in the training dataset.

[0222] 1) Input the search image Q into the first encoder backbone for processing. The backbone can use SwinTransformV2, ResNet, Vit, etc., and take the output features of the 1st to 4th layers of the backbone and send them to FPN for processing to generate a new 4-layer feature. Let the feature be recorded as F i , the width and height of these four layers of features are W i , H i , the number of channels is C f (The number of channels in each layer is the same, typically 256), the feature tensor is [B, C f , H i , W i ], where i∈[1,2,3,4];

[0223] 2) F i Scale to [W e , H e ], add them together and record them as F;

[0224] 3) Input the reference image R into the second encoder backbone for processing. The backbone uses a lightweight model such as ResNet and VGG, and takes the two-dimensional feature output of the last layer;

[0225] It should be noted that the reference image can be a category in the training dataset or not. The reference image can be one or multiple images of the same category. In this way, the model can be compatible with single-sample learning and small-sample learning.

[0226] 4) The two-dimensional features of R are averaged in the spatial dimension and batch size dimension (for smoothing the features of multiple inputs) and L2 normalized to output the final reference image features, denoted as E, whose shape is [1, C f , 1];

[0227] 5) Add position encoding to the feature F of the retrieval image;

[0228] 6) Expand F in the spatial dimension, and its shape is [1, C f, W e ×H e ], abbreviated as [1, Cf, L];

[0229] 7) Transpose F to [1, L, C f ];

[0230] 8) Use the trained token parameter T, the shape is [1, C f , 1+N o ];

[0231] 9) Concatenate the token T with the reference image feature E in the last dimension, and the resulting shape is [1, C f , 1+N o +1], transposed to shape [1, 1+N o +1, C f ]The generated features are still recorded as R;

[0232] 10) Use a decoder (such as a two-layer transform structure) to process the concatenated feature R and the transposed feature F. First, use R as the query, F as the key and value, and then use F as the query, R as the key and value. This is mainly used to perform feature matching on R and F.

[0233] 11) The above process can be repeated one or more times as needed;

[0234] 12) After feature matching, the second dimension of R is divided into two different features in turn, one is Obj, and the tensor is [1, 1, C f ], the other is Reg, the tensor is [1, N o , C f ], redundant data is ignored;

[0235] 13) After a layer of full connection processing, Obj generates a size of [1, 1, N o ] features;

[0236] 14) Instance segmentation performs the following steps:

[0237] a) Reg is multiplied by F in the form of matrix multiplication, and the output feature shape is [1, N o , L], expand the last dimension to [1, N o , H e , W e ], which is also the mask prediction value of the target object, and is directly set as mask;

[0238] b) The mask after sigmoid processing is the prediction result of the target object's location information;

[0239] c) After sigmoid processing, obj is the confidence of the mask image mask of the corresponding target object;

[0240] 15) Target detection performs the following steps:

[0241] a) After Reg is processed by a fully connected layer, the output feature shape is [1, N o ,4], whose values ​​are the values ​​of the upper left corner and lower right corner of the detection box [x0, y0, x1, y1], expressed as bboxes;

[0242] b) After sigmoid processing, obj is the confidence of the corresponding bboxes.

[0243] Among them, the steps of target detection and instance segmentation need to be consistent with those during training.

[0244] The present disclosure provides a multi-task model with a unified architecture for single-sample detection and segmentation based on a codec structure. At the same time, this architecture is compatible with few-shot learning. Compared with previous single-sample or few-sample detection and segmentation algorithms, the innovations are mainly reflected in the following aspects: when expanding new categories, there is no need to retrain the network, which makes it more convenient to use; it is compatible with both target detection and instance segmentation tasks; during the training process, multiple datasets with non-uniform labeling categories can be combined and used, and the labeling categories of different datasets do not need to be consistent; it is compatible with few-shot learning.

[0245] Since the multi-task model of the present invention only requires the reference image category to exist in the corresponding retrieval image for training during the training process, the multi-task model of the present invention can combine and use data from different training data sets for training. For example, training data set 1 only labels animals such as dogs, cats, and cows, but does not label any vehicles appearing in the image, while training data set 2 only labels motor vehicles such as cars, trucks, and buses, but does not label any animals appearing in the image. Merging training data set 1 with training data set 2 can also be used to train the multi-task model of the present invention. This training method greatly expands the amount of available data for the model and improves the practicality of the model. During prediction, the reference image can be a category that appears in the training data set or a completely new category. At the same time, the reference image can be one or multiple images of the same category, so that the multi-task model is compatible with single-sample and few-sample learning.

[0246] Based on the same inventive concept, the present disclosure also provides a multi-task model training method, as shown in FIG5 , which includes:

[0247] Step 500: Acquire a training data set, where the training data set includes a plurality of search image samples and a plurality of reference image samples;

[0248] Step 501: Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the position of the target object annotated by the retrieval image sample;

[0249] Step 502: The multi-task model to be trained is trained according to the loss function value. When the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0250] As an optional implementation, determining the loss function value based on the output result, the image category annotated by the reference image sample, and the position of the target object annotated by the retrieval image sample includes:

[0251] Inputting the retrieval image sample and the reference image sample into the multi-task model to be trained, and extracting the first feature of the retrieval image sample and the second feature of the reference image sample respectively;

[0252] Classify the second feature to output the category of the reference image sample, perform feature matching on the first feature and the second feature to obtain a matching feature, and output the location information and confidence level of the target object based on the matching feature;

[0253] Determine a first loss value according to the category of the output reference image sample and the labeled image category;

[0254] Determining a second loss value according to the marked position of the target object, the output position information of the target object, and the confidence level;

[0255] A loss function value is determined based on the first loss value and the second loss value.

[0256] As an optional implementation manner, determining the second loss value based on the marked position of the target object, the output position information of the target object, and the confidence level includes:

[0257] Determine a first sub-loss value according to the marked position of the target object and the output position information of the target object;

[0258] Calculate the intersection-over-union ratio of the marked position and the output position information, determine the confidence of the output target object based on the intersection-over-union ratio, and determine the second sub-loss value based on the confidence of the output target object and the confidence of the marked target object;

[0259] A second loss value is determined according to the first sub-loss value and the second sub-loss value.

[0260] As an optional implementation, the multi-task model includes a first encoder, a second encoder and a decoder;

[0261] Inputting the retrieval image sample into a first encoder and outputting a first feature of the retrieval image sample, and inputting the reference image sample into a second encoder and outputting a second feature of the reference image sample;

[0262] The second feature is input into the fully connected layer to output the category of the reference image sample, and the first feature and the second feature are input into the decoder to output the location information and confidence of the target object.

[0263] As an optional implementation, the first encoder is a first backbone network, the second encoder is a second backbone network, and the decoder is a neural network based on a self-attention mechanism.

[0264] As an optional implementation, the multi-task model is used to implement a target detection task; the output result includes a detection box of a target object in a retrieved image sample, and the detection box is used to represent position information of the target object in the retrieved image sample.

[0265] As an optional implementation, the multi-task model is used to implement an instance segmentation task; the output result includes a mask image of the target object in the retrieved image sample, and the mask image is used to represent the pixel-level position information of the target object in the retrieved image sample.

[0266] Based on the same inventive concept, the embodiment of the present disclosure also provides an electronic device. Since the device is the device in the target determination method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0267] As shown in FIG6 , the electronic device includes a processor 600 and a memory 601 . The memory 601 is used to store programs executable by the processor 600 . The processor 600 is used to read the programs in the memory 601 and perform the following steps:

[0268] Obtaining a retrieval image and a reference image;

[0269] Inputting the retrieval image and the reference image into a multi-task model, determining a target object in the retrieval image, wherein the target object and the reference image have the same image category;

[0270] The multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image based on the matching features.

[0271] As an optional implementation, the multi-task model includes a first encoder, a second encoder, and a decoder, and the processor 600 is specifically configured to execute:

[0272] Inputting the search image into a first encoder, extracting and outputting features of the search image, and inputting the reference image into a second encoder, extracting and outputting features of the reference image;

[0273] The features of the retrieval image and the features of the reference image are input into the decoder, and the features of the retrieval image and the reference image are matched to obtain matching features, and the target object in the retrieval image is determined based on the matching features.

[0274] As an optional implementation, the first encoder is a first backbone network, the second encoder is a second backbone network, and the decoder is a neural network based on a self-attention mechanism.

[0275] As an optional implementation, the multi-task model is used to output the location information of the target object in the retrieval image, and implement different tasks based on the location information of the target object. The different tasks correspond to different levels of granularity of the location information of the target object in the output retrieval image.

[0276] As an optional implementation, the multi-task model is used to implement a target detection task; the output target object location information includes a detection box of the target object, and the detection box is used to represent the location information of the target object in the retrieved image.

[0277] As an optional implementation manner, the processor 600 is specifically configured to output the location information of the target object in the following manner:

[0278] The matching features are input into the fully connected layer of the multi-task model, and the detection box of the target object is output on the retrieval image.

[0279] As an optional implementation, the multi-task model is used to implement an instance segmentation task; the output target object location information includes a mask image of the target object, and the mask image is used to represent the pixel-level location information of the target object in the retrieval image.

[0280] As an optional implementation manner, the processor 600 is specifically configured to output the location information of the target object in the following manner:

[0281] Multiply the matching features and the retrieval image after feature matching to output the mask image of the target object.

[0282] As an optional implementation, the processor 600 is specifically configured to train the multi-task model in the following manner:

[0283] Acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0284] Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0285] The multi-task model to be trained is trained according to the loss function value, and when the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0286] As an optional implementation manner, the processor 600 is specifically configured to execute:

[0287] Inputting the retrieval image sample and the reference image sample into the multi-task model to be trained, and extracting the first feature of the retrieval image sample and the second feature of the reference image sample respectively;

[0288] Classify the second feature to output the category of the reference image sample, perform feature matching on the first feature and the second feature to obtain a matching feature, and output the location information and confidence level of the target object based on the matching feature;

[0289] Determine a first loss value according to the category of the output reference image sample and the labeled image category;

[0290] Determining a second loss value according to the marked position of the target object, the output position information of the target object, and the confidence level;

[0291] A loss function value is determined based on the first loss value and the second loss value.

[0292] As an optional implementation, the multi-task model includes a first encoder, a second encoder, and a decoder; the processor 600 is specifically configured to execute:

[0293] Inputting the retrieval image sample into a first encoder and outputting a first feature of the retrieval image sample, and inputting the reference image sample into a second encoder and outputting a second feature of the reference image sample;

[0294] The second feature is input into the fully connected layer to output the category of the reference image sample, and the first feature and the second feature are input into the decoder to output the location information and confidence of the target object.

[0295] As an optional implementation, after outputting the first feature of the retrieved image sample and before inputting the first feature into the decoder, the processor 600 is further configured to execute:

[0296] A position code is added to the first feature, where the position code is used to represent the position information of each pixel in the retrieved image sample.

[0297] As an optional implementation manner, before inputting the second feature into the decoder, the processor 600 is further configured to execute:

[0298] Concatenate the token parameter and the second feature, input the concatenated second feature and the first feature into the decoder, and output the location information and confidence of the target object;

[0299] The token parameter is used to distinguish the location information and confidence of the output target object.

[0300] As an optional implementation manner, the processor 600 is specifically configured to execute:

[0301] Determine a first sub-loss value according to the marked position of the target object and the output position information of the target object;

[0302] Calculate the intersection-over-union ratio of the marked position and the output position information, determine the confidence of the output target object based on the intersection-over-union ratio, and determine the second sub-loss value based on the confidence of the output target object and the confidence of the marked target object;

[0303] A second loss value is determined according to the first sub-loss value and the second sub-loss value.

[0304] As an optional implementation, the reference image includes one reference image or multiple reference images of the same image category.

[0305] Based on the same inventive concept, the embodiment of the present disclosure also provides another electronic device. Since the device is the device in the model training method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0306] As shown in FIG7 , the electronic device includes a processor 700 and a memory 701 . The memory 701 is used to store programs executable by the processor 700 . The processor 700 is used to read the programs in the memory 701 and perform the following steps:

[0307] Acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0308] Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0309] The multi-task model to be trained is trained according to the loss function value, and when the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0310] Based on the same inventive concept, the embodiment of the present disclosure also provides a target determination device. Since the device is the device in the target determination method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0311] As shown in FIG8 , the device includes:

[0312] An image acquisition unit 800 is used to acquire a search image and a reference image;

[0313] The position determination unit 801 is used to input the retrieval image and the reference image into a multi-task model to determine the target object in the retrieval image, where the target object and the reference image have the same image category; wherein the multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image based on the matching features.

[0314] As an optional implementation, the multi-task model includes a first encoder, a second encoder, and a decoder, and the position determination unit 801 is specifically configured to:

[0315] Inputting the search image into a first encoder, extracting and outputting features of the search image, and inputting the reference image into a second encoder, extracting and outputting features of the reference image;

[0316] The features of the retrieval image and the features of the reference image are input into the decoder, and the features of the retrieval image and the reference image are matched to obtain matching features, and the target object in the retrieval image is determined based on the matching features.

[0317] As an optional implementation, the first encoder is a first backbone network, the second encoder is a second backbone network, and the decoder is a neural network based on a self-attention mechanism.

[0318] As an optional implementation, the multi-task model is used to output the location information of the target object in the retrieval image, and implement different tasks based on the location information of the target object. The different tasks correspond to different levels of granularity of the location information of the target object in the output retrieval image.

[0319] As an optional implementation, the multi-task model is used to implement a target detection task; the output target object location information includes a detection box of the target object, and the detection box is used to represent the location information of the target object in the retrieved image.

[0320] As an optional implementation manner, the location determination unit 801 is specifically configured to output the location information of the target object in the following manner:

[0321] The matching features are input into the fully connected layer of the multi-task model, and the detection box of the target object is output on the retrieval image.

[0322] As an optional implementation, the multi-task model is used to implement an instance segmentation task; the output target object location information includes a mask image of the target object, and the mask image is used to represent the pixel-level location information of the target object in the retrieval image.

[0323] As an optional implementation manner, the location determination unit 801 is specifically configured to output the location information of the target object in the following manner:

[0324] Multiply the matching features and the retrieval image after feature matching to output the mask image of the target object.

[0325] As an optional implementation manner, the position determination unit 801 is specifically configured to train the multi-task model in the following manner:

[0326] Acquire a training data set, wherein the training data set includes a plurality of retrieval image samples and a plurality of reference image samples;

[0327] Input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine the loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0328] The multi-task model to be trained is trained according to the loss function value, and when the loss function value meets a preset requirement or the number of training iterations reaches a threshold, a trained multi-task model is obtained.

[0329] As an optional implementation manner, the position determination unit 801 is specifically configured to:

[0330] Inputting the retrieval image sample and the reference image sample into the multi-task model to be trained, and extracting the first feature of the retrieval image sample and the second feature of the reference image sample respectively;

[0331] Classify the second feature to output the category of the reference image sample, perform feature matching on the first feature and the second feature to obtain a matching feature, and output the location information and confidence level of the target object based on the matching feature;

[0332] Determine a first loss value according to the category of the output reference image sample and the labeled image category;

[0333] Determining a second loss value according to the marked position of the target object, the output position information of the target object, and the confidence level;

[0334] A loss function value is determined based on the first loss value and the second loss value.

[0335] As an optional implementation, the multi-task model includes a first encoder, a second encoder, and a decoder; the position determination unit 801 is specifically configured to:

[0336] Inputting the retrieval image sample into a first encoder and outputting a first feature of the retrieval image sample, and inputting the reference image sample into a second encoder and outputting a second feature of the reference image sample;

[0337] The second feature is input into the fully connected layer to output the category of the reference image sample, and the first feature and the second feature are input into the decoder to output the location information and confidence of the target object.

[0338] As an optional implementation, after outputting the first feature of the retrieved image sample and before inputting the first feature into the decoder, the position determination unit 801 is further configured to:

[0339] A position code is added to the first feature, where the position code is used to represent the position information of each pixel in the retrieved image sample.

[0340] As an optional implementation manner, before inputting the second feature into the decoder, the position determination unit 801 is further configured to:

[0341] Concatenate the token parameter and the second feature, input the concatenated second feature and the first feature into the decoder, and output the location information and confidence of the target object;

[0342] The token parameter is used to distinguish the location information and confidence of the output target object.

[0343] As an optional implementation manner, the position determination unit 801 is specifically configured to:

[0344] Determine a first sub-loss value according to the marked position of the target object and the output position information of the target object;

[0345] Calculate the intersection-over-union ratio of the marked position and the output position information, determine the confidence of the output target object based on the intersection-over-union ratio, and determine the second sub-loss value based on the confidence of the output target object and the confidence of the marked target object;

[0346] A second loss value is determined according to the first sub-loss value and the second sub-loss value.

[0347] As an optional implementation, the reference image includes one reference image or multiple reference images of the same image category.

[0348] Based on the same inventive concept, the embodiment of the present disclosure also provides a multi-task model training device. Since the device is the device in the model training method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0349] As shown in FIG9 , the device includes:

[0350] A training set acquisition unit 900 is configured to acquire a training data set, wherein the training data set includes a plurality of search image samples and a plurality of reference image samples;

[0351] A loss function determination unit 901 is configured to input the retrieval image sample and the reference image sample into the multi-task model to be trained, and determine a loss function value based on the output result, the image category annotated by the reference image sample, and the location of the target object annotated by the retrieval image sample;

[0352] The training model unit 902 is used to train the multi-task model to be trained according to the loss function value. When the loss function value meets the preset requirements or the number of training iterations reaches a threshold, the trained multi-task model is obtained.

[0353] Based on the same inventive concept, an embodiment of the present disclosure provides a computer storage medium, the computer storage medium comprising: computer program code, which, when executed on a computer, causes the computer to execute any of the target determination methods or model training methods discussed above. Because the principles for solving the problems described above by the computer storage medium are similar to those of the target determination methods or model training methods, the implementation of the computer storage medium can be referred to as the implementation of the method, and any repetitions will not be repeated here.

[0354] In a specific implementation process, computer storage media may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0355] Based on the same inventive concept, embodiments of the present disclosure further provide a computer program product, comprising: computer program code, which, when executed on a computer, causes the computer to execute any of the target determination methods or model training methods discussed above. Because the principles underlying the problems solved by the aforementioned computer program products are similar to those of the target determination methods or model training methods, the implementation of the aforementioned computer program products can be referred to as the implementation of the methods, and any repetitions will not be repeated.

[0356] The computer program product can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0357] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0358] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0359] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0360] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0361] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for target determination, wherein, the method includes: obtaining a retrieval image and a reference image; inputting the retrieval image and the reference image into a multi-task model to determine a target object in the retrieval image, where the target object has the same image category as the reference image; wherein, the multi-task model is used to extract features of the retrieval image and the reference image, perform feature matching on the features of the retrieval image and the reference image to obtain matching features, and determine the target object in the retrieval image according to the matching features.

2. The method according to claim 1, wherein, the multi-task model includes a first encoder, a second encoder and a decoder, wherein: inputting the retrieval image into the first encoder to extract and output the features of the retrieval image, and inputting the reference image into the second encoder to extract and output the features of the reference image; inputting the features of the retrieval image and the features of the reference image into the decoder, performing feature matching on the features of the retrieval image and the reference image to obtain matching features, and determining the target object in the retrieval image according to the matching features.

3. The method according to claim 2, wherein, the first encoder is a first backbone network, the second encoder is a second backbone network, and the decoder is a neural network based on the self-attention mechanism.

4. The method according to claim 1, wherein, the multi-task model is used to output the position information of the target object in the retrieval image, and different tasks are implemented according to the position information of the target object, and the granularity levels of the position information of the target object output corresponding to the different tasks are different.

5. The method according to claim 4, wherein, the multi-task model is used to implement a target detection task; the output position information of the target object includes a detection box of the target object, and the detection box is used to represent the position information of the target object in the retrieval image.

6. The method according to claim 5, wherein, the position information of the target object is output in the following manner: inputting the matching features into the fully connected layer of the multi-task model, and outputting the detection box of the target object on the retrieval image.

7. The method according to claim 4, wherein, the multi-task model is used to implement an instance segmentation task; the output position information of the target object includes a mask image of the target object, and the mask image is used to represent the position information of the target object at the pixel level in the retrieval image.

8. The method according to claim 7, wherein, the position information of the target object is output in the following manner: multiplying the matching features by the retrieval image after feature matching, and outputting the mask image of the target object.

9. The method according to claim 1, wherein, the multi-task model is trained in the following manner: obtaining a training data set, where the training data set includes a plurality of retrieval image samples and a plurality of reference image samples; inputting the retrieval image samples and the reference image samples into the multi-task model to be trained, and determining the loss function value according to the output result, the image category labeled by the reference image sample, and the position of the target object labeled by the retrieval image sample. Train the multi - task model to be trained according to the loss function value. When the loss function value meets the preset requirements or the number of training iterations reaches the threshold, obtain the trained multi - task model.

10. The method according to claim 9, wherein, determining the loss function value according to the output result, the image category labeled by the reference image sample, and the position of the target object labeled by the retrieved image sample includes: Input the retrieved image sample and the reference image sample into the multi - task model to be trained, and respectively extract the first feature of the retrieved image sample and the second feature of the reference image sample; Classify the second feature to output the category of the reference image sample, and perform feature matching on the first feature and the second feature to obtain the matching feature, and output the position information and confidence of the target object according to the matching feature; Determine the first loss value according to the output category of the reference image sample and the labeled image category; Determine the second loss value according to the labeled position of the target object, the output position information of the target object, and the confidence; Determine the loss function value according to the first loss value and the second loss value.

11. The method according to claim 10, wherein, the multi - task model includes a first encoder, a second encoder, and a decoder; Input the retrieved image sample into the first encoder to output the first feature of the retrieved image sample, and input the reference image sample into the second encoder to output the second feature of the reference image sample; Input the second feature into the fully - connected layer to output the category of the reference image sample, and input the first feature and the second feature into the decoder to output the position information and confidence of the target object.

12. The method according to claim 11, wherein, after outputting the first feature of the retrieved image sample and before inputting the first feature into the decoder, further includes: Add a position encoding to the first feature, and the position encoding is used to represent the position information of each pixel in the retrieved image sample.

13. The method according to claim 11, wherein, before inputting the second feature into the decoder, further includes: Concatenate the token parameter and the second feature, and input the concatenated second feature and the first feature into the decoder to output the position information and confidence of the target object; where the token parameter is used to distinguish the output position information and confidence of the target object.

14. The method according to claim 10, wherein, determining the second loss value according to the labeled position of the target object, the output position information of the target object, and the confidence includes: Determine the first sub - loss value according to the labeled position of the target object and the output position information of the target object; Calculate the intersection - over - union ratio of the labeled position and the output position information, determine the confidence of the output target object according to the intersection - over - union ratio, and determine the second sub - loss value according to the confidence of the output target object and the confidence of the labeled target object; Determine the second loss value according to the first sub - loss value and the second sub - loss value.

15. The method according to any one of claims 1 - 14, wherein, the reference image includes one reference image or multiple reference images of the same image category.

16. An electronic device, wherein, The electronic device includes a processor and a memory, the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of claims 1 to 15.

17. A computer storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, it implements the steps of any one of claims 1 to 15.

Images