Image labeling method, device and electronic equipment

Abstract

The image labeling method, device and electronic equipment provided by the embodiments of the present application relate to the technical field of image processing. First, based on the feature map of the support image and the feature map of the query image, a first inner product transformation result with the same shape as the feature map of the query image is obtained; then, based on the first inner product transformation result, a feature mapping heat map of the target object of the same category is obtained; then, a candidate box is determined from the feature mapping heat map, and the target subgraph in the query image is intercepted by using the candidate box; then, based on the feature vector of the target subgraph and the feature vector of the support image, the feature similarity of the target object in the query image and the target object in the support image is determined; finally, the bounding box information and the category of the target object in the query image are output according to the feature similarity. The above scheme can automatically label a large number of query images through the labeled support image, and the efficiency and accuracy of labeling are higher than those of the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image labeling method and device and electronic equipment. BACKGROUND

[0002] Some specific places have clear restrictions on the target objects entering, for example, large pets are prohibited from entering high-speed rail stations. At present, the management of such target objects is mostly achieved manually, such as labeling the target objects in the station area image to identify whether the target objects have the right to enter. Specifically, the position area and corresponding category of the target objects in the station area image need to be labeled. However, due to the influence of the concentration of the labelers, the labelers will become tired over time, resulting in a decrease in labeling efficiency and an increase in errors. SUMMARY

[0003] To solve the above technical problems, the present application provides an image labeling method, device and electronic equipment.

[0004] In a first aspect, an image labeling method is provided, which comprises:

[0005] Image feature extraction is performed on support images in a support set to obtain feature vectors of the support images and feature maps of the support images, wherein the support images include labeling information, and the labeling information includes the category of a target object in the support images and the bounding box information for circumscribing the target object from the support images;

[0006] Image feature extraction is performed on query images in a query set to obtain feature maps of the query images;

[0007] Based on the feature maps of the support images and the feature maps of the query images, a first inner product transformation result with the same shape as the feature maps of the query images is obtained;

[0008] Based on the first inner product transformation result, a feature mapping heat map of target objects of the same category is obtained;

[0009] A candidate box is determined from the feature mapping heat map, and the candidate box is used to intercept a target sub-image in the query image;

[0010] A feature vector of the target sub-image is extracted, and based on the feature vector of the target sub-image and the feature vectors of the support images, a feature similarity between the target object in the query image and the target object in the support images is determined;

[0011] According to the feature similarity between the target object in the query image and the target object in the support images, the bounding box information and the category of the target object in the query image are output.

[0012] In a possible implementation, after the step of determining the candidate box from the feature mapping heat map and using the candidate box to crop the target sub-image in the query image, the method further comprises:

[0013] obtaining a second inner product transformation result in the same shape as the feature map of the target sub-image based on the feature map of the support image and the feature map of the target sub-image;

[0014] updating the feature mapping heat map of the same category based on the second inner product transformation result;

[0015] updating the candidate box based on the updated feature mapping heat map, and re-cropping the target sub-image in the query image based on the updated candidate box.

[0016] In a possible implementation, the step of performing image feature extraction on the support images in the support set to obtain the feature vectors of the support images and the feature maps of the support images comprises:

[0017] inputting the support images in the support set into an image feature extraction model to output the feature vectors of the support images and the feature maps of the support images, and setting the regions outside the region with the bounding box information in the feature maps of the support images to 0;

[0018] The step of performing image feature extraction on the query images in the query set to obtain the feature maps of the query images comprises:

[0019] inputting the query images in the query set into the image feature extraction model to output the feature maps of the query images.

[0020] In a possible implementation, the step of obtaining a first inner product transformation result in the same shape as the feature map of the query image based on the feature map of the support image and the feature map of the query image comprises:

[0021] performing vector normalization on all points in the feature map of the support image and the feature map of the query image;

[0022] calculating the inner product of the feature map of the support image and the feature map of the query image after vector normalization, wherein the dimension of the inner product is four;

[0023] subtracting the mean of the last two dimensions of the inner product from the inner product, and then averaging the first two dimensions of the inner product to obtain a first inner product transformation result in the same shape as the feature map of the query image.

[0024] In a possible implementation, the step of obtaining the feature mapping heat map of the same category target object based on the first inner product transformation result comprises:

[0025] The first inner product transformation result is processed, and points with a value less than 0 in the first inner product transformation result are clamped to 0, and points with a value greater than or equal to 0 in the first inner product transformation result are linearly transformed into a set numerical interval;

[0026] The first inner product transformation result is processed in an image interpolation manner to obtain a feature mapping heat map;

[0027] The feature mapping heat maps corresponding to the plurality of support images of the same category target object are averaged to obtain a feature mapping heat map of the same category target object.

[0028] In a possible implementation, the step of determining the candidate box from the feature mapping heat map and intercepting a target sub-image in the query image by using the candidate box comprises:

[0029] Based on the continuous highlight region in the feature mapping heat map, an outline of the target object in the feature mapping heat map is obtained;

[0030] A minimum circumscribed rectangle circumscribing the outline of the target object is calculated, and the minimum circumscribed rectangle is taken as the candidate box;

[0031] The target sub-image is intercepted from the query image by using the candidate box.

[0032] In a possible implementation, the step of extracting the feature vector of the target sub-image, determining the feature similarity of the target object in the query image and the target object in the support image based on the feature vector of the target sub-image and the feature vector of the support image comprises:

[0033] Inner products of the feature vector of the target sub-image and the feature vector of the support image are calculated to obtain a cosine distance of the feature vector of the target sub-image and the feature vector of the support image;

[0034] The cosine distance is used to represent the feature similarity of the target object in the query image and the target object in the support image, and the greater the cosine distance is, the higher the feature similarity of the target object in the query image and the target object in the support image is.

[0035] In a possible implementation, the step of outputting the bounding box information and the category of the target object in the query image according to the feature similarity of the target object in the query image and the target object in the support image comprises:

[0036] The support image corresponding to the maximum cosine distance is taken as a target support image, and the bounding box information and the category of the target support image are taken as the bounding box information and the category of the target object in the query image, and the bounding box information and the category of the target object in the query image are output.

[0037] In a second aspect, the embodiments of the present application further provide an image labeling device, the device comprising:

[0038] A first feature extraction module is configured to perform image feature extraction on support images in a support set to obtain feature vectors of the support images and feature maps of the support images, wherein the support images comprise labeling information, and the labeling information comprises a category of a target object in the support images and bounding box information for circumscribing the target object from the support images;

[0039] A second feature extraction module is configured to perform image feature extraction on a query image in a query set to obtain a feature map of the query image;

[0040] A first calculation module is configured to obtain a first inner product transformation result with the same shape as the feature map of the query image based on the feature maps of the support images and the feature map of the query image;

[0041] A second calculation module is configured to obtain a feature mapping heat map of a target object of the same category based on the first inner product transformation result;

[0042] A determination and interception module is configured to determine a candidate box from the feature mapping heat map and intercept a target sub-image in the query image using the candidate box;

[0043] An extraction and determination module is configured to extract a feature vector of the target sub-image, and determine a feature similarity between the target object in the query image and the target object in the support images based on the feature vector of the target sub-image and the feature vectors of the support images;

[0044] An output module is configured to output the bounding box information and the category of the target object in the query image according to the feature similarity between the target object in the query image and the target object in the support images.

[0045] In a third aspect, the embodiments of the present application further provide an electronic device, which comprises a processor, a communication unit and a computer readable storage medium, the processor and the computer readable storage medium are connected through a bus system, the communication unit is configured to connect with an image acquisition device to realize data interaction between each other, the computer readable storage medium is configured to store programs, instructions or codes, and the processor is configured to execute the programs, instructions or codes in the computer readable storage medium to realize the image labeling method in any one of the possible implementation manners in the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions when executed by a processor, cause an electronic device to perform the image labeling method in the first aspect or any one of the possible implementation manners of the first aspect.

[0047] Based on any one of the above aspects, the image labeling method, device and electronic device provided by the embodiments of the present application first obtain a first inner product transformation result with the same shape as the feature map of the query image based on the feature map of the support image and the feature map of the query image; then obtain a feature mapping heat map of the target object of the same category based on the first inner product transformation result; then determine a candidate box from the feature mapping heat map and use the candidate box to intercept a target sub-image in the query image; then determine the feature similarity between the target object in the query image and the target object in the support image based on the feature vector of the target sub-image and the feature vector of the support image; and finally output the bounding box information and the category of the target object in the query image according to the feature similarity. The above scheme can automatically label a large number of query images based on the labeled support images, and the efficiency and accuracy of labeling are higher than those of the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be called in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 An interactive scene diagram of the image labeling system provided by the present embodiment;

[0050] Figure 2 A flowchart of the image labeling method provided by the present embodiment;

[0051] Figure 3 A functional module diagram of the image labeling device provided by the present embodiment;

[0052] Figure 4 A possible structure diagram of the electronic device provided by the present embodiment. DETAILED DESCRIPTION

[0053] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be appreciated that the drawings in the present application are only intended to illustrate and describe the purposes, and are not used to limit the protection scope of the present application. In addition, it should be appreciated that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be appreciated that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0054] In addition, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] Please refer to Figure 1 , Figure 1 An interaction scene schematic diagram of the image labeling system 10 provided by the embodiments of the present application is exemplified. The image labeling system 10 includes an electronic device 100 and an image acquisition device 200 which are communicatively connected, wherein the electronic device 100 can be a computer device with image processing capability, and the image acquisition device 200 can be a camera with image shooting function.

[0056] The image labeling method provided by the embodiments of the present application will be exemplarily described below in combination with the application scenario shown in Figure 1 Please refer to Figure 2 The image labeling method provided by the embodiments of the present application can be executed by the electronic device 100 in Figure 1 The order of some steps in the image labeling method of the embodiments of the present application can be exchanged with each other according to actual needs, or some steps therein can be omitted or deleted. The detailed steps of the image labeling method are described as follows.

[0057] In step S11, the image feature extraction is performed on the support images in the support set, to obtain the feature vectors of the support images and the feature maps of the support images.

[0058] The support set is composed of artificially labeled support images, and the support images include label information. In detail, the label information can include the category of the target object in the support image and the bounding box information for circumscribing the target object from the support image. Each support image includes a category of target object, and the bounding box information can be the position coordinates of the four vertices of the bounding box, for example, when the bounding box is a rectangular bounding box, the bounding box information can be the position coordinates of the four vertices of the bounding box.

[0059] In this embodiment, the support images in the support set can be input into an image feature extraction model to output the feature vector of the support image and the feature map of the support image, and the region outside the bounding box information determination region in the feature map of the support image is set to 0. The image feature extraction model can be a mature Vision Transformer pre-training model in the industry.

[0060] In step S12, image feature extraction is performed on the query images in the query set to obtain the feature map of the query image.

[0061] In this embodiment, the query images in the query set can be input into an image feature extraction model to output the feature map of the query image.

[0062] In step S13, based on the feature map of the support image and the feature map of the query image, a first inner product transformation result with the same shape as the feature map of the query image is obtained.

[0063] In this step, the inner product of the feature map of the support image and the feature map of the query image can be calculated, and then the inner product is transformed to obtain a first inner product transformation result with the same shape as the feature map of the query image.

[0064] In step S14, a feature mapping heat map of the target object of the same category is obtained based on the first inner product transformation result.

[0065] In this embodiment, if there are target objects of the same category in multiple support images, multiple feature mapping heat maps can be obtained, and the feature mapping heat map of the target object of the same category is calculated based on the multiple feature mapping heat maps.

[0066] In step S15, a candidate box is determined from the feature mapping heat map, and the candidate box is used to cut a target sub-image in the query image.

[0067] In step S16, a feature vector of the target sub-image is extracted, and based on the feature vector of the target sub-image and the feature vector of the support image, the feature similarity of the target object in the query image and the target object in the support image is determined.

[0068] Step S17, according to the feature similarity of the target object in the query image and the target object in the support image, output the bounding box information and the category of the target object in the query image.

[0069] The above-mentioned scheme automatically annotates a large number of query images by using the annotated support images, and has higher annotation efficiency and accuracy than the prior art.

[0070] The inventors found that the accuracy of the feature mapping heat map obtained in the above-mentioned scheme is not high, which is caused by the excessively large size of the query image when extracting the feature map of the query image. In order to solve the technical problem, further, after step S15, the image annotation method provided by the embodiment further includes the following steps.

[0071] First, based on the feature map of the support image and the feature map of the target sub-image, a second inner product transformation result with the same shape as the feature map of the target sub-image is obtained.

[0072] The execution process of this step is the same as that of step S13, and the specific execution process is described in detail in the description of step S13.

[0073] Then, the feature mapping heat map of the same category is updated based on the second inner product transformation result.

[0074] The execution process of this step is the same as that of step S14, and the specific execution process is described in detail in the description of step S14.

[0075] Then, the candidate frame is updated based on the updated feature mapping heat map, and the target sub-image is re-cut in the query image from the updated candidate frame.

[0076] The above-mentioned scheme can obtain the feature mapping heat map by using the feature map of the target sub-image with a smaller size, and the feature map of the target sub-image is more accurate than the feature map of the query image. Therefore, the accuracy of the feature mapping heat map is higher, and the accuracy of the candidate frame is improved, and the accuracy of the re-cut target sub-image is also higher.

[0077] Further, in the embodiment, step S13 can be implemented in the following manner.

[0078] First, the vector normalization is performed on all points in the feature map of the support image and the feature map of the query image.

[0079] Then, the inner product of the vector-normalized feature map of the support image and the feature map of the query image is calculated, wherein the dimension of the inner product is four.

[0080] The inner product can correspond to the cosine distance between the normalized feature map of the support image and the feature map of the query image.

[0081] Then, the inner product is subtracted by the mean of the last two dimensions of the inner product, and the first two dimensions of the inner product are averaged to obtain a first inner product transformation result in the same shape as the feature map of the query image.

[0082] The following is described with a specific example, assuming that the feature map of the support image is S, and the feature map of the query image is Q, wherein the size of the feature map S is (h0, w0), and the size of the feature map Q is (h1, w1). After vector normalization is performed on all points in the feature map S and the feature map Q, the size of the feature map S is (h0 / 16, w0 / 16), and the size of the feature map Q is (h1 / 16, w1 / 16).

[0083] The inner product of the feature map S and the feature map Q is calculated, SQ=S×Q, and the dimension of the inner product is four, and the size of the inner product SQ is (h0 / 16, w0 / 16, h1 / 16, w1 / 16).

[0084] After the inner product SQ is subtracted by the mean of the last two dimensions of SQ, the first two dimensions of the inner product SQ are averaged to obtain a first inner product transformation result SQ' in the same shape as the feature map of the query image, and the expression can be:

[0085] SQ'=mean(S×Q-exp)

[0086] Wherein, exp is the mean of the last two dimensions of SQ, and mean is the average of the first two dimensions of the inner product SQ. The size of the first inner product transformation result SQ' is (h1 / 16, w1 / 16).

[0087] Further, in the embodiment, the step S14 can be realized by the following way.

[0088] First, the first inner product transformation result is processed, and the points with a value less than 0 in the first inner product transformation result are clamped to 0, and the points with a value greater than or equal to 0 in the first inner product transformation result are linearly transformed into a set numerical interval (such as 0-255).

[0089] Then, the first inner product transformation result is processed in the manner of image interpolation to obtain a feature mapping heat map.

[0090] Then, the feature mapping heat maps corresponding to the plurality of support images of the target object of the same category are averaged to obtain a feature mapping heat map of the target object of the same category.

[0091] In the embodiment, there can be N (N>2) support images with the same category target object, and N feature mapping heat maps can be obtained through the above steps, and the N feature mapping heat maps are averaged to be combined into one feature mapping heat map, and the combined feature mapping heat map is used as the feature mapping heat map of the target object of the same category.

[0092] Further, in the embodiment of the present application, step S15 can be implemented in the following way.

[0093] First, based on the continuous highlight area in the feature mapping heat map, the contour line of the target object in the feature mapping heat map is obtained.

[0094] Wherein, the brightness of the feature mapping heat map represents the similarity of the target object in the support image and the target object in the query image, wherein the brighter the brightness represents the more similar the two are. The continuous highlight area represents that this area is a region with high matching degree between the support image and the query image.

[0095] Next, the minimum circumscribed rectangle circumscribing the contour line of the target object is calculated, and the minimum circumscribed rectangle is taken as the candidate box.

[0096] The candidate box is used to cut the target subgraph from the query image.

[0097] Further, in the embodiment of the present application, step S16 can be implemented in the following way.

[0098] First, the inner product of the feature vector of the target subgraph and the feature vector of the support image is calculated to obtain the cosine distance of the feature vector of the target subgraph and the feature vector of the support image.

[0099] Then, the cosine distance is used to represent the feature similarity of the target object in the query image and the target object in the support image.

[0100] Wherein, the greater the cosine distance represents the higher the feature similarity of the target object in the query image and the target object in the support image.

[0101] Further, in the embodiment, step S17 can be implemented in the following way.

[0102] The support image corresponding to the maximum cosine distance is taken as the target support image, the frame information and the category of the target support image are taken as the frame information and the category of the target object in the query image, and the frame information and the category of the target object in the query image are output.

[0103] Please refer to Figure 3 , Figure 3A function module schematic diagram of the image labeling device 300 provided by the embodiment of the present application, the embodiment of the present application can divide the function modules of the image labeling device 300 according to the method embodiment executed by the server, that is, the following various function modules corresponding to the image labeling device 300 can be used to execute the various method embodiments described above. Wherein, the image labeling device 300 can include a first feature extraction module 310, a second feature extraction module 320, a first calculation module 330, a second calculation module 340, a determination and interception module 350, an extraction and determination module 360 and an output module 370, and the functions of each function module of the image labeling device 300 will be described in detail below.

[0104] The first feature extraction module 310 is configured to perform image feature extraction on the support images in the support set to obtain feature vectors of the support images and feature maps of the support images.

[0105] The support set is composed of support images labeled by human beings, and the support images include labeling information. In detail, the labeling information can include the category of the target object in the support image and the bounding box information used to enclose the target object from the support image. Each support image includes a category of target object, and the bounding box information can be the position coordinates of the target object, for example, when the bounding box is a rectangular bounding box, the bounding box information can be the position coordinates of the four vertices of the bounding box.

[0106] In the embodiment, the first feature extraction module 310 can be used to execute the step S11 described above, and the detailed implementation of the first feature extraction module 310 can be referred to the detailed description of the step S11.

[0107] The second feature extraction module 320 is configured to perform image feature extraction on the query images in the query set to obtain feature maps of the query images.

[0108] In the embodiment, the second feature extraction module 320 can be used to execute the step S12 described above, and the detailed implementation of the second feature extraction module 320 can be referred to the detailed description of the step S12.

[0109] The first calculation module 330 is configured to obtain a first inner product transformation result with the same shape as the feature map of the query image based on the feature map of the support image and the feature map of the query image.

[0110] In the embodiment, the first calculation module 330 can be used to execute the step S13 described above, and the detailed implementation of the first calculation module 330 can be referred to the detailed description of the step S13.

[0111] The second calculation module 340 is configured to obtain a feature mapping heat map of the same category target object based on the first inner product transformation result.

[0112] In this embodiment, the second calculation module 340 can be configured to perform the step S14 described above, and the detailed implementation of the second calculation module 340 can refer to the detailed description of the step S14.

[0113] The determination and interception module 350 is configured to determine a candidate box from the feature mapping heat map, and intercept a target sub-image in the query image by using the candidate box.

[0114] In this embodiment, the determination and interception module 350 can be configured to perform the step S15 described above, and the detailed implementation of the determination and interception module 350 can refer to the detailed description of the step S15.

[0115] The extraction and determination module 360 is configured to extract a feature vector of the target sub-image, and determine a feature similarity between the target object in the query image and the target object in the support image based on the feature vector of the target sub-image and the feature vector of the support image.

[0116] In this embodiment, the extraction and determination module 360 can be configured to perform the step S16 described above, and the detailed implementation of the extraction and determination module 360 can refer to the detailed description of the step S16.

[0117] The output module 370 is configured to output the bounding box information and the category of the target object in the query image according to the feature similarity between the target object in the query image and the target object in the support image.

[0118] In this embodiment, the output module 370 can be configured to perform the step S17 described above, and the detailed implementation of the output module 370 can refer to the detailed description of the step S17.

[0119] It should be noted that the division of each module in the above device or system is only a logical function division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. Moreover, all the modules can be realized in the form of software (such as open source software) that can be called by a processor; all the modules can also be realized in the form of hardware; some modules can be realized in the form of software called by a processor, and some modules can be realized in the form of hardware. As an example, the extraction and determination module 360 can be realized by a single processor, and the functions of the extraction and determination module 360 can be stored in the memory of the device or system in the form of program code, and called and executed by a processor of the device or system. The implementation of other modules is similar, and will not be described here.

[0120] Please refer to Figure 4 , Figure 4 A schematic diagram of the hardware structure of an electronic device 100 for implementing the image annotation method described above, according to an embodiment of this disclosure, is shown. Figure 4 As shown, the electronic device 100 may include a processor 110, a computer-readable storage medium 120, a bus 130, and a communication unit 140.

[0121] In a specific implementation, the processor 110 executes computer-executable instructions (e.g., instructions stored in the computer-readable storage medium 120) stored in the computer-readable storage medium 120. Figure 3 The various modules in the image annotation apparatus 300 shown above enable the processor 110 to execute the image annotation method as described in the above method embodiment, wherein the processor 110 and the computer-readable storage medium 120 can be connected via a bus 130.

[0122] The specific implementation process of processor 110 can be found in the various method embodiments executed by the above-mentioned electronic device 100. The implementation principle and technical effect are similar, and will not be repeated here in the embodiments of this application.

[0123] The computer-readable storage medium 120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 110 is used to store programs or data.

[0124] Bus 130 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the buses in the accompanying drawings of this application are not limited to only one bus or one type of bus.

[0125] In the interactive scenario provided in the embodiments of this application, the communication unit 140 can be used to communicate with the electronic device 100 and the image acquisition device 200 to realize data interaction between the electronic device 100 and the image acquisition device 200. For example, the electronic device 100 can acquire the image captured by the image acquisition device 200 through the communication unit 140.

[0126] In addition, the embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the image labeling method is realized.

[0127] To sum up, the image labeling method, device and electronic equipment provided by the embodiment of the present application, first, based on the feature map of the support image and the feature map of the query image, the first inner product transformation result with the same shape as the feature map of the query image is obtained; then, based on the first inner product transformation result, the feature mapping heat map of the target object of the same category is obtained; then, the candidate frame is determined from the feature mapping heat map, and the target subgraph is intercepted in the query image by using the candidate frame; then, based on the feature vector of the target subgraph and the feature vector of the support image, the feature similarity of the target object in the query image and the target object in the support image is determined; finally, according to the feature similarity, the bounding box information and the category of the target object in the query image are output. The above scheme can automatically label a large number of query images through the labeled support image, and the efficiency and accuracy of labeling are higher than those of the prior art.

[0128] The above-described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided in the drawings is not intended to limit the protection scope of the present application, but only represents selected embodiments of the present application. Based on this, the protection scope of the present application should be subject to the protection scope of the claims. In addition, based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor shall belong to the scope of protection of the present application.

Claims

1. An image annotation method, characterized in that, The method includes: Image features are extracted from the support images in the support set to obtain the feature vector and feature map of the support images. The support images include annotation information, which includes the category of the target object in the support image and the border information used to delineate the target object from the support image. Image features are extracted from the query images in the query set to obtain the feature map of the query images; Based on the feature map of the supporting image and the feature map of the query image, a first inner product transformation result with the same shape as the feature map of the query image is obtained; Based on the first inner product transformation result, a feature mapping heatmap of target objects of the same category is obtained; Candidate boxes are determined from the feature mapping heatmap, and target sub-images are extracted from the query image using the candidate boxes; Based on the feature map of the supporting image and the feature map of the target sub-image, a second inner product transformation result with the same shape as the feature map of the target sub-image is obtained; Update the feature map heatmap of the same category based on the result of the second inner product transformation; The candidate boxes are updated based on the updated feature map heatmap, and the target sub-image is re-extracted from the query image using the updated candidate boxes. Extract the feature vector of the target sub-image, and determine the feature similarity between the target object in the query image and the target object in the support image based on the feature vector of the target sub-image and the feature vector of the support image; Based on the feature similarity between the target object in the query image and the target object in the supporting image, the bounding box information and category of the target object in the query image are output.

2. The image annotation method as described in claim 1, characterized in that, The step of extracting image features from the support images in the support set to obtain the feature vector and feature map of the support images includes: The support image in the support set is input into the image feature extraction model, which outputs the feature vector and feature map of the support image, and sets the area outside the bounding box information defined in the feature map of the support image to 0; The step of extracting image features from the query images in the query set to obtain the feature map of the query images includes: The query images in the query set are input into the image feature extraction model, and the feature map of the query images is output.

3. The image annotation method as described in claim 1, characterized in that, The step of obtaining a first inner product transformation result with the same shape as the feature map of the query image based on the feature map of the supporting image and the feature map of the query image includes: Vector normalization is performed on all points in the feature map of the supporting image and the feature map of the query image; Calculate the inner product of the feature map of the supporting image after vector normalization and the feature map of the query image, wherein the inner product has a dimension of four. After subtracting the mean of the last two dimensions of the inner product from the inner product, the average is calculated over the first two dimensions of the inner product to obtain a first inner product transformation result with the same shape as the feature map of the query image.

4. The image annotation method as described in claim 3, characterized in that, The step of obtaining the feature mapping heatmap of target objects of the same category based on the first inner product transformation result includes: The first inner product transformation result is processed by clamping points with values ​​less than 0 to 0 and linearly transforming points with values ​​greater than or equal to 0 to a set numerical range. The first inner product transformation result is processed by image interpolation to obtain a feature mapping heatmap; The feature mapping heatmaps corresponding to multiple support images of the same type of target object are averaged to obtain the feature mapping heatmap of the same type of target object.

5. The image annotation method as described in claim 4, characterized in that, The step of determining candidate boxes from the feature mapping heatmap and using the candidate boxes to extract a target sub-image from the query image includes: Based on the continuous highlighted areas in the feature mapping heatmap, the outline of the target object in the feature mapping heatmap is obtained; Calculate the minimum bounding rectangle that circumscribes the outline of the target object, and use the minimum bounding rectangle as the candidate bounding box; The target sub-image is extracted from the query image using the candidate bounding box.

6. The image annotation method as described in claim 5, characterized in that, The step of extracting the feature vector of the target sub-image and determining the feature similarity between the target object in the query image and the target object in the support image based on the feature vector of the target sub-image and the feature vector of the support image includes: The inner product of the feature vectors of the target sub-image and the support image is used to obtain the cosine distance between the feature vectors of the target sub-image and the support image. The cosine distance is used to represent the feature similarity between the target object in the query image and the target object in the support image. The larger the cosine distance, the higher the feature similarity between the target object in the query image and the target object in the support image.

7. The image annotation method as described in claim 6, characterized in that, The step of outputting the bounding box information and category of the target object in the query image based on the feature similarity between the target object in the query image and the target object in the support image includes: The support image corresponding to the maximum cosine distance is used as the target support image. The border information and category of the target support image are used as the border information and category of the target object in the query image. The border information and category of the target object in the query image are then output.

8. An image annotation device, characterized in that, The device includes: The first feature extraction module is used to extract image features from the support images in the support set to obtain the feature vector and feature map of the support images. The support images include annotation information, which includes the category of the target object in the support image and the border information used to delineate the target object from the support image. The second feature extraction module is used to extract image features from the query images in the query set to obtain the feature map of the query images; The first calculation module is used to obtain a first inner product transformation result with the same shape as the feature map of the query image, based on the feature map of the supporting image and the feature map of the query image. The second calculation module is used to obtain a feature mapping heatmap of target objects of the same category based on the result of the first inner product transformation. The determination and cropping module is used to determine candidate boxes from the feature map heatmap and crop a target sub-image from the query image using the candidate boxes; based on the feature map of the supporting image and the feature map of the target sub-image, a second inner product transformation result with the same shape as the feature map of the target sub-image is obtained; the feature map heatmap of the same category is updated based on the second inner product transformation result; the candidate boxes are updated based on the updated feature map heatmap, and the target sub-image is cropped again from the query image using the updated candidate boxes; The extraction and determination module is used to extract the feature vector of the target sub-image and determine the feature similarity between the target object in the query image and the target object in the support image based on the feature vector of the target sub-image and the feature vector of the support image. The output module is used to output the bounding box information and category of the target object in the query image based on the feature similarity between the target object in the query image and the target object in the support image.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication unit, and a computer-readable storage medium. The processor and the computer-readable storage medium are connected via a bus system. The communication unit is used to connect with an image acquisition device to achieve data interaction between them. The computer-readable storage medium is used to store programs, instructions, or code. The processor is used to execute the programs, instructions, or code in the computer-readable storage medium to implement the image annotation method according to any one of claims 1-7.

Images