Image classification label generation method, device and electronic device

By scaling the image and extracting different receptive field features, and generating and eliminating conflicting labels, the problems of low efficiency and inequality of manual labeling are solved, and efficient and accurate generation of automatic image classification labels are achieved.

CN118279648BActive Publication Date: 2024-12-06ADDX (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410350842.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-26
Publication Date
2024-12-06
Estimated Expiration
2044-03-26

AI Technical Summary

Technical Problem

In the prior art, manual image classification labeling is inefficient and of uneven quality, especially when image size differences and complex structures, labeling errors, omissions and instance label conflicts are prone to occur.

Method used

By scaling the target image, generating a zoomed image collection, and using feature extraction models of different receptive fields to extract image features, generating instance information and tag groups, and finally excluding conflicting instance tags to realize automatic image classification tag generation .

Benefits of technology

It improves the labeling efficiency and quality, reduces labeling errors and omissions, solves the problem of instance label conflicts, and achieves more accurate image classification label generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118279648B_ABST
    Figure CN118279648B_ABST
Patent Text Reader

Abstract

The embodiments of the present disclosure disclose an image classification label generation method, device and electronic device. A specific implementation of the method includes: scaling the target image to obtain a scaled image set; extracting image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set; generating an instance information set based on the first image feature set and the second image feature set; for each instance information in the instance information set, generating an instance label group based on the instance type and instance feature map included in the instance information; removing conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image. This implementation realizes automatic image classification label generation for the target image, improving annotation efficiency and annotation quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, and electronic device for generating image classification labels. Background Art

[0002] Image classification labeling is a technology used to label images with attribute labels. By generating image classification labels, image preprocessing with label granularity can be achieved. At present, when generating image classification labels, the commonly used method is to generate image classification labels corresponding to pictures through manual annotation.

[0003] However, when the above method is adopted, the following technical problems often occur:

[0004] First, the manual labeling of image classification labels is inefficient and the quality of manual labeling varies.

[0005] Second, image sizes often vary, especially for smaller images, which are prone to labeling errors or omissions when labeling instances corresponding to instances.

[0006] Third, when the image structure is complex, that is, it contains many instances, instance label conflicts may occur, thus affecting the quality of instance label annotation.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0009] Some embodiments of the present disclosure propose image classification label generation methods, devices, and electronic devices to solve one or more of the technical problems mentioned in the above background technology section.

[0010] In a first aspect, some embodiments of the present disclosure provide a method for generating an image classification label, the method comprising: performing image scaling on a target image to obtain a scaled image set, wherein the target image is an image to be generated with a corresponding image classification label, and the scaled images in the scaled image set have different scaling factors; performing image feature extraction on each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by extracting features from the scaled image by a first feature extraction model, and the second image feature is obtained by extracting features from the first image feature set by a second feature extraction model, and there is a one-to-one correspondence between the first image feature and the second image feature; generating an instance information set based on the first image feature set and the second image feature set, wherein the instance information in the instance information set includes: an instance type and an instance feature map, and the instance feature map represents the local image features of the instance corresponding to the instance information; for each instance information in the instance information set, generating an instance label group based on the instance type and the instance feature map included in the instance information; removing conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image.

[0011] In a second aspect, some embodiments of the present disclosure provide an image classification label generating device, the device comprising: an image scaling unit, configured to perform image scaling on a target image to obtain a scaled image set, wherein the target image is an image for which a corresponding image classification label is to be generated, and the scaled images in the scaled image set have different scaling factors; an image feature extraction unit, configured to perform image feature extraction on each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by performing feature extraction on the scaled image by a first feature extraction model, and the second image feature is obtained by performing feature extraction on the first image feature set by a second feature extraction model. It is obtained by feature extraction that there is a one-to-one correspondence between the first image feature and the second image feature; the first generating unit is configured to generate an instance information set according to the above-mentioned first image feature set and the above-mentioned second image feature set, wherein the instance information in the above-mentioned instance information set includes: an instance type and an instance feature map, and the instance feature map represents the local image features of the instance corresponding to the instance information; the second generating unit is configured to generate an instance label group for each instance information in the above-mentioned instance information set according to the instance type and the instance feature map included in the above-mentioned instance information; the conflicting instance label elimination unit is configured to eliminate the conflicting instance labels of the obtained instance label group set to obtain the image classification label set corresponding to the above-mentioned target image.

[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0014] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the image label generation method of some embodiments of the present disclosure, the labeling efficiency and labeling quality are improved. Specifically, the reason for the low labeling efficiency and labeling quality is that the labeling efficiency is low when the image classification label is manually labeled, and the labeling quality of the manual labeling method is uneven. Based on this, the image classification label generation method of some embodiments of the present disclosure first scales the target image to obtain a scaled image set, wherein the target image is an image to be generated with a corresponding image classification label, and the scaled images in the scaled image set have different scaling factors. Scaling facilitates the subsequent image feature extraction from different receptive fields to improve the robustness of the feature extraction process. Secondly, image feature extraction is performed on each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by extracting the scaled image by a first feature extraction model, and the second image feature is obtained by extracting the first image feature set by a second feature extraction model, and there is a one-to-one correspondence between the first image feature and the second image feature. In this way, the first image feature set and the second image feature set for the target image are obtained. Secondly, based on the above-mentioned first image feature set and the above-mentioned second image feature set, an instance information set is generated, wherein the instance information in the above-mentioned instance information set includes: instance type and instance feature map, and the instance feature map represents the local image features of the instance corresponding to the instance information. In this way, the instance contained in the target image is identified. Then, for each instance information in the above-mentioned instance information set, an instance label group is generated according to the instance type and instance feature map included in the above-mentioned instance information. In this way, instance labels are generated (labeled) for the target image from the instance granularity. Finally, conflicting instance labels are eliminated from the obtained instance label group set to obtain the image classification label set corresponding to the above-mentioned target image. In practice, there may be instance label conflicts between different instances, so it is necessary to eliminate conflicting instance labels. In this way, automatic image classification label generation for the target image is realized, and the labeling efficiency and labeling quality are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1 is a flowchart of some embodiments of the image classification label generation method according to the present disclosure;

[0017] Figure 2 is a schematic diagram of the structure of some embodiments of the image classification label generation device according to the present disclosure;

[0018] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] refer to Figure 1, shows a process 100 of some embodiments of image classification label generation according to the present disclosure. The image classification label generation includes the following steps:

[0026] Step 101: scaling the target image to obtain a scaled image set.

[0027] In some embodiments, the execution entity (e.g., a computing device) of the image classification label generation may scale the target image to obtain a set of scaled images. The target image is an image for which a corresponding image classification label is to be generated. The image classification label is used to label the content contained in the target image. The scaled images in the scaled image set correspond to different scaling factors. The scaling factor is used to control the scaling ratio of the target image. For example, the scaling factor may be 0.25, which represents reducing the target image to 1 / 4. In practice, the above-mentioned execution entity may preset a set of scaling factors for scaling the target image. For example, the set of scaling factors may be [1, 0.75, 0.5, 0.25].

[0028] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here. It should be understood that the number of computing devices can be any number according to the implementation needs.

[0029] In some optional implementations of some embodiments, the execution subject scales the target image to obtain a scaled image set, which may include the following steps:

[0030] The first step is to determine the image size ratio between the target image and the reference image.

[0031] In practice, the execution subject may use the ratio of the image size of the target image to the image size of the reference image as the image size ratio. For example, if the image size of the target image is A and the image size of the reference image is B, the image size ratio is A / B.

[0032] In the second step, each initial scaling factor in the initial scaling factor set is updated according to the above image size ratio to generate an updated scaling factor, thereby obtaining an updated scaling factor set.

[0033] The initial scaling factor set may be a scaling factor for scaling the image with reference to the reference image. For example, the initial scaling factor set may be [2, 1.75, 1.5, 1.25, 1, 0.75, 0.5, 0.25]. In practice, the execution entity may use the ratio of the initial scaling factor to the image size ratio as the updated scaling factor. For example, the initial scaling factor may be 2, the image size ratio may be 0.8, and the updated scaling factor may be 2.5.

[0034] In the third step, for each updated scaling factor in the above updated scaling factor set, perform the following processing steps:

[0035] In a first sub-step, in response to the updated scaling factor being greater than or equal to 1, the target image is enlarged according to the updated scaling factor to obtain a scaled image in the scaled image set.

[0036] In practice, the execution subject may use the updated scaling factor as the magnification factor to magnify the target image and obtain a corresponding scaled image.

[0037] The second sub-step is, in response to the updated scaling factor being less than 1, reducing the target image according to the updated scaling factor to obtain a scaled image in the scaled image set.

[0038] In practice, the execution subject may reduce the target image by using the updated scaling factor as the reduction factor to obtain a corresponding scaled image.

[0039] Step 102: extract image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set.

[0040] In some embodiments, the execution subject may perform image feature extraction on each scaled image in the scaled image set to generate a first image feature set and a second image feature set. The first image feature is obtained by extracting features from the scaled image using a first feature extraction model. The second image feature is obtained by extracting features from the first image feature set using a second feature extraction model, and there is a one-to-one correspondence between the first image feature and the second image feature. For example, both the first feature extraction model and the second feature extraction model may be CNN (Convolutional Neural Networks) models. In practice, first, the scaled image is subjected to feature extraction using the first feature extraction model to obtain the first image feature. Then, the first image feature is further subjected to feature extraction using the second feature extraction model to obtain the second image feature.

[0041] Optionally, the first image feature extraction model includes: a first sub-image feature extraction model, a second sub-image feature extraction model and a third sub-image feature extraction model. Among them, the model receptive field of the first sub-image feature extraction model is larger than the model receptive field of the second sub-image feature extraction model. The model receptive field of the second sub-image feature extraction model is larger than the model receptive field of the third sub-image feature extraction model. The first sub-image feature extraction model, the second sub-image feature extraction model and the third sub-image feature extraction model all adopt the FPN (Feature Pyramid Networks) model architecture. Specifically, the first sub-image feature extraction model, the second sub-image feature extraction model and the third sub-image feature extraction model are set in parallel. In practice, the first sub-image feature extraction model includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer. Specifically, the model structures of the second sub-image feature extraction model, the third sub-image feature extraction model and the first sub-image feature extraction model are consistent. For example, the second sub-image feature extraction model also includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer and a fifth convolution layer.

[0042] Optionally, the second feature extraction model is a feature extraction model based on the Transformer structure. The second feature extraction model includes: an encoder and a decoder set, the number of decoders in the decoder set is consistent with the number of scaled images in the scaled image set, and there is a one-to-one correspondence between the decoder and the first image feature. In practice, the decoder set may also include one decoder, and for different first image features, the second feature extraction model may be controlled by soft weights. For example, for the first image feature A, soft weight A is used. For the first image feature B, soft weight B is used.

[0043] In some optional implementations of some embodiments, the execution subject extracts image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set, which may include the following steps:

[0044] In the first step, the scaled image is convolved through the first convolutional layer to generate a first convolutional feature.

[0045] In the second step, the first convolution feature is convolved through the second convolution layer to generate a second convolution feature.

[0046] In the third step, the second convolution feature is convolved through the third convolution layer to generate a third convolution feature.

[0047] In the fourth step, the third convolution feature is convolved through the fourth convolution layer to generate a fourth convolution feature.

[0048] In the fifth step, the fourth convolution feature is convolved through the fifth convolution layer to generate a fifth convolution feature.

[0049] In the sixth step, the first convolution feature, the second convolution feature, the third convolution feature, the fourth convolution feature and the fifth convolution feature are concatenated to obtain a first candidate convolution feature.

[0050] The seventh step is to generate a second candidate convolution feature based on the second sub-image feature extraction model and the scaled image.

[0051] Among them, the method of generating the second candidate convolution feature can refer to the first to sixth steps, which will not be repeated here.

[0052] In the eighth step, a third candidate convolution feature is generated based on the third sub-image feature extraction model and the scaled image.

[0053] Among them, the method of generating the third candidate convolution feature can refer to the first to sixth steps, which will not be repeated here.

[0054] In the ninth step, the first candidate convolution feature, the second candidate convolution feature and the third candidate convolution feature are superimposed to obtain a first image feature in the first image feature set corresponding to the scaled image.

[0055] Step 10: For each first image feature in the above first image feature set, perform the following second image feature generation step:

[0056] In the first sub-step, the encoder is used to perform feature encoding on the first image feature to obtain the encoded first image feature.

[0057] In practice, first, the execution subject can divide the first image feature into M non-overlapping patches. For example, M can be 9, that is, the first image feature is divided into 3×3, a total of nine patches. Secondly, for each of the M patches, the execution subject can convert the patch into a one-dimensional vector by horizontal scanning. Then, the one-dimensional vector corresponding to the patch is linearly projected to obtain the projected one-dimensional vector. Among them, linear projection is used to compress the feature dimension. Further, according to the obtained one-dimensional vector sequence, feature extraction is performed through N serially connected Transformer encoders included in the encoder. Specifically, the multi-head attention mechanism module is used for feature extraction between patches. The Feed Forward network is used to perform feature changes on the patch.

[0058] The second sub-step is to decode the encoded first image feature by a decoder in the decoder set corresponding to the first image feature to obtain a second image feature in the second image feature set corresponding to the first image feature.

[0059] The content of "optionally" and "in some optional implementations of some embodiments" in step 102, as an inventive point of the present disclosure, solves the second technical problem mentioned in the background technology, namely, "image sizes often differ, especially when smaller-sized images are annotated with instance labels corresponding to instances, it is very easy to have problems such as labeling errors or labeling omissions." Based on this, firstly, the present disclosure adopts a first sub-image feature extraction model, a second sub-image feature extraction model, and a third sub-image feature extraction model with different receptive fields to extract image features under different receptive fields for each scaled image in the scaled image set. Then, a second feature extraction model is set up, and by implementing the second feature extraction model based on the Transformer structure, further feature extraction of the first sub-image features under the self-attention mechanism is achieved. In this way, image feature extraction can be better performed, the quality of the extracted image features is improved, and the accuracy of subsequent instance labeling is further improved.

[0060] Step 103: Generate an instance information set according to the first image feature set and the second image feature set.

[0061] In some embodiments, the execution subject may generate an instance information set based on the first image feature set and the second image feature set. The instance information in the instance information set includes: an instance type and an instance feature map. The instance feature map represents the local image features of the instance corresponding to the instance information. The instance type represents the type of instance contained in the target image. In practice, the execution subject may generate the instance information set through a YOLO model.

[0062] In some optional implementations of some embodiments, the execution subject generates the instance information set according to the first image feature set and the second image feature set, which may include the following steps:

[0063] In the first step, for each first image feature in the first image feature set, a second image feature in the second image feature set corresponding to the first image feature is cross-multiplied with the first image feature to obtain an image feature after the cross-multiplication operation.

[0064] As an example, the first image feature set may be [first image feature A 1 , the first image feature A 3 , the first image feature A2 The second image feature set may be [second image feature B 1 , the second image feature B 3 , the second image feature B 2 ] Among them, the first image feature A 1 and the second image feature B 1 There is a corresponding relationship. The first image feature A 2 and the second image feature B 2 There is a corresponding relationship. The first image feature A 3 and the second image feature B 3 There is a corresponding relationship. The first image feature A 1 and the second image feature B 1 Cross product, get the image feature C after the cross product operation 1 The first image feature A 2 and the second image feature B 2 Cross product, get the image feature C after the cross product operation 2 The first image feature A 3 and the second image feature B 3 Cross product, get the image feature C after the cross product operation 3 .

[0065] In the second step, each post-cross-product image feature in the obtained post-cross-product image feature set is superimposed to obtain a region segmentation map.

[0066] The region segmentation map includes: a segmentation region set. The segmentation region represents an image after instance recognition and instance segmentation are performed on the target image.

[0067] The third step is to perform approximate segmentation region fusion on the segmentation regions in the above segmentation region set to obtain a fused segmentation region set.

[0068] Among them, the above-mentioned execution subject can perform approximate segmentation region fusion on the segmentation regions in the above-mentioned segmentation region set through the mask prediction branch to obtain the fused segmentation region set. In practice, the mask prediction branch includes: 5 layers of serially connected convolution layers. Specifically, the mask prediction branch includes: convolution layer A, convolution layer B, convolution layer C, convolution layer D and convolution layer E. Among them, the feature scale corresponding to convolution layer A is the same as the feature scale corresponding to convolution layer B. The feature scale corresponding to convolution layer B is smaller than the feature scale corresponding to convolution layer C. The feature scale corresponding to convolution layer C, convolution layer D and convolution layer E are the same.

[0069] Step 4: For each fused segmented region in the above fused segmented region set, perform the following instance information generation steps:

[0070] The first sub-step is to extract the local area features in the above-mentioned region segmentation map corresponding to the above-mentioned fused segmented region as the instance feature map included in the instance information corresponding to the above-mentioned fused segmented region.

[0071] In practice, the execution subject may intercept the local region features corresponding to the fused segmented region in the region segmentation map by projection as the instance feature map included in the instance information corresponding to the fused segmented region.

[0072] The second sub-step is to generate the instance type included in the instance information corresponding to the fused segmented region according to the instance feature map and instance type predictor corresponding to the fused segmented region.

[0073] Among them, the instance type predictor includes: 1 convolutional layer, 2 fully connected layers, a category classifier and a bounding box detector.

[0074] Step 104: for each instance information in the instance information set, generate an instance label group according to the instance type and instance feature graph included in the instance information.

[0075] In some embodiments, for each instance information in the instance information set, the above-mentioned execution subject can generate an instance label group according to the instance type and instance feature map included in the instance information. Among them, the instance label is an attribute label for the instance corresponding to the instance information. For example, if the instance type corresponding to the instance information is a cup contained in the target image, then the instance label group can be {cup shape: cylindrical; whether the cup has a handle: with; cup color: black}. In practice, the above-mentioned execution subject can generate a corresponding instance label group according to the instance type and instance feature map included in the instance information through multiple classifiers.

[0076] In some optional implementations of some embodiments, generating an instance label group according to the instance type and the instance feature graph included in the above instance information may include the following steps:

[0077] The first step is to perform type encoding on the above instance type to obtain the instance type code.

[0078] In practice, since instance types are limited, one-hot encoding can be used to perform type encoding on the above instance types to obtain instance type encoding.

[0079] The second step is to classify the model based on the above instance type encoding, instance feature map and instance label.

[0080] Among them, the instance label prediction model includes: a feature fusion network and an instance label classifier set. In practice, since the instance type is encoded as a one-dimensional feature vector, which is different from the feature dimension of the instance feature map, first, the above-mentioned execution subject can downsample the instance feature map through the downsampling network included in the feature fusion network to obtain the downsampled instance feature map. Then, the above-mentioned execution subject can input the downsampled instance feature map into the first fully connected layer included in the feature fusion network to obtain a one-dimensional vector. Next, the above-mentioned execution subject can concatenate the one-dimensional vector and the instance type encoding to obtain a concatenated vector, and finally, input the concatenated vector into each instance label classifier in the above-mentioned instance label classifier set to obtain an instance label group. In practice, the instance label classifier is implemented using a fully connected layer.

[0081] Step 105 , removing conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image.

[0082] In some embodiments, the above-mentioned execution subject can eliminate conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image. In practice, there may be problems of misrecognition when classifying instance labels. For example, the target image includes: an instance "water cup" and an instance "person". The instance label group corresponding to the instance "water cup" can be {cup shape: cylindrical; whether the cup has a handle: no; cup color: black}. The instance label group corresponding to the instance "person" can be {whether to grasp the handle: yes}. Among them, "whether the cup has a handle: no" and "whether to grasp the handle: yes" conflict. Therefore, it is necessary to eliminate instance labels with conflicting relationships.

[0083] In some optional implementations of some embodiments, the execution subject removes conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image, which may include the following steps:

[0084] In the first step, according to the instance label group set, perform the following conflicting instance label removal steps:

[0085] In the first sub-step, the instance tag group set is split into a first instance tag group set and a second instance tag group set.

[0086] As an example, the instance tag group set may be [instance tag group A, instance tag group B, instance tag group C], and thus the first instance tag group may be instance tag group A. The second instance tag group set may be [instance tag group B, instance tag group C].

[0087] In the second sub-step, the first instance label with instance label conflict in the first instance label set is determined as the first instance label to be eliminated through the instance label conflict classification model, thereby obtaining the first instance label set to be eliminated.

[0088] The instance label conflict classification model is a binary classification model. The classification categories of the instance label conflict classification model include: conflict category and non-conflict category.

[0089] The third sub-step is to determine the complement of the first instance label set and the first to-be-eliminated instance label set as the third instance label set.

[0090] In the fourth sub-step, the third instance label in the second instance label group set that has an instance label conflict with the third instance label set is determined through the instance label conflict classification model as the second instance label to be eliminated, thereby obtaining the second instance label set to be eliminated.

[0091] The fifth sub-step is to determine the difference between each second instance label group in the second instance label group set and the second to-be-eliminated instance label set as the fourth instance label group, thereby obtaining the fourth instance label group set.

[0092] In a sixth sub-step, in response to the fourth instance label group set including a target number of fourth instance label groups, the third instance label in the third instance label set and the fourth instance label in the fourth instance label group set are determined as image classification labels in the above-mentioned image classification label set.

[0093] Among them, the number of the above targets is 1.

[0094] In the second step, in response to the number of fourth instance label groups included in the fourth instance label group set being greater than the above target number, the fourth instance label group set is determined as the instance label group set, and the third instance label in the third instance label set is determined as the image classification label in the above image classification label set, and the above conflicting instance label elimination step is performed again.

[0095] The above-mentioned content of "in some optional implementations of some embodiments" as an inventive point of the present disclosure solves the third technical problem mentioned in the background technology, namely, "when the image structure is complex, that is, it contains many instances, the problem of instance label conflict may occur, thereby affecting the quality of instance label annotation". Therefore, firstly, the present disclosure splits the instance label group set into a first instance label group and a second instance label group set. Then, through the instance label conflict classification model, the first instance label in the first instance label set with instance label conflict is determined as the first instance label to be eliminated, and the first instance label set to be eliminated is obtained. That is, it is determined whether there are conflicting instance labels in the instance labels corresponding to an instance. Then, the complement of the first instance label set and the first instance label set to be eliminated is determined as the third instance label set. In this way, the elimination of the conflicting instance labels in an instance label is realized. Further, through the instance label conflict classification model, the third instance label in the second instance label group set with instance label conflict with the third instance label set is determined as the second instance label to be eliminated, and the second instance label set to be eliminated is obtained. In this way, it is determined whether there are conflicting instance labels between different instances. In addition, the difference between each second instance label group in the second instance label group set and the second instance label set to be eliminated is determined as the fourth instance label group, and the fourth instance label group set is obtained. By determining the difference, conflicting instance labels between different instances are eliminated. In this way, the accuracy of the image classification label set for the target image is guaranteed. In this way, the quality of the instance label annotation is guaranteed.

[0096] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the image label generation method of some embodiments of the present disclosure, the labeling efficiency and labeling quality are improved. Specifically, the reason for the low labeling efficiency and labeling quality is that the labeling efficiency is low when the image classification label is manually labeled, and the labeling quality of the manual labeling method is uneven. Based on this, the image classification label generation method of some embodiments of the present disclosure first scales the target image to obtain a scaled image set, wherein the target image is an image to be generated with a corresponding image classification label, and the scaled images in the scaled image set have different scaling factors. Scaling facilitates the subsequent image feature extraction from different receptive fields to improve the robustness of the feature extraction process. Secondly, image feature extraction is performed on each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by extracting the scaled image by a first feature extraction model, and the second image feature is obtained by extracting the first image feature set by a second feature extraction model, and there is a one-to-one correspondence between the first image feature and the second image feature. In this way, the first image feature set and the second image feature set for the target image are obtained. Secondly, based on the above-mentioned first image feature set and the above-mentioned second image feature set, an instance information set is generated, wherein the instance information in the above-mentioned instance information set includes: instance type and instance feature map, and the instance feature map represents the local image features of the instance corresponding to the instance information. In this way, the instance contained in the target image is identified. Then, for each instance information in the above-mentioned instance information set, an instance label group is generated according to the instance type and instance feature map included in the above-mentioned instance information. In this way, instance labels are generated (labeled) for the target image from the instance granularity. Finally, conflicting instance labels are eliminated from the obtained instance label group set to obtain the image classification label set corresponding to the above-mentioned target image. In practice, there may be instance label conflicts between different instances, so it is necessary to eliminate conflicting instance labels. In this way, automatic image classification label generation for the target image is realized, and the labeling efficiency and labeling quality are improved.

[0097] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image classification label generation device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the image classification label generation device can be specifically applied to various electronic devices.

[0098] like Figure 2As shown, some embodiments of the image classification label generation device 200 include: an image scaling unit 201, an image feature extraction unit 202, a first generation unit 203, a second generation unit 204 and a conflict instance label removal unit 205. The image scaling unit 201 is configured to scale the target image to obtain a scaled image set, wherein the target image is an image to be generated with a corresponding image classification label, and the scaled images in the scaled image set have different scaling factors; the image feature extraction unit 202 is configured to extract image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by extracting features from the scaled image using a first feature extraction model, and the second image feature is obtained by extracting features from the first image feature set using a second feature extraction model, and the first image feature and the second image feature are extracted from each scaled image in the scaled image set. There is a one-to-one correspondence between the features; the first generation unit 203 is configured to generate an instance information set based on the above-mentioned first image feature set and the above-mentioned second image feature set, wherein the instance information in the above-mentioned instance information set includes: instance type and instance feature map, and the instance feature map represents the local image features of the instance corresponding to the instance information; the second generation unit 204 is configured to generate an instance label group for each instance information in the above-mentioned instance information set according to the instance type and instance feature map included in the above-mentioned instance information; the conflicting instance label elimination unit 205 is configured to eliminate conflicting instance labels from the obtained instance label group set to obtain the image classification label set corresponding to the above-mentioned target image.

[0099] It can be understood that the units recorded in the image classification label generation device 200 are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image classification label generation device 200 and the units included therein, and will not be repeated here.

[0100] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. Various programs and data required for the operation of the electronic device 300 are also stored in the random access memory 303. The processing device 301, the read-only memory 302, and the random access memory 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0102] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0103] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the read-only memory 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0104] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0105] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0106] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: scales the target image to obtain a scaled image set, wherein the target image is an image for which a corresponding image classification label is to be generated, and the scaled images in the scaled image set have different scaling factors; performs image feature extraction on each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by extracting the scaled image by a first feature extraction model, and the second image feature is obtained by extracting the scaled image by a second feature extraction model. Feature extraction is performed on the first image feature set, and it is obtained that there is a one-to-one correspondence between the first image feature and the second image feature; an instance information set is generated based on the first image feature set and the second image feature set, wherein the instance information in the instance information set includes: an instance type and an instance feature map, and the instance feature map represents the local image features of the instance corresponding to the instance information; for each instance information in the instance information set, an instance label group is generated based on the instance type and the instance feature map included in the instance information; conflicting instance labels are eliminated from the obtained instance label group set to obtain an image classification label set corresponding to the target image.

[0107] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an image scaling unit, an image feature extraction unit, a first generation unit, a second generation unit, and a conflicting instance label removal unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain cases, for example, the conflicting instance label removal unit may also be described as "a unit that removes conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the above-mentioned target image."

[0110] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for generating image classification labels, comprising: Scaling the target image to obtain a scaled image set, wherein the target image is an image for which a corresponding image classification label is to be generated, and the scaled images in the scaled image set have different scaling factors; Performing image feature extraction on each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by performing feature extraction on the scaled image by a first feature extraction model, and the second image feature is obtained by performing feature extraction on the first image feature set by a second feature extraction model, and there is a one-to-one correspondence between the first image feature and the second image feature; Generate an instance information set according to the first image feature set and the second image feature set, wherein the instance information in the instance information set includes: an instance type and an instance feature map, and the instance feature map represents a local image feature of an instance corresponding to the instance information; For each instance information in the instance information set, generating an instance label group according to the instance type and instance feature graph included in the instance information; Conflicting instance labels are removed from the obtained instance label group set to obtain an image classification label set corresponding to the target image, where: The step of removing conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image includes: Based on the instance label group set, perform the following conflicting instance label removal steps: Splitting the instance tag group set into a first instance tag group and a second instance tag group set; By using the instance label conflict classification model, determining the first instance label in the first instance label group that has the instance label conflict as the first instance label to be eliminated, and obtaining the first instance label set to be eliminated; Determine a complement of the first instance label group and the first to-be-eliminated instance label set as a third instance label set; By using the instance label conflict classification model, determining the third instance label in the second instance label group set that has an instance label conflict with the third instance label set as the second instance label to be eliminated, and obtaining the second instance label set to be eliminated; Determine a difference between each second instance label group in the second instance label group set and the second to-be-eliminated instance label set as a fourth instance label group, and obtain a fourth instance label group set; In response to the fourth instance label group set including a target number of fourth instance label groups, determining the third instance label in the third instance label set and the fourth instance label in the fourth instance label group set as image classification labels in the image classification label set, wherein the target number is 1; In response to the number of fourth instance label groups included in the fourth instance label group set being greater than the target number, the fourth instance label group set is determined as the instance label group set, and the third instance label in the third instance label set is determined as the image classification label in the image classification label set, and the conflicting instance label elimination step is performed again.

2. The method according to claim 1, wherein: The step of scaling the target image to obtain a scaled image set includes: Determining an image size ratio between the target image and the reference image; According to the image size ratio, each initial scaling factor in the initial scaling factor set is updated to generate an updated scaling factor, thereby obtaining an updated scaling factor set; For each updated scale factor in the set of updated scale factors, the following processing steps are performed: In response to the updated scaling factor being greater than or equal to 1, enlarging the target image according to the updated scaling factor to obtain a scaled image in the scaled image set; In response to the updated scaling factor being less than 1, the target image is scaled down according to the updated scaling factor to obtain a scaled image in the scaled image set.

3. The method according to claim 2, wherein: The step of generating an instance label group according to the instance type and the instance feature graph included in the instance information includes: Performing type coding on the instance type to obtain an instance type code; An instance label group corresponding to the instance information is generated according to the instance type code, the instance feature map and the instance label classification model, wherein the instance label classification model includes: a feature fusion network and an instance label classifier set.

4. The method according to claim 3, wherein: The first image feature extraction model includes: a first sub-image feature extraction model, a second sub-image feature extraction model and a third sub-image feature extraction model, the model receptive field of the first sub-image feature extraction model is larger than the model receptive field of the second sub-image feature extraction model, the model receptive field of the second sub-image feature extraction model is larger than the model receptive field of the third sub-image feature extraction model, and the first sub-image feature extraction model includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer; and The extracting image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set comprises: Performing convolution processing on the scaled image through the first convolution layer to generate a first convolution feature; Performing convolution processing on the first convolution feature through the second convolution layer to generate a second convolution feature; Performing convolution processing on the second convolution feature through the third convolution layer to generate a third convolution feature; Performing convolution processing on the third convolution feature through the fourth convolution layer to generate a fourth convolution feature; Performing convolution processing on the fourth convolution feature through the fifth convolution layer to generate a fifth convolution feature; Performing feature splicing on the first convolution feature, the second convolution feature, the third convolution feature, the fourth convolution feature and the fifth convolution feature to obtain a first candidate convolution feature; Generate a second candidate convolution feature according to the second sub-image feature extraction model and the scaled image; Generate a third candidate convolution feature according to the third sub-image feature extraction model and the scaled image; The first candidate convolution feature, the second candidate convolution feature and the third candidate convolution feature are superimposed to obtain a first image feature in the first image feature set that corresponds to the scaled image.

5. The method according to claim 4, wherein: The second feature extraction model is a feature extraction model based on a Transformer structure, and the second feature extraction model includes: an encoder and a decoder set, the number of decoders in the decoder set is consistent with the number of scaled images in the scaled image set, and there is a one-to-one correspondence between the decoders and the first image features; and The step of extracting image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set further includes: For each first image feature in the first image feature set, performing the following second image feature generation step: Performing feature encoding on the first image feature by the encoder to obtain an encoded first image feature; The encoded first image feature is feature decoded by a decoder in the decoder set that corresponds to the first image feature to obtain a second image feature in the second image feature set that corresponds to the first image feature.

6. The method according to claim 5, wherein: The step of generating an instance information set according to the first image feature set and the second image feature set includes: For each first image feature in the first image feature set, performing a cross multiplication operation on a second image feature in the second image feature set that corresponds to the first image feature and the first image feature to obtain an image feature after the cross multiplication operation; Perform feature superposition on each of the image features after the cross-product operation in the obtained image feature set after the cross-product operation to obtain a region segmentation map, wherein the region segmentation map includes: a segmentation region set; Performing approximate segmentation region fusion on the segmentation regions in the segmentation region set to obtain a fused segmentation region set; For each fused segmented region in the fused segmented region set, the following instance information generation step is performed: Extracting local region features corresponding to the fused segmented region in the region segmentation map as instance feature maps included in instance information corresponding to the fused segmented region; According to the instance feature map and the instance type predictor corresponding to the fused segmented region, the instance type included in the instance information corresponding to the fused segmented region is generated.

7. An image classification label generation device, comprising: An image scaling unit is configured to scale a target image to obtain a scaled image set, wherein the target image is an image for which a corresponding image classification label is to be generated, and the scaled images in the scaled image set have different scaling factors. an image feature extraction unit configured to extract image features from each scaled image in the scaled image set to generate a first image feature set and a second image feature set, wherein the first image feature is obtained by extracting features from the scaled image using a first feature extraction model, and the second image feature is obtained by extracting features from the first image feature set using a second feature extraction model, and there is a one-to-one correspondence between the first image feature and the second image feature; A first generating unit is configured to generate an instance information set according to the first image feature set and the second image feature set, wherein the instance information in the instance information set includes: an instance type and an instance feature map, and the instance feature map represents a local image feature of an instance corresponding to the instance information; A second generating unit is configured to generate an instance label group for each instance information in the instance information set according to the instance type and the instance feature graph included in the instance information; The conflicting instance label elimination unit is configured to eliminate conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image, wherein: The step of removing conflicting instance labels from the obtained instance label group set to obtain an image classification label set corresponding to the target image includes: Based on the instance label group set, perform the following conflicting instance label removal steps: Splitting the instance tag group set into a first instance tag group and a second instance tag group set; By using the instance label conflict classification model, determining the first instance label in the first instance label group that has the instance label conflict as the first instance label to be eliminated, and obtaining the first instance label set to be eliminated; Determine a complement of the first instance label group and the first to-be-eliminated instance label set as a third instance label set; By using the instance label conflict classification model, determining the third instance label in the second instance label group set that has an instance label conflict with the third instance label set as the second instance label to be eliminated, and obtaining the second instance label set to be eliminated; Determine a difference between each second instance label group in the second instance label group set and the second to-be-eliminated instance label set as a fourth instance label group, and obtain a fourth instance label group set; In response to the fourth instance label group set including a target number of fourth instance label groups, determining the third instance label in the third instance label set and the fourth instance label in the fourth instance label group set as image classification labels in the image classification label set, wherein the target number is 1; In response to the number of fourth instance label groups included in the fourth instance label group set being greater than the target number, the fourth instance label group set is determined as the instance label group set, and the third instance label in the third instance label set is determined as the image classification label in the image classification label set, and the conflicting instance label elimination step is performed again.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.