Auxiliary marking method, device, equipment and storage medium

Abstract

The present application relates to the field of image processing technology, and proposes an auxiliary annotation method, device, equipment and storage medium. The method includes: performing multiple levels of feature extraction on each pixel of the image to be annotated through a pre-trained feature extraction model to obtain multiple levels of feature maps corresponding to each pixel; processing each level of the feature map of each pixel into the original image size and performing splicing processing to obtain the corresponding total feature map; calculating the correlation coefficient between the multiple levels of features in the total feature map, and determining the significant features based on the correlation coefficient; based on the significant features of each pixel, performing auxiliary annotation on the image to be annotated to generate an auxiliary annotation result. Through this technical solution, there is no need to annotate the image and train the model in advance, thereby improving the quality of auxiliary annotation and reducing the labor cost in the data annotation process.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an auxiliary annotation method, device, equipment and storage medium. Background Art

[0002] In real-world labeling scenarios, as models increasingly demand more data, the workload is also increasing. Furthermore, image data annotators can easily become fatigued during repetitive tasks, leading to inaccurate labeling. Existing assisted or semi-automatic labeling methods pre-annotate a portion of the data and then train the model to predict the remaining data.

[0003] Using model predictions requires pre-labeling of some data to train the model. Insufficient pre-labeled data will lead to poor prediction results. Summary of the Invention

[0004] The embodiments of the present application provide an auxiliary labeling method, apparatus, device and storage medium, which aim to solve the technical problem in related technologies that insufficient early labeling data leads to poor labeling prediction effects, thereby reducing the labor cost of the data labeling process while improving the quality of data labeling.

[0005] In a first aspect, an embodiment of the present application provides an auxiliary annotation method, comprising:

[0006] Use the pre-trained feature extraction model to perform multi-level feature extraction on each pixel of the image to be annotated, so as to obtain multi-level feature maps corresponding to each pixel.

[0007] The feature maps of each level of each pixel are processed into the original image size and spliced ​​to obtain the corresponding total feature map;

[0008] Calculating correlation coefficients between features at multiple levels in the overall feature map, and determining significant features based on the correlation coefficients;

[0009] According to the significant features of each pixel, the image to be annotated is auxiliary-annotated to generate an auxiliary-annotation result.

[0010] Optionally, the method further includes:

[0011] Acquire an image dataset, wherein the image dataset includes an open source dataset;

[0012] Self-supervised learning training is performed using a preset classification model and the image dataset to obtain the pre-trained feature extraction model.

[0013] Optionally, obtaining the image dataset includes:

[0014] Get multiple open source images;

[0015] Perform multiple affine transformations on each open source image to obtain the transformed image;

[0016] The transformed images are labeled according to the transformation parameters to obtain the image dataset.

[0017] Optionally, the preset classification model includes multiple levels, which are cascaded in sequence, each level outputs feature maps of different sizes, and the feature map output by the previous level serves as the input of the next level.

[0018] Optionally, the preset classification model includes multiple convolutional layers.

[0019] Optionally, calculating correlation coefficients between features at multiple levels in the overall feature map, and determining significant features based on the correlation coefficients, includes:

[0020] According to the total feature map, calculate the average feature corresponding to the feature map of each level of each pixel;

[0021] Calculating correlation coefficients between features at the multiple levels based on the average features;

[0022] The features whose correlation coefficients are greater than the target level of the preset coefficients are determined as the significant features.

[0023] Optionally, calculating the correlation coefficients between the features at the multiple levels based on the average feature includes:

[0024] The average features of each level are input into the preset attention mechanism model to output the correlation coefficients between features at multiple levels.

[0025] Optionally, auxiliary annotation is performed on the image to be annotated based on the significant features of each pixel, including:

[0026] Based on the salient features of each pixel, the distance between the features of each target level and the corresponding average features is calculated;

[0027] According to the distance value, auxiliary annotation is performed on the image to be annotated to generate an auxiliary annotation result.

[0028] Optionally, performing auxiliary annotation on the image to be annotated according to the distance value to generate an auxiliary annotation result includes:

[0029] Calculate the classification score of each pixel according to the distance value;

[0030] When the classification score of the pixel point is greater than or equal to the preset score, determining the classification result of the pixel point as a foreground point;

[0031] When the classification score of the pixel point is less than a preset score, determining the classification result of the pixel point as a background point;

[0032] Counting the classification results of all pixels in the image to be labeled, and determining the connected domain corresponding to the foreground point according to the classification results;

[0033] The auxiliary annotation result is generated according to the coordinate range corresponding to each connected domain.

[0034] Optionally, calculating a classification score for each pixel point according to the distance value includes:

[0035] Calculate the mean square sum of the distance values ​​corresponding to all significant features of each pixel;

[0036] The mean square sum is determined as the classification score of the pixel.

[0037] Optionally, the method further includes:

[0038] receiving a modification result of the auxiliary annotation result input by a user;

[0039] Determining the accuracy of the auxiliary annotation result according to the modification result;

[0040] When the accuracy of the auxiliary annotation result is lower than a preset threshold, the auxiliary annotation result is determined as an inaccurate sample;

[0041] The feature extraction model is trained through self-supervised learning based on the inaccurate samples and preset model updating rules to update the feature extraction model.

[0042] Optionally, the preset model update rule includes at least one of the following:

[0043] The number of inaccurate samples reaches a preset number;

[0044] The time since the last feature extraction model update reaches the preset time.

[0045] In a second aspect, an embodiment of the present application provides an auxiliary annotation device, comprising:

[0046] The extraction module is used to perform multi-level feature extraction on each pixel of the image to be annotated using a pre-trained feature extraction model to obtain a multi-level feature map corresponding to each pixel;

[0047] A processing module is used to process the feature map of each level of each pixel into the original image size and perform splicing to obtain the corresponding total feature map;

[0048] a calculation module, configured to calculate correlation coefficients between features at multiple levels in the overall feature map, and determine significant features based on the correlation coefficients;

[0049] The annotation module is used to perform auxiliary annotation on the image to be annotated according to the significant features of each pixel point and generate an auxiliary annotation result.

[0050] Optionally, the device further comprises:

[0051] An acquisition module, configured to acquire an image dataset, wherein the image dataset includes an open source dataset;

[0052] A training module is used to perform self-supervised learning training using a preset classification model and the image dataset to obtain the pre-trained feature extraction model.

[0053] Optionally, the acquisition module includes:

[0054] An image acquisition unit, used to acquire multiple open source images;

[0055] A transformation unit, used to perform multiple affine transformations on each open source image to obtain a transformed image;

[0056] The labeling unit is used to label the transformed image according to the transformation parameters to obtain the image dataset.

[0057] Optionally, the preset classification model includes multiple levels, which are cascaded in sequence, each level outputs feature maps of different sizes, and the feature map output by the previous level serves as the input of the next level.

[0058] Optionally, the preset classification model includes multiple convolutional layers.

[0059] Optionally, the calculation module includes:

[0060] A first calculation unit is used to calculate the average feature corresponding to the feature map of each level of each pixel according to the total feature map;

[0061] a second calculation unit, configured to calculate correlation coefficients between features at the multiple levels based on the average feature;

[0062] The feature determination unit is used to determine the feature of the target level with a correlation coefficient greater than a preset coefficient as the significant feature.

[0063] Optionally, the second computing unit is specifically configured to:

[0064] The average features of each level are input into the preset attention mechanism model to output the correlation coefficients between features at multiple levels.

[0065] Optionally, the marking module includes:

[0066] A third calculation unit is used to calculate the distance value between the feature of each target level and the corresponding average feature according to the significant feature of each pixel point;

[0067] A generating unit is configured to perform auxiliary annotation on the image to be annotated according to the distance value, and generate an auxiliary annotation result.

[0068] Optionally, the generating unit is specifically configured to:

[0069] Calculate the classification score of each pixel according to the distance value;

[0070] When the classification score of the pixel point is greater than or equal to the preset score, determining the classification result of the pixel point as a foreground point;

[0071] When the classification score of the pixel point is less than a preset score, determining the classification result of the pixel point as a background point;

[0072] Counting the classification results of all pixels in the image to be labeled, and determining the connected domain corresponding to the foreground point according to the classification results;

[0073] The auxiliary annotation result is generated according to the coordinate range corresponding to each connected domain.

[0074] Optionally, calculating a classification score for each pixel point according to the distance value includes:

[0075] Calculate the mean square sum of the distance values ​​corresponding to all significant features of each pixel;

[0076] The mean square sum is determined as the classification score of the pixel.

[0077] Optionally, the device further comprises:

[0078] A receiving module, configured to receive a modification result of the auxiliary annotation result input by a user;

[0079] A first determining module, configured to determine the accuracy of the auxiliary annotation result according to the modification result;

[0080] A second determination module is configured to determine the auxiliary annotation result as an inaccurate sample when the accuracy of the auxiliary annotation result is lower than a preset threshold;

[0081] A model updating module is used to perform self-supervised learning training on the feature extraction model according to the inaccurate samples and preset model updating rules to update the feature extraction model.

[0082] Optionally, the preset model update rule includes at least one of the following:

[0083] The number of inaccurate samples reaches a preset number;

[0084] The time since the last feature extraction model update reaches the preset time.

[0085] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the auxiliary annotation method when executing the computer program.

[0086] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned auxiliary annotation method are implemented.

[0087] In the scheme implemented by the above-mentioned auxiliary annotation method, device, equipment and storage medium, a feature extraction model is obtained through self-supervised learning training, and then multi-level feature extraction is automatically performed on the image to be annotated through the feature extraction model, and the display features of each pixel in the image are determined based on the extracted multi-level features, and the auxiliary annotation results are generated based on the significant features. In this way, there is no need to annotate the image and train the model in advance, thereby improving the quality of auxiliary annotation and reducing the labor cost in the data annotation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0089] Figure 1 A schematic flow chart of an auxiliary annotation method according to an embodiment of the present application is shown.

[0090] Figure 2 A schematic diagram of a pyramid feature processing process according to an embodiment of the present application is shown.

[0091] Figure 3 A schematic diagram of the processing process of the auxiliary annotation method according to an embodiment of the present application is shown.

[0092] Figure 4 A schematic diagram of a connected domain detection frame generation according to an embodiment of the present application is shown.

[0093] Figure 5 A schematic flow chart of an auxiliary annotation method according to another embodiment of the present application is shown.

[0094] Figure 6A block diagram of an auxiliary annotation device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0095] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0096] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0097] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0098] In order to solve the technical problem in the related art that insufficient early annotation data leads to poor annotation prediction effect, the present application proposes an auxiliary annotation method, device, equipment and storage medium.

[0099] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0100] It should be noted that embodiments of the present application may acquire and process relevant data based on artificial intelligence technology. Artificial intelligence refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0101] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0102] See also Figure 1 , Figure 1 A schematic flow chart of an auxiliary annotation method according to an embodiment of the present application is shown.

[0103] like Figure 1 As shown, the embodiment of the present application provides an auxiliary annotation method, including:

[0104] Step S101: performing multi-level feature extraction on each pixel of the image to be annotated using a pre-trained feature extraction model to obtain a multi-level feature map corresponding to each pixel;

[0105] The training process of the feature extraction model includes:

[0106] Obtain an image dataset, wherein the image dataset includes an open source dataset; specifically, a plurality of open source images may be obtained; multiple affine transformations may be performed on each open source image to obtain a transformed image; and the transformed image may be labeled according to transformation parameters to obtain the image dataset.

[0107] Optionally, the affine transformation includes but is not limited to flipping, rotating, translating, scaling, and shearing of the image.

[0108] Self-supervised learning is performed using a preset classification model and the image dataset to obtain the pre-trained feature extraction model. Self-supervised learning can be considered an "ideal state" of machine learning, where the model learns directly from unlabeled data without the need for labeled data.

[0109] The core of self-supervised learning lies in how to automatically generate labels for data. For example, take an image as input, randomly rotate it by a certain angle, and use the rotated image as input, with the angle of random rotation as the label. Another example is to evenly divide the input image into 3x3 grids, with the content of each grid as a patch. The order of the patches is randomly shuffled, and then the shuffled patches are used as input, with the correct order as the label. Automatic labeling like this requires no human intervention.

[0110] Optionally, the preset classification model includes multiple levels, which are cascaded in sequence, each level outputs feature maps of different sizes, and the feature map output by the previous level serves as the input of the next level.

[0111] Optionally, the preset classification model includes multiple convolutional layers.

[0112] Optionally, the preset classification model includes but is not limited to classification models such as ResNet (Residual Network). The last layer of the ResNet model is removed, that is, the fully connected layer is removed, and the model inputs a color image, while the grayscale image can be input as a color image through the copy channel.

[0113] Taking the ResNet model as an example, its network model has 4 levels of stages cascaded in sequence, and each level outputs B×N×H i ×Wi Matrix of size (B is the batch size, N is the feature dimension of each level stage output of the model setting, H represents the height, W represents the width, i∈(1,4)), B×N×H i ×W i The matrix of size is used as the extracted image features and passed to the next level stage. The previous level stage extracts the shallow information of the image, and the features extracted by the later level stage are more biased towards semantic information (abstract information). The output of multiple level stages (B×N×H1×W1~B×N×H4×W4) is used as the pyramid feature. Among them, the pyramid feature is as follows Figure 2 As shown. The multi-level pyramid features include shallow texture features and deep texture features. For example, the ResNet model with 4 levels of stages obtains four feature maps f1 to f4 of different sizes, where f i ~1×N×H i ×W i , batch size = 1.

[0114] Step S102: processing the feature map of each level of each pixel into the original image size and performing splicing processing to obtain the corresponding total feature map;

[0115] like Figure 3 As shown, the features at all levels are scaled to the original image size and spliced ​​together, and the output is 1×4N×H o ×W o Shape matrix, H o ×W o For X d The height and width of , thus obtaining the multi-level features d1~d4, d1~N of each pixel in the image.

[0116] Step S103: calculating correlation coefficients between features at multiple levels in the overall feature map, and determining significant features based on the correlation coefficients;

[0117] Optionally, the above step S103 includes:

[0118] According to the total feature map, calculate the average feature corresponding to the feature map of each level of each pixel;

[0119] Specifically, the average pixel-level feature can be obtained by averaging the width and height of the last two dimensions of the feature map at each level.

[0120] Calculating correlation coefficients between features at the multiple levels based on the average features;

[0121] Optionally, calculating the correlation coefficients between the features at the multiple levels based on the average feature includes:

[0122] The average features of each level are input into the preset attention mechanism model to output the correlation coefficients between features at multiple levels.

[0123] In this embodiment, the correlation coefficients between features at multiple levels can be determined by a preset attention mechanism model. Of course, other correlation coefficient calculation methods can also be used to calculate the correlation coefficients between features at multiple levels, and no specific restrictions are made here.

[0124] The features whose correlation coefficients are greater than the target level of the preset coefficients are determined as the significant features.

[0125] In this embodiment, a higher correlation coefficient indicates a higher degree of significance. Specifically, a preset coefficient, such as 0.5, can be set. In this way, when the correlation coefficients between a feature at a certain level and features at other levels are all greater than 0.5, the feature at that level is a significant feature, thereby determining all significant features. For example, in the above embodiment, the average features of each level are After calculation If the correlation coefficient is greater than the preset coefficient, the significant features are d2~d4.

[0126] Step S104: performing auxiliary annotation on the image to be annotated according to the salient features of each pixel point, and generating an auxiliary annotation result.

[0127] Optionally, auxiliary annotation is performed on the image to be annotated based on the significant features of each pixel, including:

[0128] Based on the salient features of each pixel, the distance between the features of each target level and the corresponding average features is calculated;

[0129] In this embodiment, the distance value between the feature of each target level and the corresponding average feature is calculated, as in the above embodiment, for example, the distance value between the feature d2 of the pixel point and the average feature d2 is calculated. The distance between them is a2, the pixel feature d3 and the average feature The distance between them is a3, the pixel feature d4 and the average feature The distance value between them is a4.

[0130] The distance calculation methods include but are not limited to Euclidean distance, cosine distance, and Mahalanobis distance.

[0131] According to the distance value, auxiliary annotation is performed on the image to be annotated to generate an auxiliary annotation result.

[0132] Optionally, performing auxiliary annotation on the image to be annotated according to the distance value to generate an auxiliary annotation result includes:

[0133] Calculate the classification score of each pixel according to the distance value;

[0134] Optionally, calculating a classification score for each pixel point according to the distance value includes:

[0135] Calculate the mean square sum of the distance values ​​corresponding to all significant features of each pixel;

[0136] The mean square sum is determined as the classification score of the pixel.

[0137] In this embodiment, the classification score of the pixel point can be calculated based on the obtained distance value. For example, the mean square sum of the distance values ​​corresponding to all significant features of each pixel point can be calculated (a2 2 +a3 2 +a4 2 ) / 3, and use the mean square sum as the classification score of the pixel. Of course, the classification score can be determined by calculating the mean square sum of the distance values, or other calculation methods such as variance.

[0138] When the classification score of the pixel point is greater than or equal to the preset score, determining the classification result of the pixel point as a foreground point;

[0139] When the classification score of a pixel point is less than a preset score, the classification result of the pixel point is determined to be a background point. According to the classification score of the pixel point, it can be determined whether the pixel point belongs to the foreground or the background, and then the auxiliary annotation result is further determined according to the foreground-background segmentation result.

[0140] In this embodiment, a preset score may also be set. For example, if the preset score is set to 0.5, when the score of a pixel point is less than 0.5, it is determined to be a background point, and when the score of a pixel point is greater than or equal to 0.5, it is determined to be a foreground point.

[0141] The classification results of all pixels in the image to be labeled are counted, and a connected domain corresponding to the foreground point is determined according to the classification results; and the auxiliary labeling result is generated according to the coordinate range corresponding to each connected domain.

[0142] In this embodiment, Figure 4 As shown in FIG, the connected domain is calculated based on the foreground and background segmentation results, and the coordinate range of each connected domain is obtained to generate the target detection result, and then auxiliary annotation is performed based on the target detection result.

[0143] like Figure 5 As shown, optionally, the method further includes:

[0144] Step S501, receiving a modification result of the auxiliary annotation result input by a user;

[0145] After automatic auxiliary annotation, the user may further manually review the auxiliary annotation results. After the user review, if the auxiliary annotation results are found to be inaccurate, the auxiliary annotation results can be modified. At this time, the user's modification of the auxiliary annotation results can be received and recorded.

[0146] Step S502: determining the accuracy of the auxiliary annotation result according to the modification result;

[0147] Specifically, the accuracy of the auxiliary annotation result can be determined based on the modification result. For example, the accuracy of the auxiliary annotation result can be calculated based on the amount of modification. The amount of modification can specifically involve the scope of the modification, the proportion of the modification scope to the total scope, etc.

[0148] Step S503: when the accuracy of the auxiliary annotation result is lower than a preset threshold, the auxiliary annotation result is determined as an inaccurate sample;

[0149] A preset threshold can also be set for the accuracy of the auxiliary annotation results, such as 0.6. When the accuracy of the auxiliary annotation result is lower than 0.6, the auxiliary annotation result is considered inaccurate. The image and the auxiliary annotation result can be saved as inaccurate samples, which facilitates further training of the feature extraction model based on the inaccurate samples in the future, thereby further improving the accuracy of the feature extraction model.

[0150] Step S504: performing self-supervised learning training on the feature extraction model according to the inaccurate samples and preset model updating rules to update the feature extraction model.

[0151] Optionally, the preset model update rule includes at least one of the following:

[0152] The number of inaccurate samples reaches a preset number;

[0153] The time since the last feature extraction model update reaches the preset time.

[0154] In this embodiment, the feature extraction model can be self-supervised and trained based on inaccurate samples and preset model update rules to update the feature extraction model. For example, when the number of inaccurate samples reaches a certain number, such as one thousand, the self-supervised training is restarted and the feature extraction model is adjusted to achieve model self-update. Of course, it is also possible to restart the self-supervised training and adjust the feature extraction model to achieve model self-update when the model reaches a preset time, such as one week, since the last update. Of course, the processing quantity and update time can also be other update rules, and are not specifically limited here.

[0155] Figure 6 A block diagram of an auxiliary annotation device according to an embodiment of the present application is shown.

[0156] like Figure 6 As shown, in a second aspect, an embodiment of the present application provides an auxiliary marking device 60, comprising:

[0157] The extraction module 61 is used to perform multiple levels of feature extraction on each pixel of the image to be annotated through a pre-trained feature extraction model to obtain multiple levels of feature maps corresponding to each pixel; the processing module 62 is used to process the feature maps of each level of each pixel into the original image size and perform splicing processing to obtain the corresponding total feature map; the calculation module 63 is used to calculate the correlation coefficient between the multiple levels of features in the total feature map, and determine the significant features based on the correlation coefficient; the annotation module 64 is used to perform auxiliary annotation on the image to be annotated based on the significant features of each pixel, and generate an auxiliary annotation result.

[0158] For the specific limitations of the auxiliary marking device, please refer to the limitations of the auxiliary marking method above, which will not be repeated here.

[0159] The auxiliary annotation device provided by the present application uses a pre-trained feature extraction model to perform multiple levels of feature extraction on each pixel of the image to be annotated through the extraction module 61 to obtain multiple levels of feature maps corresponding to each pixel, and processes the feature maps of each level of each pixel into the original image size and performs splicing processing through the processing module 62 to obtain the corresponding total feature map, calculates the correlation coefficient between the multiple levels of features in the total feature map through the calculation module 63, and determines the significant features based on the correlation coefficient, and performs auxiliary annotation on the image to be annotated based on the significant features of each pixel through the annotation module 64 to generate an auxiliary annotation result. In this way, there is no need to annotate the image and train the model in advance, thereby improving the quality of auxiliary annotation and reducing the labor cost in the data annotation process.

[0160] As an optional example, the device also includes: an acquisition module for acquiring a picture dataset, wherein the picture dataset includes an open source dataset; a training module for performing self-supervised learning training using a preset classification model and the picture dataset to obtain the pre-trained feature extraction model.

[0161] As an optional example, the acquisition module includes: an image acquisition unit for acquiring multiple open source images; a transformation unit for performing multiple affine transformations on each open source image to obtain a transformed image; and a labeling unit for labeling the transformed image according to the transformation parameters to obtain the image dataset.

[0162] As an optional example, the preset classification model includes multiple levels, which are cascaded in sequence, each level outputs feature maps of different sizes, and the feature map output by the previous level serves as the input of the next level.

[0163] As an optional example, the preset classification model includes multiple convolutional layers.

[0164] As an optional example, the calculation module 63 includes: a first calculation unit, used to calculate the average feature corresponding to the feature map of each level of each pixel point based on the total feature map; a second calculation unit, used to calculate the correlation coefficient between the features of the multiple levels based on the average feature; and a feature determination unit, used to determine the feature of the target level whose correlation coefficient is greater than the preset coefficient as the significant feature.

[0165] As an optional example, the second computing unit is specifically used to: input the average features of each level into a preset attention mechanism model to output the correlation coefficient between features of multiple levels.

[0166] As an optional example, the labeling module 64 includes: a third calculation unit, used to calculate the distance value between the feature of each target level and the corresponding average feature based on the salient features of each pixel point; a generation unit, used to assist in labeling the image to be labeled based on the distance value and generate an auxiliary labeling result.

[0167] As an optional example, the generation unit is specifically used to: calculate the classification score of each pixel point based on the distance value; when the classification score of the pixel point is greater than or equal to the preset score, determine the classification result of the pixel point as a foreground point; when the classification score of the pixel point is less than the preset score, determine the classification result of the pixel point as a background point; count the classification results of all pixels in the image to be annotated, and determine the connected domain corresponding to the foreground point based on the classification results; generate the auxiliary annotation result based on the coordinate range corresponding to each connected domain.

[0168] As an optional example, calculating the classification score of each pixel point based on the distance value includes: calculating the mean square sum of the distance values ​​corresponding to all significant features of each pixel point; and determining the mean square sum as the classification score of the pixel point.

[0169] As an optional example, the device also includes: a receiving module for receiving a modification result of the auxiliary annotation result input by a user; a first determination module for determining the accuracy of the auxiliary annotation result based on the modification result; a second determination module for determining the auxiliary annotation result as an inaccurate sample when the accuracy of the auxiliary annotation result is lower than a preset threshold; and a model updating module for performing self-supervised learning training on the feature extraction model based on the inaccurate sample and preset model updating rules to update the feature extraction model.

[0170] As an optional example, the preset model update rule includes at least one of the following:

[0171] The number of inaccurate samples reaches a preset number;

[0172] The time since the last feature extraction model update reaches the preset time.

[0173] Based on the above Figure 1 The method shown, and Figure 6 In the device embodiment shown, in order to achieve the above-mentioned purpose, the embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the steps of the above-mentioned auxiliary annotation method are implemented when the processor executes the computer program.

[0174] Based on the above Figure 1 The method shown, accordingly, an embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned auxiliary annotation method are implemented.

[0175] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0176] Based on the above Figure 1 The method shown in FIG. 1 is a method for performing the above-mentioned operation. Accordingly, the embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned operation is performed. Figure 1 The auxiliary annotation method shown.

[0177] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0178] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0179] It should be understood that although the terms "first," "second," etc. may be used to describe the setting units in the embodiments of the present application, these setting units should not be limited to these terms. These terms are merely used to distinguish the setting units from each other. For example, without departing from the scope of the embodiments of the present application, the first setting unit may also be referred to as the second setting unit, and similarly, the second setting unit may also be referred to as the first setting unit.

[0180] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interface, indirect coupling or communication connection of the device or unit, which may be electrical, mechanical or other forms.

[0182] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0183] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0184] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An auxiliary marking method, characterized in that: include: Use the pre-trained feature extraction model to perform multi-level feature extraction on each pixel of the image to be annotated, so as to obtain multi-level feature maps corresponding to each pixel. The feature maps of each level of each pixel are processed into the original image size and spliced ​​to obtain the corresponding total feature map; Calculating correlation coefficients between features at multiple levels in the overall feature map, and determining significant features based on the correlation coefficients; According to the significant features of each pixel, the image to be annotated is auxiliary-annotated to generate an auxiliary-annotation result.

2. The auxiliary marking method according to claim 1, characterized in that: The method further comprises: Acquire an image dataset, wherein the image dataset includes an open source dataset; Self-supervised learning training is performed using a preset classification model and the image dataset to obtain the pre-trained feature extraction model.

3. The auxiliary marking method according to claim 2, characterized in that: The obtaining of the image dataset includes: Get multiple open source images; Perform multiple affine transformations on each open source image to obtain the transformed image; The transformed images are labeled according to the transformation parameters to obtain the image dataset.

4. The auxiliary marking method according to claim 2, characterized in that: The preset classification model includes multiple levels, which are cascaded in sequence. Each level outputs feature maps of different sizes, and the feature map output by the previous level serves as the input of the next level.

5. The auxiliary marking method according to claim 2, characterized in that: The preset classification model includes multiple convolutional layers.

6. The auxiliary marking method according to claim 1, characterized in that: Calculating correlation coefficients between features at multiple levels in the overall feature map, and determining significant features based on the correlation coefficients, including: According to the total feature map, calculate the average feature corresponding to the feature map of each level of each pixel; Calculating correlation coefficients between features at the multiple levels based on the average features; The features of the target level with correlation coefficients greater than the preset coefficients are determined as the significant features.

7. The auxiliary marking method according to claim 6, characterized in that: Calculating correlation coefficients between features at the multiple levels based on the average feature, including: The average features of each level are input into the preset attention mechanism model to output the correlation coefficients between features at multiple levels.

8. The auxiliary marking method according to claim 6, characterized in that: Perform auxiliary annotation on the image to be annotated based on the significant features of each pixel, including: Based on the salient features of each pixel, the distance between the features of each target level and the corresponding average features is calculated; According to the distance value, auxiliary annotation is performed on the image to be annotated to generate an auxiliary annotation result.

9. The auxiliary marking method according to claim 8, characterized in that: Performing auxiliary annotation on the image to be annotated according to the distance value to generate an auxiliary annotation result, including: Calculate the classification score of each pixel according to the distance value; When the classification score of the pixel point is greater than or equal to the preset score, determining the classification result of the pixel point as a foreground point; When the classification score of the pixel point is less than a preset score, determining the classification result of the pixel point as a background point; Counting the classification results of all pixels in the image to be labeled, and determining the connected domain corresponding to the foreground point according to the classification results; The auxiliary annotation result is generated according to the coordinate range corresponding to each connected domain.

10. The auxiliary marking method according to claim 9, characterized in that: Calculating the classification score of each pixel point according to the distance value includes: Calculate the mean square sum of the distance values ​​corresponding to all significant features of each pixel; The mean square sum is determined as the classification score of the pixel.

11. The auxiliary marking method according to claim 1, characterized in that: The method further comprises: receiving a modification result of the auxiliary annotation result input by a user; Determining the accuracy of the auxiliary annotation result according to the modification result; When the accuracy of the auxiliary annotation result is lower than a preset threshold, the auxiliary annotation result is determined as an inaccurate sample; The feature extraction model is trained through self-supervised learning based on the inaccurate samples and preset model updating rules to update the feature extraction model.

12. The auxiliary marking method according to claim 11, characterized in that: The preset model update rule includes at least one of the following: The number of inaccurate samples reaches a preset number; The time since the last feature extraction model update reaches the preset time.

13. An auxiliary marking device, characterized in that: include: The extraction module is used to perform multi-level feature extraction on each pixel of the image to be annotated using a pre-trained feature extraction model to obtain a multi-level feature map corresponding to each pixel; A processing module is used to process the feature map of each level of each pixel into the original image size and perform splicing to obtain the corresponding total feature map; a calculation module, configured to calculate correlation coefficients between features at multiple levels in the overall feature map, and determine significant features based on the correlation coefficients; The annotation module is used to perform auxiliary annotation on the image to be annotated according to the significant features of each pixel point and generate an auxiliary annotation result.

14. The auxiliary marking device according to claim 13, characterized in that: The device further comprises: An acquisition module, configured to acquire an image dataset, wherein the image dataset includes an open source dataset; A training module is used to perform self-supervised learning training using a preset classification model and the image dataset to obtain the pre-trained feature extraction model.

15. The auxiliary marking device according to claim 14, characterized in that: The acquisition module includes: An image acquisition unit, used to acquire multiple open source images; A transformation unit, used to perform multiple affine transformations on each open source image to obtain a transformed image; The labeling unit is used to label the transformed image according to the transformation parameters to obtain the image dataset.

16. The auxiliary marking device according to claim 14, characterized in that: The preset classification model includes multiple levels, which are cascaded in sequence. Each level outputs feature maps of different sizes, and the feature map output by the previous level serves as the input of the next level.

17. The auxiliary marking device according to claim 14, characterized in that: The preset classification model includes multiple convolutional layers.

18. The auxiliary marking device according to claim 13, characterized in that: The calculation module includes: A first calculation unit is used to calculate the average feature corresponding to the feature map of each level of each pixel according to the total feature map; a second calculation unit, configured to calculate correlation coefficients between features at the multiple levels based on the average feature; The feature determination unit is used to determine the feature of the target level whose correlation coefficient is greater than the preset coefficient as the significant feature.

19. The auxiliary marking device according to claim 18, characterized in that: The second computing unit is specifically configured to: The average features of each level are input into the preset attention mechanism model to output the correlation coefficients between features at multiple levels.

20. The auxiliary marking device according to claim 18, characterized in that: The marking module includes: A third calculation unit is used to calculate the distance value between the feature of each target level and the corresponding average feature according to the significant feature of each pixel point; A generating unit is configured to perform auxiliary annotation on the image to be annotated according to the distance value, and generate an auxiliary annotation result.

21. The auxiliary marking device according to claim 20, characterized in that: The generating unit is specifically configured to: Calculate the classification score of each pixel according to the distance value; When the classification score of the pixel point is greater than or equal to the preset score, determining the classification result of the pixel point as a foreground point; When the classification score of the pixel point is less than a preset score, determining the classification result of the pixel point as a background point; Counting the classification results of all pixels in the image to be labeled, and determining the connected domain corresponding to the foreground point according to the classification results; The auxiliary annotation result is generated according to the coordinate range corresponding to each connected domain.

22. The auxiliary marking device according to claim 21, characterized in that: Calculating the classification score of each pixel point according to the distance value includes: Calculate the mean square sum of the distance values ​​corresponding to all significant features of each pixel; The mean square sum is determined as the classification score of the pixel.

23. The auxiliary marking device according to claim 13, characterized in that: The device further comprises: A receiving module, configured to receive a modification result of the auxiliary annotation result input by a user; A first determining module, configured to determine the accuracy of the auxiliary annotation result according to the modification result; A second determination module is configured to determine the auxiliary annotation result as an inaccurate sample when the accuracy of the auxiliary annotation result is lower than a preset threshold; A model updating module is used to perform self-supervised learning training on the feature extraction model according to the inaccurate samples and preset model updating rules to update the feature extraction model.

24. The auxiliary marking device according to claim 23, characterized in that: The preset model update rule includes at least one of the following: The number of inaccurate samples reaches a preset number; The time since the last feature extraction model update reaches the preset time.

25. A computer device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the method according to any one of claims 1 to 12.

Images