Methods, apparatus, and storage media for annotating object contours in target images

Abstract

This invention provides a method, apparatus, storage medium, and electronic device for annotating the contours of objects in a target image. The method includes acquiring target image features of the target image, wherein the target image includes a target object, and the target object is a target type; inputting the target image features into a target generator; and acquiring a target mask of the target image generated by the target generator, wherein the target mask is used to annotate the contours of the target object. This invention solves the problem of low efficiency in recognizing object contours under weak supervision, thereby improving the efficiency of object contour recognition.

Description

Technical Field

[0001] The present invention relates to the field of computers, and more specifically, to a method and apparatus for annotating the contours of objects in a target image, a storage medium, and an electronic device. Background Technology

[0002] In existing technologies, it is typically necessary to label and segment target objects in an image. For example, labeling the outline of a person in an image. Existing technologies can use weakly supervised learning to label the outline of a person.

[0003] However, existing weakly supervised learning methods typically employ image-level classification labels. Using such methods results in models with low accuracy in labeling and segmenting objects.

[0004] In other words, in the existing technology, when using weak supervision to segment and predict the contour of a target object in an image, there is a problem of low accuracy in determining the contour. Summary of the Invention

[0005] This invention provides a method, apparatus, storage medium, and electronic device for annotating the contours of objects in a target image, to at least solve the problem of low accuracy in determining the contours of target objects under weak supervision in related technologies.

[0006] According to one aspect of the present invention, a method for annotating the contours of an object in a target image is provided, comprising: acquiring target image features of the target image, wherein the target image includes a target object, and the target object is a target type; inputting the target image features into a target generator, wherein the target generator is a generator in an adversarial generative network trained using sample images, the adversarial generative network including the target generator and a discriminator, the target generator being configured to generate a first mask of the sample image after acquiring the first image features of the sample image, the discriminator being configured to identify the type of the sample object in the sample image after erasing the pixels corresponding to the first mask after receiving the sample image, the type of the sample object being used to train the parameters in the target generator; and acquiring the target mask of the target image generated by the target generator, wherein the target mask is used to annotate the contours of the target object.

[0007] In one exemplary embodiment, before inputting the target image features into the target generator, the method further includes: acquiring first image features of the sample image; inputting the first image features into the target generator to generate a first mask of the sample image; erasing pixels corresponding to the first mask to obtain a first image; inputting the first image and the sample image into the discriminator to train the discriminator; and inputting the first image into the target generator to train the target generator.

[0008] In one exemplary embodiment, the step of inputting the first image and the sample image into the discriminator to train the discriminator includes: after inputting the first image and the sample image into the discriminator, calculating a first loss of the discriminator; and adjusting the parameters in the discriminator using the first loss.

[0009] In an exemplary embodiment, inputting the first image into the target generator to train the target generator includes: obtaining a first type of a first object in the first image output by the discriminator after the first image is input; calculating a second loss of the target generator under the first type; and adjusting the parameters in the target generator using the second loss.

[0010] In one exemplary embodiment, obtaining the target image features of the target image includes: obtaining the target image; inputting the target image into a target model, wherein the target model is a model obtained after deleting the fully connected layers of a pre-trained first model; and obtaining the target image features of the target image output by the target model.

[0011] In one exemplary embodiment, before inputting the target image into the target model, the method further includes: acquiring the sample image; training a second model using the sample image to obtain a trained first model; and deleting the fully connected layers of the first model to obtain the target model, wherein the second model is the first model before training.

[0012] In one exemplary embodiment, the discriminator and the convolutional layer of the first model include dilated convolutions with different dilation coefficients.

[0013] According to another aspect of the present invention, an apparatus for annotating the contours of an object in a target image is provided, comprising: a first acquisition unit, configured to acquire target image features of a target image, wherein the target image includes a target object, and the target object is a target type; a first input unit, configured to input the target image features into a target generator, wherein the target generator is a generator in an adversarial generative network trained using sample images, the adversarial generative network including the target generator and a discriminator, the target generator being configured to generate a first mask of the sample image after acquiring the first image features of the sample image, the discriminator being configured to identify the type of the sample object in the sample image after erasing the pixels corresponding to the first mask after receiving a sample image with pixels erased, the type of the sample object being used to train parameters in the target generator; and a second acquisition unit, configured to acquire the target mask of the target image generated by the target generator, wherein the target mask is used to annotate the contours of the target object.

[0014] In one exemplary embodiment, the apparatus further includes: a third acquisition unit, configured to acquire first image features of the sample image before inputting the target image features into the target generator; a second input unit, configured to input the first image features into the target generator to generate a first mask of the sample image; an erasure unit, configured to erase pixels corresponding to the first mask to obtain a first image; a third input unit, configured to input the first image and the sample image into the discriminator to train the discriminator; and a fourth input unit, configured to input the first image into the target generator to train the target generator.

[0015] In one exemplary embodiment, the third input unit includes: a first calculation module, configured to calculate a first loss of the discriminator after inputting the first image and the sample image into the discriminator; and a first adjustment module, configured to adjust the parameters in the discriminator using the first loss.

[0016] In one exemplary embodiment, the fourth input unit includes: a first acquisition module, configured to acquire a first type of a first object in the first image output by the discriminator after the discriminator inputs the first image; a second calculation module, configured to calculate a second loss of the target generator under the first type; and a second adjustment module, configured to adjust the parameters in the target generator using the second loss.

[0017] In an exemplary embodiment, the third acquisition unit includes: a second acquisition module for acquiring the target image; an input module for inputting the target image into a target model, wherein the target model is a model obtained after deleting the fully connected layers of a pre-trained first model; and a third acquisition module for acquiring the target image features of the target image output by the target model.

[0018] In one exemplary embodiment, the third acquisition unit further includes: a fourth acquisition module, configured to acquire the sample image before inputting the target image into the target model; a training module, configured to train a second model using the sample image to obtain the trained first model; and a deletion module, configured to delete the fully connected layers of the first model to obtain the target model, wherein the second model is the first model before training.

[0019] In one exemplary embodiment, the discriminator and the convolutional layer of the first model include dilated convolutions with different dilation coefficients.

[0020] According to one aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps of any of the above method embodiments when it is run.

[0021] According to one aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0022] Through this invention, since the target generator generates a first mask for the sample image during the training process and erases the pixels corresponding to the first mask, the discriminator can consider the overall image during discrimination, thereby promoting the generation of a more accurate mask by the target generator, improving the accuracy of the target generator, and consequently improving the accuracy of the target generator in annotating the contours of target objects in the target image. Therefore, this invention solves the problem of low efficiency in recognizing object contours under weak supervision, thus achieving the effect of improving the efficiency of target object contour recognition. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of a method for annotating the contours of objects in a target image according to an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram illustrating an application scenario of another method for annotating object contours in a target image according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart illustrating a method for annotating the outline of an object in a target image according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the model structure of a method for annotating the contours of objects in a target image according to an embodiment of the present invention;

[0027] Figure 5 This is a flowchart illustrating another method for annotating object contours in a target image according to an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of the model combination structure of a method for annotating the contours of objects in a target image according to an embodiment of the present invention;

[0029] Figure 7 This is a structural block diagram of an object contour annotation device in a target image according to an embodiment of the present invention;

[0030] Figure 8 This is a structural block diagram of an object contour annotation device in a target image according to another embodiment of the present invention. Detailed Implementation

[0031] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0033] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of annotating the contours of objects in a target image according to an embodiment of the present invention. For example... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the object contour annotation method in the target image in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0036] The embodiments of this application can be run in Figure 2 In the network architecture shown, such as Figure 2 As shown, the network architecture includes: terminal 202, network 204 and server 206. Terminal 202 and server 206 can interact with each other through network 204.

[0037] This embodiment provides a method for annotating object contours in a target image running on the aforementioned mobile terminal or network architecture. Figure 3 This is a flowchart of a method for annotating the contours of objects in a target image according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0038] Step S302 obtains the target image features of the target image, wherein the target image includes a target object, and the target object is a target type;

[0039] Step S304: Input the target image features into the target generator, wherein the target generator is a generator in an adversarial generative network trained using sample images. The adversarial generative network includes the target generator and a discriminator. The target generator is used to generate a first mask of the sample image after obtaining the first image features of the sample image. The discriminator is used to identify the type of sample object in the sample image after erasing the pixels corresponding to the first mask after receiving the sample image. The type of sample object is used to train the parameters in the target generator.

[0040] Step S306: Obtain the target mask of the target image generated by the target generator, wherein the target mask is used to mark the outline of the target object.

[0041] Through the above steps, since the target generator generates a first mask for the sample image and erases the corresponding pixels during the training process, the discriminator can consider the overall image during training. This promotes the generation of more accurate masks by the target generator, improving its accuracy and consequently enhancing the accuracy of its annotation of target object contours in the target image. Therefore, the problem of low efficiency in object contour recognition can be solved, thereby improving the efficiency of object contour recognition.

[0042] The entities that perform the above steps can be base stations, terminals, servers, etc., but are not limited to these.

[0043] The target generator aims to generate a good segmentation mask for the input image, preventing the discriminator from classifying the target object in the image after the mask is removed. The discriminator, on the other hand, aims to identify the target category of the object in the image as completely as possible. In other words, for an image containing an object, the target generator's goal is to generate a sufficiently good first mask, and after erasing the pixels corresponding to this mask, the discriminator cannot determine the type of the object in the image. The discriminator's goal is to determine the type of the object in the image based on the content of the objects that have not been erased.

[0044] This application can be applied, but is not limited to, to the process of recognizing the contours of objects in an image. For example, given an input image containing a target object labeled with a target type, this application can perform semantic segmentation on the image by inputting the image into a trained generator to generate a target mask. The target mask labels the contours of the target object in the image.

[0045] For example, given an image containing a cat labeled with the cat tag, when this image is input into a target generator, the target generator generates a target mask that outlines the cat's silhouette.

[0046] Before inputting the target image features into the target generator, the method further includes: acquiring the sample image; acquiring first image features of the sample image; inputting the first image features into the generative adversarial network, whereby the target generator generates a first mask of the sample image; erasing pixels corresponding to the first mask to obtain a first image; inputting the first image and the sample image into the discriminator to train the discriminator; and inputting the first image into the target generator to train the target generator.

[0047] In other words, the target generator and discriminator in this application are pre-trained networks. During pre-training, sample images are used, each containing a first object labeled with a type. After the sample images are input into the target generator, the generator generates a mask of the sample images, marking the target locations in the sample images. Then, by erasing the pixels at the target locations, a first image is obtained. The first image and the sample images are input into the discriminator to train the discriminator. After training, the discriminator can output the type of the first object in the first image. Using this type and the first image to train the target generator, a target generator with a better output mask can be obtained.

[0048] Alternatively, this application may also obtain multiple sample images, input a portion of the sample images into a target generator, generate a first mask from the target generator, erase the pixels corresponding to the first mask of the portion of the sample images, and then input the portion of the sample images with erased pixels and the remaining sample images not input into the target generator into a discriminator to train the discriminator.

[0049] The step of inputting the first image and the sample image into the discriminator to train the discriminator includes: after inputting the first image and the sample image into the discriminator, calculating a first loss of the discriminator; and adjusting the parameters in the discriminator using the first loss. Specifically, this can be achieved by adjusting the parameters in the discriminator when the first loss is greater than a first threshold, wherein the first loss of the discriminator after parameter adjustment is less than or equal to the first threshold.

[0050] During this process, it is necessary to calculate the first loss of the discriminator. The larger the first loss, the worse the convergence of the model. Therefore, it is necessary to adjust the values ​​of the parameters in the model. By continuously adjusting the parameter values ​​and continuously calculating the loss, until the first loss is less than or equal to the first threshold, it is considered that the model parameters are appropriate and the model is sufficiently convergent.

[0051] The step of inputting the first image into the target generator to train the target generator includes: obtaining a first type of a first object in the first image output by the discriminator after inputting the first image; calculating a second loss of the target generator under the first type; and adjusting the parameters in the target generator using the second loss. Specifically, this can be done by adjusting the parameters of the target generator when the second loss is greater than a second threshold, wherein the second loss of the target generator after parameter adjustment is less than or equal to the second threshold.

[0052] The training process of the target generator also requires loss calculation. After the discriminator outputs the type of the first image, the second loss of the target generator under that type is calculated. The larger the second loss is, the worse the convergence effect of the model is. Therefore, it is necessary to adjust the values ​​of the parameters in the model. By continuously adjusting the values ​​of the parameters and continuously calculating the loss, until the second loss is less than or equal to the second threshold, it is considered that the parameters of the model are appropriate and the model is sufficiently convergent.

[0053] The step of obtaining the target image features includes: obtaining the target image; inputting the target image into a target model, wherein the target model is a model obtained after deleting the fully connected layers of a pre-trained first model; and obtaining the target image features of the target image output by the target model.

[0054] In other words, the target image features are obtained through a target model. The target model is a pre-trained model. Using this target model, after inputting a target image, the target model can obtain the target image features.

[0055] Before inputting the target image into the target model, the method further includes: acquiring the sample image; training a second model using the sample image to obtain the trained first model; and deleting the fully connected layers of the first model to obtain the target model.

[0056] In other words, the target model in this application is the model obtained by training a second model using sample images to obtain a first model, and then deleting the fully connected layers of the first model.

[0057] The following example illustrates the method for annotating the object contours in the target image mentioned above.

[0058] The idea behind this embodiment is that when a neural network is trained for image classification, the trained network often does not focus on all the features of the entire target location. Therefore, during training, an erasure method is used to delete pixels containing the features that the network is interested in from the input image. To better identify the target in the image, the neural network must then focus on the features of other parts of the image where the target is located. Through continuous iteration, the neural network eventually focuses on all the features of the entire target location, and this location distribution is consistent with the distribution of the semantic segmentation mask of the target. Thus, the target segmentation mask in the image is finally obtained through category labels. This embodiment uses a generative adversarial training method to achieve the above idea: a target generator generates a first mask for the target in the sample image, and a discriminator classifies the sample image after erasing the corresponding pixels of the first mask. During training, to compete with the discriminator, the target generator generates a better first mask to reduce the number of target pixels in the image, weakening the discriminator's perception of the target; the discriminator, in order to better identify objects in the image, gradually focuses on the features of all locations of the target in the image. After reaching Nash equilibrium, the object generator produces a sufficiently good mask that prevents the discriminator from classifying the image after the mask is removed. The Nash equilibrium described above represents an optimal combination of strategies, meaning the parameters in the object generator and discriminator are optimal. Specifically, the object generator generates a first mask that precisely occludes the object in the image. After erasing the pixels corresponding to the first mask, the object in the image is completely erased. The discriminator cannot determine the type of object because it has been erased. However, if the first mask does not completely obscure the object, the discriminator can determine the object's type by identifying the unoccluded portion.

[0059] The network structure of this application embodiment is as follows: Figure 4 The diagram can be mainly divided into three parts: a pre-trained feature extraction network, a semantic segmentation generation network (target generator), and a discriminant network (discriminator).

[0060] The pre-trained feature extraction network can be a conventional image classification network (second model) (such as Inception, ResNet, etc., which are not limited here). Pre-training is performed on the dataset. After the network converges (after obtaining the first model), the last fully connected layer is removed to obtain the target model. The convolutional features output by the target model are used as input to the target generator. During the training of the second model, dilated convolutions with different dilation parameters are used to replace the convolutions in the pre-trained feature extraction network. Since dilated convolutions have a larger receptive field than regular convolutions, the second network model can have a more comprehensive perception of the target in the image. This makes the perceptual range of the trained target network more closely resemble the semantic segmentation mask of the target, facilitating faster convergence of the subsequent semantic segmentation generation network and ensuring the stability of adversarial training.

[0061] Semantic segmentation generator networks, also known as segmentation prediction generator networks (target generators), use the convolutional features of a pre-trained feature extraction network as input. The network uses deconvolutional layers to gradually increase the width and height of the feature map until it matches the size of the image input to the pre-trained network, and finally performs semantic segmentation prediction on the target in the image.

[0062] The discriminator network also uses conventional image classification networks (such as Inception, ResNet, etc., which are not limited here) to determine the target in the image (including the background category). The input of the discriminator network mainly consists of two parts (such as... Figure 4 (As shown): Image A′ with the corresponding pixels of the predicted mask erased and the real image B. Similarly, to improve the recognition accuracy of the discriminator network, it also uses dilated convolutions with different dilation coefficients to better perceive the targets in the image more comprehensively.

[0063] The training process mainly consists of two steps: First, the feature extraction network is pre-trained on the dataset. After convergence, the fully connected layers of the network are removed, and the final convolutional activation features are used as input to the segmentation prediction network. Then, the segmentation prediction generation network and the discriminator network are trained adversarially. Figure 4 As shown, after image A passes through a feature extraction network and a segmentation prediction generation network, it predicts the mask M of the target in image A. Then, it erases the pixels corresponding to mask M in image A, obtaining image A′. First, the discriminator is trained using A′ and the real image B by minimizing the classification loss. Then, the target generator is trained using only A′ by minimizing the loss for non-current categories. Through repeated iterations, convergence is eventually achieved.

[0064] like Figure 5As shown in step S502, before training, sample images need to be acquired. During sample collection, to achieve better segmentation results during training, it is necessary to collect as many sample images as possible that may appear in the application scenario. Images can be obtained through the following channels: utilizing various publicly available indoor scene images containing people; collecting images in actual application scenarios; purchasing from third-party data companies; generating images using image generation algorithms (such as GANs); and collecting images via web crawlers for academic purposes.

[0065] S504. After data collection, it needs to be cleaned and labeled. To better train the network, the collected data needs to be validated and checked to ensure the completeness, uniformity, and accuracy of the samples. Completeness means the dataset should include as many possible scenarios as possible in the application to ensure the generalization ability of the trained model. For example, motion blur caused by rapid movement of the target in the sample image should also be included in the dataset. Uniformity means the number of samples of different categories in the dataset should be as consistent as possible, without significant differences. Accuracy means that there should be clear labeling standards in the data annotation to prevent labeling confusion.

[0066] After obtaining the sample data, step S506 can be executed to train the model. First, the second model is trained to obtain the first model, and then the target model is obtained.

[0067] First, the pre-trained feature extraction network (second model) is trained on the dataset for classification. The training uses cross-entropy for loss calculation, and the calculation formula is as follows:

[0068]

[0069] Where z is the non-softmax output predicted by the network, and c is the category of the label.

[0070] After training converges, the first model is obtained. The last fully connected layer used for classification in the first model is then removed to obtain the target model. The output of the target model is used as the input to the subsequent segmentation prediction network. If the amount of data is insufficient, data augmentation can be performed on the input data to improve the network performance.

[0071] Next, the target generator and discriminator need to be trained. After the pre-trained feature extraction network is trained, the segmentation prediction generation network and the discriminator network will be adversarially trained. At this time, the pre-trained feature extraction network and the segmentation prediction network will be trained as a whole, but the pre-trained feature extraction network layer will have a smaller learning rate compared to the segmentation prediction network.

[0072] (1) For each dataset batch, the discriminator is first trained by sending the real data and the image corresponding to the erased mask to the discriminator. After passing through the softmax function, the cross-entropy is used to calculate the loss.

[0073] (2) Then the target generator is trained. The image of the erased mask prediction pixel is fed into the discriminator and then passes through the softmin function. The loss is calculated by cross-entropy based on the type of the erased mask prediction pixel image output by the discriminator.

[0074] (3) Repeat steps (1) and (2) above until the model converges.

[0075] The above process completes the training of the model.

[0076] The S508 model can then be deployed and validated. After training, if network deployment is required, such as... Figure 6 The diagram shows how combining a feature extraction network (target network) and a segmentation prediction generation network (target generator) yields a complete segmentation prediction network. Only the original image data needs to be input to obtain the corresponding semantic segmentation prediction results. This network can be applied to most semantic segmentation application scenarios. For example, for an image containing objects, the semantic segmentation prediction result can include the outlines of the objects in the image, thus allowing the objects to be labeled.

[0077] To verify the model's actual performance, the network output mask and the manually annotated mask will be compared. The Mean Interaction of Union (MIoU) can be used to evaluate the mask prediction quality, and its definition is as follows:

[0078]

[0079] Where N+1 represents the number of categories (including empty categories), and N is an integer. p ij p represents the pixels in the image that are actually of class i but predicted to be of class j. ii Let i and j be the pixels that are actually of class i and are predicted to be of class i, where i and j are integers.

[0080] The method described in this application uses an adversarial generative network, trained by erasing pixels corresponding to the mask. Compared to supervised semantic segmentation training methods, this application significantly reduces the workload required for neural network annotation and lowers manual costs because it only uses classification labels. Furthermore, compared to other semi-supervised semantic segmentation methods based on class activation maps (CAM), this invention requires minimal prior human experience and no additional parameters. With the same amount of training data, the trained network is more robust and has better generalization, resulting in more accurate recognition of object contours in images.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0082] This embodiment also provides a device for annotating the outline of an object in a target image. This device is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0083] Figure 7 This is a structural block diagram of a device for annotating the outline of an object in a target image according to an embodiment of the present invention, such as... Figure 7 As shown, the device includes

[0084] The first acquisition unit 702 is used to acquire target image features of a target image, wherein the target image includes a target object, and the target object is a target type;

[0085] The first input unit 704 is used to input the target image features into the target generator, wherein the target generator is a generator in an adversarial generative network trained using sample images. The adversarial generative network includes the target generator and a discriminator. The target generator is used to generate a first mask of the sample image after acquiring the first image features of the sample image. The discriminator is used to identify the type of sample object in the sample image after erasing the pixels corresponding to the first mask after receiving the sample image. The type is used to train the parameters in the generator.

[0086] The second acquisition unit 706 is used to acquire the target mask of the target image generated by the target generator, wherein the target mask is used to mark the outline of the target object.

[0087] Figure 8 This is a structural block diagram of a device for annotating the outline of an object in a target image according to an embodiment of the present invention, such as... Figure 8 As shown, the device includes Figure 7 In addition to all the modules shown, it also includes:

[0088] The third acquisition unit 802 is used to acquire the sample image before inputting the target image features into the target generator;

[0089] The fourth acquisition unit 804 is used to acquire the first image feature of the sample image;

[0090] The second input unit 806 is used to input the first image features into the generative adversarial network, so that the target generator generates the first mask of the sample image;

[0091] The erasing unit 808 is used to erase the pixels corresponding to the first mask to obtain the first image;

[0092] The third input unit 810 is used to input the first image and the sample image into the discriminator to train the discriminator;

[0093] The fourth input unit 812 is used to input the first image into the target generator to train the target generator.

[0094] The third input unit includes: a first calculation module, used to calculate a first loss of the discriminator after inputting the first image and the sample image into the discriminator; and a first adjustment module, used to adjust the parameters in the discriminator when the first loss is greater than a first threshold, wherein the first loss of the discriminator after parameter adjustment is less than or equal to the first threshold.

[0095] The fourth input unit includes: a first acquisition module, used to acquire the first type of the first object in the first image output by the discriminator after inputting the first image; a second calculation module, used to calculate the second loss of the target generator under the first type; and a second adjustment module, used to adjust the parameters of the target generator when the second loss is greater than a second threshold, wherein the second loss of the target generator after parameter adjustment is less than or equal to the second threshold.

[0096] The fourth acquisition unit includes: a second acquisition module for acquiring the target image; an input module for inputting the target image into a target model, wherein the target model is a model obtained after deleting the fully connected layers of the pre-trained first model; and a third acquisition module for acquiring the target image features of the target image output by the target model.

[0097] The fourth acquisition unit further includes: a fourth acquisition module, used to acquire the sample image before inputting the target image into the target model; a training module, used to train a second model using the sample image to obtain the trained first model; and a deletion module, used to delete the fully connected layers of the first model to obtain the target model.

[0098] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0099] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0100] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0101] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0102] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0103] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0104] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for annotating the contours of objects in a target image, characterized in that, include: Obtain the target image features of the target image, wherein the target image includes a target object, and the target object is a target type; The target image features are input into a target generator, wherein the target generator is a generator in an adversarial generative network trained using sample images. The adversarial generative network includes the target generator and a discriminator. The target generator is used to generate a first mask of the sample image after acquiring the first image features of the sample image. The discriminator is used to identify the type of sample object in the sample image after erasing the pixels corresponding to the first mask after receiving the sample image. The type of sample object is used to train the parameters in the target generator. Obtain the target mask of the target image generated by the target generator, wherein the target mask is used to mark the outline of the target object; The target image features obtained from the target image include: Acquire the target image; The target image is input into the target model, wherein the target model is the model obtained after removing the fully connected layers of the pre-trained first model; Obtain the target image features of the target image output by the target model.

2. The method according to claim 1, characterized in that, Before inputting the target image features into the target generator, the method further includes: Obtain the first image feature of the sample image; The first image features are input into the target generator to generate the first mask of the sample image; Erase the pixels corresponding to the first mask to obtain the first image; The first image and the sample image are input into the discriminator to train the discriminator; The first image is input into the target generator to train the target generator.

3. The method according to claim 2, characterized in that, The step of inputting the first image and the sample image into the discriminator to train the discriminator includes: After inputting the first image and the sample image into the discriminator, the first loss of the discriminator is calculated; The parameters in the discriminator are adjusted using the first loss.

4. The method according to claim 2, characterized in that, The step of inputting the first image into the target generator to train the target generator includes: Obtain the first type of the first object in the first image output by the discriminator after the first image is input; Calculate the second loss of the target generator under the first type; The parameters in the target generator are adjusted using the second loss.

5. The method according to claim 1, characterized in that, Before inputting the target image into the target model, the method further includes: Acquire the sample image; The second model is trained using the sample images to obtain the first model after training, wherein the second model is the first model before training; The target model is obtained by deleting the fully connected layer of the first model.

6. The method according to claim 5, characterized in that, The discriminator and the convolutional layer of the first model include dilated convolutions with different dilation coefficients.

7. A device for annotating the outline of an object in a target image, characterized in that, include: The first acquisition unit is used to acquire target image features of the target image, wherein the target image includes a target object, and the target object is a target type; The first input unit is used to input the target image features into the target generator, wherein the target generator is a generator in an adversarial generative network trained using sample images. The adversarial generative network includes the target generator and a discriminator. The target generator is used to generate a first mask of the sample image after acquiring the first image features of the sample image. The discriminator is used to identify the type of sample object in the sample image after erasing the pixels corresponding to the first mask after receiving the sample image. The type of sample object is used to train the parameters in the target generator. The second acquisition unit is used to acquire the target mask of the target image generated by the target generator, wherein the target mask is used to mark the outline of the target object; The first acquisition unit is further configured to: acquire the target image; input the target image into a target model, wherein the target model is a model obtained after deleting the fully connected layers of the pre-trained first model; and acquire the target image features of the target image output by the target model.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 6.

Images