Image instance annotation method, device, equipment and medium
Through the combination of convolutional neural network and graph convolutional network, image features are extracted and instance annotated are performed, which solves the problem of inefficient image instance annotation and achieves more efficient and accurate data annotation.
Patent Information
- Application Number
- CN202210428476.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-22
AI Technical Summary
In the prior art, the image instance labeling is inefficient, especially for larger image instances, more coordinate point labeling is required, resulting in a reduced label generation efficiency.
The target image is extracted through convolutional neural network, the target prediction box is determined and the features are intercepted, and the graph convolutional network is used to perform instance category annotation and coordinate annotation, so as to streamline the complex multi-scale features in the network structure and improve the production efficiency of data annotation work .
It improves the production efficiency of data labeling, reduces the workload of manual labeling, improves the labeling accuracy, simplifies the network structure, and improves the efficiency and accuracy of the labeling process.
Smart Images

Figure CN114863120B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an image instance annotation method, device, equipment and medium. Background Art
[0002] With the rapid development and application of artificial intelligence and deep neural network models, basic tasks in the field of computer vision, such as image classification, semantic segmentation, instance segmentation, panoptic segmentation, etc., have made unprecedented progress. In order to meet the scientific research and technical practice of computer vision tasks, a large amount of manually labeled data is required. However, the working method of completely using manual visual interpretation to label data is inefficient and the accuracy is difficult to guarantee. Therefore, being able to quickly and accurately generate image instance annotation results has a positive role in promoting industrial production, scientific research and education.
[0003] One of the prior arts provides a semi-automatic image instance annotation method based on a recurrent neural network, called Polygon-RNN and its improved version Polygon-RNN++. This method requires manually giving an initial rectangular box around each image instance target, and then processing the image in the rectangular box through a deep neural network and outputting the annotation of the image instance. Although this method can effectively obtain usable image instance annotations, since the generation of instance annotations in the image is generated in sequence according to a sequence of coordinate points, more coordinate points are required for annotation of larger image instances, which greatly reduces the efficiency of annotation generation. The second prior art proposes an active contour annotation method based on a graph convolutional network, which initializes a contour consisting of a circular annotation sequence with a fixed number of vertices at the center of the rectangular box, and then predicts the offset of each vertex so that all vertices are moved simultaneously to fit the outer contour of the image instance. This method simultaneously moves all the annotated vertices to the edge of the image instance, thereby accelerating the instance annotation process and improving the efficiency of annotated data production. However, this work still requires manual selection of multiple instance objects in the image before automatic annotation of the image instance can be performed, so the above method still has defects.
[0004] Therefore, how to improve the productivity of data labeling work is an urgent problem to be solved in this field. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide an image instance annotation method, device, equipment and medium, which can improve the production efficiency of data annotation work. The specific scheme is as follows:
[0006] In a first aspect, the present application discloses an image instance annotation method, comprising:
[0007] The target image is subjected to feature extraction through a convolutional neural network to obtain extracted convolution features;
[0008] Determine one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features;
[0009] The extracted convolution features are intercepted using the target prediction box, and instance category and instance coordinate annotations are performed on the intercepted features corresponding to each target prediction box through a graph convolutional network to obtain category vectors and labeled coordinate sequences corresponding to target instances in each target prediction box in the target image.
[0010] Optionally, extracting features from the target image using a convolutional neural network to obtain extracted convolutional features includes:
[0011] The target image is operated by the Faster R-CNN pre-trained model, and features are extracted from the corresponding operation results based on the ResNet50 network to obtain the extracted convolution features.
[0012] Optionally, determining one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features includes:
[0013] Based on the extracted convolution features, one or more target prediction boxes, the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box, and the width and height of the target prediction box are determined.
[0014] Optionally, the intercepting the extracted convolutional features by using the target prediction frame includes:
[0015] The extracted convolutional features are intercepted using the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and the height of the target prediction box.
[0016] Optionally, before extracting features of the target image by using a convolutional neural network to obtain extracted convolutional features, the method further includes:
[0017] A preset training set is used to train a preset multi-task learning network framework based on a convolutional neural network and a graph convolutional neural network, and during the training process, a preset loss function in the network framework is continuously optimized and adjusted until the preset training end conditions are met.
[0018] Optionally, the preset loss function is:
[0019]
[0020] Among them, L clsis the loss function for the target instance classification task; N c is the total number of target instances to be classified; p and p * are the category labels before and after prediction respectively; L reg is the loss function of the target prediction box coordinate regression task; t and t * are the coordinate vector of the target prediction frame before and after regression, respectively; λ is the synergy coefficient between different tasks; L ins is the loss function for the target instance labeling task; i and i * are the annotations before and after the prediction of the target instance respectively; μ is the synergy coefficient of the instance annotation task.
[0021] Optionally, after the instance category labeling and instance coordinate labeling are respectively performed on the clipped features corresponding to each target prediction box by using the graph convolution network, the method further includes:
[0022] The category vector is used to search a pre-created category vector library to determine, from the category vector library, the actual category corresponding to the category vector of each target instance in all the target prediction boxes in the target image.
[0023] In a second aspect, the present application discloses an image instance annotation device, comprising:
[0024] A feature extraction module is used to extract features of a target image through a convolutional neural network to obtain extracted convolution features;
[0025] A target prediction box determination module, used to determine one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features;
[0026] A classification and labeling module is used to use the target prediction box to intercept the extracted convolution features, and to perform instance category labeling and instance coordinate labeling on the intercepted features corresponding to each target prediction box through a graph convolutional network, so as to obtain the category vector and labeling coordinate sequence corresponding to each target instance in the target prediction box in the target image.
[0027] In a third aspect, the present application discloses an electronic device, comprising:
[0028] Memory, used to store computer programs;
[0029] A processor is used to execute the computer program to implement the aforementioned disclosed image instance annotation method.
[0030] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the image instance annotation method disclosed above is implemented.
[0031] It can be seen that the present application proposes an image instance annotation method, including: extracting features of a target image through a convolutional neural network to obtain extracted convolutional features; determining one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolutional features; intercepting the extracted convolutional features using the target prediction box, and respectively performing instance category annotation and instance coordinate annotation on the intercepted features corresponding to each target prediction box through a graph convolutional network, to obtain the category vectors and annotation coordinate sequences corresponding to the target instances in each target prediction box in the target image. It can be seen that the present application, in response to the problem of low production efficiency of manual visual annotation, proposes a method of first determining a target prediction box, then intercepting the convolutional features of the target image through the target prediction box, and further reasoning about the intercepted features, thereby simplifying the complex multi-scale features in the network structure and improving the production efficiency of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0033] Figure 1 A schematic diagram of an image instance annotation principle based on multi-task learning disclosed in this application;
[0034] Figure 2 A flow chart of an image instance annotation method disclosed in this application;
[0035] Figure 3 A flowchart of a specific image instance annotation method disclosed in this application;
[0036] Figure 4 This is a schematic diagram of the structure of an image instance annotation device disclosed in this application;
[0037] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] Figure 1 The principle diagram of image instance annotation based on multi-task learning proposed in this application is Figure 1 It can be seen that the basic design idea of this application is: in the image instance annotation scenario, in order to solve the problem of manually selecting the circumscribed rectangular frame of the image instance in the method of annotating image instances based on graph convolutional networks, this application uses a target detection task as a substitute, and captures the convolutional features of the target image after the target detection task, and finally integrates the instance classification task and the instance annotation task to achieve the classification and annotation of each instance in the target image. It should be pointed out that the target detection task is to determine the target detection frame in the target image.
[0040] In order to meet the scientific research and technical practice of computer vision tasks, a large amount of manually annotated data is required. However, the work method of completely using manual visual interpretation of annotated data is inefficient and difficult to ensure accuracy. Therefore, being able to quickly and accurately generate image instance annotation results has a positive role in promoting industrial production, scientific research and education.
[0041] To this end, an embodiment of the present application proposes an image instance annotation scheme that can improve the production efficiency of data annotation work.
[0042] The present application discloses a method for labeling an image instance. Figure 2 As shown, the method includes:
[0043] Step S11: extract features of the target image through a convolutional neural network to obtain extracted convolution features.
[0044] It should be pointed out that before extracting features from the target image through the convolutional neural network, for a given target detection neural network model Faster R-CNN, the loss function that needs to be optimized is:
[0045]
[0046] Among them, L cls is the loss function for the target instance classification task; N c is the total number of target instances to be classified; p and p * are the category labels before and after prediction respectively; Lreg is the loss function of the target prediction box coordinate regression task; t and t * are the coordinate vector of the target prediction box before regression and after regression, respectively; λ is the synergy coefficient between different tasks, and in this embodiment, an empirical constant can be taken.
[0047] Step S12: determining one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features.
[0048] In this embodiment, after the extracted convolution features are determined, one or more target prediction boxes and coordinate vectors of the target prediction boxes may be determined based on the extracted convolution features.
[0049] Step S13: Using the target prediction box to intercept the extracted convolution features, and using the graph convolution network to perform instance category labeling and instance coordinate labeling on the intercepted features corresponding to each of the target prediction boxes, to obtain the category vectors and labeled coordinate sequences corresponding to the target instances in each of the target prediction boxes in the target image.
[0050] In this embodiment, a graph convolutional network (GCN) is connected to the output of the convolutional neural network so that the reverse gradient of the graph convolutional network during training can be fed back to the convolutional neural network.
[0051] In this embodiment, after adding the instance annotation task based on the graph convolutional network, a preset multi-task learning network framework is formed based on the convolutional neural network and the graph convolutional neural network. Therefore, in this embodiment, the network framework is trained using the preset training set, and the preset loss function in the network framework is continuously optimized and adjusted during the training process until the preset training end condition is met. That is, the loss function that needs to be optimized at this time in this embodiment is:
[0052]
[0053] Among them, L ins is the loss function for the target instance labeling task; i and i * are the annotations before and after the prediction of the target instance respectively; μ is the coordination coefficient of the instance annotation task, and in this embodiment, μ can be a constant of 200.
[0054] In a specific implementation, in this embodiment, the number of iterations in the training process may be 20,000, batch size = 16, the learning rate is 0.001, and a decay of 0.1 times is performed every 100 iterations.
[0055] In this embodiment, the target image is operated by using the Faster R-CNN pre-trained model trained based on the COCO (Common Objects in Contex) dataset, and the ResNet50 network is selected as the feature extraction layer of the Convolutional Neural Network (CNN) to extract features from the corresponding operation results to obtain the extracted convolution features. It should be noted that the COCO dataset is a large and rich object detection, segmentation and captioning dataset; the image size of the target image includes but is not limited to 224×224.
[0056] It is understandable that by intercepting the convolution features of the target image through the target prediction frame, and annotating the instance category and instance coordinates of the intercepted features, the category vector and the annotation coordinate sequence corresponding to each target instance in the target prediction frame in the target image can be obtained. Since after the convolution features of the target image are intercepted, only the intercepted part of the features need to be calculated, the present application simplifies the complex multi-scale features in the network structure and improves the production efficiency of data annotation. In addition, since in the present application, the manual annotation work at this time will be transformed from the basic drawing of rectangular boxes to the correction of model output, the present application not only improves work efficiency but also ensures production accuracy.
[0057] It can be seen that the present application proposes an image instance annotation method, including: extracting features of a target image through a convolutional neural network to obtain extracted convolutional features; determining one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolutional features; intercepting the extracted convolutional features using the target prediction box, and respectively performing instance category annotation and instance coordinate annotation on the intercepted features corresponding to each target prediction box through a graph convolutional network, to obtain the category vectors and annotation coordinate sequences corresponding to the target instances in each target prediction box in the target image. It can be seen that the present application, in response to the problem of low production efficiency of manual visual annotation, proposes a method of first determining a target prediction box, then intercepting the convolutional features of the target image through the target prediction box, and further reasoning about the intercepted features, thereby simplifying the complex multi-scale features in the network structure and improving the production efficiency of data annotation.
[0058] The embodiment of the present application discloses a specific method for labeling an image instance. Compared with the previous embodiment, the present embodiment further explains and optimizes the technical solution. Figure 3 As shown, specifically including:
[0059] Step S21: extract features of the target image through a convolutional neural network to obtain extracted convolution features.
[0060] Among them, the more specific working process of step S21 is shown in the aforementioned disclosed embodiment, and will not be described in detail here.
[0061] Step S22: determining one or more target prediction boxes, the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box, and the width and height of the target prediction box based on the extracted convolution features.
[0062] In this embodiment, the vector formed by the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and height of the target prediction box is also the coordinate vector of the target prediction box. It should be noted that the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and height of the target prediction box can be in pixels.
[0063] Step S23: intercepting the extracted convolutional features using the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and height of the target prediction box.
[0064] It can be understood that after determining the horizontal and vertical coordinates of the upper left corner vertex of the target prediction box and the width and height of the target prediction box, the extracted convolution feature can be intercepted based on the horizontal and vertical coordinates of the upper left corner vertex of the target prediction box and the width and height of the target prediction box.
[0065] Step S24: perform instance category labeling and instance coordinate labeling on the clipped features corresponding to each of the target prediction boxes through a graph convolutional network to obtain a category vector and a labeling coordinate sequence corresponding to each target instance in each of the target prediction boxes in the target image.
[0066] Among them, the more specific working process of step S24 is shown in the aforementioned disclosed embodiment, and will not be described in detail here.
[0067] Step S25: using the category vector to search a pre-created category vector library, so as to determine from the category vector library the actual category corresponding to the category vector of each target instance in all the target prediction boxes in the target image.
[0068] It should be pointed out that the pre-created category vector library stores the correspondence between category vectors and categories. Therefore, in this embodiment, by using the category vector to search the pre-created category vector library, the actual category corresponding to the category vector of each target instance in all the target prediction boxes in the target image can be determined.
[0069] It can be seen that the vector formed by the horizontal and vertical coordinates of the upper left corner vertex of the target prediction box and the width and height of the target prediction box in the present application is also the coordinate vector of the target prediction box. After determining the coordinate vector of the target prediction box, the extracted convolution feature can be intercepted based on the coordinate vector. In this way, the complex multi-scale features in the network structure are simplified and the production efficiency of data labeling is improved; in addition, after the instance category and instance coordinates are respectively annotated for each of the intercepted features corresponding to the target prediction box through the graph convolution network, the category vector is used to search the pre-created category vector library. In this way, the actual category corresponding to the category vector of each target instance in all the target prediction boxes in the target image can be determined from the category vector library.
[0070] Correspondingly, the present application also discloses an image instance annotation device, see Figure 4 As shown, the device comprises:
[0071] The feature extraction module 11 is used to extract features of the target image through a convolutional neural network to obtain extracted convolution features;
[0072] A target prediction box determination module 12 is used to determine one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features;
[0073] The classification and labeling module 13 is used to use the target prediction box to intercept the extracted convolution features, and to perform instance category labeling and instance coordinate labeling on the intercepted features corresponding to each target prediction box through a graph convolutional network, so as to obtain the category vector and labeling coordinate sequence corresponding to each target instance in the target prediction box in the target image.
[0074] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0075] It can be seen that the present application proposes an image instance annotation method, including: extracting features of a target image through a convolutional neural network to obtain extracted convolutional features; determining one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolutional features; intercepting the extracted convolutional features using the target prediction box, and respectively performing instance category annotation and instance coordinate annotation on the intercepted features corresponding to each target prediction box through a graph convolutional network, to obtain the category vectors and annotation coordinate sequences corresponding to the target instances in each target prediction box in the target image. It can be seen that the present application, in response to the problem of low production efficiency of manual visual annotation, proposes a method of first determining a target prediction box, then intercepting the convolutional features of the target image through the target prediction box, and further reasoning about the intercepted features, thereby simplifying the complex multi-scale features in the network structure and improving the production efficiency of data annotation.
[0076] Furthermore, an embodiment of the present application also provides an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0077] Figure 5 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the following steps:
[0078] The target image is subjected to feature extraction through a convolutional neural network to obtain extracted convolution features;
[0079] Determine one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features;
[0080] The extracted convolution features are intercepted using the target prediction box, and instance category and instance coordinate annotations are performed on the intercepted features corresponding to each target prediction box through a graph convolutional network to obtain category vectors and labeled coordinate sequences corresponding to target instances in each target prediction box in the target image.
[0081] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0082] The target image is operated by the Faster R-CNN pre-trained model, and features are extracted from the corresponding operation results based on the ResNet50 network to obtain the extracted convolution features.
[0083] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0084] Based on the extracted convolution features, one or more target prediction boxes, the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box, and the width and height of the target prediction box are determined.
[0085] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0086] The extracted convolutional features are intercepted using the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and the height of the target prediction box.
[0087] In some specific implementations, the processor may further implement the following steps by executing the computer program stored in the memory:
[0088] A preset training set is used to train a preset multi-task learning network framework based on a convolutional neural network and a graph convolutional neural network, and during the training process, a preset loss function in the network framework is continuously optimized and adjusted until the preset training end conditions are met.
[0089] In some specific implementations, the processor may execute the computer program stored in the memory, which may specifically include:
[0090] The preset loss function is:
[0091]
[0092] Among them, L cls is the loss function for the target instance classification task; N c is the total number of target instances to be classified; p and p * are the category labels before and after prediction respectively; L reg is the loss function of the target prediction box coordinate regression task; t and t * are the coordinate vector of the target prediction frame before and after regression, respectively; λ is the synergy coefficient between different tasks; L ins is the loss function for the target instance labeling task; i and i *are the annotations before and after the prediction of the target instance respectively; μ is the synergy coefficient of the instance annotation task.
[0093] In some specific implementations, the processor may further implement the following steps by executing the computer program stored in the memory:
[0094] The category vector is used to search a pre-created category vector library to determine, from the category vector library, the actual category corresponding to the category vector of each target instance in all the target prediction boxes in the target image.
[0095] In this embodiment, the power supply 26 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 24 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0096] In addition, the memory 22, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include a computer program 221, and the storage method may be temporary storage or permanent storage. In addition to including a computer program that can be used to complete the image instance annotation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 221 may further include a computer program that can be used to complete other specific tasks.
[0097] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the image instance annotation method disclosed above is implemented.
[0098] For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.
[0099] The various embodiments in this application are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0100] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0101] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0102] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0103] The above is a detailed introduction to an image instance annotation method, device, equipment, and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for labeling image instances. It is characterized in that include: The target image is subjected to feature extraction through a convolutional neural network to obtain extracted convolution features; Determine one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features; The extracted convolution features are intercepted by using the target prediction box, and instance category and instance coordinate annotation are respectively performed on the intercepted features corresponding to each target prediction box through a graph convolutional network to obtain a category vector and annotated coordinate sequence corresponding to each target instance in each target prediction box in the target image; The method of extracting features of the target image by using a convolutional neural network to obtain extracted convolutional features includes: The target image is operated by using the Faster R-CNN pre-trained model trained based on the COCO dataset, and features are extracted from the corresponding operation results based on the ResNet50 network to obtain extracted convolution features; The determining one or more target prediction boxes and the coordinate vectors of the target prediction boxes based on the extracted convolution features includes: Determine one or more target prediction boxes, the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box, and the width and height of the target prediction box based on the extracted convolution features; The method of intercepting the extracted convolutional features by using the target prediction frame includes: The extracted convolution feature is intercepted using the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and the height of the target prediction box; Before extracting features from the target image through the convolutional neural network to obtain the extracted convolution features, the method further includes: Using a preset training set to train a preset multi-task learning network framework based on a convolutional neural network and a graph convolutional neural network, and continuously optimizing and adjusting a preset loss function in the network framework during the training process until a preset training end condition is met; The preset loss function is: Among them, L cls is the loss function for the target instance classification task; N c is the total number of target instances to be classified; p and p * are the category labels before and after prediction, respectively; L reg is the loss function of the target prediction box coordinate regression task; t and t * are the coordinate vector of the target prediction frame before and after regression, respectively; λ is the synergy coefficient between different tasks; L ins is the loss function for the target instance labeling task; i and i * are the annotations before and after the prediction of the target instance respectively; μ is the synergy coefficient of the instance annotation task.
2. The image instance annotation method according to any one of claim 1, It is characterized in that After the captured features corresponding to each target prediction box are respectively labeled with instance categories and instance coordinates through the graph convolution network, the method further includes: The category vector is used to search a pre-created category vector library to determine, from the category vector library, the actual category corresponding to the category vector of each target instance in all the target prediction boxes in the target image.
3. An image instance annotation device, It is characterized in that include: A feature extraction module is used to extract features of a target image through a convolutional neural network to obtain extracted convolution features; A target prediction box determination module, used to determine one or more target prediction boxes and coordinate vectors of the target prediction boxes based on the extracted convolution features; A classification and labeling module, used to intercept the extracted convolution features using the target prediction box, and perform instance category labeling and instance coordinate labeling on the intercepted features corresponding to each target prediction box through a graph convolutional network, so as to obtain a category vector and a labeling coordinate sequence corresponding to each target instance in the target prediction box in the target image; The method of extracting features of the target image by using a convolutional neural network to obtain extracted convolutional features includes: The target image is operated by using the Faster R-CNN pre-trained model trained based on the COCO dataset, and features are extracted from the corresponding operation results based on the ResNet50 network to obtain extracted convolution features; The determining one or more target prediction boxes and the coordinate vectors of the target prediction boxes based on the extracted convolution features includes: Determine one or more target prediction boxes, the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box, and the width and height of the target prediction box based on the extracted convolution features; The method of intercepting the extracted convolutional features by using the target prediction frame includes: The extracted convolution feature is intercepted using the horizontal coordinate and the vertical coordinate of the upper left corner vertex of the target prediction box and the width and the height of the target prediction box; Before extracting features from the target image through the convolutional neural network to obtain the extracted convolution features, the method further includes: Using a preset training set to train a preset multi-task learning network framework based on a convolutional neural network and a graph convolutional neural network, and continuously optimizing and adjusting a preset loss function in the network framework during the training process until a preset training end condition is met; The preset loss function is: Among them, L cls is the loss function for the target instance classification task; N c is the total number of target instances to be classified; p and p * are the category labels before and after prediction, respectively; L reg is the loss function of the target prediction box coordinate regression task; t and t * are the coordinate vector of the target prediction frame before and after regression, respectively; λ is the synergy coefficient between different tasks; L ins is the loss function for the target instance labeling task; i and i * are the annotations before and after the prediction of the target instance respectively; μ is the synergy coefficient of the instance annotation task.
4. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the image instance annotation method as claimed in claim 1 or 2.
5. A computer-readable storage medium, It is characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the image instance annotation method according to claim 1 or 2 is implemented.
