Image labeling method, device and equipment for automatic welting of pull frame, medium and product

Abstract

The invention discloses an image annotation method and device for automatic welting of a pull frame, equipment, a medium and a product, and relates to the field of rail transit and image recognition, and the method comprises the steps: obtaining a target image; performing frame-pulling annotation on the target image to obtain a target image after frame-pulling annotation; inputting the target image subjected to frame-pulling annotation into a trained image semantic segmentation model to obtain an annotated image of the target image; according to the method, the problems that in an image annotation project, the error of the pull frame annotation is large, the polygon annotation is irregular, the polygon vertex annotation is tedious and not accurate enough, and the requirement of a deep learning-based image semantic segmentation annotation method model is high and the annotation is not accurate can be solved; and the labeling efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of rail transit and image recognition, and in particular to a method, device, equipment, medium and product for automatically labeling an image with a frame. Background Art

[0002] Traditional image object labeling is done purely manually, and images are labeled from scratch. The labeling methods mostly use box labeling, polygon labeling, and labeling based on deep learning image semantic segmentation.

[0003] The box annotation method is currently the most widely used image annotation method. It can quickly frame the specified target object in the image or video data using a rectangular box in a relatively simple and convenient way. However, the box annotation method has a large annotation error. It is difficult to accurately annotate irregularly shaped target objects. It can only be framed, which will inevitably include some background pixels that do not belong to the object, resulting in large errors. As samples, it will affect the effect of machine learning model training.

[0004] Polygonal annotation refers to the use of polygonal boxes to mark irregular target objects in static images. Compared with rectangular box annotation, polygonal annotation can frame the target more accurately and is more targeted for irregular objects. However, polygonal annotation has low annotation efficiency and low accuracy. It requires the use of polygon tools to manually mark the target entity and adjust the control points multiple times to draw the boundaries of the annotated object, which is time-consuming and labor-intensive, and it is difficult to achieve pixel-level accuracy.

[0005] The annotation method based on deep learning image semantic segmentation uses the deep learning semantic segmentation method to pre-annotate the entire image, and then uses the deep learning-based target detection algorithm again to detect the target category and target location area, and further semantic segmentation is performed within the area, and finally the annotator uses the tool to make corrections. The annotation method based on deep learning image semantic segmentation has high requirements for the model to annotate the entire image, and the annotated results are box selections, with low accuracy, and the degree of manual intervention depends on the accuracy of the model pre-annotation. For complex images or scenes with low model accuracy, more manual intervention is required.

[0006] Therefore, a method for automatically sticking to the edges of a frame is needed to improve the efficiency and accuracy of image annotation, so as to solve the problems of large errors in frame annotation, irregular polygon annotation, cumbersome and inaccurate polygon vertex annotation, and high model requirements and inaccurate annotation based on deep learning image semantic segmentation annotation methods in image annotation projects. Summary of the invention

[0007] The purpose of this application is to provide an image annotation method, device, equipment, medium and product for automatic frame-binding, which can solve the problems of large frame annotation errors, irregular polygon annotation, cumbersome and inaccurate polygon vertex annotation, and high model requirements and inaccurate annotation of deep learning image semantic segmentation annotation methods in image annotation projects, so as to improve annotation efficiency and accuracy.

[0008] To achieve the above-mentioned objectives, the present application provides the following solutions: In the first aspect, the present application provides an image annotation method with automatic frame alignment, comprising: acquiring a target image; annotating the target image with a frame to obtain a target image after frame annotation; inputting the target image with frame annotation into a trained image semantic segmentation model to obtain an annotated image of the target image; the annotation box in the annotated image is located inside the frame of the frame annotation.

[0009] In a second aspect, the present application provides an image annotation device that automatically sticks to the edges of a frame, including: an acquisition module, used to acquire a target image; a frame annotation module, used to annotate the target image with a frame to obtain the target image after the frame annotation; an annotation module, used to input the target image after the frame annotation into a trained image semantic segmentation model to obtain an annotated image of the target image; the annotation box in the annotated image is located inside the frame of the frame annotation.

[0010] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for automatically labeling images by pulling a frame.

[0011] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for automatically labeling an image by pulling a frame.

[0012] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for automatically labeling images by pulling a frame.

[0013] According to the specific embodiments provided by the present application, the present application has the following technical effects: the present application provides an image annotation method, device, equipment, medium and product with automatic frame-drawing and edge-adapting, firstly draw a frame to annotate the target image to obtain the target image after the frame-drawing annotation; the target image after the frame-drawing annotation is input into the trained image semantic segmentation model to obtain the annotated image of the target image, firstly use the frame-drawing annotation to determine the approximate area to be annotated of the target image, and then annotate the image, so that the trained image semantic segmentation model recognizes fewer objects, which can solve the problem of high model requirements of deep learning image semantic segmentation annotation methods, and the obtained annotation frame fits the edge of the object, which can solve the problem of inaccurate annotation of the model based on deep learning image semantic segmentation annotation methods. Fully automatic annotation is adopted, and the obtained annotated image annotation frame fits the edge of the object, which can solve the problems of large error in frame-drawing annotation, irregular polygon annotation, and cumbersome and inaccurate polygon vertex annotation in image annotation projects. The annotation efficiency and accuracy are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flowchart of an image annotation method for automatically framing an image provided by an embodiment of the present application is provided.

[0016] Figure 2 A flowchart of a training image semantic segmentation model provided in one embodiment of the present application.

[0017] Figure 3 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] In an exemplary embodiment, Figure 1As shown, a method for automatically labeling an image by drawing a frame, comprising the following steps 201 to 203.

[0021] Step 201: Acquire a target image.

[0022] Step 202: Mark the target image with a box to obtain the marked target image.

[0023] Step 203: input the target image after the box annotation into the trained image semantic segmentation model to obtain an annotated image of the target image; the annotated box in the annotated image is located inside the box of the box annotation.

[0024] Among them, step 203 calls the trained image semantic segmentation model for the image in the selected rectangular frame (target segmentation image), inputs the target segmentation image into the trained image semantic segmentation model, and the trained image semantic segmentation model analyzes the image in the rectangular frame after the frame is marked, and automatically adjusts the size and position of the rectangular frame to fit the edge of the object in the image, thereby realizing the automatic fitting of the rectangular frame to the image object. The marked frame in the marked image is the final marking result of the target image, and the drawn frame in the drawn frame marking is only a preliminary marking, and the drawn frame is not displayed in the final marked image.

[0025] By implementing the above steps 201 to 203, the present application can solve the problems in image annotation projects such as large errors in box annotation, irregular polygon annotation, cumbersome and inaccurate polygon vertex annotation, and high model requirements and inaccurate annotation based on deep learning image semantic segmentation annotation methods, so as to improve annotation efficiency and accuracy.

[0026] In another exemplary embodiment of the present application, the image semantic segmentation model is a DeepLabv3+ model. DeepLabv3+ model: uses an encoder and decoder structure.

[0027] The encoder part is composed of a backbone network and an atrous spatial pyramid pooling (ASPP) module to generate feature representations to provide a decoder with 3×3 convolutions to produce the final predicted image. The backbone network is a deep convolutional neural network (DCNN). After the image enters the backbone network, two feature layers are obtained. The shallow feature layer directly enters the decoder for channel compression (through 1×1 convolution) to reduce the proportion of the low level; the deep features enter the ASPP module of the encoder. The ASPP module includes 1 1×1 convolution, 3 3×3 atrous convolutions of different ratios (6, 12, 18), and a global pooling operation of the image. These operations are followed by a 1×1 convolution. The features obtained by the encoder have richer information and a higher proportion.

[0028] The ASPP module consists of the following key parts: Atrous Convolutions, Global Average Pooling, 1×1 convolution, and feature fusion and normalization.

[0029] The ASPP module uses multiple dilated convolutions and sets different dilation rates for each convolution operation, so that each convolution kernel has a different receptive field. These dilated convolutions act on the input feature map in parallel and can capture features at different scales. Global average pooling is used to capture global contextual information, that is, the background or scene information of the entire image. Global average pooling aggregates the information in the entire feature map into a single global feature, then adjusts the dimension through a 1×1 convolution, and finally upsamples and combines it with other feature maps. 1×1 convolution is used to capture the most basic local information. The role of 1×1 convolution is to reduce the number of channels in the feature map, thereby reducing the amount of calculation, and it helps to fuse features from different convolutional layers. Feature fusion and normalization are batch normalization and activation functions (such as ReLU). The feature maps generated by different branches of the ASPP module are merged by concatenation. These merged feature maps are then integrated through a 1×1 convolution and further processed through batch normalization and activation functions (such as ReLU) to ensure that these features can be effectively used by subsequent layers.

[0030] The decoder part is responsible for restoring the spatial information of the image to obtain accurate segmentation results. The decoder receives the low-resolution feature map from the encoder and the high-resolution feature map from the middle layer of the backbone network. Then, the number of channels of the high-resolution feature map is adjusted by convolution operation, and it is concatenated with the upsampled low-resolution feature map. Finally, the final segmentation result is obtained through a series of convolution operations.

[0031] Although the DeepLabv3 model captures rich multi-scale contextual information through the dilated convolution and ASPP modules, the downsampling process in the convolution operation (i.e., the feature map has low resolution) leads to insufficient performance on the details of the object boundary, which easily leads to blurred or inaccurate edge segmentation. The decoder module is designed to restore the spatial information by upsampling these low-resolution feature maps and fuse the low-level features extracted in the encoder, thereby improving the segmentation effect of the object boundary while maintaining high accuracy.

[0032] DeepLabv3+ uses an improved Xception model: In the improved version of Xception, all max pooling operations are replaced by depthwise separable convolutions. Depthwise separable convolutions can also achieve downsampling when the convolution kernel slides, thus replacing max pooling. In this way, Xception reduces information loss while retaining finer-grained spatial features and improving the resolution of the network.

[0033] In another exemplary embodiment of the present application, Figure 2 As shown in the figure, the process of training the image semantic segmentation model is as follows: obtaining a data set; the data set includes multiple pictures and annotated images of each picture; each picture is annotated by a draw frame, the annotated frame of the annotated image is located inside the draw frame of the corresponding picture, and the annotated frame in the annotated image of the picture fits the edge of the object to be annotated in the picture. Common data sets include PASCAL VOC, Cityscapes, etc. You can also select videos of video collection points at specific stations, and capture pictures containing valid samples in the video as images to be annotated, such as large luggage items, gatherings of people, etc. (depending on the actual application scenario of image recognition).

[0034] The image semantic segmentation model is iteratively trained with the pictures in the data set as input and the labeled images of each picture in the data set as output.

[0035] In another exemplary embodiment of the present application, the image semantic segmentation model is iteratively trained with the pictures in the data set as input and the annotated images of each picture in the data set as output, and the specific steps are: the pictures in the data set and the annotated images are sequentially subjected to image normalization and data enhancement processing to obtain an expanded data set. Image normalization normalizes the pixel values ​​of the image to a specific range, usually [0,1] or [-1,1]. For example, for RGB images, each pixel value can be divided by 255 to achieve normalization to [0,1]. In order to increase the diversity of the data set and improve the generalization ability of the model, data enhancement technology can be used. Common data enhancement methods include random rotation, random flipping, random cropping, color jittering, etc.

[0036] The image semantic segmentation model is iteratively trained according to the expanded data set.

[0037] In another exemplary embodiment of the present application, the iterative training of the image semantic segmentation model according to the expanded data set is specifically as follows: the pictures in the expanded data set are input into the image semantic segmentation model, the image semantic segmentation model performs feature extraction and pixel-level classification on the pictures, and outputs the category prediction result of each pixel point, that is, the pixel point of the classification number, to achieve image segmentation, and the categories include people, vehicles, and suitcases, etc.

[0038] Calculate the value of the loss function based on the actual annotations and the model prediction results. Select an appropriate loss function (such as the cross entropy loss function) to measure the difference between the model prediction results and the actual annotations.

[0039] Use an optimization algorithm to update the model's parameters so that the loss function value gradually decreases. Select an optimizer (such as stochastic gradient descent) to update the model's weight parameters.

[0040] Repeat the above process until the model converges or reaches the preset number of training times.

[0041] In another exemplary embodiment of the present application, after inputting the target segmented image into a trained image semantic segmentation model to obtain an annotated image of the target image, it also includes: using an image annotation tool to annotate the automatically fitted object border, specifically including adding labels, descriptions and other information.

[0042] The marking method of the present application has the following advantages.

[0043] 1. Input the target image into the trained image semantic segmentation model to obtain the target segmentation image, classify each pixel in the image, and then perform region division. After pixel classification, the boundaries of different objects or regions can be clearly demarcated, making the annotation more accurate. Image pixel classification and region division can greatly improve the accuracy of image annotation, better realize the automated annotation process, and provide important basic information foundation for subsequent image analysis and processing. For example, in tasks such as target detection, image segmentation, and image recognition, accurate region division and pixel classification can help the algorithm better understand the image content and improve the performance of the task.

[0044] 2. The annotator only needs to locate or select the approximate region of interest (ROI) of the object in the image to quickly and automatically annotate the object contour to complete the image annotation task - the approximate area is directly selected manually, and the image semantic segmentation model based on deep learning is used to identify the target object contour to achieve pixel-level annotation.

[0045] 3. It can adapt to target objects of different shapes, reduce the workload of manual adjustment of annotation frames, and improve the efficiency of annotation.

[0046] Based on the same inventive concept, the embodiment of the present application also provides an image annotation device for automatically pasting the frame for implementing the image annotation method for automatically pasting the frame involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the image annotation device for automatically pasting the frame provided below can refer to the limitations of the image annotation method for automatically pasting the frame above, and will not be repeated here.

[0047] In an exemplary embodiment, the present application provides an image annotation device for automatically framing an image, including the following modules.

[0048] The acquisition module is used to acquire the target image.

[0049] The category determination module is used to input the target image into a trained image semantic segmentation model to obtain the category of each pixel in the target image.

[0050] The frame marking module is used to mark the target image with a frame to obtain the target image after the frame marking.

[0051] The annotation module is used to input the target image after the draw-box annotation into the trained image semantic segmentation model to obtain the annotated image of the target image; the annotation box in the annotated image is located inside the draw-box annotation.

[0052] In an exemplary embodiment, the image annotation device for automatically sticking to the edges of the frame also includes: an image semantic segmentation model training module, which is used to obtain a data set, take the pictures in the data set as input, and take the annotated images of each picture in the data set as output, and iteratively train the image semantic segmentation model; the data set includes multiple pictures and annotated images of each picture; each picture is annotated with a frame, and the annotated image of the picture includes an annotation box, wherein the annotation box is located inside the draw box of the corresponding picture.

[0053] In an exemplary embodiment, the image semantic segmentation model training module specifically includes: an expansion submodule, which is used to perform image normalization processing and data enhancement processing on the pictures and annotated images in the data set in sequence to obtain an expanded data set.

[0054] The training submodule is used to iteratively train the image semantic segmentation model according to the expanded data set.

[0055] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image annotation data for automatic bordering of the draw frame. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for image annotation of automatic bordering of the draw frame is implemented.

[0056] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned method embodiments when executing the computer program.

[0057] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0058] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0059] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0060] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0061] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0062] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0063] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for automatically marking an image by drawing a frame, characterized in that: The image annotation method of automatically sticking to the edge of the frame includes: Get the target image; Perform frame marking on the target image to obtain the target image after frame marking; The target image after the draw-box annotation is input into the trained image semantic segmentation model to obtain the annotated image of the target image; the annotated box in the annotated image is located inside the draw-box annotation.

2. The image annotation method for automatically framing the frame according to claim 1, characterized in that: The image semantic segmentation model is the DeepLabv3+ model.

3. The image annotation method for automatically framing the frame according to claim 1, characterized in that: The process of training the image semantic segmentation model is: Acquire a data set; the data set includes multiple pictures and annotated images of each picture; each picture is annotated by a frame, and the annotated image of the picture includes an annotated frame, wherein the annotated frame is located inside the frame of the corresponding picture, and the annotated frame is attached to the edge of the object to be annotated in the picture; The image semantic segmentation model is iteratively trained with the pictures in the data set as input and the labeled images of each picture in the data set as output.

4. The image annotation method for automatically framing the frame according to claim 3 is characterized in that: Taking the pictures in the data set as input and the annotated images of each picture in the data set as output, the image semantic segmentation model is iteratively trained, specifically including: The images and annotated images in the data set are subjected to image normalization and data enhancement processing in turn to obtain an expanded data set; The image semantic segmentation model is iteratively trained according to the expanded data set.

5. An image annotation device for automatically attaching a frame to an edge, characterized in that: The image annotation device for automatically attaching the frame to the edge comprises: An acquisition module, used for acquiring a target image; A frame marking module is used to mark the target image with a frame to obtain the target image after the frame marking; The annotation module is used to input the target image after the draw-box annotation into the trained image semantic segmentation model to obtain the annotated image of the target image; the annotation box in the annotated image is located inside the draw-box annotation.

6. The image annotation device for automatically attaching a frame to an edge according to claim 5, characterized in that: The image annotation device for automatically attaching the frame to the edge also includes: The image semantic segmentation model training module is used to obtain a data set, take the pictures in the data set as input, and take the annotated images of each picture in the data set as output, and iteratively train the image semantic segmentation model; the data set includes multiple pictures and annotated images of each picture; each picture is annotated with a draw frame, and the annotated image of the picture includes an annotated frame, wherein the annotated frame is located inside the draw frame of the corresponding picture.

7. The image annotation device for automatically attaching a frame to an edge according to claim 6, characterized in that: Image semantic segmentation model training module, including: An expansion submodule, used to perform image normalization processing and data enhancement processing on the pictures and annotated images in the data set in sequence to obtain an expanded data set; The training submodule is used to iteratively train the image semantic segmentation model according to the expanded data set.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image annotation method for automatically pasting edges of a frame as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image annotation method for automatically framing the frame according to any one of claims 1 to 4 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image annotation method for automatically framing the frame according to any one of claims 1 to 4 is implemented.

Images