Image annotation method and device

Abstract

The present application provides an image annotation method and apparatus. The method comprises: generating an image to be annotated and a segmentation map of the image to be annotated based on a three-dimensional model of a target, wherein the image to be annotated includes the target, and the segmentation map includes regional information of the region where the target is located on the image to be annotated; determining the position coordinates of an annotation frame for the target on the image to be annotated based on the regional information; and drawing the annotation frame on the image to be annotated based on the position coordinates to obtain an annotated image. The present application can improve the accuracy of the position of the annotation frame for image annotation by utilizing the precise regional information on the segmentation map obtained from the three-dimensional model to determine the position coordinates of the annotation frame and draw the annotation frame.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image annotation method and device. Background Art

[0002] Deep learning algorithms are widely used for object recognition in images. Trained deep learning models can be used to classify, identify, or segment objects in input images. Training deep learning models requires a large number of training samples, each of which includes a training image and its annotated counterpart.

[0003] Generally, after obtaining a training image, a person manually checks the target in the training image and draws a labeling box for the target, thereby labeling the training image and obtaining a labeled image of the training image.

[0004] However, when manually labeling training images, the accuracy of the positions of the annotation boxes drawn is poor, which in turn affects subsequent model training. Summary of the Invention

[0005] Embodiments of the present application provide an image annotation method and apparatus to solve the problem of poor position accuracy of annotated frames for annotating images.

[0006] In a first aspect, an embodiment of the present application provides an image annotation method, comprising:

[0007] Generate an image to be labeled and a segmentation map of the image to be labeled according to the three-dimensional model of the target, wherein the image to be labeled contains the target, and the segmentation map contains regional information of the region where the target is located on the image to be labeled;

[0008] Determining the position coordinates of the marking frame of the target on the image to be marked according to the region information;

[0009] The annotation frame is drawn on the image to be annotated according to the position coordinates to obtain an annotated image.

[0010] In a second aspect, an embodiment of the present application provides an image annotation device, comprising:

[0011] a generating module, configured to generate an image to be annotated and a segmentation map of the image to be annotated based on a three-dimensional model of the target, wherein the image to be annotated includes the target, and the segmentation map includes regional information of a region where the target is located on the image to be annotated;

[0012] A processing module, configured to determine the position coordinates of a marking frame of the target on the image to be marked based on the region information;

[0013] A drawing module is used to draw the annotation frame on the image to be annotated according to the position coordinates to obtain an annotated image.

[0014] In a third aspect, an embodiment of the present application provides an image annotation device, comprising: at least one processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image annotation method as described in the first aspect and various possible implementations of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the image annotation method described in the first aspect and various possible implementations of the first aspect is implemented.

[0018] The image annotation method and apparatus provided in the embodiments of the present application generate an image to be annotated and a segmentation map of the image to be annotated based on a three-dimensional model of the target, wherein the image to be annotated contains the target, and the segmentation map contains regional information of the area where the target is located on the image to be annotated; based on the regional information, the position coordinates of the annotation box of the target on the image to be annotated are determined; based on the position coordinates, the annotation box is drawn on the image to be annotated to obtain an annotated image. By utilizing the precise regional information on the segmentation map obtained from the three-dimensional model to determine the position coordinates of the annotation box and draw the annotation box, the accuracy of the position of the annotation box for image annotation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A schematic diagram of a scenario of an image annotation method provided in one embodiment of the present application;

[0021] Figure 2 A flowchart of an image annotation method provided in one embodiment of the present application;

[0022] Figure 3A The image to be labeled includes experimental equipment on a laboratory table provided in an embodiment of the present application;

[0023] Figure 3B for Figure 3A Segmentation map of

[0024] Figure 3C Based on Figure 3B right Figure 3A Annotated images obtained by annotation;

[0025] Figure 4 A flowchart of an image annotation method provided in another embodiment of the present application;

[0026] Figure 5A An image to be labeled containing pedestrians on the street provided in an embodiment of the present application;

[0027] Figure 5B for Figure 5A Segmentation map of

[0028] Figure 5C Based on Figure 5B right Figure 5A Annotated images obtained by annotation;

[0029] Figure 6 A flowchart of an image annotation method provided in another embodiment of the present application;

[0030] Figure 7A The image to be labeled includes beverage bottles in a freezer provided in an embodiment of the present application;

[0031] Figure 7B for Figure 7A Segmentation map of

[0032] Figure 7C Based on Figure 7B right Figure 7A Annotated images obtained by annotation;

[0033] Figure 8 A flowchart of an image annotation method provided in yet another embodiment of the present application;

[0034] Figure 9 for Figures 3A-3C a flow chart of the example shown;

[0035] Figure 10 for Figures 5A-5C a flow chart of the example shown;

[0036] Figure 11 for Figures 7A-7C a flow chart of the example shown;

[0037] Figure 12 A schematic diagram of the structure of an image annotation device provided in one embodiment of the present application;

[0038] Figure 13A schematic diagram of the hardware structure of an image annotation device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] The embodiments of the present application construct a three-dimensional model of the target, utilize optical tracking technology to obtain an image to be annotated and a segmentation map of the image to be annotated based on the virtual model and environment, and then utilize the regional information of the target area contained in the segmentation map to determine the position coordinates of the annotation box, thereby drawing the annotation box on the image to be annotated. By utilizing the precise regional information on the segmentation map obtained by mapping the three-dimensional virtual data onto a two-dimensional plane to draw the annotation box, the accuracy of the position of the annotation box for image annotation can be improved, and the annotation efficiency can be improved compared to manual annotation.

[0041] Figure 1 This is a scene diagram of the image annotation method provided in one embodiment of the present application. Figure 1 As shown, the scenario includes an image annotation device 11 and a server 12. The image annotation device 11 can be an electronic device such as a desktop computer, laptop, mobile phone, tablet computer, vehicle-mounted terminal, or server. The image annotation device 11 is used to annotate images, generate training samples, and train a deep learning-based object recognition model, uploading the trained object recognition model to the server 12. The server 12 receives the trained object recognition model and recognizes the image to be processed based on the object recognition model to obtain a recognition result.

[0042] The process of annotating an image by the image annotation device 11 may be: the image annotation device 11 may automatically construct a three-dimensional model of the target and a three-dimensional environment model based on images, videos, laser scanning data, etc. containing the target and the environment, and deploy the three-dimensional model of the target in the three-dimensional environment model, thereby obtaining a scene model. Alternatively, the image annotation device 11 may also draw a three-dimensional model of the target and a three-dimensional environment model according to the user's control operation, and deploy the three-dimensional model of the target in the three-dimensional environment model according to the user's instructions, thereby obtaining a scene model. The image annotation device 11 adjusts the lighting parameters, camera parameters, target identification, etc. of the scene model according to the environmental configuration parameters set by the user, and maps the scene model to a two-dimensional plane through ray tracing technology to obtain an image to be annotated and a segmentation map of the image to be annotated. The image annotation device 11 determines the position coordinates of the annotation box of the target on the image to be annotated based on the area information of the area where the target is located in the segmentation map, and then draws the annotation box on the image to be annotated based on the position coordinates, and adds the target identification to the corresponding position of the annotation box, thereby obtaining an annotated image. Reference Figure 1 The target in the dotted box below the image annotation device 11 is a water cup, and the 3D model of the target is a water cup model. The 3D environment model can be set as required, such as a shelf, a dining table, etc. The image annotation device 11 can use a virtual camera to map the scene model containing the water cup to a two-dimensional plane, and obtain the image to be annotated containing the water cup and the segmentation map of the image to be annotated ( Figure 1 (not shown). Based on the regional information of the area where the water cup is located in the segmentation map, the position coordinates of the annotation box of the water cup on the image to be annotated are determined, and then based on the position coordinates, the annotation box of the water cup is drawn on the image to be annotated, thereby obtaining an annotated image. After obtaining the annotated image, the image annotation device 11 forms a training sample with the image to be annotated and its corresponding annotated image. The embodiment of the present application draws the annotation box by utilizing the precise regional information on the segmentation map obtained by mapping the three-dimensional virtual data to a two-dimensional plane. This can improve the accuracy of the position of the annotation box for image annotation, improve annotation efficiency, and thereby improve the efficiency of generating training samples and the training effect.

[0043] It should be noted that Figure 1 The scenario is only used as an example. The image annotation method provided in the embodiment of the present application can also be used in other scenarios and is not limited here.

[0044] Figure 2 This is a flow chart of an image annotation method provided in one embodiment of the present application. The execution subject of this method is an image annotation device. Figure 2 As shown, the method includes:

[0045] S201 : Generate an image to be labeled and a segmentation map of the image to be labeled according to a three-dimensional model of the target, wherein the image to be labeled contains the target, and the segmentation map contains region information of a region where the target is located on the image to be labeled.

[0046] In this embodiment, the three-dimensional model of the target is a virtual three-dimensional data model. The three-dimensional model of the target can be automatically generated by the image annotation device, or it can be constructed by the image annotation device according to the control operation of the user. The target can be determined according to actual needs. For example, the target can be a pedestrian, and the target recognition model trained by the labeled training samples is used to identify pedestrians on the street. The segmentation map can be obtained in a variety of ways. For example, the segmentation map can be obtained from the three-dimensional model of the target through an optical tracking algorithm, or by performing semantic segmentation on the image to be annotated to obtain the segmentation map. The segmentation map is consistent with the area where the target is located in the image to be annotated. The pixel values ​​of the pixels in the area where the same target is located in the segmentation map can be the same value. The area where the target is located on the image to be annotated can be determined based on the segmentation map.

[0047] S202: Determine the position coordinates of the marking frame of the target on the image to be marked according to the region information.

[0048] In this embodiment, the shape of the annotation frame is not limited and may be, for example, a rectangle, a trapezoid, a circle, an irregular shape, etc. The position coordinates of the annotation frame may include the position coordinates of each boundary line of the annotation frame and / or the position coordinates of multiple points on the annotation frame. The area enclosed by the annotation frame may include all or part of the area where the object is located on the image to be annotated.

[0049] S203 : Draw a labeling frame on the image to be labeled according to the position coordinates to obtain a labeled image.

[0050] In this embodiment, the annotated image is the image obtained by automatically annotating the image to be annotated. The annotated image and the image to be annotated form a training sample for training the object recognition model. After determining the position coordinates of the annotation box, the image annotation device draws the annotation box on the image to be annotated, thereby obtaining the annotated image.

[0051] In an embodiment of the present application, an image to be annotated and a segmentation map of the image to be annotated are generated based on a three-dimensional model of the target, wherein the image to be annotated contains the target, and the segmentation map contains regional information of the area where the target is located on the image to be annotated; based on the regional information, the position coordinates of the annotation box of the target on the image to be annotated are determined; based on the position coordinates, the annotation box is drawn on the image to be annotated to obtain an annotated image. By utilizing the precise regional information on the segmentation map obtained from the three-dimensional model to determine the position coordinates of the annotation box and draw the annotation box, the accuracy of the position of the annotation box of the image annotation can be improved.

[0052] Optionally, the types of annotation frames include direction annotation frames and axis-aligned annotation frames, wherein both the direction annotation frame and the axis-aligned annotation frame are rectangular frames. The main axis of the direction annotation frame is not parallel to the coordinate axis of the image, the two boundary lines of the direction annotation frame are parallel to the main axis, and the other two boundary lines are perpendicular to the main axis. The main axis of the axis-aligned annotation frame is parallel to the coordinate axis of the image, the two boundary lines of the axis-aligned annotation frame are parallel to the first axis of the image along the height direction, and the other two boundary lines are parallel to the second axis of the image along the width direction. The following two embodiments respectively illustrate the method for determining the position coordinates of the direction annotation frame and the method for determining the position coordinates of the axis-aligned annotation frame.

[0053] As an embodiment of the present application, Figure 2 Based on the embodiment, the annotation box includes a direction annotation box;

[0054] S202 may include:

[0055] Determine the main axis of the target's orientation annotation box;

[0056] The position coordinates of the target's direction annotation box are determined according to the region information and the main axis, wherein the two boundary lines of the direction annotation box are parallel to the main axis, and the direction annotation box is the minimum circumscribed rectangle of the target area.

[0057] In this embodiment, for the direction marking box, the main axis of the direction marking box can be first determined based on the three-dimensional model or segmentation map, and then based on the main axis and the area information of the target area on the segmentation map, a rectangular box with two boundary lines parallel to the main axis and the minimum circumscribed rectangle of the target area is determined as the direction standard box.

[0058] Optionally, the main axis of the direction annotation frame may be determined in one of the following two ways:

[0059] In the first implementation, the principal axis of the directional annotation box can be determined based on the coordinates of designated points on the 3D model. Specifically, the 3D coordinates of two designated points on the 3D model are mapped onto the image to be annotated to obtain the coordinates of two target points corresponding to the two points. The line connecting the two target points is then determined as the principal axis.

[0060] In this implementation, the designated locations can be set by the user based on actual needs, using two key feature points on the target's 3D model as designated locations. The image annotation device uses a ray tracing algorithm to map the 3D coordinates of the two designated locations on the 3D model onto the image to be annotated, obtaining the coordinates of the target location. The line connecting the coordinates of the two target locations becomes the principal axis of the directional annotation box.

[0061] In this embodiment, by mapping the designated position point on the target 3D model to the image to be annotated to determine the main axis of the direction annotation frame, the main axis of the determined direction annotation frame can be made consistent with the spatial position of the target, thereby improving the accuracy of the annotation frame.

[0062] In the second implementation, the principal axis of the orientation annotation box can be determined using a principal component analysis (PCA) algorithm. Specifically, the PCA algorithm processes the coordinates of each pixel within the target region on the segmentation map to obtain a feature vector; the principal axis is then determined based on the feature vector.

[0063] The PCA algorithm is a data dimensionality reduction method that uses linear transformations to transform raw data into a set of linearly independent representations, thereby extracting the data's primary linear components. In this implementation, the image annotation device constructs a dataset of the (X, Y) coordinates of each pixel within the target region of the segmented image. The covariance matrix of this dataset is then calculated, along with the eigenvectors of this covariance matrix. The direction of these eigenvectors is the principal axis of the orientation calibration box.

[0064] This embodiment uses the PCA algorithm to perform principal component analysis on the target area in the segmentation image to determine the main axis of the direction calibration frame, so that the main axis of the direction calibration frame can represent the main direction of the target area, thereby improving the accuracy of the annotation frame.

[0065] The following is an example of labeling experimental equipment on a laboratory table. Figure 3A The image to be labeled includes experimental equipment on the laboratory table provided in the embodiment of the present application. Figure 3B for Figure 3A The segmentation map, Figure 3C Based on Figure 3B right Figure 3AThe labeled image obtained by labeling. In this example, the experiment is an electrical experiment, and the experimental equipment on the table includes switch 31, sliding rheostat 32, voltmeter 33, ammeter 34, wire terminal 35, wire 36, lamp 37, etc. Wire terminal 35 is connected to the wiring knob 38 of switch 31, sliding rheostat 32, voltmeter 33, ammeter 34, etc. The target is the experimental equipment. Among them, the labeling boxes of switch 31, voltmeter 33, and ammeter 34 are axisymmetric labeling boxes; the labeling boxes of sliding rheostat 32, wire terminal 35, and wire 36 are directional labeling boxes. Taking the direction marking frame of the sliding rheostat 32 as an example, if the first implementation method is adopted, the user can select two designated position points on the three-dimensional model of the sliding rheostat, such as selecting a designated position point at each end of the sliding axis. The image annotation device maps the designated position points at both ends of the sliding axis in the three-dimensional model to the image to be annotated to determine two corresponding target position points, and connects the two target position points to determine the main axis; if the second implementation method is adopted, the image annotation device can map the designated position points at both ends of the sliding axis in the three-dimensional model to the image to be annotated to determine two corresponding target position points. Figure 3B The coordinates of each pixel point in the area where the sliding resistor is located in the segmentation diagram shown are subjected to principal component analysis to obtain a eigenvector; and the principal axis is determined based on the eigenvector.

[0066] As an embodiment of the present application, Figure 2 According to an embodiment, the annotation box comprises an axis-aligned annotation box.

[0067] S202 may include:

[0068] Obtain the coordinates of each pixel point in the target area on the segmentation map; the coordinates include the first axis coordinate value along the height direction of the segmentation map and the second axis coordinate value along the width direction of the segmentation map;

[0069] Determine the maximum and minimum values ​​of the first-axis coordinates of all pixels, as well as the maximum and minimum values ​​of the second-axis coordinates;

[0070] The position coordinates of the axis-aligned annotation box of the target are determined according to the maximum and minimum values ​​of the first axis coordinate values ​​and the maximum and minimum values ​​of the second axis coordinate values, wherein two boundary lines of the axis-aligned annotation box are parallel to the height direction.

[0071] In this embodiment, the coordinate system of the segmentation map is constructed by a first axis along the height direction and a second axis along the width direction. The coordinates of each pixel point include the first axis coordinate value and the second axis coordinate value. The image annotation device compares the first axis coordinate values ​​of all pixel points in the target area to obtain the maximum and minimum values ​​therein; compares the second axis coordinate values ​​of all pixel points in the target area to obtain the maximum and minimum values ​​therein. The image annotation device can determine the straight line passing through the position point corresponding to the maximum coordinate value of the first axis and the straight line passing through the position point corresponding to the minimum coordinate value of the first axis among the straight lines parallel to the second axis as the two boundary lines of the axisymmetric annotation frame; can determine the straight line passing through the position point corresponding to the maximum coordinate value of the second axis and the straight line passing through the position point corresponding to the minimum coordinate value of the second axis among the straight lines parallel to the first axis as the other two boundary lines of the axisymmetric annotation frame.

[0072] This embodiment can accurately determine the position coordinates of the target's axis-aligned annotation box by segmenting the coordinates of each pixel point in the area where the target is located on the image, including the maximum and minimum values ​​in the first-axis coordinate values ​​and the maximum and minimum values ​​in the second-axis coordinate values, thereby making the drawn axis-aligned annotation box more accurate.

[0073] Based on any of the above embodiments, as an embodiment of the present application, there is at least one target, and each target corresponds to an identifier and a label box. The method further includes:

[0074] For each target, an association relationship between the target's identifier and the target's annotation box is established.

[0075] In this embodiment, after marking each target in the image to be annotated with a box, the image annotation system can add a marker for each target within a designated location within the box. Alternatively, the marker box can be drawn with the line color corresponding to each marker, with each line color corresponding to the marker. The association between the marker and the box can also be implemented in other ways, which are not limited here.

[0076] Figure 4 This is a flow chart of an image annotation method provided by another embodiment of the present application. Based on any of the above embodiments, in this embodiment, there are multiple targets, and each target corresponds to a position in the three-dimensional environment model. Figure 4 As shown, the method includes:

[0077] S401: Arrange the three-dimensional model of each target at a corresponding position in the three-dimensional environment model to obtain a first scene model, and generate an image to be labeled containing each target and a segmentation map of the image to be labeled according to the first scene model.

[0078] In this embodiment, the image to be annotated includes not only objects but also a background. When constructing the model, a 3D model of each object and a 3D environment model serving as the background are constructed. The 3D models of each object are placed at corresponding locations within the 3D environment model to generate a first scene model. The image to be annotated and the segmentation map of the image to be annotated generated based on this first scene model include all objects.

[0079] S402. For each target, the three-dimensional model of the target is configured at a corresponding position in the three-dimensional environment model to obtain a second scene model, and based on the second model, an independent image containing the target and a segmentation map of the independent image are generated, and based on the area information of the area where the target is located on the segmentation map of the independent image, the position coordinates of the annotation box of the target on the independent image are determined.

[0080] In this embodiment, the 3D model of each target is placed at a corresponding position in the 3D environment model to obtain a second scene model. Each target corresponds to a second scene model, which includes only that target and no other targets. The independent image generated based on the second scene model and the segmentation map of the independent image only include that target and no other targets.

[0081] It should be noted that, in this embodiment, the execution order of S401 and S402 is not limited. S401 may be executed first and then S402, or S402 may be executed first and then S401, or they may be executed in parallel.

[0082] S403 : Determine the position coordinates of the annotation boxes of the respective objects on the respective independent images as the position coordinates of the annotation boxes of the respective objects on the image to be annotated.

[0083] S404: Draw a labeling frame on the image to be labeled according to the position coordinates to obtain a labeled image.

[0084] In this embodiment, each independent image contains only one target, while the image to be annotated includes all targets. The coordinates of the target's annotation box in each independent image are used as the coordinates of the target's annotation box in the image to be annotated. This determines the coordinates of each target's annotation box in the image to be annotated, and then draws the annotation box for each target.

[0085] In some application scenarios, the image to be annotated includes multiple targets, and there is mutual occlusion between the targets. When the annotation box to be drawn is required to include the obscured part of the target, the method in this embodiment can be adopted. That is, the three-dimensional models of each target are respectively configured at the corresponding positions in the three-dimensional environment model, and each target corresponds to a second scene model, and then an independent image containing only a single target and a segmentation map are obtained, and the position of the annotation box of the target on the independent image is determined. The position of the annotation box of the target on each independent image is then determined as the position of each target on the image to be annotated, so that even if the target is partially obscured, a annotation box containing all parts of the target can be drawn.

[0086] The following is an example of labeling pedestrians on the street. Figure 5A is an image to be labeled containing pedestrians on the street, Figure 5B for Figure 5A The segmentation map, Figure 5C Based on Figure 5B right Figure 5A The labeled image is obtained by labeling. In this example, the target is a pedestrian and the street is the background. Some pedestrians’ bodies are blocked by other pedestrians, and the labeling box drawn is required to include the blocked part of the pedestrian. Figure 4 In the embodiment, an independent image may be generated for each pedestrian and street, the position of the pedestrian's annotation box in the independent image may be determined, and then the annotation box of each pedestrian may be drawn on the image to be annotated according to the position of the annotation box of each pedestrian in each independent image.

[0087] Figure 6 This is a flow chart of an image annotation method provided by another embodiment of the present application. Based on any of the above embodiments, in this embodiment, the target is multiple. Figure 6 As shown, the method includes:

[0088] S601 : Generate an image to be labeled and a segmentation map of the image to be labeled according to a three-dimensional model of the target, wherein the image to be labeled contains the target, and the segmentation map contains region information of a region where the target is located on the image to be labeled.

[0089] S602: Determine the position coordinates of the marking frame of the target on the image to be marked according to the region information.

[0090] S603: Draw a labeling frame on the image to be labeled according to the position coordinates.

[0091] S604: Determine the ratio of the length of the first designated target on the designated boundary line of the annotation box of the first designated target to the total length of the designated boundary line. The first designated target is a target whose annotation box overlaps with the annotation boxes of other targets.

[0092] S605: Adjust the size of the annotation frame of the first designated object according to the ratio to obtain an annotated image.

[0093] In this embodiment, the designated boundary line can be determined according to actual needs and is not limited here. For example, the annotation box is a rectangle, and the designated boundary line can be the left boundary line, the right boundary line, and the bottom boundary line of the rectangle, or it can also be the left boundary line and the right boundary line of the rectangle. After drawing the annotation box on the image to be annotated, the first designated target among all targets can be determined. Then the annotation box of each first designated target is adjusted in a targeted manner. Specifically, for a first designated target, the proportion of the length of the first designated target on the designated boundary line of its annotation box to the total length of the designated boundary line can be calculated. According to the proportion, the size of the annotation box of the first designated target is adjusted so that the adjusted annotation box meets the requirements.

[0094] Optionally, S605 may include reducing the annotation box of the first designated target when the ratio is less than a preset ratio threshold, until the ratio is equal to or greater than the preset ratio threshold, or until the annotation box of the first designated target does not overlap with the annotation boxes of other targets.

[0095] In this embodiment, in one possible implementation, when the ratio is less than a preset ratio threshold, the labeling box of the first designated target is reduced until the ratio is equal to or greater than the preset ratio threshold. In another possible implementation, when the ratio is less than a preset ratio threshold, the labeling box of the first designated target is reduced until the labeling box of the first designated target and the labeling boxes of other targets no longer overlap. The preset ratio threshold can be set according to actual needs and is not limited here. For example, the preset ratio threshold can be 0.9, 0.8, etc. Taking 0.9 as an example, when the ratio is less than 0.9, the labeling box of the first designated target is reduced, such as reducing all or part of the side length of the labeling box, etc., until the ratio is equal to 0.9, or the labeling box of the first designated target is reduced until the labeling box of the first designated target and the labeling boxes of other targets no longer overlap.

[0096] The image to be annotated includes multiple targets, and there is mutual occlusion between the targets. Therefore, after drawing the annotation box of a certain target, there may be areas of other targets on the boundary line of the annotation box, that is, the annotation boxes of two or more targets may overlap. In application scenarios where the boundary line of the annotation box is required to include fewer other targets, the annotation box can be further adjusted using the method in this embodiment, that is, based on the length of the first designated target on the designated boundary line of the annotation box of the first designated target and the proportion of the length of the first designated target in the total length of the designated boundary line, the annotation box is adjusted, so that the boundary line of the annotation box contains fewer other targets and overlaps less or does not overlap with the annotation boxes of other targets, thereby meeting the needs of the application scenario.

[0097] The following is an example of labeling beverage bottles in a freezer. Figure 7A is an image to be labeled containing beverage bottles in a freezer. Figure 7B for Figure 7A The segmentation map, Figure 7C Based on Figure 7B right Figure 7A The labeled image obtained by labeling. In this example, the target is a beverage bottle. The drawn labeling box is required to include the bottle cap and part of the bottle body of the beverage bottle. The labeling box of each beverage bottle should contain less area of ​​other beverage bottles and should not overlap or overlap less with the labeling boxes of other beverage bottles. Figure 6 In this embodiment, the labeling boxes of some beverage bottles are adjusted to meet the requirements. Specifically, in this example, the designated boundary lines are the left boundary line, right boundary line, and bottom boundary line of the rectangle. The preset ratio threshold is set to 0.95. Since the top boundary line is the area where the bottle cap is located, the top boundary line will not be lowered during the adjustment process to prevent the bottle cap from being divided outside the top boundary line. The ratio can be adjusted by translating the left boundary line, right boundary line, and bottom boundary line.

[0098] As an embodiment of the present application, based on any of the above embodiments, determining the position coordinates of the annotation box of the target on the image to be annotated according to the region information may include:

[0099] determining at least two targets to be combined as combined targets;

[0100] According to the region information of the combined target, the position coordinates of the combined annotation frame of the combined target on the image to be annotated are determined.

[0101] In this embodiment, in certain application scenarios, it is necessary to draw the same annotation box for multiple targets. Among them, the targets to be combined can be specified by the user. The image annotation device can determine at least two targets to be combined based on the identifiers of the targets to be combined specified by the user, and determine the at least two targets to be combined as a combined target. According to the area information of the combined target, the position coordinates of the combined annotation box of the combined target on the image to be annotated are determined. For a combined target, it is processed as a target and a annotation box is drawn. The method for determining the position coordinates of the combined annotation box of the combined target can adopt the method for determining the position coordinates of the annotation box of the target in any of the above embodiments, which is not limited here. Figures 3A-3C In the example, the wire terminal 35 and the wiring knob 38 connected thereto may be determined as a combination target, and a combination annotation box including the wire terminal 35 and the wiring knob 38 connected thereto may be drawn.

[0102] This embodiment determines the position coordinates of the combined annotation frame of the combined object on the image to be annotated according to the region information of the combined object, thereby achieving the drawing of the combined annotation frame.

[0103] Based on any of the above embodiments, Figure 8 This is a flow chart of an image annotation method provided in another embodiment of the present application. Figure 8 As shown, the method includes:

[0104] S801: Construct a three-dimensional model of the target and a three-dimensional environment model.

[0105] S802: placing the three-dimensional model of the target in the three-dimensional environment model, and generating an image to be labeled and a segmentation map of the image to be labeled by using a ray tracing algorithm.

[0106] In this embodiment, the image annotation device automatically constructs a 3D model of the target and a 3D environment model based on images, videos, laser scanning data, etc. containing the target and environment, and deploys the 3D model of the target in the 3D environment model, thereby obtaining a scene model. Alternatively, the image annotation device may draw the 3D model of the target and the 3D environment model based on user control operations, and deploy the 3D model of the target in the 3D environment model according to user instructions, thereby obtaining a scene model. The image annotation device may use a ray tracing algorithm to map the scene model to generate an image to be annotated and a segmentation map of the image to be annotated.

[0107] S803: Determine the position coordinates of the marking frame of the target on the image to be marked according to the region information.

[0108] S804: Draw a labeling frame on the image to be labeled according to the position coordinates to obtain a labeled image.

[0109] S805: The image to be annotated and the annotated image are combined into a training sample.

[0110] In this embodiment, the image to be annotated and the annotated image can be combined into a training sample. By adjusting lighting parameters, camera parameters, etc., multiple images to be annotated can be generated from the same scene model, thereby obtaining multiple training samples, thereby greatly improving the efficiency of generating training samples.

[0111] Below through Figures 3A-3C , Figures 5A-5C , Figures 7A-7C Three application scenarios are used to illustrate the image annotation method provided in the embodiments of this application.

[0112] Figure 9 for Figures 3A-3CFlowchart of the example shown. First, construct three-dimensional models of targets such as switch 31, sliding rheostat 32, voltmeter 33, ammeter 34, wire terminal 35, wire 36, and lamp 37, as well as three-dimensional environmental models such as the test bench and the ground, and configure the three-dimensional models of the targets in the three-dimensional environmental model to obtain a scene model of the application scenario. Then configure the lighting parameters, camera parameters, identification parameters of each target, and other parameters, and obtain the image to be annotated and the segmentation map through rendering processing. The annotation boxes in this application scenario include direction annotation boxes, axisymmetric annotation boxes, and combined annotation boxes. The position coordinates of each annotation box are determined and each annotation box is drawn in accordance with the above-mentioned corresponding embodiments to obtain an annotated image.

[0113] Figure 10 for Figures 5A-5C Flowchart of the example shown. First, construct three-dimensional models of multiple pedestrians, as well as three-dimensional environmental models such as streets, walls, and lawns. Place the three-dimensional models of all pedestrians in the three-dimensional environmental model to obtain a scene model of the application scenario. And configure parameters such as lighting parameters, camera parameters, and identification of each target, and obtain the image to be labeled through rendering. And for each pedestrian, place the pedestrian separately in the three-dimensional environmental model to obtain an independent image and segmentation map of the pedestrian, and determine the annotation box of the pedestrian in the independent image based on the segmentation map. Draw the annotation box of each pedestrian to the corresponding position in the image to be labeled, so as to obtain an annotated image. The annotation box in this application scenario includes an axisymmetric annotation box, which is required to contain the obscured part of the pedestrian. Therefore, the position coordinates of each annotation box can be determined and each annotation box can be drawn in the above manner to obtain an annotated image.

[0114] Figure 11 for Figures 7A-7C The flowchart of the example shown is as follows. First, 3D models of multiple beverage bottles and a 3D environment model of a freezer are constructed. The 3D models of the multiple beverage bottles are then placed within the 3D environment model to obtain a scene model for the application scenario. Lighting parameters, camera parameters, target identification, and other parameters are then configured. Rendering is then performed to obtain the image to be annotated and a segmentation map. The annotation boxes in this application scenario include directional annotation boxes, requiring that the boundaries of a beverage bottle annotation box should be minimally contiguous with the bodies of other beverage bottles. In this example, the center of each bottle's cap and base are used as designated points and mapped onto the image to be annotated, thereby determining the principal axis of each bottle's annotation box. The annotation box for each bottle is then determined based on the principal axis and the segmentation map. After the annotation box is determined, the size of the annotation box is adjusted based on the proportion of the bottle's length on the left, right, and bottom boundaries of the annotation box relative to the total length of the boundaries, thereby obtaining a labeled image.

[0115] Figure 12 This is a schematic diagram of the structure of an image annotation device provided in one embodiment of the present application. Figure 12As shown, the image annotation device 120 includes: a generating module 1201 , a processing module 1202 and a drawing module 1203 .

[0116] The generating module 1201 is used to generate an image to be labeled and a segmentation map of the image to be labeled according to the 3D model of the target, wherein the image to be labeled contains the target and the segmentation map contains the region information of the region where the target is located on the image to be labeled.

[0117] The processing module 1202 is configured to determine the position coordinates of the annotation box of the target on the image to be annotated according to the region information.

[0118] The drawing module 1203 is used to draw a labeling frame on the image to be labeled according to the position coordinates to obtain a labeled image.

[0119] Optionally, the annotation box includes a direction annotation box;

[0120] The processing module 1202 is configured to:

[0121] Determine the main axis of the target's orientation annotation box;

[0122] The position coordinates of the target's direction annotation box are determined according to the region information and the main axis, wherein the two boundary lines of the direction annotation box are parallel to the main axis, and the direction annotation box is the minimum circumscribed rectangle of the target area.

[0123] Optionally, the processing module 1202 is configured to:

[0124] Map the three-dimensional coordinates of two specified position points on the three-dimensional model to the image to be annotated to obtain the position coordinates of two target position points corresponding to the two position points;

[0125] The line connecting the two target position points is determined as the main axis.

[0126] Optionally, the processing module 1202 is configured to:

[0127] The principal component analysis algorithm is used to process the coordinates of each pixel in the target area on the segmentation map to obtain a feature vector.

[0128] Based on the eigenvectors, the principal axis is determined.

[0129] Optionally, the callout box comprises an axis-aligned callout box;

[0130] The processing module 1202 is configured to:

[0131] Obtain the coordinates of each pixel point in the target area on the segmentation map; the coordinates include the first axis coordinate value along the height direction of the segmentation map and the second axis coordinate value along the width direction of the segmentation map;

[0132] Determine the maximum and minimum values ​​of the first-axis coordinates of all pixels, as well as the maximum and minimum values ​​of the second-axis coordinates;

[0133] The position coordinates of the axis-aligned annotation box of the target are determined according to the maximum and minimum values ​​of the first axis coordinate values ​​and the maximum and minimum values ​​of the second axis coordinate values, wherein two boundary lines of the axis-aligned annotation box are parallel to the height direction.

[0134] Optionally, there is at least one target, and each target corresponds to an identifier and a marked box;

[0135] The processing module 1202 is further configured to:

[0136] For each target, an association relationship between the target's identifier and the target's annotation box is established.

[0137] Optionally, there are multiple targets, each of which corresponds to a position in the three-dimensional environment model;

[0138] The generating module 1201 is used to:

[0139] Arrange the three-dimensional model of each target at a corresponding position in the three-dimensional environment model to obtain a first scene model, and generate an image to be labeled containing each target and a segmentation map of the image to be labeled according to the first scene model;

[0140] The processing module 1202 is configured to:

[0141] Determine the position coordinates of the annotation box of the target on the image to be annotated based on the region information, including:

[0142] For each target, a three-dimensional model of the target is placed at a corresponding position in the three-dimensional environment model to obtain a second scene model, and based on the second model, an independent image containing the target and a segmentation map of the independent image are generated, and the position coordinates of the annotation box of the target in the independent image are determined based on the regional information of the region where the target is located on the segmentation map of the independent image;

[0143] The position coordinates of the marking box of each target on each independent image are determined as the position coordinates of the marking box of each target on the image to be marked.

[0144] Optionally, the target is multiple;

[0145] The processing module 1202 is further configured to:

[0146] Determine the ratio of the length of the first designated target on the designated boundary line of the annotation box of the first designated target to the total length of the designated boundary line; wherein the first designated target is a target whose annotation box overlaps with the annotation boxes of other targets.

[0147] Adjust the size of the callout box of the first specified target according to the scale.

[0148] Optionally, the processing module 1202 is configured to:

[0149] When the ratio is less than the preset ratio threshold, the labeling box of the first designated target is reduced until the ratio is equal to or greater than the preset ratio threshold, or until the labeling box of the first designated target does not overlap with the labeling boxes of other targets.

[0150] Optionally, there are multiple targets; the annotation box includes a combined annotation box;

[0151] The processing module 1202 is configured to:

[0152] determining at least two targets to be combined as combined targets;

[0153] According to the region information of the combined target, the position coordinates of the combined annotation frame of the combined target on the image to be annotated are determined.

[0154] Optionally, the generating module 1201 is configured to:

[0155] Construct a 3D model of the target and a 3D environment model;

[0156] The 3D model of the target is configured in the 3D environment model, and the image to be labeled and the segmentation map of the image to be labeled are generated by the ray tracing algorithm;

[0157] The processing module 1202 is further configured to:

[0158] The image to be labeled and the labeled image form a training sample.

[0159] The image annotation device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0160] Figure 13 This is a hardware structure diagram of an image annotation device provided in one embodiment of the present application. Figure 13 As shown, the image annotation device 130 provided in this embodiment includes: at least one processor 1301 and a memory 1302. The image annotation device 130 also includes a communication component 1303. The processor 1301, the memory 1302 and the communication component 1303 are connected via a bus 1304.

[0161] In a specific implementation process, at least one processor 1301 executes the computer-executable instructions stored in the memory 1302 , so that the at least one processor 1301 performs the above image annotation method.

[0162] The specific implementation process of the processor 1301 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0163] In the above Figure 13 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules in the processor.

[0164] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0165] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0166] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the above image annotation method is implemented.

[0167] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0168] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image annotation method, characterized in that: include: Generate an image to be annotated and a segmentation map of the image to be annotated based on the three-dimensional model of the target, wherein the image to be annotated includes the target, and the segmentation map includes regional information of a region where the target is located on the image to be annotated, wherein there are multiple targets, and the regional information of the region where each target is located includes multiple pixels with the same pixel value; Determining the position coordinates of the marking frame of the target on the image to be marked according to the region information; Drawing the annotation frame on the image to be annotated according to the position coordinates to obtain an annotated image; If the annotation box includes a direction annotation box, determining the position coordinates of the annotation box of the target on the image to be annotated according to the region information includes: Mapping the three-dimensional coordinates of two designated position points on the three-dimensional model to the image to be annotated to obtain the position coordinates of two target position points corresponding to the two designated position points; Determine the line connecting the two target position points as the main axis of the target direction annotation frame; The position coordinates of the direction mark box of the target are determined according to the area information and the main axis, wherein two boundary lines of the direction mark box are parallel to the main axis, and the direction mark box is the minimum circumscribed rectangle of the area where the target is located.

2. The method according to claim 1, characterized in that The callout box includes an axis-aligned callout box; Determining the position coordinates of the annotation box of the target on the image to be annotated according to the region information includes: Obtaining coordinates of each pixel point in the area where the target is located on the segmentation map; the coordinates include a first axis coordinate value along the height direction of the segmentation map and a second axis coordinate value along the width direction of the segmentation map; Determine the maximum and minimum values ​​of the first-axis coordinates of all pixels, as well as the maximum and minimum values ​​of the second-axis coordinates; The position coordinates of the axis-aligned annotation box of the target are determined according to the maximum and minimum values ​​of the first axis coordinate values ​​and the maximum and minimum values ​​of the second axis coordinate values, wherein two boundary lines of the axis-aligned annotation box are parallel to the height direction.

3. The method according to claim 1 or 2, characterized in that Each target corresponds to a logo and a label box; The method further comprises: For each target, an association relationship between the target's identifier and the target's annotation box is established.

4. The method according to claim 1 or 2, characterized in that Each target corresponds to a position in the three-dimensional environment model; Generating an image to be labeled and a segmentation map of the image to be labeled according to the three-dimensional model of the target, including: Arrange the three-dimensional model of each target at a corresponding position in the three-dimensional environment model to obtain a first scene model, and generate an image to be labeled containing each target and a segmentation map of the image to be labeled according to the first scene model; Determining the position coordinates of the annotation box of the target on the image to be annotated according to the region information includes: For each target, a three-dimensional model of the target is placed at a corresponding position in the three-dimensional environment model to obtain a second scene model, and based on the second scene model, an independent image containing the target and a segmentation map of the independent image are generated, and based on the regional information of the region where the target is located on the segmentation map of the independent image, the position coordinates of the annotation box of the target on the independent image are determined; The position coordinates of the marking frame of each target on each independent image are determined as the position coordinates of the marking frame of each target on the image to be marked.

5. The method according to claim 1 or 2, characterized in that After drawing a labeling frame on the image to be labeled according to the position coordinates, the method further includes: Determining, on a designated boundary line of a labeled box of a first designated target, the ratio of the length of the first designated target to the total length of the designated boundary line; wherein the first designated target is a target whose labeled box overlaps with the labeled boxes of other targets; The size of the annotation box of the first designated object is adjusted according to the ratio.

6. The method according to claim 5, characterized in that Adjusting the size of the annotation box of the first designated target according to the ratio includes: When the ratio is smaller than a preset ratio threshold, the annotation box of the first designated target is reduced until the ratio is equal to or greater than the preset ratio threshold, or until the annotation box of the first designated target does not overlap with the annotation boxes of other targets.

7. The method according to claim 1 or 2, characterized in that There are multiple targets; the marking box includes a combined marking box; Determining the position coordinates of the annotation box of the target on the image to be annotated according to the region information includes: determining at least two targets to be combined as combined targets; The position coordinates of the combined labeling frame of the combined object on the image to be labeled are determined according to the region information of the combined object.

8. The method according to claim 1 or 2, characterized in that Generating an image to be labeled and a segmentation map of the image to be labeled according to the three-dimensional model of the target, including: Construct a 3D model of the target and a 3D environment model; arranging the three-dimensional model of the target in the three-dimensional environment model, and generating the image to be labeled and a segmentation map of the image to be labeled by a ray tracing algorithm; After obtaining the annotated image, the method further includes: The image to be annotated and the annotated image are combined to form a training sample.

9. An image annotation device, characterized in that: include: a generation module, configured to generate an image to be annotated and a segmentation map of the image to be annotated based on a three-dimensional model of the target, wherein the image to be annotated includes the target, and the segmentation map includes regional information of a region where the target is located on the image to be annotated, wherein there are multiple targets, and the regional information of the region where each target is located includes multiple pixels with the same pixel value; A processing module, configured to determine the position coordinates of a marking frame of the target on the image to be marked based on the region information; a drawing module, configured to draw the annotation frame on the image to be annotated according to the position coordinates, so as to obtain an annotated image; If the annotation frame includes a direction annotation frame, the processing module is specifically configured to map the three-dimensional coordinates of two designated position points on the three-dimensional model to the image to be annotated, so as to obtain the position coordinates of two target position points corresponding to the two designated position points; The line connecting the two target position points is determined as the principal axis of the direction annotation box of the target; and the position coordinates of the direction annotation box of the target are determined based on the area information and the principal axis, wherein two boundary lines of the direction annotation box are parallel to the principal axis, and the direction annotation box is the minimum circumscribed rectangle of the area where the target is located.

Images