A continuous image annotation method

Abstract

This invention relates to a continuous image annotation method, comprising the following steps: S10. Determining the annotation type of the continuous image and obtaining a variable group corresponding to the annotation type; S20. Extracting multiple frames of images from the continuous image at intervals, and the annotation engineer annotating each object to be annotated in each frame of the image according to the determined annotation type, and determining the number of annotations; S30. Generating a preprocessing model for each variable in the variable group based on the number of annotations; S40. Obtaining the state of the above variables in the remaining frames of the continuous image through the preprocessing model; S50. Describing the state of the label together with all the variable states in each frame of the image, thus completing the annotation of the continuous image. This invention only requires annotating a small number of frames in the continuous image, and then quickly generating a preprocessing model. The preprocessing model will automatically complete the annotation type of the object to be annotated in the remaining frames of the image. Afterwards, the annotation engineer confirms the annotation, requiring no adjustment or only minor adjustments, greatly improving the annotation efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for annotating continuous images. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence, "data," as one of the three essential elements of AI, has received increasing attention from the industry. High-quality data is a crucial prerequisite for the successful implementation of AI products. It should be noted that currently, AI primarily requires structured data, that is, data with truth-based annotations.

[0003] Currently, the generation of labeled data is mainly done manually, which has problems such as long training cycles, inconsistent standards, and low labeling efficiency. Therefore, excellent labeling tools and methods are needed to assist labeling engineers and improve labeling efficiency.

[0004] Video data can usually be converted into a continuous image sequence for processing. Currently, annotation engineers use various types of annotation for images, such as points, line segments, 2D bounding boxes, 3D cubes, and closed polygons. Since there is a lot of redundant information between adjacent frames in a continuous image sequence, if the continuous image sequence is treated as a general image and annotated manually using the above-mentioned annotation types, the efficiency is very low. How to utilize the inter-frame information is the key to improving the efficiency of continuous image annotation.

[0005] Meanwhile, as the main production tool for annotation engineers, annotation tools need to be convenient and fast, and the algorithms supporting the functions behind them should not be too complex or time-consuming, otherwise the user experience will be greatly reduced. Summary of the Invention

[0006] This invention addresses the problems of low annotation efficiency and complex, time-consuming algorithms in processing continuous image sequences, where a large amount of redundant information exists between adjacent frames. It also addresses the issue of treating continuous image sequences as general images and manually annotating them using various image annotation types such as points, line segments, 2D bounding boxes, 3D cubes, and closed polygons. The invention provides a continuous image annotation method.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0008] A continuous image annotation method includes the following steps:

[0009] S10. Determine the annotation type of consecutive images and obtain the variable group corresponding to the annotation type;

[0010] S20. Extract multiple frames of images at intervals from consecutive images. The annotation engineer annotates each object to be annotated in each frame of the image according to the determined annotation type, and determines the number of annotations.

[0011] S30. For each variable in the variable group, generate a preprocessed model based on the number of annotations;

[0012] S40. Obtain the state of the above variables in the remaining frames of consecutive images through a preprocessing model;

[0013] S50. The label state is described by combining all the variable states of each frame of the image, thus completing the annotation of consecutive images.

[0014] Furthermore, as a preferred technical solution, generating the preprocessing model specifically includes:

[0015] Define a polynomial function equation for a single frame image based on the number of image frames, the number of annotations, and variables;

[0016] Transform the polynomial function equation into mean square error;

[0017] Construct a least squares problem to obtain the matrix equation;

[0018] Solve the matrix equations to obtain the parameters of the preprocessing model, thus obtaining the preprocessing model.

[0019] Furthermore, as a preferred technical solution, the polynomial function equation for a single frame image is defined as follows:

[0020] Given a set of data based on the number of image frames, the number of annotations, and variables: (j i y i ), i = 0, 1, ..., M-1;

[0021] The polynomial function equation for a single frame image: f(j) = a0 + a1j + ... + a n j n ;

[0022] Where M is the number of annotations, j is the image frame number, y is the state of the variable in the variable group at image frame j, n is the highest degree of the polynomial, and a0, a1, ..., a n The preprocessed model parameters are to be solved.

[0023] Furthermore, as a preferred technical solution, the mean square error of the conversion is:

[0024]

[0025] Furthermore, as a preferred technical solution, the resulting matrix equation is:

[0026]

[0027] in,

[0028] Furthermore, as a preferred technical solution, the obtained preprocessing model parameters are:

[0029]

[0030] Furthermore, as a preferred technical solution, the annotation type includes points, line segments, 2D detection boxes, 3D cubes, or closed polygons.

[0031] Furthermore, as a preferred technical solution, the variable group is a set of multiple variables of the label type in the image coordinate system, which is used to describe the state of the label;

[0032] The variable set of the point includes: the horizontal axis coordinate of the point and the vertical axis coordinate of the point;

[0033] The variable set of the line segment includes: the horizontal axis coordinate of the point and the vertical axis coordinate of the point, wherein the line segment is composed of two points;

[0034] The variable set of the 2D detection frame includes: the horizontal axis coordinate value of the center point of the 2D detection frame, the vertical axis coordinate value of the center point of the 2D detection frame, the width of the 2D detection frame, and the height of the 2D detection frame;

[0035] The variable set of the 3D cube includes: the horizontal axis coordinates of the points and the vertical axis coordinates of the points, wherein the 3D cube is composed of eight points;

[0036] The variable set of the closed polygon includes: the horizontal axis coordinates of the points and the vertical axis coordinates of the points, wherein the closed polygon is composed of multiple points.

[0037] Furthermore, as a preferred technical solution, the number of extracted image frames is 3-5 frames.

[0038] Furthermore, as a preferred technical solution, the number of objects to be labeled on each frame of the image is at least one.

[0039] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0040] This invention only requires labeling a small number of frames in a series of images, and then quickly generates a preprocessed model. The preprocessed model will automatically complete the labeling type of the object to be labeled in the remaining frames. After that, the labeling engineer confirms it, and no adjustment is needed or only minor adjustments are required, which greatly improves the labeling efficiency. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0042] Figure 2 This is a schematic diagram of continuous image annotation in Embodiment 1 of the present invention.

[0043] Figure 3This describes the generation process of the preprocessing model in Embodiment 1 of the present invention.

[0044] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. The same or similar reference numerals correspond to the same or similar components. The terms describing positional relationships in the drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. Detailed Implementation

[0045] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer definition of the scope of protection of the present invention.

[0046] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper", "lower", "left", "right", "top", "bottom", "inner", "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0047] Furthermore, if terms such as "first" or "second" are used for descriptive purposes only, they are mainly used to distinguish different devices, components or parts (the specific types and structures may be the same or different), and are not used to indicate or imply the relative importance or quantity of the indicated devices, components or parts, and should not be construed as indicating or implying relative importance.

[0048] Example 1

[0049] This embodiment discloses a continuous image annotation method, such as... Figure 1 As shown, it includes the following steps:

[0050] S10. Determine the annotation type of consecutive images and obtain the variable group corresponding to the annotation type.

[0051] In this invention, the types of annotation commonly used by annotation engineers for images include points, line segments, 2D bounding boxes, 3D cubes, and closed polygons. Therefore, when annotating an image, the annotation type to be used is determined first, and then the variable group is determined according to the selected annotation type.

[0052] In this embodiment, the variable group is a set of multiple variables of the annotation type in the image coordinate system, which is used to describe the state of the label, and different annotation types have different variable groups.

[0053] In this step, the selected annotation type for annotating continuous images is a 2D bounding box. The variable group of this 2D bounding box includes four variables: the x-axis coordinate value of the center point of the 2D bounding box. 2D detection frame center point y-axis coordinate value 2D detection box width w (j) and 2D detection frame height h (j) , where j is the frame number, representing the state of the variable in frame j.

[0054] S20. Extract multiple frames of images at intervals from consecutive images. The annotation engineer annotates each object to be annotated in each frame of the image according to the determined annotation type, and determines the number of annotations.

[0055] The specific steps are as follows: Suppose the annotation engineer needs to use 2D bounding boxes to annotate the objects to be annotated in N consecutive images. If there is only one object to be annotated in each frame, then M non-consecutive frames are selected from the N consecutive images. In this case, 3≤M≤5. The annotation engineer uses 2D bounding boxes to annotate the objects to be annotated in the M frames, and annotates them a total of M times. If there are multiple objects to be annotated in each frame, then each frame needs to be annotated multiple times, that is, each object to be annotated needs to be annotated once.

[0056] In this embodiment, there is only one object to be labeled in each frame.

[0057] For further examples illustrating this step, see [link to example]. Figure 2 The number of consecutive images is 10 frames, and M is 3. That is, 3 non-consecutive images were selected from the 10 consecutive images, with frame numbers 0, 5 and 10 respectively. Each image has only 1 object to be labeled, so it is labeled 3 times in total. If each image has only 2 objects to be labeled, it needs to be labeled 2*3=6 times. If each image has multiple objects to be labeled, the number of labeling times will be calculated accordingly.

[0058] S30. For each variable in the variable group, generate a preprocessed model based on the number of annotations.

[0059] Since this embodiment uses a 2D bounding box annotation type, its variable group includes four variables: the x-axis coordinate value of the center point of the 2D bounding box. 2D detection frame center point y-axis coordinate value 2D detection box width w (j) and 2D detection frame height h (j) .

[0060] Therefore, in this step, four preprocessed models are generated, each representing the x-axis coordinates of the 2D detection box center point. The preprocessed model, with respect to the ordinate values ​​of the center point of the 2D detection box. The preprocessed model, regarding the 2D detection box width w (j) The preprocessing model and information about the 2D detection box height h (j) The preprocessing model.

[0061] In this step, the generation of the preprocessing model is described in [reference needed]. Figure 3 Specifically:

[0062] S301. Select a variable and define a polynomial function equation for a single frame image based on the number of image frames, the number of annotations, and the variable.

[0063] In this embodiment, it is assumed that the first selected variable is the horizontal axis coordinate value of the center point of the detection box.

[0064] In this step, we first need to provide a set of data based on the number of image frames, the number of annotations, and variables: (j i y i ), i = 0, 1, ..., M-1;

[0065] The defined polynomial function equation for a single frame image is: f(j) = a0 + a1j + ... + a n j n ;

[0066] Where M is the number of annotations, j is the image frame number, y is the state of the variable in the variable group at image frame j, n is the highest degree of the polynomial, and a0, a1, ..., a n The preprocessed model parameters are to be solved.

[0067] S302. Transform the polynomial function equation into mean square error.

[0068] In this step, the mean square error of the transformation is:

[0069]

[0070] S303. Construct a least squares problem to obtain the matrix equation.

[0071] In this step, the obtained matrix equation is:

[0072]

[0073] in,

[0074] S304. Solve the matrix equation to obtain the preprocessing model parameters, thereby obtaining the preprocessing model.

[0075] In this step, the preprocessing model parameters obtained are:

[0076]

[0077] Where a0, a1, ..., a n The preprocessed model parameters are to be solved.

[0078] The horizontal axis coordinates of the center point of the 2D detection box are obtained through steps S301-S304. The preprocessing model is obtained by repeating steps S301-S304 for each variable in the variable group to obtain the preprocessing model for the corresponding variable.

[0079] That is, the y-axis coordinate value of the center point of the 2D detection box in the variable group. 2D detection box width w (j) and the height h of the 2D detection frame (j) Repeat steps S301-S304 to obtain the ordinate values ​​of the center point of the 2D detection box. The preprocessed model, regarding the 2D detection box width w (j) The preprocessing model and information about the 2D detection box height h (j) The preprocessing model.

[0080] S40. Obtain the state of the above variables in the remaining frames of the continuous images through the preprocessing model.

[0081] Specifically, this step involves using the preprocessed model generated in step S30 to automatically complete the 2D detection bounding box of the object to be labeled in the remaining frames of consecutive images.

[0082] That is, four preprocessed models were generated in step S30, which are the x-axis coordinates of the center point of the 2D detection box. The preprocessed model, with respect to the ordinate values ​​of the center point of the 2D detection box. The preprocessed model, regarding the 2D detection box width w (j) The preprocessing model and information about the 2D detection box height h (j) The preprocessing model.

[0083] This step illustrates the following example: If 10 consecutive images are used to generate a preprocessing model, and M of these images are used, then the remaining 10-M frames will be automatically labeled using the four preprocessing models generated in step S30. That is, the preprocessing model for each variable will automatically label the objects to be labeled in the 10-M frames, thereby obtaining the state of that variable in each frame and thus automatically completing the 2D detection box of the object to be labeled in the 10-M frames.

[0084] S50. The label state is described by combining all the variable states of each frame of the image, thus completing the annotation of consecutive images.

[0085] Specifically, this step involves using the preprocessing model of variable groups already employed in step S40 to perform 2D bounding box annotation for each object to be labeled in each frame of the continuous images. Therefore, the state of the label is described by the state of all variables in each frame, thus completing the 2D bounding box annotation of the continuous images.

[0086] The method in this embodiment only needs to annotate a small number of frames in a series of images, and then quickly generate a preprocessed model. The preprocessed model will automatically complete the annotation type of the object to be annotated in the remaining frames. After that, the annotation engineer confirms it, and no adjustment is needed or only minor adjustments are required, which greatly improves the annotation efficiency.

[0087] Example 2

[0088] This embodiment discloses a continuous image annotation method. The difference between this embodiment and Embodiment 1 is that the annotation type used is different.

[0089] In this embodiment, the selected annotation type for annotating continuous images is points, and its variable group includes two variables: the x-axis coordinate of the point in the image coordinate system. j And in the image coordinate system, the y-axis coordinate of a point j Where j is the frame number, representing the state of the variable in frame j; therefore, the generated preprocessing model is also different.

[0090] In this embodiment, the generation of the preprocessing model in step S30 specifically involves:

[0091] S301. Select a variable, the x-axis coordinate of the point, and define a polynomial function equation about a single frame image based on the number of image frames, the number of annotations, and the variable. See Example 1 for the generated polynomial function equation.

[0092] S302. Transform the polynomial function equation into mean square error. See Example 1 for the transformed mean square error.

[0093] S303. Construct the least squares problem to obtain the matrix equation, which can be found in Example 1.

[0094] S304. Solve the matrix equation to obtain the preprocessing model parameters. The obtained preprocessing model parameters are shown in Example 1, thus obtaining the preprocessing model.

[0095] The preprocessed model obtained in this step is the x-axis coordinate of the point. j The preprocessing model.

[0096] In this embodiment, the x-axis coordinates of the point are obtained through the above steps S301-S304. j The preprocessed model is then used to select another variable from the variable group, the y-axis coordinate of the point. j Repeat steps S301-S304 to obtain the y-axis coordinate of the point. j The preprocessing model.

[0097] In this embodiment, two preprocessing models were generated. For subsequent image annotation, please refer to Embodiment 1. This step will not be repeated here.

[0098] Example 3

[0099] This embodiment discloses a continuous image annotation method. The difference between this embodiment and Embodiment 1 is that the annotation type used is different.

[0100] In this embodiment, the selected annotation type for annotating continuous images is line segments. Since a line segment has two points, and the variables for each point are as described in Embodiment 2, they are: the x-axis coordinate of the point in the image coordinate system. j And in the image coordinate system, the y-axis coordinate of a point j Therefore, the variable group of the line segment has 4 variables, and the preprocessing model generated by the preprocessing model generation method of Example 1 also has 4 variables.

[0101] In this embodiment, a total of 4 preprocessing models were generated. For subsequent image annotation, please refer to Embodiment 1. This step will not be repeated here.

[0102] Example 4

[0103] This embodiment discloses a continuous image annotation method. The difference between this embodiment and Embodiment 1 is that the annotation type used is different.

[0104] In this embodiment, the selected annotation type for annotating continuous images is a 3D cube. Since a 3D cube has 8 points, and the variables for each point are described in Embodiment 2, the variable set of the 3D cube includes: the horizontal coordinate of point k in the image coordinate system. In the image coordinate system, the ordinate of point k is... Where: 0 < k <= 8, k is the k-th point, and j is the frame number, representing the state of the variable in the j-th frame. This variable group has 16 variables, and the preprocessing models generated by using the preprocessing model generation method of Embodiment 1 are also 16.

[0105] In this embodiment, a total of 16 preprocessing models are generated. For the subsequent annotation of pictures, refer to Embodiment 1, and this step will not be elaborated here.

[0106] Embodiment 5

[0107] A continuous picture annotation method disclosed in this embodiment. The difference between this embodiment and Embodiment 1 lies in the different annotation types used.

[0108] In this embodiment, the selected annotation type for annotating continuous pictures is a closed polygon. Assume that this closed polygon has k points, and for the variables of each point, refer to Embodiment 2; therefore, the variable group of the closed polygon includes: in the image coordinate system, the horizontal axis coordinate of point k In the image coordinate system, the vertical axis coordinate of point k Where: 0 < k <= k, k is the k-th point, and j is the frame number, representing the state of the variable in the j-th frame. This variable group has 2k variables, and the preprocessing models generated by using the preprocessing model generation method of Embodiment 1 are also 2k.

[0109] In this embodiment, a total of 2k preprocessing models are generated. For the subsequent annotation of pictures, refer to Embodiment 1, and this step will not be elaborated here.

[0110] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for annotating continuous images, characterized in that, Includes the following steps: S10. Determine the annotation type of consecutive images and obtain the variable group corresponding to the annotation type; S20. Extract multiple frames of images at intervals from consecutive images. The annotation engineer annotates each object to be annotated in each frame of the image according to the determined annotation type, and determines the number of annotations. S30. For each variable in the variable group, generate a preprocessed model based on the number of annotations; S40. Obtain the state of the above variables in the remaining frames of consecutive images through a preprocessing model; S50. Describe the label's state by combining all the variable states of each frame of the image, thus completing the annotation of consecutive images; The generation of preprocessed models specifically includes: Define a polynomial function equation for a single frame image based on the number of image frames, the number of annotations, and variables; Transform the polynomial function equation into mean square error; Construct a least squares problem to obtain the matrix equation; Solve the matrix equations to obtain the parameters of the preprocessing model, thus obtaining the preprocessing model; The annotation types include points, line segments, 2D detection boxes, 3D cubes, or closed polygons; The variable group is a set of multiple variables of the label type in the image coordinate system, which is used to describe the state of the label; The variable set of the point includes: the horizontal axis coordinate of the point and the vertical axis coordinate of the point; The variable set of the line segment includes: the horizontal axis coordinate of the point and the vertical axis coordinate of the point, wherein the line segment is composed of two points; The variable set of the 2D detection frame includes: the horizontal axis coordinate value of the center point of the 2D detection frame, the vertical axis coordinate value of the center point of the 2D detection frame, the width of the 2D detection frame, and the height of the 2D detection frame; The variable set of the 3D cube includes: the horizontal axis coordinates of the points and the vertical axis coordinates of the points, wherein the 3D cube is composed of eight points; The variable set of the closed polygon includes: the horizontal axis coordinates of the points and the vertical axis coordinates of the points, wherein the closed polygon is composed of multiple points.

2. The continuous image annotation method according to claim 1, characterized in that, The defined polynomial function equation for a single frame image is: Given a set of data based on the number of image frames, the number of annotations, and variables: ; The polynomial function equation for a single frame image: ; Where M is the number of annotations, j is the image frame number, y is the state of the variable in the variable group at image frame j, and n is the highest degree of the polynomial. The preprocessed model parameters are to be solved.

3. The continuous image annotation method according to claim 2, characterized in that, The mean square error of the transformation is: 。 4. The continuous image annotation method according to claim 3, characterized in that, The resulting matrix equation is: ； in, .

5. The continuous image annotation method according to claim 4, characterized in that, The obtained preprocessing model parameters are: 。 6. The continuous image annotation method according to claim 1, characterized in that, The extracted image frames are 3-5 frames.

7. The continuous image annotation method according to claim 1, characterized in that, There must be at least one object to be labeled in each frame.

Images